Coding method and apparatus, decoding method and apparatus, and device, storage medium and program product

Abstract

The present application belongs to the field of data processing. Disclosed are a coding method and apparatus, a decoding method and apparatus, and a device, a storage medium and a program product. The coding method comprises: determining a first rendering intermediate result corresponding to a target area in a virtual view plane and first illumination coefficients corresponding to pixels in the target area; on the basis of the first illumination coefficients corresponding to the pixels, processing the first rendering intermediate result, so as to obtain a second rendering intermediate result; and coding the second rendering intermediate result into a code stream. The present application can reduce the occupancy of cloud computing resources without coding the whole rendering image into a code stream, thereby ensuring a smaller code rate for transmission between a cloud platform and a client.

Description

Coding and decoding method, device, equipment, storage medium and program product

[0001] This application claims priority to Chinese patent application No. 202410199353.2, filed on February 22, 2024, entitled “Encoding and decoding method, device, equipment, storage medium and program product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of data processing, and in particular to a coding and decoding method, apparatus, device, storage medium and program product. Background Art

[0003] With advances in technology and the development of cloud computing, cloud rendering has become a vital tool in industries such as gaming, design, architecture, and film and television. Cloud rendering involves using a cloud platform to render 3D models and other related data to generate rendered images, which are then encoded into a bitstream and transmitted to the client. This not only reduces client hardware and maintenance costs, but also leverages the powerful performance and elastic scalability of the cloud platform to improve rendering efficiency and quality.

[0004] Taking the gaming industry as an example, cloud gaming, leveraging cloud rendering technology, is becoming increasingly popular. Cloud gaming involves offloading computing and rendering tasks to a cloud platform. Specifically, the cloud platform renders the game images required by the client based on information such as the 3D game scene and the player's actions. The rendered images are then encoded into a bitstream and transmitted to the client, where they are decoded and displayed.

[0005] However, this approach requires the cloud platform to transmit the entire rendered image to the client, which results in a high bitrate between the cloud platform and the client. Furthermore, when the network is unstable, the client may have difficulty receiving the bitstream sent by the cloud platform in a timely manner, resulting in an inability to display the rendered image, impacting the user experience. For example, in gaming scenarios, an unstable network can prevent the client from displaying the game image in a timely manner, preventing players from playing the game normally. Summary of the Invention

[0006] This application provides a coding and decoding method, apparatus, device, storage medium, and program product that can solve the problem of high bit rate transmission between cloud platforms and clients in related technologies. The technical solution is as follows:

[0007] In a first aspect, a coding method is provided, which is applied to a cloud platform, wherein the cloud platform includes multiple three-dimensional models; the method includes: determining a first rendering intermediate result corresponding to a target area in a virtual viewing plane and a first illumination coefficient corresponding to each pixel in the target area; wherein a first visible area corresponding to the target area in the multiple three-dimensional models has reflection and / or refraction characteristics, the first rendering intermediate result indicates a second visible area in the multiple three-dimensional models that is reflected and / or refracted by the first visible area, the first illumination coefficient refers to the illumination coefficient of an intersection of a light ray emitted from a virtual viewpoint and passing through the pixel and the first visible area, the virtual viewing plane is located between the virtual viewpoint and the multiple three-dimensional models, and the first illumination coefficient indicates the degree of contribution of the second visible area to the display content of each pixel in the target area; based on the first illumination coefficient corresponding to each pixel, processing the first rendering intermediate result to obtain a second rendering intermediate result, the number of bits occupied by the second rendering intermediate result during encoding being less than the number of bits occupied by the first rendering intermediate result during encoding; and encoding the second rendering intermediate result into a bitstream.

[0008] Since determining the refraction / reflection data of the target area requires high device performance, the present application determines the first rendering intermediate result corresponding to the target area in the virtual visual plane and the first illumination coefficient corresponding to each pixel in the target area through the cloud platform, and then processes the first rendering intermediate result based on the first illumination coefficient corresponding to each pixel to obtain the second rendering intermediate result. Since the first rendering intermediate result indicates the second visible area reflected and / or refracted by the first visible area in the multiple three-dimensional models, the first rendering intermediate result can indicate the reflection and / or refraction data of the first visible area. Compared with the related technology that requires the cloud platform to render the entire image, the cloud platform in the present application only needs to determine the reflection and / or refraction data and the first illumination coefficient of the first visible area, which can reduce the occupancy of cloud computing resources. The cloud platform can also encode the second rendering intermediate result into the code stream. Compared with the related technology that requires the entire rendered image to be encoded into the code stream, the present application does not need to encode the entire rendered image into the code stream, which can ensure that the code rate transmitted between the cloud platform and the client is relatively low. Since the number of bits occupied by the second rendering intermediate result during encoding is smaller than the number of bits occupied by the first rendering intermediate result during encoding, the bit rate of transmission between the cloud platform and the client can be further reduced.

[0009] Optionally, the first illumination coefficient indicates the reflectivity and / or refractive index of the first visible area. A greater reflectivity indicates that more light incident on the first visible area can be reflected to the second visible area, and therefore, the second visible area contributes more to the display content of the pixel. Similarly, a greater refractive index indicates that more light incident on the first visible area can be refracted to the second visible area, and therefore, the second visible area contributes more to the display content of the pixel.

[0010] Before determining a first intermediate rendering result corresponding to the target area in the virtual viewing plane and a first illumination coefficient corresponding to each pixel in the target area, the cloud platform can receive an image rendering instruction, wherein the rendering instruction includes a virtual viewpoint pose (or camera pose). Based on the camera pose and the multiple three-dimensional models, the cloud platform determines the first intermediate rendering result corresponding to the target area in the virtual viewing plane and a first illumination coefficient corresponding to each pixel in the target area.

[0011] Optionally, the reflection result includes at least one of a reflection map and reflection vertex data, and the refraction result includes at least one of a refraction map and refraction vertex data. The reflection map and the refraction map may be collectively referred to as map data, and the reflection vertex data and the refraction vertex data may be collectively referred to as vertex data.

[0012] Optionally, the vertex data is the face element number of the second face element, or the vertex number of the second face element, or the coordinates of the visible point in the multiple three-dimensional models, the second face element is the face element corresponding to the corresponding pixel in the second visible area, and the visible point is the point in the second face element corresponding to the corresponding pixel.

[0013] Before determining the first rendering intermediate result corresponding to the target area in the virtual viewing plane and the first illumination coefficient corresponding to each pixel in the target area, the cloud platform can also obtain first information, which includes end-side device information and / or attribute information of the target area, the end-side device information indicates the performance of the client, and the attribute information indicates the reflection and / or refraction properties of the target area.

[0014] Optionally, the end-side device information indicates whether the client supports the encoding and decoding method provided in this application, the method for determining the illumination coefficient supported by the client, the method for determining the second rendering intermediate result supported by the client, the client's frame rate requirements, the client's latency requirements, and at least one of the client's power consumption.

[0015] If the first information includes terminal-side device information, the client can send its own device information to the cloud platform before the cloud platform obtains the terminal-side device information, and the cloud platform can receive the client's device information (ie, terminal-side device information).

[0016] If the first information includes attribute information of the target area, the attribute information of the target area includes the wave coefficient corresponding to the target area and at least one of the first indication information, and the first indication information indicates the method for determining the second rendering intermediate result, the cloud platform needs to determine the attribute information of the target area before obtaining the attribute information of the target area.

[0017] When the first face element corresponding to the target area is a water surface, the attribute information of the target area includes the wave coefficient corresponding to the target area. At this time, the wave coefficient of the water surface is stored in the cloud platform. At this time, the cloud platform can directly determine the wave coefficient of the water surface as the wave coefficient corresponding to the target area. The wave coefficient indicates the degree of fluctuation of the water surface. The larger the wave coefficient, the greater the degree of fluctuation of the water surface.

[0018] Optionally, rendering indication information is determined based on the first information, and the rendering indication information includes cloud-side indication information, and the cloud-side indication information indicates a method for determining the first rendering intermediate result and the first illumination coefficient, and / or a method for determining the second rendering intermediate result.

[0019] Optionally, after the cloud platform determines the terminal-side indication information, the cloud platform can send the terminal-side indication information to the client.

[0020] Since the first illumination coefficient is usually a number less than 1, in this case, the first rendering intermediate result is processed based on the first illumination coefficient corresponding to each pixel in the target area, and the data bit amount of the obtained second rendering intermediate result is smaller than the data bit amount of the first rendering intermediate result. In this way, during encoding, the number of bits occupied by the second rendering intermediate result is smaller than the number of bits occupied by the first rendering intermediate result, thereby achieving effective compression of the first rendering intermediate result and further reducing the bit rate transmitted between the cloud platform and the client.

[0021] There are multiple implementation methods for processing the first rendering intermediate result based on the first illumination coefficient corresponding to each pixel in the target area to obtain the second rendering intermediate result. Two of the implementation methods are introduced below.

[0022] In a first implementation, when the first rendering intermediate result includes the mapping data of each pixel in the target area, the mapping data corresponding to each pixel in the target area is multiplied by the first illumination coefficient corresponding to each pixel to obtain the second rendering intermediate result.

[0023] If the map data includes a reflection map, the first illumination coefficient includes a reflection illumination coefficient. At this time, the reflection map corresponding to each pixel in the target area is multiplied by the corresponding reflection illumination coefficient to obtain the reflection map corresponding to each pixel after multiplication (for the sake of convenience of description, the reflection map after multiplication is called the second reflection map), and the second reflection map corresponding to each pixel is used as the second rendering intermediate result.

[0024] Since the reflection illumination coefficient is typically less than 1, in this case, for any pixel in the target area, the value obtained by multiplying the number less than 1 by the pixel's texture data is less than the original texture data of the pixel, thereby reducing the number of data bits in the reflection map of the pixel. Furthermore, since data encoding typically requires quantization, the quantized data will suffer from data loss compared to the pre-quantized data. For the same reflection map, if different reflection illumination coefficients are used to process the reflection map, the data loss after quantization will be different. A second reflection map is obtained after processing with a larger reflection illumination coefficient, while a second reflection map is obtained after processing with a smaller reflection illumination coefficient. Thus, the second reflection map obtained after processing with the smaller reflection illumination coefficient is closer to the quantization step size, resulting in a greater data loss for the second reflection map obtained after processing with the smaller reflection illumination coefficient. In other words, the larger the reflection illumination coefficient, the less data loss there is for the reflection map. This results in less data loss for reflection maps with a greater contribution, thereby reducing the loss of valid data while ensuring effective data compression.

[0025] If the map data includes a refraction map, the first illumination coefficient includes a refraction illumination coefficient. At this time, the refraction map corresponding to each pixel in the target area is multiplied by the corresponding refraction illumination coefficient to obtain the refraction map corresponding to each pixel after multiplication (for the sake of convenience of description, the refraction map after multiplication is called the second refraction map), and the second refraction map corresponding to each pixel is used as the second rendering intermediate result.

[0026] Since the refraction illumination coefficient is typically less than 1, in this case, for any pixel in the target area, the value obtained by multiplying the pixel's map data by the number less than 1 is less than the original map data for that pixel, thereby reducing the number of data bits in the refraction map for that pixel. Furthermore, since data encoding typically requires quantization, the quantized data suffers from data loss compared to the pre-quantized data. For the same refraction map, processing it with different refraction illumination coefficients results in different data loss after quantization. A second refraction map is obtained after processing with a larger refraction illumination coefficient, while a second refraction map is obtained after processing with a smaller refraction illumination coefficient. Thus, the second refraction map obtained with the smaller refraction illumination coefficient is closer to the quantization step size, resulting in a greater data loss for the second refraction map obtained with the smaller refraction illumination coefficient. In other words, the larger the refraction illumination coefficient, the less data loss there is in the refraction map. This results in less data loss for refraction maps with a greater contribution, thereby reducing the loss of valid data while ensuring effective data compression.

[0027] The second implementation method is to divide the target area into multiple blocks, each block includes multiple pixels, and based on the first illumination coefficient corresponding to each pixel in the multiple blocks, determine the second illumination coefficient corresponding to the multiple blocks respectively; based on the second illumination coefficient corresponding to the multiple blocks respectively, process the first rendering intermediate result to obtain the second rendering intermediate result.

[0028] Optionally, the first rendering intermediate result includes the map data of each pixel in the target area. In this case, the cloud platform processes the first rendering intermediate result based on the second illumination coefficients corresponding to multiple blocks respectively, and determines that there are multiple implementation methods for the second rendering intermediate result. Two of the implementation methods are introduced below.

[0029] The first method is to multiply the second illumination coefficient corresponding to each block in the pair of blocks by the reflection map of the pixels included in the block to obtain a second rendering intermediate result.

[0030] The second method is to determine the convolution kernel sizes corresponding to the multiple blocks based on the second illumination coefficients corresponding to the multiple blocks, and perform convolution operations on the map data of the pixels in the multiple blocks based on the convolution kernel sizes corresponding to the multiple blocks to obtain the second rendering intermediate result.

[0031] It should be noted that since the first illumination coefficient indicates the contribution of the second visible area to the display content of each pixel in the target area, a larger second illumination coefficient indicates a greater contribution of the second visible area in the block to the display content of each pixel in the block. When performing convolution on the block, it is necessary to reduce the data loss caused by the convolution, so that the data loss of the texture data with a large contribution is minimized. Therefore, the size of the convolution kernel is inversely proportional to the second illumination coefficient: the larger the second illumination coefficient, the smaller the size of the convolution kernel.

[0032] Optionally, the first rendering intermediate result includes vertex data of each pixel in the target area. In this case, the cloud platform processes the first rendering intermediate result based on the second illumination coefficients corresponding to multiple blocks. There are multiple ways to obtain the second rendering intermediate result. Two of the implementation methods are introduced below.

[0033] The first method is to determine the filtering radius corresponding to the multiple blocks based on the second illumination coefficients corresponding to the multiple blocks, and filter the vertex data of the pixels in the multiple blocks based on the filtering radius corresponding to the multiple blocks to obtain a second rendering intermediate result.

[0034] Similarly, since the first illumination coefficient indicates the contribution of the second visible area to the display content of each pixel in the target area, a larger second illumination coefficient indicates a greater contribution of the second visible area in the block to the display content of each pixel in the block. When performing convolution on the block, it is necessary to reduce the data loss caused by the convolution, so that the loss of vertex data with a large contribution is minimized. Therefore, the filter radius is inversely proportional to the second illumination coefficient: the larger the second illumination coefficient, the smaller the filter radius.

[0035] The second method is to determine the data filtering ratios corresponding to the multiple blocks based on the second illumination coefficients corresponding to the multiple blocks, and the data filtering ratios indicate the filtering status of the vertex data of the pixels in the blocks. Based on the data filtering ratios corresponding to the multiple blocks, the vertex data of the pixels in the multiple blocks are filtered to obtain a second rendering intermediate result.

[0036] It should be noted that the data screening ratio can be a data deletion ratio or a data retention ratio.

[0037] Similarly, since the first illumination coefficient indicates the contribution of the second visible area to the display content of each pixel in the target area, the larger the second illumination coefficient is, the greater the contribution of the second visible area in the block to the display content of each pixel in the block. When performing convolution on the block, it is necessary to reduce the data loss caused by the convolution so that the data loss of vertex data with a large contribution is smaller. Therefore, in the case where the data screening ratio is the data deletion ratio, the data deletion ratio is inversely proportional to the second illumination coefficient. The larger the second illumination coefficient is, the smaller the data deletion ratio is. In the case where the data screening ratio is the data retention ratio, the data retention ratio is directly proportional to the second illumination coefficient. The larger the second illumination coefficient is, the larger the data retention ratio is.

[0038] In a second aspect, a decoding method is provided, which is applied to a client, the client including multiple three-dimensional models. The method includes: decoding from a code stream to obtain a second rendering intermediate result corresponding to a target area in a virtual viewing plane; wherein a first visible area corresponding to the target area in the multiple three-dimensional models has reflection and / or refraction characteristics, the second rendering intermediate result is obtained by processing the first rendering intermediate result of the target area by a first illumination coefficient corresponding to each pixel in the target area, the number of bits occupied by the second rendering intermediate result during encoding is less than the number of bits occupied by the first rendering intermediate result during encoding, the first rendering intermediate result indicates a second visible area in the multiple three-dimensional models that is reflected and / or refracted by the first visible area, the first illumination coefficient refers to the illumination coefficient of an intersection of a light ray emitted from a virtual viewpoint and passing through the pixel and the first visible area, the virtual viewing plane is located between the virtual viewpoint and the multiple three-dimensional models, and the first illumination coefficient indicates the degree of contribution of the second visible area to the display content of each pixel in the target area; and generating a rendered image corresponding to the virtual viewing plane based on the second rendering intermediate result and the multiple three-dimensional models.

[0039] In the present application, the client can parse the code stream encoded with the second rendering intermediate result, and thus render the image based on the second rendering intermediate result. That is, in the present application, the cloud platform does not need to render the final image, but only needs to determine the reflection and / or refraction data, and the work of rendering the image is performed by the client. In this way, even in the case of unstable network, the terminal side can perform basic image rendering, thereby effectively improving the user experience.

[0040] Optionally, the second intermediate rendering result includes at least one of a reflection result of pixels in the target area and a refraction result of pixels in the target area. The reflection result includes at least one of a reflection map and reflection vertex data, and the refraction result includes at least one of a refraction map and refraction vertex data. The reflection map and the refraction map may be collectively referred to as map data, and the reflection vertex data and the refraction vertex data may be collectively referred to as vertex data.

[0041] Optionally, the second intermediate rendering result is not a target intermediate rendering result. The target intermediate rendering result refers to an intermediate rendering result that can be directly used for rendering without the need for restoration by the client. In this case, the process of generating a rendered image corresponding to the virtual viewing plane based on the second intermediate rendering result and multiple three-dimensional models by the client may include the following steps (1)-(2).

[0042] (1) The client restores the second rendering intermediate result to obtain a reconstructed value of the first rendering intermediate result.

[0043] The client determines a first illumination coefficient corresponding to each pixel in the target area, and restores the second rendering intermediate result based on the first illumination coefficient corresponding to each pixel to obtain a reconstructed value of the first rendering intermediate result.

[0044] Optionally, the second rendering intermediate result may include the map data of each pixel in the target area, and the second rendering intermediate result may also include the vertex data of some pixels in the target area. In different situations, the implementation methods of restoring the second rendering intermediate result and obtaining the reconstructed value of the first rendering intermediate result are different, which will be introduced below.

[0045] In the case where the second rendering intermediate result includes the map data of each pixel in the target area, if the cloud platform processes the first rendering intermediate result based on the first illumination coefficient corresponding to each pixel in the target area and adopts the first implementation method to obtain the second rendering intermediate result, then for the reflection map of any pixel in the target area in the second rendering intermediate result, the client can divide the reflection map of the pixel by the reflection illumination coefficient corresponding to the pixel to obtain the reflection map of the pixel after the division process. Each pixel in the target area is processed in the same manner to obtain the reflection map of each pixel after the division process, so as to restore the second rendering intermediate result and obtain the reconstructed value of the first rendering intermediate result.

[0046] If the cloud platform processes the first rendering intermediate result based on the first illumination coefficient corresponding to each pixel in the target area and adopts the second implementation method to obtain the second rendering intermediate result, the client determines the second illumination coefficients corresponding to the multiple blocks based on the first illumination coefficients corresponding to each pixel in the multiple blocks included in the target area, and restores the second rendering intermediate result based on the second illumination coefficients corresponding to the multiple blocks to obtain the reconstructed value of the first rendering intermediate result.

[0047] Optionally, the client can determine the pixels included in each block of the target area, and then determine the second illumination coefficients corresponding to the multiple blocks based on the first illumination coefficients corresponding to the pixels in the multiple blocks included in the target area.

[0048] Optionally, if the cloud platform processes the first rendering intermediate result based on the second illumination coefficients corresponding to the multiple blocks and adopts the second implementation method when determining the second rendering intermediate result, the client restores the second rendering intermediate result based on the second illumination coefficients corresponding to the multiple blocks, and the implementation process of obtaining the reconstructed value of the first rendering intermediate result includes: the client determines the convolution kernel sizes corresponding to the multiple blocks based on the second illumination coefficients corresponding to the multiple blocks, and performs deconvolution operations on the map data of the pixels in the multiple blocks based on the convolution kernel sizes corresponding to the multiple blocks to obtain the reconstructed value of the first rendering intermediate result.

[0049] In the case where the second rendering intermediate result includes vertex data of some pixels in the target area, based on the vertex data of each pixel included in the second rendering intermediate result, the vertex data corresponding to each pixel in the multiple blocks included in the target area is determined to obtain the reconstructed value of the first rendering intermediate result.

[0050] Among them, the vertex data is the face element number of the second face element, or the vertex number of the second face element, or the coordinates of the visible point in multiple three-dimensional models, the second face element is the face element corresponding to the corresponding pixel in the second visible area, and the visible point is the point in the second face element corresponding to the corresponding pixel.

[0051] Optionally, the client can determine the pixels included in each block of the target area, and then determine the vertex data corresponding to each pixel in the multiple blocks included in the target area based on the vertex data of each pixel included in the second rendering intermediate result.

[0052] Optionally, the client can obtain end-side indication information, which indicates at least one of the following: whether the second rendering intermediate result is the target rendering intermediate result, the method for determining the first illumination coefficient, the method for determining the second illumination coefficient, the method for determining the second rendering intermediate result, the method for dividing multiple blocks, whether the second rendering intermediate result contains reflection results, whether it contains refraction results, and the type of vertex data.

[0053] (2) Based on the reconstructed value of the first rendering intermediate result, the target illumination coefficient corresponding to each pixel in the target area and multiple three-dimensional models, a rendering image corresponding to the virtual viewing plane is generated, and the target illumination coefficient is the illumination coefficient adopted by the cloud platform when determining the normal vector of the first surface element corresponding to each pixel in the target area, and is based on the illumination coefficient determined by the normal vector of the first surface element.

[0054] Optionally, the client determines the color corresponding to each pixel in the target area based on the reconstructed value of the first rendering intermediate result, processes the color corresponding to each pixel in the target area based on the target illumination coefficient corresponding to each pixel in the target area, and obtains a third rendering intermediate result, and generates a rendered image corresponding to the virtual viewing plane based on the third rendering intermediate result and the multiple three-dimensional models.

[0055] Optionally, the reconstructed value of the first rendering intermediate result includes texture data of each pixel in the target area and / or vertex data of each pixel in the target area.

[0056] If the reconstructed value of the first rendering intermediate result includes the texture data of each pixel in the target area, in this case, the client can determine the color in the texture data of each pixel in the target area as the color corresponding to each pixel in the target area.

[0057] If the reconstructed value of the first rendering intermediate result includes vertex data of each pixel in the target area, in this case, based on the vertex data of each pixel in the target area, according to the relevant algorithm, the color corresponding to each pixel in the target area is determined from multiple three-dimensional models.

[0058] Optionally, the second intermediate rendering result is a target intermediate rendering result, which refers to an intermediate rendering result that the client does not need to restore and can be directly used for rendering. In this case, the client can directly determine the second intermediate rendering result as the third intermediate rendering result and, based on the third intermediate rendering result and the multiple 3D models, generate a rendered image corresponding to the virtual view plane according to a relevant algorithm. In other words, when the second intermediate rendering result is the target intermediate rendering result, the client can directly generate a rendered image corresponding to the virtual view plane based on the target intermediate rendering result and the multiple 3D models.

[0059] When the second rendering intermediate result is the target rendering intermediate result, the client can directly generate a rendering image corresponding to the virtual viewing plane based on the target rendering intermediate result and the multiple three-dimensional models, without having to restore the second rendering intermediate result to obtain the reconstructed value of the first rendering intermediate result, nor without having to process the color corresponding to each pixel in the target area based on the target illumination coefficient. In this way, the efficiency of the client in rendering the image can be effectively improved.

[0060] In a third aspect, a coding device is provided, wherein the coding device has the function of implementing the coding method described in the first aspect. The coding device includes at least one module, wherein the at least one module is configured to implement the coding method described in the first aspect.

[0061] In a fourth aspect, a decoding device is provided, wherein the decoding device has the function of implementing the decoding method described in the second aspect. The decoding device includes at least one module, wherein the at least one module is used to implement the decoding method described in the second aspect.

[0062] In a fifth aspect, an encoding device is provided, comprising a processor and a memory, wherein the memory is configured to store a computer program for executing the encoding method provided in the first aspect. The processor is configured to execute the computer program stored in the memory to implement the encoding method described in the first aspect.

[0063] Optionally, the encoding device may further include a communication bus, which is used to establish a connection between the processor and the memory.

[0064] In a sixth aspect, a decoding device is provided, comprising a processor and a memory, wherein the memory is configured to store a computer program for executing the decoding method provided in the second aspect. The processor is configured to execute the computer program stored in the memory to implement the decoding method described in the second aspect.

[0065] Optionally, the decoding device may further include a communication bus, which is used to establish a connection between the processor and the memory.

[0066] In the seventh aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the steps of the encoding method described in the first aspect or the decoding method described in the second aspect.

[0067] In an eighth aspect, a computer program product is provided, comprising computer instructions that, when executed by a computer or processor, cause the computer or processor to perform the steps of the encoding method described in the first aspect, or the steps of the decoding method described in the second aspect. Alternatively, a computer program is provided that, when executed on a computer or processor, causes the computer or processor to perform the steps of the encoding method described in the first aspect, or the steps of the decoding method described in the second aspect.

[0068] In the ninth aspect, a coding and decoding system is provided, which includes a coding device and a decoding device, the coding device is used to implement the steps of the coding method described in the first aspect above, and the decoding device is used to implement the steps of the decoding method described in the second aspect above.

[0069] In a tenth aspect, an encoded code stream is provided, wherein the code stream is generated according to the encoding method described in the first aspect.

[0070] In an eleventh aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a code stream generated according to the encoding method described in the first aspect.

[0071] In a twelfth aspect, a device for storing a code stream is provided, the device comprising: a receiver and at least one storage medium, the receiver being used to receive a code stream generated according to the encoding method described in the first aspect above, and the at least one storage medium being used to store the code stream.

[0072] In the thirteenth aspect, a device for transmitting a code stream is provided, the device comprising: a transmitter and a receiver, the receiver being used to receive the code stream generated by the encoding method described in the first aspect above, and the transmitter being used to send the code stream to an end-side device via a transmission medium.

[0073] In a fourteenth aspect, a device for transmitting a code stream is provided, the device comprising: a transmitter and at least one storage medium, the at least one storage medium being used to store the code stream generated according to the encoding method described in the first aspect above, the transmitter being used to obtain the code stream from the storage medium and send the code stream to an end-side device via a transmission medium.

[0074] In a fifteenth aspect, a system for distributing code streams is provided, the system comprising: at least one storage medium for storing code streams generated according to the encoding method described in the first aspect above, a streaming media device for obtaining a target code stream from the at least one storage medium and sending the target code stream to an end-side device, wherein the streaming media device comprises a content server or a content distribution server.

[0075] The technical effects obtained in the above-mentioned third to fifteenth aspects are similar to the technical effects obtained by the corresponding technical means in the first and second aspects, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] FIG1 is a schematic diagram of a rendering structure under a single virtual viewpoint provided in an embodiment of the present application;

[0077] FIG2 is a schematic diagram of dividing a surface of a three-dimensional model into multiple facets according to an embodiment of the present application;

[0078] FIG3 is a schematic diagram of a facet provided in an embodiment of the present application;

[0079] FIG4 is a flow chart of a cloud rendering method provided by an embodiment of the present application;

[0080] FIG5 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0081] FIG6 is a schematic diagram of the structure of a client provided in an embodiment of the present application;

[0082] FIG7 is a flowchart of an encoding method provided in an embodiment of the present application;

[0083] FIG8 is a schematic diagram of a virtual viewing plane provided in an embodiment of the present application;

[0084] FIG9 is a flowchart of a decoding method provided in an embodiment of the present application;

[0085] FIG10 is a flow chart of a coding and decoding method provided in an embodiment of the present application;

[0086] FIG11 is a schematic structural diagram of an encoding device provided in an embodiment of the present application;

[0087] FIG12 is a schematic structural diagram of a decoding device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0088] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0089] For ease of understanding, before explaining in detail the encoding and decoding method provided in the embodiment of the present application, the nouns, application scenarios and implementation environment involved in the embodiment of the present application are first introduced.

[0090] First, the nouns involved in the embodiments of the present application are introduced.

[0091] Planar reflection, also known as specular reflection or direct reflection, is a phenomenon in rendering where light waves are reflected back upon reaching a medium's interface. When rendering materials like smooth surfaces (e.g., metal, smooth floors) and water surfaces (e.g., lakes, ponded water), adding planar reflection can significantly enhance the quality of rendered images.

[0092] Screen space reflection (SSR): is a technology that calculates the reflection effect of the object surface in graphics rendering in real time. The core idea of ​​SSR is to perform reflection calculations in screen space (screen coordinate system) rather than in world space (world coordinate system). This can reduce the amount of calculations and increase rendering speed. However, SSR has some limitations, such as being limited by screen information, prone to errors, and noise problems. Among them, being limited by screen information means that SSR cannot achieve reflection effects for objects outside the screen. Errors mean that since SSR performs reflection calculations in screen space, distortion or errors may occur for reflections over large areas or long distances. The noise problem refers to the possibility of noise problems in the implementation of SSR, which will affect the clarity and authenticity of the reflection effect.

[0093] Intermediate rendering results: Intermediate data generated by the graphics rendering engine during the generation of images / videos to be displayed. These include reflection plane coordinates, normal maps, albedo maps, specular intensity maps, mesh identifiers (Mesh IDs), refraction maps, reflection maps, and other information generated during the rendering process.

[0094] Reflection map: In the field of computer graphics, it is an efficient method to simulate complex mirror surfaces using pre-calculated texture images. The stored texture image is called a reflection texture or reflection map for subsequent rendering.

[0095] Refraction map: also called refraction texture, stores the refraction characteristics of the object surface in the form of an image for subsequent rendering.

[0096] Filtering: A method of processing data such as images and point clouds. Filtering is done by applying a specific function to specific regions of the data to enhance or reduce specific features. For example, it can be used to increase image edges, reduce noise, or smooth an image or point cloud.

[0097] A facet, also known as a patch, is the smallest planar unit in two- or three-dimensional space. In rendering, models are typically divided into countless tiny planes, called facets. These facets can be any polygon, with triangles and quadrilaterals being common. The intersection of the edges of these facets is the vertex of each facet.

[0098] Pixels: The screens we typically see are composed of individual pixels. A pixel's red, green, and blue (RGB) value is determined by the color of the bin it covers. The meaning of the bins covered by a pixel will be explained in detail later, so I won't go into detail here.

[0099] Rendering Engine: The rendering engine is deployed on the client or cloud platform (also known as the cloud side). The rendering engine can be a computer program product or a physical device. Optionally, when the rendering engine is deployed on the cloud platform, the rendering engine can be a computing device cluster consisting of one or more computing devices.

[0100] Rendering Application: A rendering application includes at least one 3D model and is software or a program used to reconstruct the at least one 3D model. The rendering application can be deployed on a client and / or a cloud platform and can render images by requesting a rendering engine.

[0101] In order to make the technical solution of the present application clearer and easier to understand, before explaining the encoding and decoding method provided in the embodiment of the present application in detail, the relationship between the facets, vertices and pixels involved in the encoding and decoding method is first introduced.

[0102] Please refer to FIG1 , which is a schematic diagram of a rendering structure under a single virtual viewpoint provided by an embodiment of the present application. The rendering structure at least includes a virtual viewpoint 101 , a virtual viewing plane 102 , a three-dimensional model 103 , and a virtual light source 104 .

[0103] Virtual viewpoint 101 is a simulated human eye or eyes in space, used to perceive three-dimensional structures. Based on the number of viewpoints included, virtual viewpoint 101 is categorized as monocular, binocular, or multi-viewpoint. Binocular or multi-viewpoint refers to capturing two or more images from two or more different viewpoints, and then reconstructing a three-dimensional model from these two or more images. This virtual viewpoint can also be considered a camera.

[0104] The virtual viewing plane 102 is a display screen simulated in space. The construction of the virtual viewing plane 102 depends on two factors: the distance between the virtual viewpoint 101 and the virtual viewing plane 102 and the screen resolution of the virtual viewing plane 102 .

[0105] The distance between the virtual viewpoint 101 and the virtual viewing plane 102 refers to the vertical distance from the virtual viewpoint 101 to the virtual viewing plane 102. Optionally, the vertical distance can be set according to different requirements.

[0106] The screen resolution of the virtual viewing plane 102 refers to the number of pixels included in the virtual viewing plane 102. In other words, the virtual viewing plane 102 includes one or more pixels (in FIG. 1 , the virtual viewing plane 102 is schematically illustrated as including 9 pixels).

[0107] The display screen of the user terminal (also referred to as the client) corresponding to the virtual viewing plane 102 is used to output the rendering result. The screen resolution of the display screen may be equal to or different from the screen resolution of the virtual viewing plane 102. If the screen resolution of the display screen is equal to the screen resolution of the virtual viewing plane 102, the rendering result on the virtual viewing plane 102 is output to the display screen at a 1:1 ratio. If the screen resolution of the display screen is different from the screen resolution of the virtual viewing plane 102, the rendering result on the virtual viewing plane 102 is output to the display screen at a certain ratio.

[0108] Typically, a rendering application includes at least one three-dimensional model 103, that is, software or a program is used to determine which three-dimensional models 103 are included in the rendering result on the virtual viewing plane 102. The rendering result is determined based on the pose of the virtual viewpoint 101 (or camera pose / user perspective) and the three-dimensional model 103.

[0109] In some embodiments, before rendering the at least one 3D model 103 by a rendering engine, the at least one 3D model 103 needs to be divided into a plurality of facets, wherein the size and shape of each facet in the plurality of facets may be the same or different.

[0110] For example, please refer to Figure 2, which is a schematic diagram of an embodiment of the present application providing a method of dividing a face of a three-dimensional model 103 into multiple facets. In Figure 2, a face of the three-dimensional model 103 is divided into six triangular facets of different sizes, namely facets 1-6. Among them, the vertices of facet 1 are D0, D1, and D2, the vertices of facet 2 are D0, D2, and D3, the vertices of facet 3 are D0, D3, and D4, the vertices of facet 4 are D0, D4, and D5, the vertices of facet 5 are D0, D5, and D6, and the vertices of facet 6 are D0, D1, and D6.

[0111] Optionally, the rendering application determines, based on the camera pose and rasterization technology, the bins covered by the projection area of ​​each pixel in the virtual viewing plane on the three-dimensional model 103. These covered bins are referred to as visible bins, and each pixel in the virtual viewing plane corresponds to one visible bin. For ease of description, the projection area of ​​the pixels in the virtual viewing plane on the three-dimensional model 103 is subsequently referred to as the visible area corresponding to the virtual viewing plane. This visible area includes at least one visible bin.

[0112] In actual applications, there are usually areas with reflection and refraction characteristics in the at least one three-dimensional model 103. Therefore, for the visible area corresponding to the virtual viewing plane, the visible area may include areas with reflection and refraction characteristics (for the sake of convenience of description, the area with reflection and / or refraction characteristics in the visible area will be referred to as the first visible area in the following), and the first visible area includes at least one first facet.

[0113] The rendering result on the virtual viewing plane 102 is pixel-by-pixel. The rendering result of each pixel (i.e., the color of each pixel) is determined based on the color of the pixel's visible bin. For any pixel in the virtual viewing plane, if the pixel's visible bin is the first bin, a sampling ray emitted from the virtual viewpoint and passing through the pixel will intersect with the first bin after contacting the first bin. The ray will be refracted and / or reflected at this intersection. The reflected and / or refracted ray of the sampling ray can continue to propagate and intersect with other bins in the three-dimensional model.

[0114] For ease of description, the intersection of the reflected and / or refracted light of the sampling ray and other facets in the three-dimensional model will be referred to as a visible point. The facet in the three-dimensional model that contacts the reflected ray will be referred to as a reflection facet. The facet in the three-dimensional model that contacts the refracted ray will be referred to as a refraction facet. The reflection facets and refraction facets will be collectively referred to as second facets, and the area formed by the second facets will be referred to as the second visible area. Thus, at this virtual viewpoint, each pixel in the virtual viewing plane has a corresponding first facet, a second facet, and a visible point. The color of the first facet is determined based on the material properties of the first facet and the refraction and / or reflection data corresponding to the first facet (i.e., the data of the second facet corresponding to the first facet).

[0115] For example, referring to FIG3 , if ray 1 originates from a virtual viewpoint and passes through pixel 1 (not shown in FIG3 ) on virtual viewing plane 102, contacts bin 1 and is reflected at intersection point 1 on bin 1. The reflected ray of ray 1 is then reflected at intersection point 3 on bin 3 (not shown in FIG3 ), then bin 3 is a reflective bin. If ray 1 is also refracted at intersection point 1 on bin 1, and the refracted ray of ray 1 is then refracted at intersection point 2 on bin 2, then bin 2 is a refractive bin. In this case, the color of bin 1 is determined based on the material properties of bin 1, the data of bin 2, and the data of bin 3.

[0116] Virtual light sources 104 are simulated light sources within a space, used to generate the lighting environment within the space. Virtual light sources 104 can be any of the following: point light sources, surface light sources, and line light sources. Optionally, a space can include one or more virtual light sources 104. When a space includes multiple virtual light sources 104, these multiple virtual light sources 104 can be of the same or different types.

[0117] Next, the application scenarios involved in the embodiments of this application are introduced.

[0118] With advances in technology and the development of cloud computing, cloud rendering has become a vital tool in industries such as gaming, design, architecture, and film and television. As shown in Figure 4, cloud rendering involves using a high-quality rendering engine on a cloud platform to render a rendered image based on 3D models and other related data. This image is then encoded into a bitstream, which is then transmitted over the network to the client, where it is parsed and displayed. This not only reduces client hardware and maintenance costs, but also leverages the powerful performance and elastic scalability of the cloud platform to improve rendering efficiency and quality.

[0119] Taking the gaming industry as an example, in some game scenes (also known as the three-dimensional model of the game), there is often a first visible area with reflective and / or refractive properties, such as the water surface or the floor of a room in the game scene. The rendering effect of this first visible area in the game image will affect the gaming experience of the customer (also known as the game player), and the more realistic the rendering effect of this first visible area, the better the gaming experience of the game player.

[0120] Since the first visible area is rendered based on the perspective of the game player, the changes in the player's posture in the game, and the skill effects triggered by the game player, and the player's perspective, posture, and skill effects will change frequently during the game, the first visible area is updated more frequently in the game image. In addition, since the rendering of the first visible area is usually achieved by simulating the propagation of light in a three-dimensional model, the incident angle of the light, the material properties of the surface element, and the reflection and / or refraction data of the first visible area need to be considered during the rendering process. Among them, determining the reflection and / or refraction data of the first visible area requires high computing power. At present, the computing power of the client is difficult to accurately determine the reflection and / or refraction data of the first visible area, thereby achieving high-quality rendering of the first visible area. Therefore, cloud gaming with the help of cloud rendering technology is entering people's lives. Cloud gaming refers to transferring the computing and rendering tasks of the game to the cloud platform for execution. That is, the cloud platform renders the game images required by the client based on information such as the three-dimensional game scene and the operations of the game players, encodes the rendered game images into a bit stream, and then transmits the bit stream to the client so that the client can decode the bit stream and display the game images.

[0121] However, the above method requires the cloud platform to occupy a large amount of cloud computing resources to render the game image, resulting in a high rendering cost. In addition, the cloud platform also needs to transmit the entire rendered image to the client. Transmitting the rendered image requires a high bandwidth, which will result in a high bit rate for transmission between the cloud platform and the client. Moreover, when the network is unstable, it will make it difficult for the client to receive the bit stream sent by the cloud platform in a timely manner, and thus it will not be able to display the rendered image in time, affecting the user experience. For example, in a gaming scenario, an unstable network will cause the client to be unable to display the game image in time, making it impossible for gamers to play the game normally.

[0122] Based on this, an embodiment of the present application provides a coding and decoding method. Since determining the refraction / reflection data of the target area has high requirements for device performance, the embodiment of the present application determines the first rendering intermediate result corresponding to the target area in the virtual visual plane and the first illumination coefficient corresponding to each pixel in the target area through the cloud platform, and then processes the first rendering intermediate result based on the first illumination coefficient corresponding to each pixel to obtain a second rendering intermediate result. Since the first rendering intermediate result indicates the second visible area reflected and / or refracted by the first visible area in the multiple three-dimensional models, the first rendering intermediate result can indicate the reflection and / or refraction data of the first visible area. Compared with the related technology that the cloud platform needs to render the entire image, the cloud platform in the embodiment of the present application only needs to determine the reflection and / or refraction data and the first illumination coefficient of the first visible area, which can reduce the occupancy of cloud computing resources. The cloud platform can also encode the second rendering intermediate result into the code stream. Compared with the related technology that requires the entire rendered image to be encoded into the code stream, the embodiment of the present application does not need to encode the entire rendered image into the code stream, which can ensure that the code rate transmitted between the cloud platform and the client is relatively low. Since the number of bits occupied by the second rendering intermediate result during encoding is less than the number of bits occupied by the first rendering intermediate result during encoding, the bit rate of transmission between the cloud platform and the client can be further reduced. Moreover, in an embodiment of the present application, the client can parse the code stream encoded with the second rendering intermediate result, and thus render the image based on the second rendering intermediate result. That is, in an embodiment of the present application, the cloud platform does not need to render the final image, but only needs to determine the reflection and / or refraction data, and the work of rendering the image is performed by the client. In this way, even in the case of unstable network, the terminal side can perform basic image rendering, thereby effectively improving the user experience.

[0123] It should be noted that the above is only an example of a game scenario. In actual applications, the encoding and decoding method provided in the embodiment of the present application can be applied not only to the game field, but also to other fields that use cloud rendering technology, such as virtual reality (VR) field, digital human field, etc., and the embodiment of the present application is not limited to this.

[0124] Next, the implementation environment involved in the embodiments of this application is introduced.

[0125] Please refer to Figure 5, which is a schematic diagram of an implementation environment provided by an embodiment of the present application. The implementation environment includes a source device 10, a destination device 20, a link 30, and a storage device 40. Source device 10 can generate an encoded image, i.e., a code stream. Therefore, source device 10 can also be referred to as an encoding device. Destination device 20 can decode the code stream generated by source device 10. Therefore, destination device 20 can also be referred to as a decoding device. Link 30 can receive the encoded image generated by source device 10 and transmit it to destination device 20. Storage device 40 can receive the encoded image generated by source device 10 and store it. In this case, destination device 20 can directly obtain the encoded image from storage device 40. Alternatively, storage device 40 can correspond to a file server or another intermediate storage device that can store the encoded image generated by source device 10. In this case, destination device 20 can stream or download the encoded image stored by storage device 40.

[0126] The source device 10 and the destination device 20 may each include one or more processors and a memory coupled to the one or more processors. The memory may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or any other medium that can be used to store desired program code in the form of computer-accessible instructions or data structures. For example, the source device 10 may be a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or a cloud computing service center. The destination device 20 may include a cell phone, a smartphone, a personal digital assistant (PDA), a wearable device, a pocket PC (PPC), a tablet computer, a smart car computer, a smart TV, a smart speaker, a desktop computer, a mobile computing device, a notebook (e.g., laptop) computer, a tablet computer, a set-top box, a telephone handset such as a so-called "smart" phone, a television, a camera, a display device, a digital media player, a video game console, an in-vehicle computer, or the like.

[0127] Link 30 may include one or more media or devices capable of transmitting encoded images from source device 10 to destination device 20. In one possible implementation, link 30 may include one or more communication media that enable source device 10 to send encoded images directly to destination device 20 in real time. In an embodiment of the present application, source device 10 may modulate the encoded images based on a communication standard, such as a wireless communication protocol, and may transmit the modulated images to destination device 20. The one or more communication media may include wireless and / or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (e.g., the Internet). The one or more communication media may include routers, switches, base stations, or other devices that facilitate communication from source device 10 to destination device 20, although this embodiment of the present application does not specifically limit this.

[0128] In one possible implementation, the storage device 40 may store the received encoded image sent by the source device 10, and the destination device 20 may directly obtain the encoded image from the storage device 40. Under such conditions, the storage device 40 may include any of a variety of distributed or locally accessible data storage media, for example, any of the various distributed or locally accessible data storage media may be a hard disk drive, a Blu-ray disc, a digital versatile disc (DVD), a compact disc read-only memory (CD-ROM), a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing a code stream.

[0129] In one possible implementation, storage device 40 may correspond to a file server or another intermediate storage device that can store the codestream generated by source device 10. Destination device 20 may stream or download the images stored on storage device 40. The file server may be any type of server capable of storing and transmitting encoded images to destination device 20. In one possible implementation, the file server may include a network server, a file transfer protocol (FTP) server, a network attached storage (NAS) device, or a local disk drive. Destination device 20 may obtain the encoded images via any standard data connection, including an internet connection. Any standard data connection may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of both suitable for obtaining encoded images stored on a file server. The transmission of the encoded images from storage device 40 may be streaming, downloading, or a combination of both.

[0130] The implementation environment shown in FIG5 is only one possible implementation method, and the technology of the embodiment of the present application is applicable not only to the source device 10 that can encode images and the destination device 20 that can decode encoded images shown in FIG5 , but also to other devices that can encode images and decode code streams. The embodiment of the present application does not make specific limitations on this.

[0131] In the implementation shown in FIG5 , source device 10 includes a data source 120, an encoder 100, and an output interface 140. In some embodiments, output interface 140 may include a modem and / or a transmitter, where the transmitter may also be referred to as a transmitter. Data source 120 may include an image capture device (e.g., a camera), an archive containing previously captured images, a feed interface for receiving images from an image content provider, and / or a computer graphics system for generating images, or a combination of these sources of images.

[0132] The data source 120 may send an image to the encoder 100, and the encoder 100 may encode the image received from the data source 120 to generate an encoded image. The encoder may send the encoded image to an output interface. In some embodiments, the source device 10 directly sends the encoded image to the destination device 20 via the output interface 140. In other embodiments, the encoded image may also be stored on the storage device 40 for later retrieval by the destination device 20 for decoding and / or display.

[0133] In the implementation environment shown in FIG5 , the destination device 20 includes an input interface 240, a decoder 200, and a display device 220. In some embodiments, the input interface 240 includes a receiver and / or a modem. The input interface 240 may receive encoded images via the link 30 and / or from the storage device 40, and then transmit the encoded images to the decoder 200. The decoder 200 may decode the received encoded images to obtain decoded images. The decoder may transmit the decoded images to the display device 220. The display device 220 may be integrated with the destination device 20 or may be external to the destination device 20. Generally, the display device 220 displays the decoded images. The display device 220 may be any of a variety of types of display devices, for example, the display device 220 may be a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.

[0134] Although not shown in FIG5 , in some aspects, the encoder 100 and decoder 200 can be integrated with an encoder and decoder, respectively, and can include appropriate multiplexer-demultiplexer (MUX-DEMUX) units or other hardware and software for encoding both audio and video in a common data stream or in separate data streams. In some embodiments, the MUX-DEMUX units can conform to the ITU H.223 multiplexer protocol, or other protocols such as the User Datagram Protocol (UDP), if applicable.

[0135] The encoder 100 and the decoder 200 can each be any of the following circuits: one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. If the technology of the embodiments of the present application is implemented in part by software, the device can store instructions for the software in a suitable non-volatile computer-readable storage medium, and can use one or more processors to execute the instructions in hardware to implement the technology of the embodiments of the present application. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) can be regarded as one or more processors. Each of the encoder 100 and the decoder 200 can be included in one or more encoders or decoders, and any of the encoders or decoders can be integrated as part of a combined encoder / decoder (encoder / decoder) in the corresponding device.

[0136] Embodiments of the present application may generally refer to encoder 100 as "signaling" or "sending" certain information to another device, such as decoder 200. The terms "signaling" or "sending" may generally refer to the transmission of syntax elements and / or other data used to decode a compressed image. This transmission may occur in real time or near real time. Alternatively, this communication may occur over time, such as when the syntax elements are stored in the encoded bitstream to a computer-readable storage medium during encoding, and the decoding device may then retrieve the syntax elements at any time after they are stored to this medium.

[0137] The encoding and decoding methods provided in the embodiments of the present application can be applied to a variety of scenarios. In various scenarios, the images encoded and decoded can be images included in image files or images included in video files. It should be noted that, in conjunction with the implementation environment shown in Figure 5, any of the encoding methods below can be executed by the encoder 100 in the source device 10, and the encoder 100 is implemented by software, hardware, or a combination of both to become part or all of the cloud platform in the embodiments of the present application. Any of the decoding methods below can be executed by the decoder 200 in the destination device 20, and the decoder is implemented by software, hardware, or a combination of both to become part or all of the client in the embodiments of the present application.

[0138] For example, a cloud platform can be a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or a cloud computing service center. The cloud platform can implement the encoding method provided in the embodiment of Figure 7 below.

[0139] Please refer to Figure 6, which is a schematic diagram of a client structure according to an embodiment of the present application. The client includes at least one processor 601, a communication bus 602, a memory 603, and at least one communication interface 604, and has an image rendering function.

[0140] The processor 601 may be a general-purpose central processing unit (CPU), a network processor (NP), a microprocessor, or one or more integrated circuits for implementing the solution of the present application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0141] Communication bus 602 is used to transmit information between the above components. Communication bus 602 can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or one type of bus.

[0142] The memory 603 may be a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compact disc, a laser disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 603 may exist independently and be connected to the processor 601 via the communication bus 602. The memory 603 may also be integrated with the processor 601.

[0143] The communication interface 604 uses any device such as a transceiver to communicate with other devices or communication networks. The communication interface 604 includes a wired communication interface and may also include a wireless communication interface. The wired communication interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless communication interface may be a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0144] In a specific implementation, as an embodiment, the processor 601 may include one or more CPUs, such as CPU0 and CPU1 shown in FIG6 .

[0145] In a specific implementation, as an example, the client may include multiple processors, such as processor 601 and processor 605 shown in FIG6 . Each of these processors may be a single-core processor or a multi-core processor. The processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0146] In a specific implementation, as an embodiment, the client may further include an output device 606 and an input device 607. The output device 606 communicates with the processor 601 and can display information in a variety of ways. For example, the output device 606 can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device 607 communicates with the processor 601 and can receive user input in a variety of ways. For example, the input device 607 can be a mouse, a keyboard, a touch screen device, or a sensor device.

[0147] In some embodiments, the memory 603 is used to store program code 610 for executing the solution of the present application, and the processor 601 can execute the program code 610 stored in the memory 603. The program code 610 may include one or more software modules, and the client can implement the decoding method provided in the embodiment of Figure 9 below through the processor 601 and the program code 610 in the memory 603.

[0148] FIG7 is a flow chart of an encoding method provided in an embodiment of the present application, wherein the method is applied to a cloud platform including multiple three-dimensional models. Referring to FIG7 , the method includes the following steps.

[0149] Step 701: Determine a first rendering intermediate result corresponding to a target area in a virtual viewing plane and a first illumination coefficient corresponding to each pixel in the target area, wherein a first visible area corresponding to the target area in a plurality of three-dimensional models has reflection and / or refraction characteristics, and the first rendering intermediate result indicates a second visible area in the plurality of three-dimensional models that is reflected and / or refracted by the first visible area, and the first illumination coefficient refers to an illumination coefficient of an intersection of a light ray emitted from a virtual viewpoint and passing through a corresponding pixel and the first visible area, and the virtual viewing plane is located between the virtual viewpoint and the plurality of three-dimensional models, and the first illumination coefficient indicates a degree of contribution of the second visible area to the display content of each pixel in the target area.

[0150] It should be noted that the visible surface elements corresponding to the pixels in the target area include the first surface element, and the area composed of the first surface elements corresponding to the pixels in the target area is the first visible area. The first visible area has reflection and / or refraction characteristics, or in other words, the first surface element has reflection and / or refraction characteristics.

[0151] Optionally, the first illumination coefficient indicates the reflectivity and / or refractive index of the first visible area. A greater reflectivity indicates that more light incident on the first visible area can be reflected to the second visible area, and therefore, the second visible area contributes more to the display content of the pixel. Similarly, a greater refractive index indicates that more light incident on the first visible area can be refracted to the second visible area, and therefore, the second visible area contributes more to the display content of the pixel.

[0152] For example, please refer to Figure 8, which is a schematic diagram of a virtual viewing plane provided in an embodiment of the present application. The cloud platform includes multiple three-dimensional models of a house, a tree, a river, and stones under the river surface (not shown in Figure 8). Since the surface of the river has the characteristics of refraction and reflection, the first visible area corresponding to the virtual viewing plane is the surface of the river. Please refer to the thicker dotted box in the virtual viewing plane in Figure 8. The dotted box represents the target area. The surface of the river corresponds to the target area in the virtual viewing plane, or in other words, the projection area of ​​the target area in the virtual viewing plane on the multiple three-dimensional models is the surface of the river.

[0153] In practical applications, before determining the first intermediate rendering result corresponding to the target area in the virtual viewing plane and the first illumination coefficient corresponding to each pixel in the target area, the cloud platform can receive an image rendering instruction, wherein the rendering instruction includes the pose of the virtual viewpoint (or the camera pose). Based on the camera pose and the multiple three-dimensional models, the cloud platform determines the first intermediate rendering result corresponding to the target area in the virtual viewing plane and the first illumination coefficient corresponding to each pixel in the target area.

[0154] In some embodiments, a user changes the camera pose by interacting with the client. In this case, the rendering instruction includes the user's interactive operation. The cloud platform determines the camera pose based on the user's interactive operation, and then determines a first intermediate rendering result corresponding to the target area in the virtual viewing plane and a first illumination coefficient corresponding to each pixel in the target area based on the camera pose and the multiple three-dimensional models.

[0155] It should be noted that the interactive operation can be triggered by the user through voice, click, sliding and other actions. The interactive operation is used to change the camera posture, so that the client presents images of the multiple three-dimensional models in different camera postures.

[0156] Based on the above description, the first visible area has reflective and / or refractive characteristics. Therefore, in actual applications, the first rendering intermediate result includes at least one of the reflection results of each pixel in the target area and the refraction results of each pixel in the target area. The cloud platform determines the reflection results and the refraction results based on the camera posture and the multiple three-dimensional models. The implementation methods are different, which will be introduced separately below.

[0157] When the first intermediate rendering result includes the reflection results of each pixel in the target area, the cloud platform determines the first bin corresponding to each pixel based on the camera pose and the multiple 3D models according to the relevant algorithm. For any pixel in the target area, a mirror camera is placed below the first bin corresponding to the pixel. Based on the perspective of the mirror camera, the multiple 3D models are raster sampled to obtain the reflection result corresponding to the pixel. Each pixel in the target area is processed in the same manner to obtain the reflection result of each pixel in the target area.

[0158] When the first rendering intermediate result includes the refraction results for each pixel in the target area, for any pixel in the target area, the cloud platform performs raster sampling on the multiple 3D models based on the camera pose to obtain the refraction result corresponding to the pixel. Each pixel in the target area is processed in the same manner to obtain the refraction result for each pixel in the target area.

[0159] In actual applications, the cloud platform may also use a screen space reflection (SSR) algorithm to determine the reflection and refraction results based on the camera pose and the multiple three-dimensional models, which is not limited in the embodiments of the present application.

[0160] Optionally, the reflection result includes at least one of a reflection map and reflection vertex data, and the refraction result includes at least one of a refraction map and refraction vertex data. The reflection map and the refraction map may be collectively referred to as map data, and the reflection vertex data and the refraction vertex data may be collectively referred to as vertex data.

[0161] Optionally, the vertex data is the face element number of the second face element, or the vertex number of the second face element, or the coordinates of the visible point in the multiple three-dimensional models, the second face element is the face element corresponding to the corresponding pixel in the second visible area, and the visible point is the point in the second face element corresponding to the corresponding pixel.

[0162] In some embodiments, a normal vector of a first bin corresponding to each pixel in the target area is determined. For any pixel in the target area, a first illumination coefficient corresponding to the pixel is determined based on the normal vector of the first bin corresponding to the pixel and the camera pose according to a relevant algorithm. Each pixel in the target area is processed in the same manner to obtain the first illumination coefficient corresponding to each pixel in the target area.

[0163] In actual applications, there are many ways for the cloud platform to determine the normal vector of the first surface element corresponding to each pixel in the target area. Three of the implementation methods will be introduced in detail below.

[0164] The first implementation method is that for each pixel in the target area, if a normal map exists for the first surface element corresponding to the pixel, the normal vector of the first surface element distorted by the normal map is determined as the normal vector of the first surface element; if a normal map does not exist for the first surface element corresponding to the pixel, the original normal vector of the first surface element is determined as the normal vector of the first surface element.

[0165] In practical applications, to achieve more realistic rendering of bins, a 3D model may contain bins with normal maps. This normal map can render the bins with more texture detail. Therefore, when determining the normal vector of a first bin, the normal vector of the first bin, distorted by the normal map, can be used as the normal vector of the first bin. This improves the accuracy of the subsequent illumination coefficient determination. For first bins without normal maps, the original normal vector of the first bin (i.e., the first bin's own normal vector) can be used as the normal vector of the first bin.

[0166] In a second implementation, for each pixel in the target area, the original normal vector of the first surface element is directly determined as the normal vector of the first surface element.

[0167] The second implementation differs from the first in that it directly uses the original normal vector of the first panel as the normal vector of the first panel, without considering the normal map of the first panel. In other words, even if a normal map exists for the first panel, the second implementation directly uses the original normal vector of the first panel as the normal vector of the first panel, thereby significantly reducing the computational effort.

[0168] In a third implementation, the target region includes at least one target subregion, each target subregion including at least one pixel. When the target subregion includes multiple pixels, at least one pixel in each target subregion has at least one pixel adjacent to the pixel. In this case, for each target subregion, a fixed value of the normal vector corresponding to the target subregion is determined based on the first bin corresponding to at least one pixel in the target subregion. This fixed value of the normal vector corresponding to the target subregion is then determined as the normal vector of the first bin for each pixel in the target subregion. Each target subregion in the target region is processed in the same manner to obtain the normal vector of the first bin corresponding to each pixel in the target region.

[0169] If any pixel in the target sub-region has at least one pixel adjacent to the pixel, it can be understood that the positions of the pixels in the target sub-region are continuous. In some embodiments, the position of each pixel in the virtual viewing plane can be represented by a two-dimensional coordinate indicating the position of the pixel in the x-axis direction and the position in the y-axis direction. In this case, the above-mentioned continuous position means that the positions of the pixels in the target sub-region are continuous in the x-axis direction and / or the positions of the pixels in the target sub-region are continuous in the y-axis direction.

[0170] There are multiple ways to determine the fixed value of the normal vector corresponding to the target sub-region based on the first bin corresponding to at least one pixel in the target sub-region, two of which are described below.

[0171] The first method is to determine the surface composed of the at least one first surface element based on the at least one first surface element, determine the normal vector corresponding to the tangent plane of the surface, and determine the normal vector corresponding to the tangent plane as the fixed value of the normal vector corresponding to the target sub-area.

[0172] The second method is to determine a first normal vector of the at least one first surface element, and determine a fixed value of a normal vector corresponding to the target sub-region based on an average value of the at least one first normal vector.

[0173] For each pixel in the target area, if a normal map exists for the first surface element corresponding to the pixel, the normal vector of the first surface element distorted by the normal map is determined as the first normal vector of the first surface element; if a normal map does not exist for the first surface element corresponding to the pixel, the original normal vector of the first surface element is determined as the first normal vector of the first surface element.

[0174] In some embodiments, the average value of the at least one first normal vector can be determined as the fixed normal value corresponding to the target sub-region. In practical applications, the fixed normal value corresponding to the target sub-region can also be determined by other means. For example, any one of the at least one first normal vector can be used as the fixed normal value corresponding to the target sub-region. This embodiment of the present application is not limited to this.

[0175] For the sake of convenience of description, the normal vector of the first surface element determined by the first implementation method can be referred to as the high-quality normal vector of the first surface element, the normal vector of the first surface element determined by the second implementation method can be referred to as the normal vector of the first surface element without distortion by the normal map, and the normal vector of the first surface element determined by the third implementation method can be referred to as the fixed value of the normal vector of the first surface element.

[0176] Based on the above description, the first rendering intermediate result includes at least one of the reflection results of each pixel in the target area and the refraction results of each pixel in the target area. In this case, the first illumination coefficient also includes at least one of the reflection illumination coefficient and the refraction illumination coefficient. When the first rendering intermediate result includes the reflection results of each pixel in the target area, the first illumination coefficient includes the reflection illumination coefficient, which indicates the degree of contribution of the reflection result of the pixel to the display content of the pixel, or in other words, the reflection illumination coefficient indicates the reflectivity of the first surface element corresponding to the pixel.

[0177] When the first rendering intermediate result includes the refraction results of each pixel in the target area, the first illumination coefficient includes a refraction illumination coefficient, which indicates the contribution degree of the refraction result of the pixel to the display content of the pixel, or in other words, the refraction illumination coefficient indicates the refractive index of the first surface element corresponding to the pixel.

[0178] It should be noted that for any first bin, the sum of the refractive index and reflectivity of the first bin is 1. The larger the reflection illumination coefficient, the greater the contribution of the pixel's reflection result to the pixel's display content. Similarly, the larger the refraction illumination coefficient, the greater the contribution of the pixel's refraction result to the pixel's display content. In other words, the first illumination coefficient is proportional to the contribution.

[0179] In actual applications, before determining the first rendering intermediate result corresponding to the target area in the virtual viewing plane and the first illumination coefficient corresponding to each pixel in the target area, the cloud platform can also obtain first information, which includes end-side device information and / or attribute information of the target area, the end-side device information indicates the performance of the client, and the attribute information indicates the reflection and / or refraction properties of the target area.

[0180] Optionally, the end-side device information indicates whether the client supports the encoding and decoding method provided in the embodiment of the present application, the method for determining the illumination coefficient supported by the client, the method for determining the second rendering intermediate result supported by the client, the client's frame rate requirements, the client's latency requirements, and at least one of the client's power consumption.

[0181] Methods for determining the illumination coefficient include: using the high-quality normal vector of the first bin, using the normal vector of the first bin without normal map distortion, and using a fixed normal vector of the first bin. There are multiple methods for determining the second rendering intermediate result, which will be discussed in detail later and are not discussed here.

[0182] If the first information includes terminal-side device information, the client can send its own device information to the cloud platform before the cloud platform obtains the terminal-side device information, and the cloud platform can receive the client's device information (ie, terminal-side device information).

[0183] If the first information includes attribute information of the target area, the attribute information of the target area includes the wave coefficient corresponding to the target area and at least one of the first indication information, and the first indication information indicates the method for determining the second rendering intermediate result, the cloud platform needs to determine the attribute information of the target area before obtaining the attribute information of the target area.

[0184] When the first face element corresponding to the target area is a water surface, the attribute information of the target area includes the wave coefficient corresponding to the target area. At this time, the wave coefficient of the water surface is stored in the cloud platform. At this time, the cloud platform can directly determine the wave coefficient of the water surface as the wave coefficient corresponding to the target area. The wave coefficient indicates the degree of fluctuation of the water surface. The larger the wave coefficient, the greater the degree of fluctuation of the water surface.

[0185] If the attribute information of the target area includes the first indication information, in this case, the cloud platform can determine the first indication information in a variety of ways, two of which are described below.

[0186] In the first implementation method, preset information corresponding to at least one surface element in the three-dimensional model is stored in the cloud platform, and the preset information indicates a method for determining the second rendering intermediate result. The cloud platform can determine whether there is corresponding preset information for the first surface element corresponding to the target area. If the proportion of the candidate surface element in the first surface element of the target area is greater than a first proportion threshold, the first indication information is determined based on the preset information of the candidate surface element in the first surface element. The candidate surface element is a surface element that has corresponding preset information in the first surface element corresponding to the target area.

[0187] In some embodiments, the preset information indicates whether to multiply the texture data corresponding to each pixel in the target area by the first illumination coefficient corresponding to each pixel to obtain the second intermediate rendering result. If the proportion of candidate bins with the same preset information in the preset information of the candidate bins is greater than a second ratio threshold, the preset information of the candidate bins with the same preset information is used as the first indication information. If the proportion of candidate bins with the same preset information in the preset information of the candidate bins is not greater than the second ratio threshold, it is determined that the first indication information indicates not to multiply the texture data corresponding to each pixel in the target area by the first illumination coefficient corresponding to each pixel to obtain the second intermediate rendering result.

[0188] The first ratio threshold and the second ratio threshold are set in advance. For example, the first ratio threshold can be set to 80 percent, and the second ratio threshold can be set to 90 percent. They can be adjusted according to different situations.

[0189] It should be noted that multiplying the map data corresponding to each pixel in the target area with the corresponding first illumination coefficient to obtain the second rendering intermediate result is one of the multiple methods of determining the above-mentioned second rendering intermediate result. The detailed implementation process of this method will be introduced in detail later and will not be elaborated here.

[0190] In a second implementation, the cloud platform determines a first illumination coefficient for each pixel in the target area to obtain a plurality of first illumination coefficients, and determines frequency domain characteristics of the first illumination coefficients of the plurality of pixels according to a related frequency domain algorithm. The frequency domain characteristics indicate whether the image corresponding to the target area has a lot of high-frequency information. If it is determined that the image corresponding to the target area has a lot of high-frequency information, the first indication information is determined to indicate that the texture data corresponding to each pixel in the target area is not multiplied by the respective first illumination coefficients to obtain a second rendering intermediate result. Otherwise, the first indication information is determined to indicate that the texture data corresponding to each pixel in the target area is not multiplied by the respective first illumination coefficients to obtain a second rendering intermediate result.

[0191] The implementation method of determining the first illumination coefficient of each pixel in the target area has been described in detail above and will not be repeated here.

[0192] Since the frequency domain feature indicates whether the image corresponding to the target area has more high-frequency information, when it is determined that the image corresponding to the target area has more high-frequency information, the map data corresponding to each pixel in the target area is multiplied with the respective corresponding first illumination coefficients to obtain a second rendering intermediate result, and the efficiency of compressing the second rendering intermediate result is low. Therefore, when it is determined that the image corresponding to the target area has more high-frequency information, it is determined not to multiply the map data corresponding to each pixel in the target area with the respective corresponding first illumination coefficients to obtain the second rendering intermediate result.

[0193] It should be noted that, in the above description of the implementation method for obtaining the attribute information of the target area, the cloud platform also needs to determine the attribute information of the target area. However, in actual applications, the execution entity of determining the attribute information of the target area can also be the client, and the embodiments of the present application do not limit this. If the execution entity of determining the attribute information of the target area is the client, then after determining the attribute information of the target area, the client can send the attribute information of the target area to the cloud platform, so that the cloud platform can obtain the attribute information of the target area.

[0194] In some embodiments, rendering indication information is determined based on the first information, and the rendering indication information includes cloud-side indication information, and the cloud-side indication information indicates a method for determining the first rendering intermediate result and the first lighting coefficient, and / or a method for determining the second rendering intermediate result.

[0195] If the first information includes any one of the wave coefficient corresponding to the target area and the first indication information corresponding to the target area, the first rendering intermediate result and the first illumination coefficient may be determined in any of the above-described ways.

[0196] If the first information includes a wave coefficient corresponding to the target area, and the wave coefficient is greater than the wave coefficient threshold, then the method for processing the first intermediate rendering result is determined not to be multiplying the texture data corresponding to each pixel in the target area by the corresponding first illumination coefficient to obtain the second intermediate rendering result. If the wave coefficient is not greater than the wave coefficient threshold, then the method for determining the second intermediate rendering result can be any of the methods provided below.

[0197] Among them, the wave coefficient threshold is set in advance and can be modified according to needs in different situations.

[0198] If the first information includes first indication information corresponding to the target area, and the first indication information indicates that the texture data corresponding to each pixel in the target area should not be multiplied by the first illumination coefficient corresponding to each pixel to obtain the second rendering intermediate result, the second rendering intermediate result is determined not by multiplying the texture data corresponding to each pixel in the target area by the first illumination coefficient corresponding to each pixel to obtain the second rendering intermediate result. If the first indication information indicates that the texture data corresponding to each pixel in the target area should be multiplied by the first illumination coefficient corresponding to each pixel to obtain the second rendering intermediate result, the second rendering intermediate result may be determined by multiplying the texture data corresponding to each pixel in the target area by the first illumination coefficient corresponding to each pixel to obtain the second rendering intermediate result, or by any one of the multiple methods for determining the second rendering intermediate result provided subsequently.

[0199] Optionally, the client's frame rate requirement, the client's latency requirement, and the client's power consumption are used to indicate the client's performance requirements for image rendering. Among them, the higher the client's frame rate requirement, the higher the rate at which the client expects rendered images to be generated per second, and the higher the client's latency requirement, the faster the client expects the rendered images to respond. In this case, the method of the embodiment of the present application can be used to reduce the bit rate transmitted between the cloud platform and the client, thereby effectively improving the stability of the rendered image. The lower the client's power consumption, the less electrical energy the client consumes, which generally means that the device has higher energy efficiency and excellent performance. Therefore, the method of the embodiment of the present application can be used to perform image rendering on the client.

[0200] It should be noted that the cloud-side instruction information is used to instruct the cloud platform on which specific encoding method to use.

[0201] In some embodiments, the rendering indication information also includes end-side indication information, which indicates at least one of the following: whether the second rendering intermediate result is the target rendering intermediate result, the method for determining the first illumination coefficient, the method for determining the second illumination coefficient, the method for determining the second rendering intermediate result, and the method for dividing multiple blocks. The target rendering intermediate result refers to the rendering intermediate result that the client does not need to restore and can be directly used for rendering.

[0202] The division method of the multiple blocks indicates the size of each block. The details will be introduced later and will not be explained here.

[0203] Among them, the target rendering intermediate result refers to the cloud platform using the first implementation method to determine the normal vector of the first surface element corresponding to each pixel in the target area, and subsequently processing the first rendering intermediate result based on the first illumination coefficient corresponding to each pixel in the target area, and using the second rendering intermediate result obtained by the first implementation method.

[0204] Since the embodiments of the present application provide multiple methods for determining the first illumination coefficient, the second illumination coefficient, the second intermediate rendering result, and the multiple block division methods, the terminal-side indication information indicates the specific method used by the cloud platform during the encoding process, or indicates the specific method used by the client during the decoding process. In other words, the terminal-side indication information is used to guide the subsequent client decoding process.

[0205] Based on the above description, the first rendering intermediate result includes at least one of the reflection result of each pixel in the target area and the refraction result of each pixel in the target area. In this case, the end-side indication information is also used to indicate whether the second rendering intermediate result includes a reflection result and whether it includes a refraction result.

[0206] For example, when the second rendering intermediate result includes a reflection result, the field in the terminal-side indication information indicating whether the second rendering intermediate result includes a reflection result may be 1. When the second rendering intermediate result does not include a reflection result, the field in the terminal-side indication information indicating whether the second rendering intermediate result includes a reflection result may be 0. When the second rendering intermediate result includes a refraction result, the field in the terminal-side indication information indicating whether the second rendering intermediate result includes a refraction result may be 1. When the second rendering intermediate result does not include a refraction result, the field in the terminal-side indication information indicating whether the second rendering intermediate result includes a refraction result may be 0.

[0207] Optionally, if the second intermediate rendering result includes a reflection result, the terminal-side indication information is further used to indicate the type of the reflection result, which includes a reflection map and reflection vertex data. If the second intermediate rendering result includes a refraction result, the terminal-side indication information is further used to indicate the type of the refraction result, which includes a refraction map and refraction vertex data.

[0208] For example, when the reflection result type is a reflection map, the field indicating the type of the reflection result in the end-side indication information may be 1. When the reflection result type is reflection vertex data, the field indicating the type of the reflection result in the end-side indication information may be 0. When the refraction result type is a refraction map, the field indicating the type of the refraction result in the end-side indication information may be 1. When the refraction result type is refraction vertex data, the field indicating the type of the refraction result in the end-side indication information may be 0.

[0209] Optionally, the terminal-side indication information is further used to indicate a method for determining the first intermediate rendering result.

[0210] For example, when the method for determining the first rendering intermediate result is the SSR algorithm, the field for indicating the method for determining the first rendering intermediate result in the terminal-side indication information may be 1. When the first rendering intermediate result includes the reflection result of each pixel in the target area, and the method for determining the reflection result of each pixel in the target area is to place a mirror camera, the field for indicating the method for determining the first rendering intermediate result in the terminal-side indication information may be 0.

[0211] Optionally, in the case where the first rendering intermediate result includes vertex data of each pixel in the target area, since the type of the vertex data can be the face number of the second face, the vertex number of the second face, or any one of the coordinates of the visible point in the multiple three-dimensional models, the end-side indication information is also used to indicate the type of the vertex data.

[0212] For example, when the type of vertex data is the bin number of the second bin, the field used to indicate the type of the vertex data in the terminal-side indication information may be 00. When the type of vertex data is the vertex number of the second bin, the field used to indicate the type of the vertex data in the terminal-side indication information may be 01. When the type of vertex data is the coordinates of the visible point in the multiple three-dimensional models, the field used to indicate the type of the vertex data in the terminal-side indication information may be 02.

[0213] Optionally, after the cloud platform determines the terminal-side indication information, the cloud platform can send the terminal-side indication information to the client.

[0214] Step 702: Based on the first illumination coefficient corresponding to each pixel in the target area, the first rendering intermediate result is processed to obtain a second rendering intermediate result, and the number of bits occupied by the second rendering intermediate result during encoding is less than the number of bits occupied by the first rendering intermediate result during encoding.

[0215] Since the first illumination coefficient is usually a number less than 1, in this case, the first rendering intermediate result is processed based on the first illumination coefficient corresponding to each pixel in the target area, and the data bit amount of the obtained second rendering intermediate result is smaller than the data bit amount of the first rendering intermediate result. In this way, during encoding, the number of bits occupied by the second rendering intermediate result is smaller than the number of bits occupied by the first rendering intermediate result, thereby achieving effective compression of the first rendering intermediate result and further reducing the bit rate transmitted between the cloud platform and the client.

[0216] There are multiple implementation methods for processing the first rendering intermediate result based on the first illumination coefficient corresponding to each pixel in the target area to obtain the second rendering intermediate result. Two of the implementation methods are introduced below.

[0217] In a first implementation, when the first rendering intermediate result includes the mapping data of each pixel in the target area, the mapping data corresponding to each pixel in the target area is multiplied by the first illumination coefficient corresponding to each pixel to obtain the second rendering intermediate result.

[0218] If the map data includes a reflection map, the first illumination coefficient includes a reflection illumination coefficient. At this time, the reflection map corresponding to each pixel in the target area is multiplied by the corresponding reflection illumination coefficient to obtain the reflection map corresponding to each pixel after multiplication (for the sake of convenience of description, the reflection map after multiplication is called the second reflection map), and the second reflection map corresponding to each pixel is used as the second rendering intermediate result.

[0219] Since the reflection illumination coefficient is typically less than 1, in this case, for any pixel in the target area, the value obtained by multiplying the number less than 1 by the pixel's texture data is less than the original texture data of the pixel, thereby reducing the number of data bits in the reflection map of the pixel. Furthermore, since data encoding typically requires quantization, the quantized data will suffer from data loss compared to the pre-quantized data. For the same reflection map, if different reflection illumination coefficients are used to process the reflection map, the data loss after quantization will be different. A second reflection map is obtained after processing with a larger reflection illumination coefficient, while a second reflection map is obtained after processing with a smaller reflection illumination coefficient. Thus, the second reflection map obtained after processing with the smaller reflection illumination coefficient is closer to the quantization step size, resulting in a greater data loss for the second reflection map obtained after processing with the smaller reflection illumination coefficient. In other words, the larger the reflection illumination coefficient, the less data loss there is for the reflection map. This results in less data loss for reflection maps with a greater contribution, thereby reducing the loss of valid data while ensuring effective data compression.

[0220] For example, if the target area includes two pixels, pixel 1 and pixel 2, the reflection illumination coefficient 1 corresponding to pixel 1 is 0.8, the reflection illumination coefficient 2 corresponding to pixel 2 is 0.2, and the reflection map 1 corresponding to pixel 1 and the reflection map 2 corresponding to pixel 2 have the same data, that is, both reflection map 1 and reflection map 2 are (200, 0, 0). In this case, the second reflection map 1 obtained by multiplying the reflection map 1 of pixel 1 by the reflection illumination coefficient 1 is (160, 0, 0), and the second reflection map 2 obtained by multiplying the reflection map 2 of pixel 2 by the reflection illumination coefficient 2 is (40, 0, 0).

[0221] If the quantization step size is 32, for the second reflection map 2, since 160 can be directly divided by 32, the quotient after divisibility is 5, therefore, the data of the second reflection map 1 after quantization is (5, 0, 0), and the data of the second reflection map 1 after dequantization after quantization is (160, 0, 0), so there is no data loss. For the second reflection map 2, since the quotient after 40 divided by 32 is 1 and the remainder is 8, the data of the second reflection map 2 after quantization is (1, 0, 0), and the data of the second reflection map 2 after dequantization after quantization is (32, 0, 0), and the data after dequantization after quantization is data loss compared to the second reflection map 2. In this way, it is not difficult to see that for the same reflection map, the larger the reflection illumination coefficient is, the smaller the data loss of the reflection map is, and the smaller the reflection illumination coefficient is, the greater the data loss of the reflection map is.

[0222] If the map data includes a refraction map, the first illumination coefficient includes a refraction illumination coefficient. At this time, the refraction map corresponding to each pixel in the target area is multiplied by the corresponding refraction illumination coefficient to obtain the refraction map corresponding to each pixel after multiplication (for the sake of convenience of description, the refraction map after multiplication is called the second refraction map), and the second refraction map corresponding to each pixel is used as the second rendering intermediate result.

[0223] Since the refraction illumination coefficient is typically less than 1, in this case, for any pixel in the target area, the value obtained by multiplying the pixel's map data by the number less than 1 is less than the original map data for that pixel, thereby reducing the number of data bits in the refraction map for that pixel. Furthermore, since data encoding typically requires quantization, the quantized data suffers from data loss compared to the pre-quantized data. For the same refraction map, processing it with different refraction illumination coefficients results in different data loss after quantization. A second refraction map is obtained after processing with a larger refraction illumination coefficient, while a second refraction map is obtained after processing with a smaller refraction illumination coefficient. Thus, the second refraction map obtained with the smaller refraction illumination coefficient is closer to the quantization step size, resulting in a greater data loss for the second refraction map obtained with the smaller refraction illumination coefficient. In other words, the larger the refraction illumination coefficient, the less data loss there is in the refraction map. This results in less data loss for refraction maps with a greater contribution, thereby reducing the loss of valid data while ensuring effective data compression.

[0224] The second implementation method is to divide the target area into multiple blocks, each block includes multiple pixels, and based on the first illumination coefficient corresponding to each pixel in the multiple blocks, determine the second illumination coefficient corresponding to the multiple blocks respectively; based on the second illumination coefficient corresponding to the multiple blocks respectively, process the first rendering intermediate result to obtain the second rendering intermediate result.

[0225] It should be noted that the size of each block is consistent, and the size of the block may be an integer multiple of the size of a coding unit (CU) for image coding.

[0226] Since the first rendering intermediate result includes at least one of the reflection result and the refraction result of each pixel in the target area, the first illumination coefficient also includes at least one of the reflection illumination coefficient and the refraction illumination coefficient. If the first rendering intermediate result includes the reflection result of each pixel in the target area, the first illumination coefficient includes the reflection illumination coefficient. In this case, based on the reflection illumination coefficient corresponding to each pixel in the multiple blocks, the second illumination coefficient corresponding to each of the multiple blocks is determined. Based on the second illumination coefficient corresponding to each of the multiple blocks, the reflection result of each pixel in the target area is processed to obtain the second rendering intermediate result. If the first rendering intermediate result includes the refraction result of each pixel in the target area, the first illumination coefficient includes the refraction illumination coefficient. In this case, based on the refraction illumination coefficient corresponding to each pixel in the multiple blocks, the second illumination coefficient corresponding to each of the multiple blocks is determined. Based on the second illumination coefficient corresponding to each of the multiple blocks, the refraction result of each pixel in the target area is processed to obtain the second rendering intermediate result.

[0227] For the sake of convenience of description, the following will be introduced by taking the example that the first rendering intermediate result includes the reflection result of each pixel in the target area, and the first illumination coefficient includes the reflection illumination coefficient. The case where the first rendering intermediate result includes the refraction result of each pixel in the target area, and the first illumination coefficient includes the refraction illumination coefficient is similar to the case where the first rendering intermediate result includes the reflection result of each pixel in the target area, and the first illumination coefficient includes the reflection illumination coefficient, so it will not be repeated.

[0228] Optionally, for each block in the pair of blocks, an average value, a maximum value or a minimum value of the reflection illumination coefficient corresponding to each pixel in the block is used as the second illumination coefficient corresponding to the block.

[0229] In some embodiments, the first rendering intermediate result includes the mapping data of each pixel in the target area. In this case, the cloud platform processes the first rendering intermediate result based on the second illumination coefficients corresponding to multiple blocks respectively, and determines that there are multiple implementation methods for determining the second rendering intermediate result. Two of the implementation methods are introduced below.

[0230] The first method is to multiply the second illumination coefficient corresponding to each block in the pair of blocks by the reflection map of the pixels included in the block to obtain a second rendering intermediate result.

[0231] The second method is to determine the convolution kernel sizes corresponding to the multiple blocks based on the second illumination coefficients corresponding to the multiple blocks, and perform convolution operations on the map data of the pixels in the multiple blocks based on the convolution kernel sizes corresponding to the multiple blocks to obtain the second rendering intermediate result.

[0232] It should be noted that since the first illumination coefficient indicates the contribution of the second visible area to the display content of each pixel in the target area, a larger second illumination coefficient indicates a greater contribution of the second visible area in the block to the display content of each pixel in the block. When performing convolution on the block, it is necessary to reduce the data loss caused by the convolution, so that the data loss of the texture data with a large contribution is minimized. Therefore, the size of the convolution kernel is inversely proportional to the second illumination coefficient: the larger the second illumination coefficient, the smaller the size of the convolution kernel.

[0233] Among them, the template of the convolution kernel can be a Gaussian filter template, a mean filter template, a median filter module, a maximum filter template, a minimum filter template, etc., and the embodiment of the present application does not limit this.

[0234] In some embodiments, the first rendering intermediate result includes vertex data of each pixel in the target area. In this case, the cloud platform processes the first rendering intermediate result based on the second illumination coefficients corresponding to multiple blocks. There are multiple implementation methods to obtain the second rendering intermediate result, and two of them are introduced below.

[0235] The first method is to determine the filtering radius corresponding to the multiple blocks based on the second illumination coefficients corresponding to the multiple blocks, and filter the vertex data of the pixels in the multiple blocks based on the filtering radius corresponding to the multiple blocks to obtain a second rendering intermediate result.

[0236] Similarly, since the first illumination coefficient indicates the contribution of the second visible area to the display content of each pixel in the target area, a larger second illumination coefficient indicates a greater contribution of the second visible area in the block to the display content of each pixel in the block. When performing convolution on the block, it is necessary to reduce the data loss caused by the convolution, so that the loss of vertex data with a large contribution is minimized. Therefore, the filter radius is inversely proportional to the second illumination coefficient: the larger the second illumination coefficient, the smaller the filter radius.

[0237] For each of the multiple blocks, based on the vertex data of the pixels in the block, the first positions corresponding to the vertex data of each pixel in the block are determined, and based on the first position of each pixel and the filtering radius corresponding to the block, radius filtering is performed on the first positions of the pixels in the block, and the vertex data corresponding to the first position after radius filtering is determined as the second rendering intermediate result.

[0238] When the vertex data is the face number of the second face, or the vertex number of the second face, for the vertex data of each pixel in the block, the position of the center of gravity of the second face of the pixel or the position of any point on the second face can be determined as the first position corresponding to the vertex data.

[0239] When the vertex data is the coordinates of the visible point in the multiple three-dimensional models, for the vertex data of each pixel in the block, the coordinates of the pixel visible point in the multiple three-dimensional models can be directly determined as the first position corresponding to the vertex data.

[0240] The second method is to determine the data filtering ratios corresponding to the multiple blocks based on the second illumination coefficients corresponding to the multiple blocks, and the data filtering ratios indicate the filtering status of the vertex data of the pixels in the blocks. Based on the data filtering ratios corresponding to the multiple blocks, the vertex data of the pixels in the multiple blocks are filtered to obtain a second rendering intermediate result.

[0241] It should be noted that the data screening ratio can be a data deletion ratio or a data retention ratio, and the embodiment of the present application does not limit this.

[0242] Similarly to the above, since the first illumination coefficient indicates the contribution of the second visible area to the display content of each pixel in the target area, the larger the second illumination coefficient is, the greater the contribution of the second visible area in the block to the display content of each pixel in the block is. When performing convolution on the block, it is necessary to reduce the data loss caused by the convolution so that the data loss of vertex data with a large contribution is smaller. Therefore, in the case where the data screening ratio is the data deletion ratio, the data deletion ratio is inversely proportional to the second illumination coefficient. The larger the second illumination coefficient is, the smaller the data deletion ratio is. In the case where the data screening ratio is the data retention ratio, the data retention ratio is proportional to the second illumination coefficient. The larger the second illumination coefficient is, the larger the data retention ratio is.

[0243] When the data screening ratio is the data deletion ratio, for each of the multiple blocks, the vertex data of the pixels in the block are deleted, and the ratio of the deleted vertex data to all the vertex data of the block is the data deletion ratio. The vertex data of the pixels in each block after the data deletion is determined as the second rendering intermediate result.

[0244] When the data screening ratio is the data retention ratio, for each of the multiple blocks, the vertex data of the pixels in the block are deleted, and the ratio of the retained vertex data to all the vertex data of the block is the data retention ratio. The vertex data of the pixels in each block after the data deletion is determined as the second rendering intermediate result.

[0245] Step 703: Encode the second rendering intermediate result into the bitstream.

[0246] Based on the above description, the first intermediate rendering result includes the texture data for each pixel in the target area. The first intermediate rendering result may also include the vertex data for each pixel in the target area. Therefore, when the first intermediate rendering result includes the texture data for each pixel in the target area, the second intermediate rendering result includes the texture data for each pixel in the target area. When the first intermediate rendering result includes the vertex data for each pixel in the target area, the second intermediate rendering result includes the vertex data for some pixels in the target area. In different situations, the method of encoding the second intermediate rendering result into the codestream varies, and each will be described below.

[0247] When the second rendering intermediate result includes the texture data of each pixel in the target area, the texture data of each pixel can be spliced ​​according to the position of each pixel, and then the spliced ​​texture data can be encoded into the bitstream according to the relevant image or video encoding algorithm.

[0248] It should be noted that, since the map data may include a reflection map and a refraction map, the reflection map corresponding to each pixel in the second rendering intermediate result and the refraction map corresponding to each pixel in the target area may be spliced ​​together to obtain a spliced ​​reflection map and a spliced ​​refraction map, and then, according to a relevant image or video encoding algorithm, the spliced ​​reflection map and the spliced ​​refraction map may be encoded respectively to encode the spliced ​​reflection map and the spliced ​​refraction map into a bitstream. Of course, in actual applications, after obtaining the spliced ​​reflection map and the spliced ​​refraction map, the spliced ​​reflection map and the spliced ​​refraction map may be spliced ​​together again to obtain a spliced ​​total map, and then, according to a relevant image encoding algorithm, the spliced ​​total map may be encoded into a bitstream, but this embodiment of the present application does not limit this.

[0249] When the second rendering intermediate result includes vertex data of each pixel in the target area, the vertex data of each pixel in the target area can be encoded into the code stream according to a relevant encoding algorithm.

[0250] Based on the above description, in some embodiments, the cloud platform can send the terminal-side indication information to the client. In this case, the cloud platform can encode the terminal-side indication information into a bitstream and send it to the client.

[0251] It should be noted that the code stream encoded with the terminal-side indication information and the code stream encoded with the second rendering intermediate result can be the same code stream or different code streams, and this embodiment of the present application does not limit this.

[0252] Since determining the refraction / reflection data of the target area requires high device performance, the embodiment of the present application can determine the first rendering intermediate result corresponding to the target area in the virtual visual plane and the first illumination coefficient corresponding to each pixel in the target area through the cloud platform, and then process the first rendering intermediate result based on the first illumination coefficient corresponding to each pixel to obtain a second rendering intermediate result. Since the first rendering intermediate result indicates the second visible area reflected and / or refracted by the first visible area in the multiple three-dimensional models, the first rendering intermediate result can indicate the reflection and / or refraction data of the first visible area. Compared with the related technology that the cloud platform needs to render the entire image, the cloud platform in the embodiment of the present application only needs to determine the reflection and / or refraction data and the first illumination coefficient of the first visible area, which can reduce the occupancy of cloud computing resources. The cloud platform can also encode the second rendering intermediate result into the code stream. Compared with the related technology that requires the entire rendered image to be encoded into the code stream, the embodiment of the present application does not need to encode the entire rendered image into the code stream, which can ensure that the code rate transmitted between the cloud platform and the client is relatively low. Since the number of bits occupied by the second rendering intermediate result during encoding is less than the number of bits occupied by the first rendering intermediate result during encoding, the bit rate of transmission between the cloud platform and the client can be further reduced. In addition, since quantization is usually required when encoding data, the quantized data will have data loss compared to the data before quantization. For the same texture data, if different illumination coefficients are used to process the texture data, the data loss of the texture data after quantization will be different. The texture data obtained after processing with a larger illumination coefficient is greater than the texture data obtained after processing with a smaller illumination coefficient. In this way, the texture data obtained after processing with a smaller illumination coefficient is closer to the quantization step size, resulting in a greater data loss for the texture data obtained after processing with a smaller illumination coefficient. In other words, the larger the illumination coefficient, the smaller the data loss of the texture data, so that the data loss of the texture data with a larger contribution is smaller, thereby reducing the loss of valid data while ensuring effective data compression.

[0253] FIG9 is a flowchart of a decoding method provided by an embodiment of the present application, wherein the method is applied to a client, and the client includes multiple three-dimensional models. Referring to FIG9 , the method includes the following steps.

[0254] Step 901: Decode from the bitstream to obtain a second rendering intermediate result corresponding to a target area in a virtual viewing plane, wherein a first visible area corresponding to the target area in a plurality of three-dimensional models has reflection and / or refraction characteristics, and the second rendering intermediate result is obtained by processing the first rendering intermediate result of the target area by using a first illumination coefficient corresponding to each pixel in the target area. The number of bits occupied by the second rendering intermediate result during encoding is smaller than the number of bits occupied by the first rendering intermediate result during encoding, and the first rendering intermediate result indicates a second visible area in the plurality of three-dimensional models that is reflected and / or refracted by the first visible area, and the first illumination coefficient refers to an illumination coefficient of an intersection of a light ray emitted from a virtual viewpoint and passing through a pixel and the first visible area. The virtual viewing plane is located between the virtual viewpoint and the plurality of three-dimensional models, and the first illumination coefficient indicates a degree of contribution of the second visible area to the display content of each pixel in the target area.

[0255] The second intermediate rendering result includes at least one of a reflection result of pixels in the target area and a refraction result of pixels in the target area. The reflection result includes at least one of a reflection map and reflection vertex data, and the refraction result includes at least one of a refraction map and refraction vertex data. The reflection map and the refraction map may be collectively referred to as map data, and the reflection vertex data and the refraction vertex data may be collectively referred to as vertex data.

[0256] Step 902: Generate a rendering image corresponding to the virtual viewing plane based on the second rendering intermediate result and multiple three-dimensional models.

[0257] For the sake of convenience of description, the following will take the case where the second rendering intermediate result includes the reflection result of the pixels in the target area as an example. The case where the second rendering intermediate result includes the refraction result of the pixels in the target area is similar to the case where the second rendering intermediate result includes the reflection result of the pixels in the target area, so it will not be explained in detail.

[0258] In some embodiments, the second intermediate rendering result is not a target intermediate rendering result. The target intermediate rendering result refers to an intermediate rendering result that can be directly used for rendering without the need for restoration by the client. In this case, the process of generating a rendered image corresponding to the virtual viewing plane based on the second intermediate rendering result and multiple three-dimensional models by the client may include the following steps (1)-(2).

[0259] (1) The client restores the second rendering intermediate result to obtain a reconstructed value of the first rendering intermediate result.

[0260] The client determines a first illumination coefficient corresponding to each pixel in the target area, and restores the second rendering intermediate result based on the first illumination coefficient corresponding to each pixel to obtain a reconstructed value of the first rendering intermediate result.

[0261] In some embodiments, a normal vector of a first bin corresponding to each pixel in the target area is determined. For any pixel in the target area, a first illumination coefficient corresponding to the pixel is determined based on the normal vector of the first bin corresponding to the pixel and the camera pose according to a relevant algorithm. Each pixel in the target area is processed in the same manner to obtain the first illumination coefficient corresponding to each pixel in the target area.

[0262] The implementation method of the client determining the normal vector of the first surface element corresponding to each pixel in the target area is similar to the implementation method of the above-mentioned cloud platform determining the normal vector of the first surface element corresponding to each pixel in the target area. For details, please refer to the relevant content above and will not be repeated here.

[0263] Based on the above description, the first illumination coefficient also includes at least one of a reflection illumination coefficient and a refraction illumination coefficient. When the first rendering intermediate result includes the reflection result of each pixel in the target area, the first illumination coefficient includes the reflection illumination coefficient, which indicates the contribution of the reflection result of the pixel to the display content of the pixel, or in other words, the reflection illumination coefficient indicates the reflectivity of the first bin of the pixel. When the first rendering intermediate result includes the refraction result of each pixel in the target area, the first illumination coefficient includes the refraction illumination coefficient, which indicates the contribution of the refraction result of the pixel to the display content of the pixel, or in other words, the refraction illumination coefficient indicates the refractive index of the first bin of the pixel.

[0264] In summary, there are three implementation methods for the client to determine the normal vector of the first bin corresponding to each pixel in the target area. In some embodiments, the client can select one of these three implementation methods as the implementation method ultimately adopted by the client. The implementation method selected by the client is consistent with the implementation method adopted by the cloud platform when determining the normal vector of the first bin. In other words, the client and the cloud platform both adopt the same implementation method when determining the normal vector of the first bin corresponding to each pixel in the target area.

[0265] In actual applications, the second rendering intermediate result may include the mapping data of each pixel in the target area, and the second rendering intermediate result may also include the vertex data of some pixels in the target area. In different situations, the implementation methods of restoring the second rendering intermediate result and obtaining the reconstructed value of the first rendering intermediate result are different, which will be introduced below.

[0266] In the case where the second rendering intermediate result includes the map data of each pixel in the target area, if the cloud platform processes the first rendering intermediate result based on the first illumination coefficient corresponding to each pixel in the target area and adopts the first implementation method to obtain the second rendering intermediate result, then for the reflection map of any pixel in the target area in the second rendering intermediate result, the client can divide the reflection map of the pixel by the reflection illumination coefficient corresponding to the pixel to obtain the reflection map of the pixel after the division process. Each pixel in the target area is processed in the same manner to obtain the reflection map of each pixel after the division process, so as to restore the second rendering intermediate result and obtain the reconstructed value of the first rendering intermediate result.

[0267] If the cloud platform processes the first rendering intermediate result based on the first illumination coefficient corresponding to each pixel in the target area and adopts the second implementation method to obtain the second rendering intermediate result, the client determines the second illumination coefficients corresponding to the multiple blocks based on the first illumination coefficients corresponding to each pixel in the multiple blocks included in the target area, and restores the second rendering intermediate result based on the second illumination coefficients corresponding to the multiple blocks to obtain the reconstructed value of the first rendering intermediate result.

[0268] Optionally, the client can determine the pixels included in each block of the target area, and then determine the second illumination coefficients corresponding to the multiple blocks based on the first illumination coefficients corresponding to the pixels in the multiple blocks included in the target area.

[0269] In some embodiments, the cloud platform can also send block division information of the target area to the client, where the block division information indicates the pixels included in each block of the target area. The client can receive the block division information and, based on the block division information, determine the pixels included in each block. In other implementations, the client obtains a division method for multiple blocks, and then the client can divide the target area into multiple blocks according to the division method for multiple blocks, and then the client can determine the pixels included in each block. The division method for multiple blocks refers to the method by which the cloud platform determines the division of the multiple blocks.

[0270] Optionally, the client obtains a method for determining the second illumination coefficient, and determines, based on the first illumination coefficient corresponding to each pixel in the plurality of blocks, the second illumination coefficient corresponding to each of the plurality of blocks according to the method for determining the second illumination coefficient. The method for determining the second illumination coefficient refers to a method for determining the second illumination system by the cloud platform.

[0271] In some embodiments, if the cloud platform processes the first rendering intermediate result based on the second illumination coefficients corresponding to the multiple blocks respectively and adopts the second implementation method when determining the second rendering intermediate result, the client restores the second rendering intermediate result based on the second illumination coefficients corresponding to the multiple blocks respectively, and the implementation process of obtaining the reconstructed value of the first rendering intermediate result includes: the client determines the convolution kernel sizes corresponding to the multiple blocks respectively based on the second illumination coefficients corresponding to the multiple blocks respectively, and performs deconvolution operation on the map data of the pixels in the multiple blocks based on the convolution kernel sizes corresponding to the multiple blocks respectively, to obtain the reconstructed value of the first rendering intermediate result.

[0272] Similar to the above, the convolution kernel size is inversely proportional to the second illumination coefficient. The larger the second illumination coefficient, the smaller the convolution kernel size. The convolution kernel size used by the client is consistent with the convolution kernel size used by the cloud platform, and the convolution kernel template used by the client is also consistent with the convolution kernel template used by the cloud platform.

[0273] In the case where the second rendering intermediate result includes vertex data of some pixels in the target area, based on the vertex data of each pixel included in the second rendering intermediate result, the vertex data corresponding to each pixel in the multiple blocks included in the target area is determined to obtain the reconstructed value of the first rendering intermediate result.

[0274] Among them, the vertex data is the face element number of the second face element, or the vertex number of the second face element, or the coordinates of the visible point in multiple three-dimensional models, the second face element is the face element corresponding to the corresponding pixel in the second visible area, and the visible point is the point in the second face element corresponding to the corresponding pixel.

[0275] Optionally, the client can determine the pixels included in each block of the target area, and then determine the vertex data corresponding to each pixel in the multiple blocks included in the target area based on the vertex data of each pixel included in the second rendering intermediate result.

[0276] In some embodiments, the cloud platform can also send block division information of the target area to the client, where the block division information indicates the pixels included in each block of the target area. The client can receive the block division information and, based on the block division information, determine the pixels included in each block. In other implementations, the client obtains a division method for multiple blocks, and then the client can divide the target area into multiple blocks according to the division method for multiple blocks, and then the client can determine the pixels included in each block. The division method for multiple blocks refers to the method by which the cloud platform determines the division of the multiple blocks.

[0277] In some embodiments, the client interpolates the pixels in the target area that do not have vertex data based on the vertex data of each pixel included in the second rendering intermediate result according to a relevant algorithm to obtain the vertex data of each pixel in the target area, and uses the vertex data of each pixel in the target area after interpolation processing as the reconstructed value of the first rendering intermediate result.

[0278] In other embodiments, the client determines the position of the vertex data of each pixel in the multiple three-dimensional models based on the vertex data of each pixel included in the second rendering intermediate result, divides the distance between any two first pixels in the target area by the distance between the vertex data of the two first pixels to obtain a first ratio, where the first pixel is the pixel for which vertex data exists in the second rendering intermediate result.

[0279] For any pixel in the target area that does not have vertex data, the distance between the pixel and any first pixel is divided by the first distance to obtain a target length. The position of the point whose distance from the vertex data of the first pixel is the target length and is located at a first orientation is determined as the vertex data of the pixel, where the first orientation is the orientation of the pixel relative to the first pixel. All pixels in the target area that do not have vertex data are processed in the same manner to obtain the vertex data of each pixel in the target area, thereby obtaining a reconstructed value of the first rendering intermediate result.

[0280] In actual applications, before determining the rendered image corresponding to the virtual viewing plane based on the second rendering intermediate result and multiple three-dimensional models, the client can also determine whether the second rendering intermediate result is the target rendering intermediate result, the method for determining the first lighting coefficient, the method for determining the second lighting coefficient, the method for determining the second rendering intermediate result, and the method for dividing multiple blocks.

[0281] In some embodiments, the client stores information indicating a target intermediate rendering result, as well as information on how to determine the first illumination coefficient, how to determine the second illumination coefficient, how to determine the second intermediate rendering result, how to divide the multiple blocks, whether the second intermediate rendering result includes a reflection result, and whether it includes a refraction result. The information indicating the target intermediate rendering result indicates whether the second intermediate rendering result is the target intermediate rendering result. In this case, the client can directly determine whether the second intermediate rendering result is the target intermediate rendering result, how to determine the first illumination coefficient, how to determine the second illumination coefficient, how to determine the second intermediate rendering result, and how to divide the multiple blocks.

[0282] In other embodiments, the client is able to obtain end-side indication information, which indicates at least one of the following: whether the second rendering intermediate result is the target rendering intermediate result, the method for determining the first illumination coefficient, the method for determining the second illumination coefficient, the method for determining the second rendering intermediate result, the method for dividing multiple blocks, whether the second rendering intermediate result contains reflection results, whether it contains refraction results, and the type of vertex data.

[0283] Since the first illumination coefficient is determined based on the normal vector of the first surface element, in some embodiments, the method for determining the first illumination coefficient can indicate the method for determining the normal vector of the first surface element.

[0284] The cloud platform can determine the terminal side indication information and send the terminal side indication information to the client, and the client can receive the terminal side indication information.

[0285] It should be noted that, since the embodiment of the present application introduces a variety of methods for determining the first illumination coefficient, the second illumination coefficient, the second rendering intermediate result, and the division of multiple blocks, etc. when introducing the encoding method, the end-side indication information indicates the specific method adopted by the cloud platform during the encoding process, or indicates the specific method adopted by the client during the decoding process. In other words, the end-side indication information is used to guide the decoding process of the client, so that when the second rendering intermediate result is not the target rendering intermediate result, the client restores the second rendering intermediate result sent by the cloud platform to obtain the reconstructed value of the first rendering intermediate result, and then renders the image. When the second rendering intermediate result is the target rendering intermediate result, the image is rendered based on the target rendering intermediate result.

[0286] (2) Based on the reconstructed value of the first rendering intermediate result, the target illumination coefficient corresponding to each pixel in the target area and multiple three-dimensional models, a rendering image corresponding to the virtual viewing plane is generated, and the target illumination coefficient is the illumination coefficient adopted by the cloud platform when determining the normal vector of the first surface element corresponding to each pixel in the target area, and is based on the illumination coefficient determined by the normal vector of the first surface element.

[0287] In some embodiments, the client determines the color corresponding to each pixel in the target area based on the reconstructed value of the first rendering intermediate result, processes the color corresponding to each pixel in the target area based on the target illumination coefficient corresponding to each pixel in the target area, and obtains a third rendering intermediate result, and generates a rendered image corresponding to the virtual viewing plane based on the third rendering intermediate result and the multiple three-dimensional models.

[0288] Optionally, the reconstructed value of the first rendering intermediate result includes texture data of each pixel in the target area and / or vertex data of each pixel in the target area.

[0289] If the reconstructed value of the first rendering intermediate result includes the texture data of each pixel in the target area, in this case, the client can determine the color in the texture data of each pixel in the target area as the color corresponding to each pixel in the target area.

[0290] If the reconstructed value of the first rendering intermediate result includes vertex data of each pixel in the target area, in this case, based on the vertex data of each pixel in the target area, according to the relevant algorithm, the color corresponding to each pixel in the target area is determined from multiple three-dimensional models.

[0291] It should be noted that, for any pixel, if the reconstructed value of the first rendering intermediate result includes the texture data and vertex data of the pixel, the client can determine the color corresponding to the pixel based on any one of the texture data and vertex data. Of course, in actual applications, the color of the pixel can also be determined by other means, for example, according to the pixel texture data, the first color corresponding to the pixel is determined, according to the vertex data of the pixel, the second color corresponding to the pixel is determined, and then the average of the first color and the second color is used as the color corresponding to the pixel. This embodiment of the present application does not limit this.

[0292] Optionally, the target illumination coefficient corresponding to each pixel in the target area is multiplied by the corresponding color to obtain the target color corresponding to each pixel, thereby obtaining a third rendering intermediate result.

[0293] In some embodiments, the client can generate a rendered image corresponding to the virtual viewing plane based on the third rendering intermediate result, the multiple three-dimensional models, and the camera perspective according to a relevant image rendering algorithm.

[0294] In some embodiments, the second intermediate rendering result is a target intermediate rendering result, which refers to an intermediate rendering result that the client can directly use for rendering without restoration. In this case, the client can directly determine the second intermediate rendering result as the third intermediate rendering result and, based on the third intermediate rendering result and the multiple 3D models, generate a rendered image corresponding to the virtual view plane according to a relevant algorithm. In other words, if the second intermediate rendering result is the target intermediate rendering result, the client can directly generate a rendered image corresponding to the virtual view plane based on the target intermediate rendering result and the multiple 3D models.

[0295] It should be noted that in the case of an unstable network, the client may not be able to receive the code stream sent by the cloud platform in a timely manner. In this case, the client can perform basic image rendering based on the camera perspective and multiple three-dimensional models to obtain the rendered image corresponding to the virtual viewing plane. In this way, even in the case of an unstable network, the client side can perform basic image rendering, thereby effectively improving the user experience.

[0296] In some embodiments, the basic image rendering means that the client can use an algorithm with less computational effort (such as the SSR algorithm) to determine the reflection and / or refraction data of the target area, and then obtain a rendered image corresponding to the virtual viewing plane, or the client does not perform the step of determining the reflection and / or refraction data of the target area, and directly generates a rendered image corresponding to the virtual viewing plane.

[0297] When the client uses a computationally inefficient algorithm to determine the reflection and / or refraction data of the target area, the accuracy of the reflection and / or refraction data determined by the client is lower than that determined by the cloud platform, resulting in lower clarity and realism in the rendered image. When the client does not determine the reflection and / or refraction data and directly generates a rendered image corresponding to the virtual viewing plane, the rendered image will not have any reflection and / or refraction effects.

[0298] In an embodiment of the present application, the client can parse the code stream encoded with the second rendering intermediate result, and thus render the image based on the second rendering intermediate result. That is, in the embodiment of the present application, the cloud platform does not perform rendering, but is only used to determine the reflection and / or refraction data. The rendering work is performed by the client. In this way, even in the case of unstable network, the terminal side can perform basic image rendering, thereby effectively improving the user experience. Moreover, in the case where the second rendering intermediate result is the target rendering intermediate result, the client can directly generate the rendered image corresponding to the virtual viewing plane based on the target rendering intermediate result and the multiple three-dimensional models, without the need to restore the second rendering intermediate result to obtain the reconstructed value of the first rendering intermediate result, and without the need to process the color corresponding to each pixel in the target area based on the target illumination coefficient. In this way, the efficiency of the client in rendering the image can be effectively improved.

[0299] Next, the encoding and decoding method provided in the embodiment of the present application will be introduced again through Figure 10.

[0300] If the first visible area corresponding to the target area in multiple three-dimensional models has reflection and refraction characteristics, in this case, the cloud platform can determine the first rendering intermediate result corresponding to the target area in the virtual viewing plane and the first illumination coefficient corresponding to each pixel in the target area. The first rendering intermediate result includes the reflection map and the refraction map corresponding to each pixel in the target area, and the first illumination coefficient includes the reflection illumination coefficient and the refraction illumination coefficient. Then, the cloud platform multiplies the reflection map corresponding to each pixel in the target area with the corresponding reflection illumination coefficient to obtain the second reflection map corresponding to each pixel. The cloud platform multiplies the refraction map corresponding to each pixel in the target area with the corresponding refraction illumination coefficient to obtain the second refraction map corresponding to each pixel, and uses the second reflection map and the second refraction map corresponding to each pixel as the second rendering intermediate result. The second reflection map corresponding to each pixel in the target area and the second refraction map corresponding to each pixel in the target area are spliced ​​to obtain a spliced ​​second reflection map and a spliced ​​second refraction map, respectively. The spliced ​​second reflection map and the spliced ​​second refraction map are respectively encoded into the bitstream. The cloud platform can also send end-side indication information to the client, indicating that the second rendering intermediate result is the target rendering intermediate result. The client can parse the bitstream to obtain the second rendering intermediate result corresponding to the target area in the virtual view plane. The second rendering intermediate result includes the reflection result of the pixels in the target area and the refraction result of the pixels in the target area. The reflection result includes the second reflection map, and the refraction result includes the second refraction map. The client parses the end-side indication information sent by the cloud platform and determines that the second rendering intermediate result is the target rendering intermediate result. The target rendering intermediate result refers to the rendering intermediate result that the client does not need to restore and can be directly used for rendering. In this case, the client can directly generate a rendered image corresponding to the virtual view plane based on the second rendering intermediate result and the multiple three-dimensional models.

[0301] FIG11 is a schematic diagram of the structure of an encoding device provided in an embodiment of the present application. The encoding device can be implemented as part or all of the above-mentioned encoder by software, hardware, or a combination of both. Referring to FIG11 , the device includes: a first determination module 1101, a processing module 1102, and a first encoding module 1103.

[0302] The first determination module 1101 is used to determine the first rendering intermediate result corresponding to the target area in the virtual viewing plane and the first illumination coefficient corresponding to each pixel in the target area. The first visible area corresponding to the target area in multiple three-dimensional models has reflection and / or refraction characteristics. The first rendering intermediate result indicates the second visible area in the multiple three-dimensional models that is reflected and / or refracted by the first visible area. The first illumination coefficient refers to the illumination coefficient of the intersection of the light emitted from the virtual viewpoint and passing through the pixel and the first visible area. The virtual viewing plane is located between the virtual viewpoint and the multiple three-dimensional models. The first illumination coefficient indicates the contribution of the second visible area to the display content of each pixel in the target area. The detailed implementation process refers to the corresponding content in the above embodiments and will not be repeated here.

[0303] Processing module 1102 is configured to process the first intermediate rendering result based on the first illumination coefficient corresponding to each pixel to obtain a second intermediate rendering result, where the number of bits occupied by the second intermediate rendering result during encoding is smaller than the number of bits occupied by the first intermediate rendering result during encoding. The detailed implementation process is described in the corresponding embodiments above and will not be repeated here.

[0304] The first encoding module 1103 is configured to encode the second rendering intermediate result into a bitstream. The detailed implementation process is described in the corresponding contents of the above embodiments and will not be repeated here.

[0305] Optionally, the first rendering intermediate result includes texture data of each pixel in the target area;

[0306] The processing module 1102 is specifically configured to:

[0307] The texture data corresponding to each pixel in the target area is multiplied by the first illumination coefficient corresponding to each pixel to obtain a second rendering intermediate result.

[0308] Optionally, the processing module 1102 is specifically configured to:

[0309] Divide the target area into multiple blocks, each block includes multiple pixels;

[0310] Determining second illumination coefficients corresponding to the plurality of blocks respectively based on first illumination coefficients corresponding to pixels in the plurality of blocks;

[0311] The first rendering intermediate result is processed based on the second illumination coefficients corresponding to the plurality of blocks to obtain a second rendering intermediate result.

[0312] Optionally, the first rendering intermediate result includes texture data of each pixel in the target area; the processing module 1102 is specifically configured to:

[0313] The second illumination coefficients corresponding to the multiple blocks are multiplied by the texture data of the pixels included in each block to obtain a second rendering intermediate result.

[0314] Optionally, the first rendering intermediate result includes texture data of each pixel in the target area; the processing module 1102 is specifically configured to:

[0315] Determining convolution kernel sizes corresponding to the multiple blocks based on the second illumination coefficients corresponding to the multiple blocks respectively;

[0316] Based on the convolution kernel sizes corresponding to the multiple blocks, a convolution operation is performed on the map data of the pixels in the multiple blocks to obtain a second rendering intermediate result.

[0317] Optionally, the first rendering intermediate result includes vertex data of each pixel in the target area; the processing module 1102 is specifically configured to:

[0318] Determining filtering radii corresponding to the plurality of blocks based on second illumination coefficients corresponding to the plurality of blocks;

[0319] Based on the filtering radius corresponding to each of the multiple blocks, the vertex data of the pixels in the multiple blocks are filtered to obtain a second rendering intermediate result.

[0320] Optionally, the first rendering intermediate result includes vertex data of each pixel in the target area; the processing module 1102 is specifically configured to:

[0321] Determining data filtering ratios corresponding to the plurality of blocks based on second illumination coefficients corresponding to the plurality of blocks, the data filtering ratios indicating filtering conditions of vertex data of pixels in the blocks;

[0322] Based on the data screening ratios corresponding to the multiple blocks, the vertex data of the pixels in the multiple blocks are screened to obtain a second rendering intermediate result.

[0323] Optionally, the device further comprises:

[0324] an acquisition module, configured to acquire first information, the first information including terminal-side device information and / or attribute information of a target area, the terminal-side device information indicating performance of the client, and the attribute information indicating reflection and / or refraction properties of the target area;

[0325] The second determination module is used to determine rendering indication information based on the first information, where the rendering indication information includes cloud-side indication information, where the cloud-side indication information indicates a method for determining the first rendering intermediate result and the first illumination coefficient, and / or a method for determining the second rendering intermediate result.

[0326] Optionally, the rendering indication information further includes terminal-side indication information, where the terminal-side indication information indicates at least one of the following: whether the second rendering intermediate result is a target rendering intermediate result, a method for determining the first illumination coefficient, a method for determining the second illumination coefficient, a method for determining the second rendering intermediate result, and a method for dividing the multiple blocks. The target rendering intermediate result refers to a rendering intermediate result that the client does not need to restore and can be directly used for rendering.

[0327] The device also includes: a second encoding module, configured to encode the terminal side indication information into a code stream and send the code stream.

[0328] In an embodiment of the present application, since determining the refraction / reflection data of the target area requires high device performance, the embodiment of the present application can determine the first rendering intermediate result corresponding to the target area in the virtual visual plane and the first illumination coefficient corresponding to each pixel in the target area through the cloud platform, and then process the first rendering intermediate result based on the first illumination coefficient corresponding to each pixel to obtain a second rendering intermediate result. Since the first rendering intermediate result indicates the second visible area reflected and / or refracted by the first visible area in the multiple three-dimensional models, the first rendering intermediate result can indicate the reflection and / or refraction data of the first visible area. Compared with the related technology that the cloud platform needs to render the entire image, the cloud platform in the embodiment of the present application only needs to determine the reflection and / or refraction data and the first illumination coefficient of the first visible area, which can reduce the occupancy of cloud computing resources. The cloud platform can also encode the second rendering intermediate result into the code stream. Compared with the related technology that requires the entire rendered image to be encoded into the code stream, the embodiment of the present application does not need to encode the entire rendered image into the code stream, which can ensure that the code rate transmitted between the cloud platform and the client is relatively low. Since the number of bits occupied by the second rendering intermediate result during encoding is less than the number of bits occupied by the first rendering intermediate result during encoding, the bit rate of transmission between the cloud platform and the client can be further reduced. In addition, since quantization is usually required when encoding data, the quantized data will have data loss compared to the data before quantization. For the same texture data, if different illumination coefficients are used to process the texture data, the data loss of the texture data after quantization will be different. The texture data obtained after processing with a larger illumination coefficient is greater than the texture data obtained after processing with a smaller illumination coefficient. In this way, the texture data obtained after processing with a smaller illumination coefficient is closer to the quantization step size, resulting in a greater data loss for the texture data obtained after processing with a smaller illumination coefficient. In other words, the larger the illumination coefficient, the smaller the data loss of the texture data, so that the data loss of the texture data with a larger contribution is smaller, thereby reducing the loss of valid data while ensuring effective data compression.

[0329] It should be noted that the encoding device provided in the above embodiment is merely illustrated by the division of the above functional modules during encoding. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the encoding device provided in the above embodiment and the encoding method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0330] FIG12 is a schematic diagram of the structure of a decoding device provided in an embodiment of the present application, which can be implemented as part or all of the above decoder by software, hardware, or a combination of both. Referring to FIG12 , the device includes: a parsing module 1201 and a generating module 1202 .

[0331] Parsing module 1201 is configured to decode from a bitstream a second intermediate rendering result corresponding to a target area in a virtual viewing plane, wherein the target area corresponds to a first visible area in multiple three-dimensional models and has reflective and / or refractive properties. The second intermediate rendering result is obtained by processing the first intermediate rendering result of the target area using a first illumination coefficient corresponding to each pixel in the target area. The number of bits occupied by the second intermediate rendering result during encoding is smaller than the number of bits occupied by the first intermediate rendering result during encoding. The first intermediate rendering result indicates a second visible area in the multiple three-dimensional models that is reflected and / or refracted by the first visible area. The first illumination coefficient refers to the illumination coefficient of the intersection of a ray emitted from a virtual viewpoint and passing through a pixel and the first visible area. The virtual viewing plane is located between the virtual viewpoint and the multiple three-dimensional models. The first illumination coefficient indicates the contribution of the second visible area to the display content of each pixel in the target area. The detailed implementation process refers to the corresponding content in the above-mentioned embodiments and will not be repeated here.

[0332] The generating module 1202 is configured to generate a rendering image corresponding to the virtual viewing plane based on the second intermediate rendering result and the multiple three-dimensional models. The detailed implementation process is referred to the corresponding content of the above embodiments and will not be repeated here.

[0333] Optionally, the second rendering intermediate result is not a target rendering intermediate result, and the target rendering intermediate result refers to a rendering intermediate result that the client does not need to restore and can be directly used for rendering;

[0334] The generation module 1202 is specifically used for:

[0335] Restoring the second rendering intermediate result to obtain a reconstructed value of the first rendering intermediate result;

[0336] A rendered image corresponding to the virtual viewing plane is generated based on the reconstructed value of the first rendering intermediate result, the target illumination coefficient corresponding to each pixel in the target area, and the multiple three-dimensional models.

[0337] Optionally, the second rendering intermediate result includes texture data of each pixel in the target area; the generating module 1202 is specifically configured to:

[0338] Determine a first illumination coefficient corresponding to each pixel in the target area;

[0339] The second rendering intermediate result is restored based on the first illumination coefficient corresponding to each pixel to obtain a reconstructed value of the first rendering intermediate result.

[0340] Optionally, the generating module 1202 is specifically configured to:

[0341] Determining second illumination coefficients corresponding to the plurality of blocks respectively based on first illumination coefficients corresponding to pixels in the plurality of blocks included in the target area;

[0342] The second rendering intermediate result is restored based on the second illumination coefficients respectively corresponding to the plurality of blocks to obtain a reconstructed value of the first rendering intermediate result.

[0343] Optionally, the generation module 1202 is specifically configured to:

[0344] Determining convolution kernel sizes corresponding to the multiple blocks based on the second illumination coefficients corresponding to the multiple blocks respectively;

[0345] Based on the convolution kernel sizes corresponding to the multiple blocks, a deconvolution operation is performed on the map data of the pixels in the multiple blocks to obtain a reconstructed value of the first rendering intermediate result.

[0346] Optionally, the second rendering intermediate result includes vertex data of some pixels in the target area; the generating module 1202 is specifically configured to:

[0347] Based on the vertex data of each pixel included in the second rendering intermediate result, vertex data corresponding to each pixel in a plurality of blocks included in the target area is determined to obtain a reconstructed value of the first rendering intermediate result.

[0348] Optionally, the generating module 1202 is specifically configured to:

[0349] Determine the color corresponding to each pixel in the target area based on the reconstructed value of the first rendering intermediate result;

[0350] Based on the target illumination coefficient corresponding to each pixel in the target area, the color corresponding to each pixel in the target area is processed to obtain a third rendering intermediate result;

[0351] Based on the third rendering intermediate result and the three-dimensional model, a rendering image corresponding to the virtual viewing plane is generated.

[0352] Optionally, the reconstructed value of the first rendering intermediate result includes texture data and / or vertex data of each pixel in the target area; the generating module 1202 is specifically configured to:

[0353] Determine the color in the mapping data of each pixel in the target area as the color corresponding to each pixel in the target area; or

[0354] Based on vertex data of each pixel in the target area, a color corresponding to each pixel in the target area is determined from the plurality of three-dimensional models.

[0355] Optionally, the vertex data is the face element number of the second face element, or the vertex number of the second face element, or the coordinates of the visible point in multiple three-dimensional models, the second face element is the face element corresponding to the pixel in the second visible area, and the visible point is the point corresponding to the pixel in the second face element.

[0356] Optionally, the device further comprises:

[0357] An acquisition module is used to obtain end-side indication information, which indicates at least one of the following: whether the second rendering intermediate result is the target rendering intermediate result, a method for determining the first illumination coefficient, a method for determining the second illumination coefficient, a method for determining the second rendering intermediate result, and a method for dividing multiple blocks.

[0358] In an embodiment of the present application, the client can parse the code stream encoded with the second rendering intermediate result, and thus render the image based on the second rendering intermediate result. That is, in the embodiment of the present application, the cloud platform does not perform rendering, but is only used to determine the reflection and / or refraction data. The rendering work is performed by the client. In this way, even in the case of unstable network, the terminal side can perform basic image rendering, thereby effectively improving the user experience. Moreover, in the case where the second rendering intermediate result is the target rendering intermediate result, the client can directly generate the rendered image corresponding to the virtual viewing plane based on the target rendering intermediate result and the multiple three-dimensional models, without the need to restore the second rendering intermediate result to obtain the reconstructed value of the first rendering intermediate result, and without the need to process the color corresponding to each pixel in the target area based on the target illumination coefficient. In this way, the efficiency of the client in rendering the image can be effectively improved.

[0359] It should be noted that the decoding device provided in the above embodiment is merely illustrated by the division of the above functional modules during decoding. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the decoding device provided in the above embodiment and the decoding method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0360] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or processor executes the steps of the encoding method described in the above embodiment, or executes the steps of the decoding method described in the above embodiment.

[0361] The present application also provides a computer program product comprising computer instructions that, when executed by a computer or processor, cause the computer or processor to perform the steps of the encoding method described in the above embodiments, or to perform the steps of the decoding method described in the above embodiments. Alternatively, a computer program is provided that, when executed on a computer or processor, causes the computer or processor to perform the steps of the encoding method described in the above embodiments, or to perform the steps of the decoding method described in the above embodiments.

[0362] An embodiment of the present application also provides a coding and decoding system, which includes a coding device and a decoding device. The coding device is used to implement the steps of the coding method described in the above embodiment, and the decoding device is used to implement the steps of the decoding method described in the above embodiment.

[0363] The embodiment of the present application further provides an encoded code stream, which is generated according to the encoding method described in the above embodiment.

[0364] An embodiment of the present application further provides a computer-readable storage medium, which stores a code stream generated by the encoding method described in the above embodiment.

[0365] An embodiment of the present application further provides a device for storing a code stream, the device comprising a receiver and at least one storage medium, the receiver being configured to receive a code stream generated by the encoding method described in the above embodiment, and the at least one storage medium being configured to store the code stream.

[0366] An embodiment of the present application also provides a device for transmitting a code stream, the device comprising a transmitter and a receiver, the receiver being used to receive a code stream generated by the encoding method described in the above embodiment, and the transmitter being used to send the code stream to an end-side device via a transmission medium.

[0367] An embodiment of the present application also provides a device for transmitting a code stream, the device comprising a transmitter and at least one storage medium, the at least one storage medium being used to store the code stream generated by the encoding method described in the above embodiment, the transmitter being used to obtain the code stream from the storage medium and send the code stream to an end-side device via a transmission medium.

[0368] An embodiment of the present application further provides a system for distributing a code stream, the system comprising at least one storage medium and a streaming media device, the at least one storage medium being used to store a code stream generated according to the encoding method described in the above embodiment, the streaming media device being used to obtain a target code stream from the at least one storage medium and send the target code stream to a terminal-side device, wherein the streaming media device comprises a content server or a content distribution server.

[0369] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of the present application may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0370] It should be understood that the "plurality" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0371] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the first rendering intermediate result corresponding to the target area in the virtual viewing plane and the first illumination coefficient corresponding to each pixel in the target area involved in the embodiments of this application are both obtained with full authorization.

[0372] The above description is an embodiment provided for this application and is not intended to limit this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A coding method, characterized in that: Applied to a cloud platform, the cloud platform includes multiple three-dimensional models; the method includes: Determine a first intermediate rendering result corresponding to a target area in a virtual viewing plane and a first illumination coefficient corresponding to each pixel in the target area; The target area corresponds to a first visible area in the multiple three-dimensional models having reflective and / or refractive properties, the first intermediate rendering result indicates a second visible area in the multiple three-dimensional models that is reflected and / or refracted by the first visible area, the first illumination coefficient refers to an illumination coefficient of an intersection of a light ray emitted from a virtual viewpoint and passing through the pixel and the first visible area, the virtual viewing plane is located between the virtual viewpoint and the multiple three-dimensional models, and the first illumination coefficient indicates a contribution of the second visible area to the display content of each pixel in the target area; Processing the first intermediate rendering result based on the first illumination coefficient corresponding to each pixel to obtain a second intermediate rendering result, wherein the number of bits occupied by the second intermediate rendering result during encoding is smaller than the number of bits occupied by the first intermediate rendering result during encoding; Encode the second rendering intermediate result into a bitstream.

2. The method according to claim 1, wherein The first rendering intermediate result includes the texture data of each pixel in the target area; The processing of the first rendering intermediate result based on the first illumination coefficient corresponding to each pixel to obtain a second rendering intermediate result includes: The second rendering intermediate result is obtained by multiplying the map data corresponding to each pixel in the target area by the first illumination coefficient corresponding to each pixel.

3. The method according to claim 1, wherein The processing of the first rendering intermediate result based on the first illumination coefficient corresponding to each pixel to obtain a second rendering intermediate result includes: Dividing the target area into a plurality of blocks, each block including a plurality of pixels; Determining second illumination coefficients corresponding to the plurality of blocks respectively based on first illumination coefficients corresponding to pixels in the plurality of blocks; The first rendering intermediate result is processed based on the second illumination coefficients respectively corresponding to the multiple blocks to obtain the second rendering intermediate result.

4. The method according to claim 3, wherein The first rendering intermediate result includes texture data of each pixel in the target area; and the processing of the first rendering intermediate result based on the second illumination coefficients corresponding to the plurality of blocks to obtain the second rendering intermediate result includes: The second illumination coefficients corresponding to the plurality of blocks are multiplied by the texture data of the pixels included in the blocks to obtain the second rendering intermediate result.

5. The method according to claim 3, wherein The first rendering intermediate result includes the texture data of each pixel in the target area; The processing of the first rendering intermediate result based on the second illumination coefficients respectively corresponding to the plurality of blocks to obtain the second rendering intermediate result includes: Determining, based on the second illumination coefficients respectively corresponding to the plurality of blocks, the convolution kernel sizes respectively corresponding to the plurality of blocks; Based on the convolution kernel sizes corresponding to the multiple blocks, a convolution operation is performed on the map data of the pixels in the multiple blocks to obtain the second rendering intermediate result.

6. The method according to claim 3, wherein The first rendering intermediate result includes vertex data of each pixel in the target area; The processing of the first rendering intermediate result based on the second illumination coefficients respectively corresponding to the plurality of blocks to obtain the second rendering intermediate result includes: Determining filtering radii corresponding to the plurality of blocks respectively based on second illumination coefficients corresponding to the plurality of blocks respectively; Based on the filtering radiuses respectively corresponding to the multiple blocks, vertex data of pixels in the multiple blocks are filtered to obtain the second rendering intermediate result.

7. The method according to claim 3, wherein The first rendering intermediate result includes vertex data of each pixel in the target area; The processing of the first rendering intermediate result based on the second illumination coefficients respectively corresponding to the plurality of blocks to obtain the second rendering intermediate result includes: Determining data filtering ratios corresponding to the plurality of blocks respectively based on second illumination coefficients corresponding to the plurality of blocks respectively, the data filtering ratios indicating filtering conditions of vertex data of pixels in the blocks; Based on the data screening ratios respectively corresponding to the multiple blocks, the vertex data of the pixels in the multiple blocks are screened to obtain the second rendering intermediate result.

8. The method according to any one of claims 1 to 7, wherein: Before determining the first intermediate rendering result corresponding to the target area in the virtual viewing plane and the first illumination coefficient corresponding to each pixel in the target area, the method further includes: Acquire first information, where the first information includes terminal-side device information and / or attribute information of the target area, where the terminal-side device information indicates performance of the client, and the attribute information indicates reflection and / or refraction properties of the target area; Rendering indication information is determined based on the first information, the rendering indication information including cloud-side indication information, the cloud-side indication information indicating a method for determining the first rendering intermediate result and the first illumination coefficient, and / or a method for determining the second rendering intermediate result.

9. The method according to claim 8, wherein The rendering instruction information further includes terminal-side instruction information, where the terminal-side instruction information indicates at least one of the following: whether the second intermediate rendering result is a target intermediate rendering result, a method for determining the first illumination coefficient, a method for determining the second illumination coefficient, a method for determining the second intermediate rendering result, and a method for dividing the multiple blocks; the target intermediate rendering result refers to an intermediate rendering result that the client does not need to restore and can be directly used for rendering; The method further includes: encoding the terminal side indication information into the code stream and sending the code stream.

10. A decoding method, characterized in that: Applied to a client, the client includes multiple three-dimensional models; the method includes: Decoding the bitstream to obtain a second rendering intermediate result corresponding to the target area in the virtual viewing plane; wherein, a first visible area corresponding to the target area in the multiple three-dimensional models has reflective and / or refractive characteristics, the second rendering intermediate result is obtained by processing the first rendering intermediate result of the target area through a first illumination coefficient corresponding to each pixel in the target area, the number of bits occupied by the second rendering intermediate result when encoded is smaller than the number of bits occupied by the first rendering intermediate result when encoded, the first rendering intermediate result indicates a second visible area in the multiple three-dimensional models that is reflected and / or refracted by the first visible area, the first illumination coefficient refers to an illumination coefficient of an intersection of a light ray emitted from a virtual viewpoint and passing through the pixel and the first visible area, the virtual viewing plane is located between the virtual viewpoint and the multiple three-dimensional models, and the first illumination coefficient indicates a degree of contribution of the second visible area to the display content of each pixel in the target area; A rendered image corresponding to the virtual viewing plane is generated based on the second intermediate rendering result and the multiple three-dimensional models.

11. The method according to claim 10, wherein The second rendering intermediate result is not a target rendering intermediate result. The target rendering intermediate result refers to a rendering intermediate result that the client does not need to restore and can be directly used for rendering; Generating a rendered image corresponding to the virtual viewing plane based on the second intermediate rendering result and the three-dimensional model includes: Restoring the second rendering intermediate result to obtain a reconstructed value of the first rendering intermediate result; A rendered image corresponding to the virtual viewing plane is generated based on a reconstructed value of the first rendering intermediate result, a target illumination coefficient corresponding to each pixel in the target area, and the multiple three-dimensional models.

12. The method according to claim 11, wherein The second rendering intermediate result includes the texture data of each pixel in the target area; and restoring the second rendering intermediate result to obtain a reconstructed value of the first rendering intermediate result includes: Determine a first illumination coefficient corresponding to each pixel in the target area; The second rendering intermediate result is restored based on the first illumination coefficient corresponding to each pixel to obtain a reconstructed value of the first rendering intermediate result.

13. The method according to claim 12, wherein: The restoring the second rendering intermediate result based on the first illumination coefficient corresponding to each pixel to obtain a reconstructed value of the first rendering intermediate result includes: Determining second illumination coefficients corresponding to each pixel in a plurality of blocks included in the target area based on first illumination coefficients corresponding to each pixel in the plurality of blocks respectively included in the target area; The second rendering intermediate result is restored based on the second illumination coefficients respectively corresponding to the multiple blocks to obtain a reconstructed value of the first rendering intermediate result.

14. The method according to claim 13, wherein Restoring the second rendering intermediate result based on the second illumination coefficients respectively corresponding to the plurality of blocks to obtain a reconstructed value of the first rendering intermediate result includes: Determining, based on the second illumination coefficients respectively corresponding to the plurality of blocks, the convolution kernel sizes respectively corresponding to the plurality of blocks; Based on the convolution kernel sizes corresponding to the multiple blocks, a deconvolution operation is performed on the map data of the pixels in the multiple blocks to obtain a reconstructed value of the first rendering intermediate result.

15. The method according to claim 11, wherein The second rendering intermediate result includes vertex data of some pixels in the target area; and restoring the second rendering intermediate result to obtain a reconstructed value of the first rendering intermediate result includes: Based on the vertex data of each pixel included in the second rendering intermediate result, vertex data corresponding to each pixel in a plurality of blocks included in the target area is determined to obtain a reconstructed value of the first rendering intermediate result.

16. The method according to claim 11, wherein The generating, based on the reconstructed value of the first rendering intermediate result, the target illumination coefficient corresponding to each pixel in the target area, and the multiple three-dimensional models, a rendered image corresponding to the virtual viewing plane includes: Determining a color corresponding to each pixel in the target area based on a reconstructed value of the first rendering intermediate result; Processing the color corresponding to each pixel in the target area based on the target illumination coefficient corresponding to each pixel in the target area to obtain a third rendering intermediate result; A rendered image corresponding to the virtual viewing plane is generated based on the third intermediate rendering result and the three-dimensional model.

17. The method according to claim 16, wherein The reconstructed value of the first rendering intermediate result includes texture data and / or vertex data of each pixel in the target area; and determining the color corresponding to each pixel in the target area based on the reconstructed value of the first rendering intermediate result includes: Determine the color in the mapping data of each pixel in the target area as the color corresponding to each pixel in the target area; or Based on vertex data of each pixel in the target area, a color corresponding to each pixel in the target area is determined from the multiple three-dimensional models.

18. The method according to claim 17, wherein The vertex data is the face element number of the second face element, or the vertex number of the second face element, or the coordinates of the visible point in the multiple three-dimensional models, the second face element is the face element corresponding to the pixel in the second visible area, and the visible point is the point in the second face element corresponding to the pixel.

19. The method according to claim 13, wherein Before generating the rendered image corresponding to the virtual viewing plane based on the second intermediate rendering result and the three-dimensional model, the method further includes: Obtain end-side indication information, where the end-side indication information indicates at least one of the following: whether the second rendering intermediate result is the target rendering intermediate result, a method for determining the first illumination coefficient, a method for determining the second illumination coefficient, a method for determining the second rendering intermediate result, and a method for dividing the multiple blocks.

20. An encoding device, characterized in that: Applied to a cloud platform, the cloud platform includes multiple three-dimensional models; the device includes: A first determining module is configured to determine a first intermediate rendering result corresponding to a target area in a virtual viewing plane and a first illumination coefficient corresponding to each pixel in the target area; The target area corresponds to a first visible area in the multiple three-dimensional models having reflective and / or refractive properties, the first intermediate rendering result indicates a second visible area in the multiple three-dimensional models that is reflected and / or refracted by the first visible area, the first illumination coefficient refers to an illumination coefficient of an intersection of a light ray emitted from a virtual viewpoint and passing through the pixel and the first visible area, the virtual viewing plane is located between the virtual viewpoint and the multiple three-dimensional models, and the first illumination coefficient indicates a contribution of the second visible area to the display content of each pixel in the target area; a processing module, configured to process the first intermediate rendering result based on the first illumination coefficient corresponding to each pixel to obtain a second intermediate rendering result, wherein the number of bits occupied by the second intermediate rendering result during encoding is smaller than the number of bits occupied by the first intermediate rendering result during encoding; The first encoding module is used to encode the second rendering intermediate result into a bit stream.

21. A decoding device, characterized in that: Applied to a client, the client includes multiple three-dimensional models; the device includes: A parsing module, configured to decode the bitstream to obtain a second intermediate rendering result corresponding to the target area in the virtual viewing plane; wherein, a first visible area corresponding to the target area in the multiple three-dimensional models has reflective and / or refractive characteristics, the second rendering intermediate result is obtained by processing the first rendering intermediate result of the target area through a first illumination coefficient corresponding to each pixel in the target area, the number of bits occupied by the second rendering intermediate result when encoded is smaller than the number of bits occupied by the first rendering intermediate result when encoded, the first rendering intermediate result indicates a second visible area in the multiple three-dimensional models that is reflected and / or refracted by the first visible area, the first illumination coefficient refers to an illumination coefficient of an intersection of a light ray emitted from a virtual viewpoint and passing through the pixel and the first visible area, the virtual viewing plane is located between the virtual viewpoint and the multiple three-dimensional models, and the first illumination coefficient indicates a degree of contribution of the second visible area to the display content of each pixel in the target area; A generating module is configured to generate a rendered image corresponding to the virtual viewing plane based on the second intermediate rendering result and the multiple three-dimensional models.

22. A coding device, characterized in that The encoding device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the steps of the method according to any one of claims 1 to 9.

23. A decoding device, characterized in that The decoding device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the steps of any one of the methods of claims 10-19.

24. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when run on a computer or a processor, enables the computer or the processor to execute the method according to any one of claims 1 to 9, or the method according to any one of claims 10 to 19.

25. A computer program product, characterized in that The computer program product comprises computer instructions, which, when executed by a computer or a processor, cause the steps of the method according to any one of claims 1 to 9 to be performed, or the steps of the method according to any one of claims 10 to 19 to be performed.

26. A coded bitstream, characterized in that The code stream is generated according to the method of any one of claims 1 to 9.

27. A coded bitstream, characterized in that The code stream includes a second intermediate rendering result corresponding to the target area in the virtual viewing plane, where the second intermediate rendering result is obtained by processing the first intermediate rendering result corresponding to the target area based on the first illumination coefficient corresponding to each pixel in the target area. Among them, the number of bits occupied by the second rendering intermediate result during encoding is smaller than the number of bits occupied by the first rendering intermediate result during encoding, the first visible area corresponding to the target area in the multiple three-dimensional models included in the cloud platform has reflection and / or refraction characteristics, the first rendering intermediate result indicates the second visible area in the multiple three-dimensional models that is reflected and / or refracted by the first visible area, the first illumination coefficient refers to the illumination coefficient of the intersection of the light emitted from the virtual viewpoint and passing through the pixel and the first visible area, the virtual viewing plane is located between the virtual viewpoint and the multiple three-dimensional models, and the first illumination coefficient indicates the degree of contribution of the second visible area to the display content of each pixel in the target area.

28. A computer-readable storage medium, characterized in that The computer-readable storage medium stores the code stream according to any one of claims 1 to 9.

29. A device for storing a code stream, characterized in that: The device comprises: a receiver and at least one storage medium, The receiver is used to receive the code stream according to any one of claims 1 to 9; The at least one storage medium is used to store the code stream.

30. A device for transmitting a code stream, characterized in that: The device comprises: a transmitter and a receiver, The receiver is used to receive the code stream according to any one of claims 1 to 9; The transmitter is used to send the code stream to the terminal side device through a transmission medium.

31. A device for transmitting a code stream, characterized in that: The device includes: a transmitter and at least one storage medium, The at least one storage medium is used to store the code stream according to any one of claims 1 to 9; The transmitter is configured to obtain the code stream from the storage medium and send the code stream to the terminal side device via a transmission medium.

32. A system for distributing code streams, characterized in that: The system comprises: At least one storage medium, configured to store at least one code stream according to any one of claims 1 to 9; The streaming media device is configured to obtain a target code stream from the at least one storage medium and send the target code stream to the terminal side device, wherein the streaming media device includes a content server or a content distribution server.

Images