Three-dimensional scene cloud rendering method and system based on asynchronous transmission data

By using asynchronous transmission and local residual information processing, the problem of high-quality rendering on mobile devices under low latency and high frame rate was solved, and image quality was improved in poor network environments.

CN115147530BActive Publication Date: 2024-11-19LIGHT CLOUD(HANGZHOU)TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202210757039.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-29
Publication Date
2024-11-19
Estimated Expiration
2042-06-29

AI Technical Summary

Technical Problem

On mobile devices, existing technologies struggle to achieve high-quality image rendering with low latency and high frame rates, especially in poor network conditions where the collaborative rendering effect between the cloud and the local device cannot be guaranteed.

Method used

A cloud-based 3D scene rendering method with asynchronous data transmission is adopted. High-quality results are rendered in the cloud and historical moments are transmitted. Combined with pixel-by-pixel shading and residual information processing on the local end, the local rendering quality is improved.

Benefits of technology

In situations with poor network conditions, asynchronous transmission and local residual information processing can significantly improve the rendering quality of mobile devices, enabling low-latency and high-frame-rate image display.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115147530B_ABST
    Figure CN115147530B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and device for cloud rendering of three-dimensional scenes based on asynchronously transmitted data, comprising the following steps: the cloud uses a rendering pipeline to render three-dimensional scene data to obtain a high-quality rendering result, and transmits the historical moment of obtaining the high-quality rendering result to the local end; the local end uses the three-dimensional scene data to color each pixel to obtain a low-quality rendering result; the local end compares the received high-quality rendering result with the low-quality rendering result at the same historical moment, and calculates the residual information of the low-quality rendering result relative to the high-quality rendering result; the local end uses the residual information to enhance the colored pixels at the current moment to improve the local rendering quality. The method and system allow the local device to asynchronously receive the high-quality results rendered by the cloud, and improve the low-quality results calculated locally through the high-quality results of the cloud, thereby improving the image quality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of real-time rendering, and in particular relates to a three-dimensional scene cloud rendering method and system based on asynchronous transmission data. Background Art

[0002] With the popularization of various personal terminals and the improvement of their computing power, the demand for real-time rendering of realistic images on various devices has gradually increased. But at the same time, the continuous development of graphics rendering technology has also put forward higher and higher requirements on the computing power of devices. The growth of computing performance of devices has not kept up with people's pursuit of higher quality images. Under the existing hardware and graphics rendering technology, practitioners have to balance the picture quality, picture resolution and latency. For products such as games, ensuring low latency and stable frame rate is the first thing to ensure, so many products on portable devices have to sacrifice picture quality and resolution to ensure players a smoother gaming experience.

[0003] In order to provide mobile device users with a better experience, some people have thought of utilizing the powerful computing power of the cloud. For example, patent document CN1856819A proposes a method of completing drawing operations on a server with strong computing power and then compressing and transmitting the results to the end. This method solves the problem of low image quality and low resolution in real-time rendering of mobile devices, but the technical solution does not solve the problem of how to obtain low-latency, high-frame-rate images.

[0004] Patent document CN 113440838 A proposes an end-to-end image rendering fusion method and system, which integrates cloud and terminal capabilities based on cloud graphics application reuse scenes’ cloud rendering / data packet distribution, and terminal-side character equipment graphics rendering / resource integration / light and shadow synchronization / control processing, so as to share cloud computing resources and general scene rendering data based on network transmission guarantee. However, in actual application, the image rendering fusion achieved by this solution cannot be guaranteed when network transmission cannot be guaranteed.

[0005] Transmitting every frame rendered by the cloud to the local device means that the communication between the cloud and the local device must be able to ensure good real-time performance and have a large bandwidth. In actual application scenarios, it is impossible to guarantee that all users have a high-quality network that meets the requirements. How to complete end-cloud collaborative rendering in a poor network environment is a problem that needs to be solved. Summary of the invention

[0006] In view of the above, the purpose of the present invention is to provide a three-dimensional scene cloud rendering method and system based on asynchronous data transmission, allowing local devices to asynchronously receive high-quality results rendered in the cloud, and improve the low-quality results obtained by local calculation through the high-quality results in the cloud, thereby improving image quality.

[0007] To achieve the above-mentioned purpose of the invention, an embodiment provides a 3D scene cloud rendering method based on asynchronous transmission data, comprising the following steps:

[0008] The cloud uses a rendering pipeline to render the 3D scene data to obtain high-quality rendering results, and transmits the historical moment of obtaining the high-quality rendering results to the local end;

[0009] The local end uses the 3D scene data to color each pixel, obtaining a low-quality rendering result;

[0010] The local end compares the received high-quality rendering result with the low-quality rendering result at the same historical moment, and calculates the residual information of the low-quality rendering result relative to the high-quality rendering result;

[0011] The local side uses the residual information to enhance the currently colored pixels to improve the local rendering quality.

[0012] In one embodiment, the local end uses the three-dimensional scene data for pixel-by-pixel shading, including preprocessing the three-dimensional scene data and using the preprocessed data for pixel-by-pixel shading;

[0013] The preprocessing of the three-dimensional scene data includes preprocessing of illumination information, three-dimensional scene geometry information and motion information. The preprocessed illumination information is a parameter representation of a light source or a light encoding obtained after the light source parameter information is input into a neural network; the preprocessed three-dimensional scene geometry information includes a position map, a depth map, a normal map and a material map; the preprocessed motion information is represented by a motion vector, which records the motion of an object corresponding to each pixel in a series of images, and is used to track the corresponding relationship between pixels in images at different historical moments;

[0014] The pixel-by-pixel shading using the preprocessed data includes using the preprocessed illumination information and the three-dimensional scene geometry information to perform pixel-by-pixel shading to obtain a low-quality rendering result;

[0015] Preferably, the shading process uses a shader in a rendering engine or a shader built based on a neural network.

[0016] In one embodiment, each pixel has an independent residual value, and between different frame images, historical residuals are reused according to motion vectors, that is, two pixels connected by a motion vector use the same historical residual to enhance the rendering result.

[0017] In one embodiment, after receiving a high-quality rendering result, a low-quality rendering result at the same historical moment is searched based on the historical moment corresponding to the high-quality rendering, and the low-quality rendering result at the same historical moment is subtracted from the high-quality rendering result to obtain residual information for each pixel. Before receiving the high-quality rendering result downloaded from the cloud next time, the local end uses the residual information to enhance the low-quality rendering result at the current moment.

[0018] In one embodiment, the local end uses the residual information to enhance the current moment colored pixels to improve the local rendering quality, including:

[0019] A monotonically increasing function w(Δt) with a value range of [0, 1] is used as a weight, and the product of the weight and the residual information is used as the first enhanced pixel. The first enhanced pixel is superimposed on the low-quality colored pixel to achieve colored pixel enhancement. The formula is expressed as:

[0020] g1(x t )=f(x t )-f′(x t )

[0021] h1(x t ′)=f′(x t′ )+w(t′-t)g1(x t )

[0022] Among them, x is the illumination information and 3D scene geometry information at a certain pixel, f(x t ) and f′(x t ) represent the historical moment t for x t High-quality rendering results and low-quality rendering results, g1(x t ) represents the residual information calculated at the historical moment t, f′(x t′ ) represents the current time t′ for x t′ The low-quality rendering result, w(t′-t) represents the weight to be used when the residual at time t′ is superimposed at time t, h1(x t′ ) represents the current time t′ for x t′ The enhanced result of the colored pixels.

[0023] In one embodiment, the local end uses the residual information to enhance the current moment colored pixels to improve the local rendering quality, including:

[0024] A monotonically increasing function w(Δt) with a value range of [0, 1] is used as a weight, and the product of the weight and the corrected residual information is used as the second enhanced pixel, and the second enhanced pixel is superimposed on the low-quality colored pixel to achieve colored pixel enhancement. The formula is expressed as:

[0025] g1(xt )=f(x t )-f′(x t )

[0026] g2(x t′ )=F(x t′ ;x t , g1(x t ))

[0027] h2(x t′ )=f′(x t′ )+w(t′-t)g2(x t′ )

[0028] Among them, x is the illumination information and 3D scene geometry information at a certain pixel, f(x t ) and f′(x t ) represent the historical moment t for x t High-quality rendering results and low-quality rendering results, g1(x t ) represents the residual information calculated at the historical moment t, g2(x t′ ) represents the corrected residual information, F(x t′ ;x t , g1(t)) means that the illumination information and 3D scene geometry information at a certain pixel at the current time t′ are input into the correction model based on the neural network, and x t , g1(t) is input into the correction model as auxiliary information, and the corrected residual information obtained by model prediction is f′(x t′ ) represents the current time t′ for x t′ The low-quality rendering result, w(t′-t) represents the weight to be used when the residual at time t′ is superimposed at time t, h2(x t′ ) represents the current time t′ for x t′ The enhanced result of the colored pixels.

[0029] In one embodiment, the local end uses the residual information to enhance the future time-lapse colored pixels to improve the local rendering quality, including:

[0030] Get a historical moment sequence consisting of a series of historical moments {t i : i∈[1,I]}, and its corresponding residual information sequence Lighting information and 3D scene geometry information Combining the three into As the input of the shading residual model built based on the neural network, the geometric information and lighting information x of the given three-dimensional scene at the current time t′ are calculated and predicted. t′ The residual information g3(xt′ ), the residual information g3(x t′ ) is used as an enhancement pixel to be superimposed on the low-quality colored pixel to achieve colored pixel enhancement. The formula is expressed as:

[0031]

[0032]

[0033] h3(x t′ )=f′(x t′ )+g3(x t′ )

[0034] Among them, x is the illumination information and three-dimensional scene geometry information at a certain pixel, and Represent the historical moment t i against The high-quality rendering results and low-quality rendering results of f′(x t′ ) represents the current time t′ for x t′ The low-quality rendering result of h3(x t′ ) represents the current time t′ for x t′ The color pixel enhancement result is Represents the illumination information and three-dimensional scene geometry information x at a certain pixel at the current time t′ t′ Input to the coloring residual model built based on the neural network, and at the same time Input the coloring residual model, and the residual information obtained after model prediction, i represents the index of the historical moment, and I is the total number of historical moments.

[0035] To achieve the above-mentioned object of the invention, an embodiment provides a 3D scene cloud rendering method based on asynchronous transmission data, comprising the following steps:

[0036] The cloud uses a rendering pipeline to render the 3D scene data to obtain high-quality rendering results, and transmits the historical moment of obtaining the high-quality rendering results to the local end;

[0037] The local end uses the 3D scene data to color each pixel, and obtains a low-quality rendering result composed of the low-quality rendering results of all pixels;

[0038] The entire low-quality rendering result F′(X t′ ), 3D scene geometry information and lighting information X t′ And sports information composition As input to the augmentation model built on neural networks;

[0039] A historical moment sequence consisting of a series of historical moments i : i∈[1, I]}, and its corresponding high-quality rendering result Lighting information and 3D scene geometry information Motion Vector Composition As the auxiliary input of the enhancement model, the enhanced rendering result output at the current moment t′ is calculated and output, where i represents the index of the historical moment and I is the total number of historical moments.

[0040] To achieve the above-mentioned purpose of the invention, an embodiment provides a 3D scene cloud rendering system based on asynchronous data transmission, including a cloud end and a local end, wherein the cloud end is used to render the 3D scene data using a rendering pipeline to obtain a high-quality rendering result, and transmit the historical moment of obtaining the high-quality rendering result to the local end;

[0041] The local end is used to use the three-dimensional scene data to color each pixel to obtain a low-quality rendering result; it is also used to compare the received high-quality rendering result with the low-quality rendering result at the same historical moment, and calculate the residual information of the low-quality rendering result relative to the high-quality rendering result; it is also used to use the residual information to enhance the colored pixels at the current moment to improve the local rendering quality.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] By constructing the residual information of each pixel based on the high-quality rendering results of the cloud and the low-quality rendering results of the local end at the same historical moment, and using the residual information to enhance the pixel rendering results of the current moment, it is possible to use part of the image transmitted from the cloud to supplement the local image to improve the local rendering quality even when efficient and synchronous data transmission with the cloud is not possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 is a flow chart of a three-dimensional scene cloud rendering method based on asynchronous transmission data provided by an embodiment;

[0046] Figure 2 It is a flowchart of a three-dimensional scene cloud rendering method based on asynchronous transmission data provided by an embodiment;

[0047] Figure 3 is a flowchart of a three-dimensional scene cloud rendering method based on asynchronous transmission data provided by another embodiment;

[0048] Figure 4 This is a flowchart of a three-dimensional scene cloud rendering method based on asynchronous transmission data provided by another embodiment. DETAILED DESCRIPTION

[0049] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0050] In order to solve the problem that the cloud cannot perform efficient and synchronous data transmission with the local end, how can the local end use some high-quality rendering results transmitted at some historical moments to enhance the local low-quality rendering results? The embodiments provide a 3D scene cloud rendering method and system based on asynchronously transmitted data.

[0051] Figure 1 1 is a flow chart of a 3D scene cloud rendering method based on asynchronous transmission data provided by an embodiment. Figure 1 As shown, the three-dimensional scene cloud rendering method based on asynchronous transmission data provided in the embodiment includes the following steps:

[0052] Step 1: The cloud uses a rendering pipeline to render the 3D scene data to obtain high-quality rendering results.

[0053] The cloud has powerful computing power, so it can use the complete rendering pipeline to obtain high-quality images in real time, that is, high-quality rendering results. High-quality images refer to high-resolution images that contain global illumination information.

[0054] Step 2: The cloud will transmit the historical moments with high-quality rendering results to the local end.

[0055] In asynchronous transmission, there is no guarantee as to when and in what order the high-quality rendering results will be transmitted from the cloud to the local end. Therefore, when the cloud transmits the high-quality rendering results, it must also record the historical moment corresponding to the high-quality rendering result of the frame and transmit it to the local end, so that the local device can distinguish which historical moment the high-quality rendering results from the cloud come from and adopt different strategies according to different historical moments.

[0056] In an embodiment, when the cloud transmits the high-quality rendering results, it can simultaneously transmit multiple high-quality rendering results corresponding to multiple historical moments.

[0057] Step 3: The local end uses the 3D scene data to color each pixel to obtain a low-quality rendering result.

[0058] The local end uses the three-dimensional scene data for pixel-by-pixel shading, including preprocessing of the three-dimensional scene data and using the preprocessed data for pixel-by-pixel shading.

[0059] Among them, the preprocessing of 3D scene data includes the preprocessing of illumination information, 3D scene geometry information and motion information. The preprocessed illumination information can be a parameter representation of the light source or a light encoding obtained by inputting the light source parameter information into a neural network according to the selected shader. The preprocessed 3D scene geometry information contains information such as position, depth, normal vector, material, etc., which can be represented by a position map, depth map, normal map, and material map obtained by a simple rendering pipeline. The preprocessed motion information is represented by a motion vector, which records the motion of the object corresponding to each pixel in a series of images, and is used to track the corresponding relationship between pixels in images at different historical moments.

[0060] In an embodiment, using the preprocessed data to perform pixel-by-pixel shading includes using the preprocessed lighting information and the three-dimensional scene geometry information to perform pixel-by-pixel shading to obtain a low-quality rendering result.

[0061] In the embodiment, the shading process uses a shader in a rendering engine or a shader built based on a neural network. The pre-processed illumination information and the three-dimensional scene geometry information are used as inputs to the shader. When the shader is selected as a shader in a rendering engine, the illumination information is a parameter representation of the light source. When the shader is selected as a shader built based on a neural network, the illumination information is a light encoding obtained after the light source parameter information is input into the neural network.

[0062] In the embodiment, since different pixels correspond to different global information of objects, each pixel has a unique shader, and between different frames, the shader can be reused using motion vectors, so that two pixels connected by a motion vector use the same shader.

[0063] In step 4, the local end compares the received high-quality rendering result with the low-quality rendering result at the same historical moment, and calculates the residual information of the low-quality rendering result relative to the high-quality rendering result.

[0064] In the embodiment, after the local end receives the high-quality rendering result transmitted from the cloud, it is necessary to find the low-quality rendering result of the local same historical moment through the historical moment transmitted together with the high-quality rendering result, and subtract the two to obtain residual information. Before receiving the high-quality rendering result transmitted from the cloud next time, the local end uses the residual information to enhance the low-quality rendering result at the current moment. It should be noted that the high-quality rendering result of calculating the residual information is the high-quality rendering result obtained by rendering the historical moment closest to the current moment.

[0065] Step 5: The local end uses the residual information to enhance the colored pixels at the current moment to improve the local rendering quality.

[0066] In the embodiment, the motion vector is used to track the movement of the pixel, and the residual is appropriately superimposed on the corresponding pixel to enhance the rendering quality of the subsequent picture.

[0067] In one embodiment, if Figure 2 As shown, the local end uses the residual information to enhance the current shading pixels to improve the local rendering quality, including:

[0068] A monotonically increasing function w(Δt) with a value range of [0, 1] is used as a weight, and the product of the weight and the residual information is used as the first enhanced pixel. The first enhanced pixel is superimposed on the low-quality colored pixel to achieve colored pixel enhancement. The formula is expressed as:

[0069] g1(x t )=f(x t )-f′(x t ) (1)

[0070] h1(x t′ )=f′(x t′ )+w(t′-t)g1(x t ) (2)

[0071] Among them, x is the illumination information and 3D scene geometry information at a certain pixel, f(x t ) and f′(x t ) represent the historical moment t for x t High-quality rendering results and low-quality rendering results, g1(x t ) represents the residual information calculated at the historical moment t, f′(x t′ ) represents the current time t′ for x t′ The low-quality rendering result, w(t′-t) represents the weight to be used when the residual at time t′ is superimposed at time t, h1(x t′ ) represents the current time t′ for x t′ The enhanced result of the colored pixels.

[0072] In one embodiment, instead of directly using the residual information of the pixel corresponding to the high-quality rendering result and the low-quality rendering result at the most recent historical moment, the influence of the change of the 3D scene geometry information and illumination information at the pixel on the residual information is considered, and a neural network can be used to fit the change of the residual with the change of the 3D geometry information and illumination information. This method is introduced because the 3D scene geometry information and illumination information at the same pixel may change over time, and directly using the previous residual cannot reflect this change. Based on this, Figure 3 As shown, the local end uses the residual information to enhance the current shading pixels to improve the local rendering quality, including:

[0073] A monotonically increasing function w(Δt) with a value range of [0, 1] is used as a weight, and the product of the weight and the corrected residual information is used as the second enhanced pixel, and the second enhanced pixel is superimposed on the low-quality colored pixel to achieve colored pixel enhancement. The formula is expressed as:

[0074] g1(x t )=f(x t )-f′(x t ) (3)

[0075] g2(x t′ )=F(x t′ ;x t , g1(x t )) (4)

[0076] h2(x t′ )=f′(x t′ )+w(t′-t)g2(x t′ ) (5)

[0077] Among them, x is the illumination information and 3D scene geometry information at a certain pixel, f(x t ) and f′(x t ) represent the historical moment t for x t High-quality rendering results and low-quality rendering results, g1(x t ) represents the residual information calculated at the historical moment t, g2(x t′ ) represents the corrected residual information, F(x t′ ;x t , g1(t)) means that the illumination information and 3D scene geometry information at a certain pixel at the current time t′ are input into the correction model based on the neural network, and x t , g1(t) is input into the correction model as auxiliary information, and the corrected residual information obtained by model prediction is f′(x t′) represents the current time t′ for x t′ The low-quality rendering result, w(t′-t) represents the weight to be used when the residual at time t′ is superimposed at time t, h2(x t′ ) represents the current time t′ for x t′ The enhanced result of the colored pixels.

[0078] When constructing the correction model, a simple MLP network can be selected as the neural network. The specific model optimization process includes rendering a series of high-quality and high-resolution images, and simultaneously obtaining the three-dimensional scene geometry information, lighting information, and residual information corresponding to these images. Select a part of a series of images as key frames (corresponding to historical moments), and the remaining part as prediction frames (corresponding to the current moment). For each prediction frame, the most recent key frame is used to calculate the color pixel enhancement result of the prediction frame at the current moment according to formula (5), and the difference between the reinforcement result of the prediction frame and the actual high-quality rendering result is minimized as the optimization goal, and the neural network is optimized to obtain the correction model.

[0079] In one embodiment, the influence of timing and the influence of changes in 3D scene geometry and illumination information on the rendering result are comprehensively considered. Figure 4 As shown in the figure, the local end uses the residual information to enhance the pixels shaded in the future to improve the local rendering quality, including:

[0080] Get a historical moment sequence consisting of a series of historical moments {t i : i∈[1,I]}, and its corresponding residual information sequence Lighting information and 3D scene geometry information Combining the three into As the input of the shading residual model built based on the neural network, the geometric information and lighting information x of the given three-dimensional scene at the current time t′ are calculated and predicted. t′ The residual information g3(x t′ ), the residual information g3(x t′ ) is used as an enhancement pixel to be superimposed on the low-quality colored pixel to achieve colored pixel enhancement. The formula is expressed as:

[0081] g1(x t )=f(x t )-f′(x t ) (6)

[0082]

[0083] h3(x t′ )=f′(x t′ )+g3(x t′ ) (8)

[0084] Among them, x is the illumination information and three-dimensional scene geometry information at a certain pixel, and Represent the historical moment t i against The high-quality rendering results and low-quality rendering results of f′(x t′ ) represents the current time t′ for x t′ The low-quality rendering result of h3(x t′ ) represents the current time t′ for x t′ The color pixel enhancement result is Represents the illumination information and three-dimensional scene geometry information x at a certain pixel at the current time t′ t′ Input to the coloring residual model built based on the neural network, and at the same time Input the coloring residual model, and the residual information obtained after model prediction, i represents the index of the historical moment, and I is the total number of historical moments.

[0085] When constructing the shading residual model, the neural network selects a neural network with time series information, such as LSTM, RNN, etc. The specific model optimization process includes rendering a series of high-quality and high-resolution pictures, and simultaneously obtaining the three-dimensional scene geometry information, lighting information and residual information corresponding to these pictures. Select a part of a series of pictures as key frames (corresponding to historical moments), and the remaining part as predicted frames (corresponding to the current moment). For each predicted frame, all the key frames before it are used as the input of the neural network, and the shading pixel enhancement result of the predicted frame at the current moment is calculated according to formula (8), and the difference between the reinforcement result of the predicted frame and the real high-quality rendering result is minimized as the optimization goal, and the neural network is optimized to obtain a residual prediction model.

[0086] The embodiment also provides a three-dimensional scene cloud rendering method based on asynchronous transmission data, comprising the following steps:

[0087] Step 1: The cloud uses a rendering pipeline to render the three-dimensional scene data to obtain a high-quality rendering result, and transmits the historical moment of obtaining the high-quality rendering result to the local end;

[0088] Step 2: The local end uses the 3D scene data to color each pixel one by one, and obtains a whole low-quality rendering result composed of low-quality rendering results of all pixels;

[0089] Step 3: Take the entire low-quality rendering result F′(X t′ ), 3D scene geometry information and lighting information X t′ And sports information composition As input to the augmentation model built on neural networks;

[0090] Step 4: Take a historical moment sequence {t i : i∈[1, I]}, and its corresponding high-quality rendering result Lighting information and 3D scene geometry information Motion Vector Composition As the auxiliary input of the enhancement model, the enhanced rendering result output at the current time t′ is output after calculation.

[0091] When constructing the enhancement model, the neural network selects a neural network with time series information, such as LSTM, RNN, etc. The specific model optimization process includes rendering a series of high-quality and high-resolution images, and simultaneously obtaining the three-dimensional scene geometry information, lighting information, and residual information corresponding to these images. Select a part of a series of images as key frames (corresponding to historical moments), and the remaining part as predicted frames (corresponding to the current moment). For each predicted frame, all the key frames before it are used as the input of the neural network. According to the above process, the picture enhancement result of the predicted frame at the current moment is calculated, and the difference between the enhanced result of the predicted frame and the actual high-quality rendering result is minimized as the optimization goal, and the neural network is optimized to obtain an enhanced model.

[0092] Based on the same inventive concept, an embodiment also provides a three-dimensional scene cloud rendering system based on asynchronously transmitted data, including a cloud end and a local end, wherein the cloud end is used to use a rendering pipeline to render the three-dimensional scene data to obtain a high-quality rendering result, and transmit the historical moment of obtaining the high-quality rendering result to the local end; the local end is used to use the three-dimensional scene data to color pixel by pixel to obtain a low-quality rendering result; it is also used to compare the received high-quality rendering result with the low-quality rendering result of the same historical moment, and calculate the residual information of the low-quality rendering result relative to the high-quality rendering result; it is also used to use the residual information to enhance the colored pixels at the current moment to improve the local rendering quality.

[0093] The three-dimensional scene cloud rendering method and system based on asynchronous data transmission provided in the above embodiments can still use part of the picture transmitted from the cloud to supplement the local picture under the premise that efficient and synchronous data transmission with the cloud is impossible.

[0094] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A three-dimensional scene cloud rendering method based on asynchronous transmission data, characterized in that: The following steps are involved: The cloud uses a rendering pipeline to render the 3D scene data to obtain high-quality rendering results, and transmits the historical moment of obtaining the high-quality rendering results to the local end; The local end uses the 3D scene data to color each pixel, obtaining a low-quality rendering result; The local end compares the received high-quality rendering result with the low-quality rendering result at the same historical moment, and calculates the residual information of each pixel of the low-quality rendering result relative to the high-quality rendering result; The local end uses the residual information to enhance the colored pixels at the current moment to improve the local rendering quality. Each pixel has independent residual information. Between different frame images, the historical residual is reused according to the motion vector. That is, the two pixels connected by the motion vector use the same historical residual to enhance the rendering result. Before receiving the high-quality rendering result uploaded from the cloud next time, the local end uses the residual information to enhance the low-quality rendering result at the current moment.

2. The three-dimensional scene cloud rendering method based on asynchronous transmission data according to claim 1, characterized in that: The local end uses the three-dimensional scene data for pixel-by-pixel coloring, including preprocessing the three-dimensional scene data and using the preprocessed data for pixel-by-pixel coloring; The preprocessing of the three-dimensional scene data includes preprocessing of illumination information, three-dimensional scene geometry information and motion information, wherein the preprocessed illumination information is a parameter representation of a light source or a light encoding obtained by inputting the light source parameter information into a neural network; The pre-processed 3D scene geometry information includes position map, depth map, normal map, and material map; The preprocessed motion information is represented by motion vectors, which record the motion of the object corresponding to each pixel in a series of images and are used to track the corresponding relationship between pixels in images at different historical moments. The pixel-by-pixel shading using the preprocessed data includes using the preprocessed illumination information and the three-dimensional scene geometry information to perform pixel-by-pixel shading to obtain a low-quality rendering result; Preferably, the shading process uses a shader in a rendering engine or a shader built based on a neural network.

3. The three-dimensional scene cloud rendering method based on asynchronous transmission data according to claim 1, characterized in that: The local end uses the residual information to enhance the current moment colored pixels to improve the local rendering quality, including: A monotonically increasing function w(Δt) with a value range of [0,1] is used as the weight, and the product of the weight and the residual information is used as the first enhanced pixel. The first enhanced pixel is superimposed on the low-quality colored pixel to achieve colored pixel enhancement. The formula is expressed as: g1(x t )=f(x t )-f′(x t ) h1(x t′ )=f′(x t′ )+w(t′-t)g1(x t ) Among them, x is the illumination information and 3D scene geometry information at a certain pixel, f(x t ) and f′(x t ) represent the historical moment t for x t High-quality rendering results and low-quality rendering results, g1(x t ) represents the residual information calculated at the historical moment t, f′(x t′ ) represents the current time t′ for x t′ The low-quality rendering result, w(t′-t) represents the weight to be used when the residual at time t′ is superimposed at time t, h1(x t′ ) represents the current time t′ for x t′ The enhanced result of the colored pixels.

4. The three-dimensional scene cloud rendering method based on asynchronous transmission data according to claim 1, characterized in that: The local end uses the residual information to enhance the current moment colored pixels to improve the local rendering quality, including: A monotonically increasing function w(Δt) with a value range of [0,1] is used as a weight, and the product of the weight and the corrected residual information is used as the second enhanced pixel. The second enhanced pixel is superimposed on the low-quality colored pixel to achieve colored pixel enhancement. The formula is expressed as: g1(x t )=f(x t )-f′(x t ) g2(x t′ )=F(x t′ ;x t ,g1(x t )) h2(x t′ )=f′(x t′ )+w(t′-t)g2(x t′ ) Among them, x is the illumination information and 3D scene geometry information at a certain pixel, f(x t ) and f′(x t ) represent the historical moment t for x t High-quality rendering results and low-quality rendering results, g1(x t ) represents the residual information calculated at the historical moment t, g2(x t′ ) represents the corrected residual information, F(x t′ ;x t , g1(t)) represents the illumination information of a pixel at the current time t′ and the three-dimensional scene geometry information x t′ Input into the correction model based on neural network, and at the same time t ,g1(t) is input into the correction model as auxiliary information, and the corrected residual information obtained by model prediction, f′(x t′ ) represents the low-quality rendering result of x at the current time t′, w(t′-t) represents the weight to be used when the residual at time t′ is superimposed at time t, h2(x t′ ) represents the current time t′ for x t′ The enhanced result of the colored pixels.

5. The three-dimensional scene cloud rendering method based on asynchronous transmission data according to claim 1, characterized in that: The local end uses the residual information to enhance the pixels colored at the future moment to improve the local rendering quality, including: Get a historical moment sequence consisting of a series of historical moments {t i :i∈[1,I]}, and its corresponding residual information sequence Lighting information and 3D scene geometry information Combining the three into As the input of the shading residual model built based on the neural network, the geometric information and lighting information x of the given three-dimensional scene at the current time t′ are calculated and predicted. t′ The residual information g3(x t′ ), the residual information g3(x t′ ) is used as an enhancement pixel to be superimposed on the low-quality colored pixel to achieve colored pixel enhancement. The formula is expressed as: h3(x t′ )=f′(x t′ )+g3(x t′ ) Among them, x is the illumination information and three-dimensional scene geometry information at a certain pixel, and Represent the historical moment t i against The high-quality rendering results and low-quality rendering results of f′(x t′ ) represents the current time t′ for x t′ The low-quality rendering result of h3(x t′ ) represents the current time t′ for x t′ The color pixel enhancement result is Represents the illumination information and three-dimensional scene geometry information x at a certain pixel at the current time t′ t′ Input to the coloring residual model built based on the neural network, and at the same time Input the coloring residual model, and the residual information obtained after model prediction, i represents the index of the historical moment, and I is the total number of historical moments.

6. A three-dimensional scene cloud rendering method based on asynchronous transmission data, characterized in that: The following steps are involved: The cloud uses a rendering pipeline to render the 3D scene data to obtain high-quality rendering results, and transmits the historical moment of obtaining the high-quality rendering results to the local end; The local end uses the 3D scene data to color each pixel, and obtains a low-quality rendering result composed of the low-quality rendering results of all pixels; The entire low-quality rendering result F′(X t′ ), 3D scene geometry information and lighting information X t′ And sports information composition As input to the augmentation model built on neural networks; A historical moment sequence consisting of a series of historical moments i ,i∈[1,I]}, and its corresponding high-quality rendering result Lighting information and 3D scene geometry information Motion Vector Composed of As the auxiliary input of the enhancement model, the enhanced rendering result output at the current moment t′ is calculated and output, where i represents the index of the historical moment and I is the total number of historical moments.

7. A three-dimensional scene cloud rendering system based on asynchronous data transmission, comprising a cloud end and a local end, characterized in that: The cloud end is used to render the three-dimensional scene data using a rendering pipeline to obtain a high-quality rendering result, and transmit the historical moment of obtaining the high-quality rendering result to the local end; The local end is used to color the three-dimensional scene data pixel by pixel to obtain a low-quality rendering result; It is also used to compare the received high-quality rendering result with the low-quality rendering result at the same historical moment, and calculate the residual information of each pixel of the low-quality rendering result relative to the high-quality rendering result; It is also used to use residual information to enhance the colored pixels at the current moment to improve the local rendering quality. Each pixel has independent residual information. Between different frame images, historical residuals are reused according to motion vectors. That is, two pixels connected by motion vectors use the same historical residual to enhance the rendering result. Before receiving the high-quality rendering result uploaded from the cloud next time, the local end uses the residual information to enhance the low-quality rendering result at the current moment.

Citation Information

Cited By

  • End-cloud collaborative rendering method and system based on mixing of object stream and video stream

    CN120416534A