Rendering model training method, rendering method, device, equipment and storage medium

Abstract

The present invention relates to a rendering model training method, a rendering method, an apparatus, a device and a storage medium, the method comprising: inputting an image of a second scene into a first machine learning module to extract corresponding high-dimensional features; inputting the high-dimensional features into a second machine learning module to obtain corresponding lighting parameters according to the high-dimensional features; inputting the lighting parameters and the data to be rendered of the first scene into a differentiable rendering module, and using a differentiable rendering method to render the first scene according to the lighting parameters to obtain a lighting-fused rendering image; calculating the loss according to the rendered image and the label image, calculating a gradient map for the loss, and performing back propagation to update the weights of the first machine learning module and the second machine learning module using a gradient descent method. The rendering model training method and rendering method of the present invention are advantageous for accurately extracting lighting parameters, and can make the lighting of the overall image obtained by rendering consistent, and obtain a highly realistic result image.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a rendering model training method, a rendering method, a device, a equipment and a storage medium. Background Art

[0002] The statements herein merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Rendering models can be divided into local illumination models and global illumination models. The local illumination model only considers the illumination effect of the light source on the surface of the object, while the global illumination model not only considers the illumination effect of the light source on the object, but also needs to consider the mutual influence between objects. The global illumination model generally uses a physics-based method and uses the laws of the physical world to simulate and calculate the propagation and intensity of light everywhere, so that the rendering results of the global illumination model have a very high sense of reality. Classic global illumination algorithms include radiosity, path tracing, and photon mapping.

[0004] The acquisition and application of lighting parameters are essential in the rendering process. In order to achieve high-quality and realistic rendering results, the existing rendering methods still have inconveniences and defects in the acquisition and application of lighting parameters, and further improvement is urgently needed. Summary of the invention

[0005] The purpose of the present invention is to provide a new rendering model training method, rendering method, device, equipment and storage medium.

[0006] The purpose of the present invention is achieved by adopting the following technical solutions. A rendering model training method proposed in the present invention includes the following steps: obtaining one or more training data, wherein the training data includes data to be rendered of a first scene, an image of a second scene with lighting, and a labeled image of the first scene; inputting the image of the second scene into a first machine learning module, the first machine learning module is used to extract high-dimensional features corresponding to the image of the second scene, wherein the high-dimensional features are used to characterize information about objects and / or environments contained in the image of the second scene; inputting the high-dimensional features into a second machine learning module, the second machine learning module is used to obtain lighting parameters corresponding to the second scene according to the high-dimensional features; comparing the lighting parameters with the first scene; The data to be rendered of the scene is input into a differentiable rendering module, and the differentiable rendering module is used to render the first scene according to the lighting parameters using a differentiable rendering method to obtain a lighting-fused rendering image; the loss is calculated according to the lighting-fused rendering image and the label image, a gradient map is calculated for the loss, and backpropagation is performed on the differentiable rendering module, the first machine learning module, and the second machine learning module to update the weights of the first machine learning module and the second machine learning module using a gradient descent method; after multiple rounds of training iterations, after the target conditions are met, a rendering model including the first machine learning module, the second machine learning module, and the differentiable rendering module is obtained.

[0007] The purpose of the present invention is also achieved by the following technical solutions. According to a rendering method proposed in the present application, the method comprises: inputting the data to be rendered of the third scene into a differentiable rendering module in a trained rendering model obtained according to the aforementioned rendering model training method, inputting the image of the fourth scene into a first machine learning module in the rendering model, extracting illumination parameters from the image of the fourth scene and rendering the third scene according to the illumination parameters, and obtaining an illumination-fused rendering image of the third scene.

[0008] The object of the present invention is also achieved by adopting the following technical solutions. A rendering model training device proposed in the present application includes: an acquisition module and a training module;

[0009] The acquisition module is used to: acquire one or more training data, wherein the training data includes data to be rendered of a first scene, an image of a second scene with illumination, and a label image of the first scene;

[0010] The training module is used to: input the image of the second scene into the first machine learning module, the first machine learning module is used to extract high-dimensional features corresponding to the image of the second scene, wherein the high-dimensional features are used to characterize the information of the object and / or environment contained in the image of the second scene; input the high-dimensional features into the second machine learning module, the second machine learning module is used to obtain the lighting parameters corresponding to the second scene according to the high-dimensional features; input the lighting parameters and the data to be rendered of the first scene into the differentiable rendering module, the differentiable rendering module is used to render the first scene according to the lighting parameters using a differentiable rendering method to obtain a lighting-fused rendering image; calculate the loss based on the lighting-fused rendering image and the label image, calculate the gradient map of the loss, and perform back propagation on the differentiable rendering module, the first machine learning module and the second machine learning module to update the weights of the first machine learning module and the second machine learning module using a gradient descent method; after multiple rounds of training iterations, after reaching the target condition, a rendering model including the first machine learning module, the second machine learning module and the differentiable rendering module is obtained.

[0011] The purpose of the present invention is also achieved by the following technical solutions. A rendering device proposed in the present application includes an inference module, which is used to: input the data to be rendered of the third scene into the differentiable rendering module in the trained rendering model obtained by the above-mentioned rendering model training device, input the image of the fourth scene into the first machine learning module in the rendering model, extract the illumination parameters of the image of the fourth scene and render the third scene according to the illumination parameters, so as to obtain the illumination-fused rendering image of the third scene.

[0012] The purpose of the present invention is also achieved by the following technical solution. A rendering device proposed in this application includes: a memory for storing non-temporary computer-readable instructions; and a processor for running the computer-readable instructions, so that the processor implements any of the aforementioned rendering model training methods or rendering methods when executing.

[0013] The purpose of the present invention is also achieved by the following technical solution: A computer-readable storage medium proposed in this application is used to store non-temporary computer-readable instructions, and when the non-temporary computer-readable instructions are executed by a computer, the computer executes any of the aforementioned rendering model training methods or rendering methods.

[0014] The present invention has obvious advantages and beneficial effects compared with the prior art. By means of the above technical scheme, the rendering model training method, rendering method, device, equipment and storage medium proposed in the present invention have at least the following advantages and beneficial effects:

[0015] 1. The present invention proposes a method for obtaining lighting parameters and corresponding lighting fusion rendering based on differentiable rendering.

[0016] 2. After adopting the present invention, lighting information can be extracted from real scenes, lighting fusion can be performed on the reconstructed or rendered scenes, and rendering can be performed through path tracing and other methods to achieve consistent overall image lighting and highly realistic result images, streaming media, or synthesized short videos.

[0017] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following specifically cites a preferred embodiment and describes it in detail with the accompanying drawings as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flowchart of a rendering model training method according to an embodiment of the present invention;

[0019] Figure 2 is a flowchart of a rendering method according to an embodiment of the present invention;

[0020] Figure 3 is a flowchart of a rendering method according to another embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of the structure of a first machine learning module provided by an embodiment of the present invention;

[0022] Figure 5 is a schematic diagram of the structure of a second machine learning module provided by an embodiment of the present invention;

[0023] Figure 6 is a schematic diagram of the structure of a neural network block in a second machine learning module provided by an embodiment of the present invention;

[0024] Figure 7 is a schematic diagram of a rendering model training device according to an embodiment of the present invention;

[0025] Figure 8 is a schematic diagram of a rendering device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the specific implementation methods, structures, features and effects of the rendering model training method, rendering method, device, equipment and storage medium proposed in the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0027] It should be noted that, in this article, relational terms such as "first", "second", etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. In addition, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0028] It should be noted that the images mentioned in this application can be independent pictures or video frames.

[0029] Figure 1 FIG. 1 is a schematic flow chart of an embodiment of the rendering model training method of the present invention. In some embodiments of the present invention, please refer to Figure 1 The rendering model training method of the present invention mainly includes the following steps:

[0030] Step S11, obtaining one or more training data. The training data includes the data to be rendered of the first scene, the image of the second scene, and the label image of the first scene. The data to be rendered of the first scene and the image of the second scene are used as training inputs, and the label image of the first scene is used as a training label.

[0031] The data to be rendered of the first scene may include a three-dimensional model of the first scene and other necessary parameters; optionally, the data to be rendered of the first scene may not necessarily include illumination parameters.

[0032] The image of the second scene is an image with lighting. In fact, only with lighting can the environment and objects in the scene be seen, otherwise it will be pitch black. Optionally, the second scene may not have specific objects, but only the background environment.

[0033] It should be noted that the images mentioned in this article generally refer to two-dimensional images.

[0034] In some optional examples, the first scene includes virtual objects. Optionally, the first scene may even consist of only virtual objects. Or in other optional examples, the first scene includes both virtual objects and real objects in the real scene.

[0035] In some optional examples, the second scene includes a real environment and / or a real object in the real scene. Optionally, the second scene may be a completely real scene consisting of a real object and a real environment.

[0036] It should be noted that there is a corresponding relationship between the label and the input of the model. Specifically, a picture of a real scene collected by shooting or the like (for example, the training data is a reconstruction of the real scene), or a picture obtained by artificial rendering or artificial adjustment of the rendering with high realism (at least the lighting is convenient for high realism) (for example, the training data includes virtual objects that do not exist in the real scene) can be used as a label image. Optionally, during the training process, the training input and the virtual objects or real objects / backgrounds involved in the label are matched. For example, the input can be a combination of virtual objects and real scenes, or a combination of pure virtual objects, and the label is the actual photographed / high-realistic picture video corresponding to these (that is, the combination of virtual objects and real scenes, or pure virtual objects).

[0037] Generally speaking, labeled images include all objects, environments, and scenes (such as indoor scenes, wilderness scenes, etc.) in rendered images, but are more realistic than rendered images.

[0038] It should be noted that the label image of the first scene may include virtual objects or real objects in the real scene. For example, the label image may include an image of a real scene obtained by means of shooting, etc., and may include a virtual 3D object designed artificially, etc. Generally speaking, the label image is a specific object scene that combines the real and virtual.

[0039] It should be noted that the objects in this article are not limited to "things" in the usual sense, but can also be "people", or a part of a person (such as a face), or even the environment (such as space, floor, wall, etc.).

[0040] It should be noted that the present invention does not impose any specific restrictions on virtual objects. For example, the virtual object can be placed in the scene by any means, including but not limited to artificially designed virtual objects, or virtual objects obtained by reconstructing real objects, such as reconstructed virtual people, faces, objects or scenes.

[0041] Optionally, the first scene and the second scene may be associated, or the first scene and the second scene may be unrelated.

[0042] Step S12, input the image of the second scene into the first machine learning module (also called the image feature extraction module), and the first machine learning module is used to extract high-dimensional features corresponding to the image of the second scene. The high-dimensional features are used to characterize the information of the objects and / or environment contained in the image of the second scene. In some examples, the high-dimensional features can reflect the situation of the three-dimensional space corresponding to the two-dimensional image, but generally speaking, the high-dimensional features cannot directly represent the three-dimensional model of the second scene.

[0043] Optionally, the high-dimensional features obtained in this step are not vectors, but feature maps.

[0044] Step S13: input the high-dimensional features into a second machine learning module, where the second machine learning module is used to obtain lighting parameters corresponding to the second scene based on the high-dimensional features.

[0045] The lighting parameters are used to represent the properties of the lighting. Optionally, the lighting parameters include, but are not limited to, one or more of: the number of light sources, the lighting type of each light source (e.g., point light source or surface light source), the position of the light source, the lighting intensity (or luminous flux), color, direction, size, etc. In fact, only with lighting can the environment and objects in the scene be seen, otherwise it will be pitch black.

[0046] Step S14, inputting the data to be rendered of the first scene into the differentiable rendering module, and inputting the lighting parameters extracted in step S13 into the differentiable rendering module, the differentiable rendering module is used to render the first scene according to the lighting parameters using a differentiable rendering method to obtain a lighting-fused rendering image.

[0047] It should be noted that the lighting information can be fused (or migrated) to any target scene for rendering during the rendering stage. Specifically, the aforementioned fusion includes but is not limited to: directly using the extracted lighting parameters, or replacing at least some of the original lighting parameters of the target scene with the extracted lighting parameters, or superimposing at least some of the original lighting parameters of the target scene with the extracted lighting parameters.

[0048] Optionally, the aforementioned rendering of the first scene according to the illumination parameters specifically includes: performing global illumination rendering of the first scene according to the extracted illumination parameters. Optionally, the realistic rendering performed by the present invention refers to: a physics-based global illumination model in computer graphics. It should be noted that the present invention does not limit the specific method of global illumination rendering, including but not limited to path tracing rendering (ray tracing rendering), radiosity rendering or photon mapping rendering.

[0049] It should be noted that in a general rendering process, by inputting the data of the scene to be rendered into the rendering module, a rendered image can be output. In step S14, during rendering, by integrating the extracted lighting parameters into the scene to be rendered, a more realistic rendering result can be obtained.

[0050] Step S15, calculate the loss based on the lighting-fused rendering image of the first scene and the label image of the first scene, calculate the gradient map of the loss, and perform back propagation on the differentiable rendering module, the first machine learning module and the second machine learning module to update the weights (or parameters of the machine learning module) of the first machine learning module and the second machine learning module by gradient descent. Thus, the first machine learning module and the second machine learning module are optimized as a whole at the same time.

[0051] The gradient graph refers to the visualized gradient result obtained during the gradient descent process.

[0052] Among them, the aforementioned loss is also called a loss function or a cost function.

[0053] Note that when using the labeled image to calculate the loss, the label is fused with the image obtained by lighting rendering to calculate the loss, so in practice, the labeled image may not be related to the image of the second scene.

[0054] Step S16, after multiple rounds of training iterations, after reaching the target conditions, a rendering model including a first machine learning module, a second machine learning module and a differentiable rendering module is obtained.

[0055] The aforementioned target reaching conditions include but are not limited to: reaching a preset number of iterations, and / or reaching a convergence condition, etc. The purpose is to minimize the loss function as much as possible.

[0056] In some embodiments of the present application, the rendering of the first scene according to the illumination parameters using a differentiable rendering method in the aforementioned step S14 includes: when calculating the objective function (e.g., spherical harmonics), calculating the integral relative to the input parameters (also referred to as the parameters to be rendered) to obtain the output result (also referred to as the rendering result). The back-propagation of the differentiable rendering module, the first machine learning module, and the second machine learning module in the aforementioned step S15 includes: when back-propagating the objective function, differentiating the loss corresponding to the output result relative to the input parameters.

[0057] About differentiable rendering: The differentiable rendering module is not a neural network model built with perceptron as the basic unit in the traditional sense. For ease of understanding, it can be intuitively understood as a module that can perform back propagation similar to a neural network. Generally speaking, the differentiable rendering module does not need to train its parameters, nor does it have a node tree structure similar to machine learning or deep learning visualization. Taking global illumination rendering as an example, spherical harmonics appear in global illumination rendering as the basis of global illumination rendering, and the calculation of spherical harmonics involves complex integration links, which can be considered similar to the forward propagation of machine learning models. Differentiability refers to the inverse process of the previous rendering, that is, the differentiation process of the rendering result relative to the renderer input parameters, which can be considered similar to the back propagation of the machine learning model. Because traditional renderers often introduce more complex probability-based sampling links in order to simplify the integration link, the calculation process becomes a discrete summation and cannot be differentiated.

[0058] It should be noted that unlike the general machine learning / deep learning method that uses parallel numerical operations to calculate gradients, in the training process of differentiable rendering, the gradient descent method obtains data results through the process of image to numerical operation (graph-compute).

[0059] Therefore, in the solution of the example of this application, when gradient descent is performed through loss function calculation, the first machine learning module, the second machine learning module, and the differentiable rendering module will all participate and perform. All the "loss calculation + gradient descent" used for back propagation here are unified at the computational level, that is, they can be passed to each other for a series of subsequent operations.

[0060] The rendering model training method proposed in this application has the following three advantages.

[0061] First, in the traditional fields of computer vision (CV) and computer graphics (CG), machine learning / deep learning (ml / dl) and renderers are separate processes and uses. For example, ml / dl is used alone to obtain the corresponding parameters in the picture or video, and then the parameters are applied to the pipeline of the traditional renderer alone. This is a multi-stage task that is relatively independent of each stage and requires manual intervention and manual adjustment. In the existing method, the acquisition process of the illumination parameters is separated from the rendering process, that is, the illumination parameters are acquired independently first, for example, a neural network model for acquiring illumination parameters is trained separately, and then the acquired illumination parameters are directly used for rendering. In the solution of the example of this application, whether it is the training process or the forward use process, multiple modules are integrated into a unified whole, input image / video-output corresponding image / video. When additional virtual objects are introduced, a series of parameters obtained in the middle, such as illumination parameters, need to be used. For the training process, this application adopts end-to-end (also called input end to output end) training.

[0062] Second, in the scheme of the example of this application, the data is interpretable and transferable across modules. The differentiable rendering module used by the method shown in this application is reverse differentiable, and the gradient parameters of the back-propagation of the differentiable renderer can be directly passed to the previous first machine learning module / second machine learning module, thereby eliminating the need for manual participation. Specifically, existing machine learning and deep learning focus on the value of the data corresponding to the image, rather than the visualized result; while the input and output of the model proposed in the present invention are both visualized, wherein the output obtained through the differentiable renderer module is an image, but during reverse propagation, the input part is converted into corresponding parameters, such as data and lighting parameters corresponding to the image, and is seamlessly passed directly to the previously trained module (no intermediate layer processing such as data regularization or data scaling is required, because the size of the gradient data and the updated weights are uniform), through the image to the front-end data flow.

[0063] In other words, the differentiable renderer obtains an image, and it does not need to care what form the data in it is in. When calculating and performing reverse propagation, the image is reversed to obtain the corresponding number. The differentiable renderer is used to: convert the image into a corresponding gradient map, and then convert the gradient map into a corresponding number (for example, int8 format). However, the traditional renderer does not include the process of reverse gradient propagation because it adopts a probability-based sampling method. Therefore, when calculating the rendering equation (for example, the bidirectional reflectance distribution function brdf), when it comes to complex integration links, the probability-based sampling method of the traditional renderer causes the integration process to be irreversible. The differentiable renderer of the present application can perform the inverse process of the integration process.

[0064] Third, in the solution of the example in this application, the graph-compute is correct and feasible from image to data, and it can also reflect the unity of training + forward use as a whole.

[0065] It should be noted that the focus of the present invention is not on the design of the differentiable renderer itself, so the present invention does not limit the specific structure of the differentiable rendering module, but can adopt any differentiable rendering module.

[0066] As an optional example of the first machine learning module of the present invention, Figure 4 As shown, the first machine learning module (image feature extraction module) can be a deep learning module, and the specific structure of the first machine learning module can include k downsampling layers and 2k circular convolution layers, where k is a natural number.

[0067] As an optional example of the second machine learning module of the present invention, Figure 5 and Figure 6 As shown, the second machine learning module is a multi-layer perceptron module, which may include three neural network blocks, and each neural network block shares parameters; wherein each neural network block includes a first hidden layer, a first activation layer, a second hidden layer, a second activation layer and a third hidden layer arranged in sequence.

[0068] It should be noted that, generally speaking, rendering is to obtain a two-dimensional image based on three-dimensional data, while inverse rendering is to obtain three-dimensional data based on a two-dimensional image. In the method exemplified in the present invention, inverse rendering is achieved by using a differentiable rendering module, and the "inverse" process essentially refers to the back propagation of the gradient, that is, the differentiability of the rendering calculation process. The back propagation process is an executable capability of a specific module (such as a differentiable rendering module) here, which is used to reversely transmit the gradient data of gradient descent, etc.

[0069] The embodiment of the present invention also provides a rendering method, see Figure 2 The rendering method of the present invention mainly includes the following steps:

[0070] Step S21, inputting the to-be-rendered data of the third scene into the differentiable rendering module in the trained rendering model, inputting the image of the fourth scene into the first machine learning module in the rendering model, using the first machine learning module and the second machine learning module in the rendering model to extract illumination parameters from the image of the fourth scene, and using the differentiable rendering module in the rendering model to render the third scene according to the illumination parameters, to obtain an illumination-fused rendering image of the third scene. The rendering model is obtained according to any of the aforementioned rendering model training methods.

[0071] The third scene includes virtual objects. Optionally, the third scene may include only virtual objects, or include both virtual objects and real objects in the real scene.

[0072] The fourth scene includes real objects. Optionally, the fourth scene may even only include real objects.

[0073] Optionally, the third scene and the fourth scene may be associated, or the third scene and the fourth scene may be unrelated.

[0074] It should be noted that the lighting information can be fused into any target scene rendering during the rendering stage. Specifically, the aforementioned fusion includes but is not limited to: directly using the extracted lighting parameters, or replacing at least some of the original lighting parameters of the target scene with the extracted lighting parameters, or superimposing at least some of the original lighting parameters of the target scene with the extracted lighting parameters.

[0075] By using the rendering method proposed in the present invention, lighting fusion is achieved. For example, when rendering the third scene, an image of the fourth scene obtained by shooting the real scene and possible virtual objects (such as faces, people, objects, scenes, etc.) that may be artificially added or placed in the third scene by various means are used to obtain a rendering result with a realistic lighting effect.

[0076] The present invention proposes: an overall training method and use method for a first machine learning module F (also called an inference module), a second machine learning module M (also called an illumination parameter extraction module), and a differentiable rendering module renderer. In the rendering stage, the extracted illumination information can be transferred to any scene rendering, and after reconstruction, a realistic rendering image with illumination angle, brightness, color temperature, etc. that fits the input real scene image can be obtained. The method proposed in the present invention can be used to transfer the illumination effect to a variety of scenes, such as faces, human bodies, environmental reconstruction and other application targets, and ensure the consistency of the overall image illumination.

[0077] Figure 3 It is the process setting for infer (model inference). The third scene (scene to be rendered) can include "artificially added virtual objects", that is, one or more virtual objects to be rendered that do not exist in reality can be artificially added. The role of the fourth scene (real scene) here is to provide information about the acquisition of lighting parameters. Please refer to Figure 3In one embodiment of the present invention, a rendering method provided by the present invention includes: firstly, performing a reverse rendering operation on an input real scene image to obtain a high-dimensional feature corresponding to the image; then, extracting illumination parameters from the image as desired illumination information through a multi-layer perceptron module; finally, in the rendering stage, the illumination information can be transferred to any scene rendering, and after reconstruction, a realistic rendering image with illumination angle, brightness, color temperature, etc. that fits the input real scene image can be obtained. Through this method, the illumination effect can be transferred to a variety of scenes, such as face, human body, environment reconstruction and other application targets, and the consistency of the overall image illumination can be ensured.

[0078] Extract lighting information from real scenes, perform lighting fusion on reconstructed or rendered scenes, and render them through path tracing and other methods.

[0079] In some embodiments of the present invention, the rendering method of the present invention includes:

[0080] Step S1, based on the experience of deep learning algorithm model design, a set of image feature extraction modules (inference module F) based on deep learning algorithm model is designed. The structure of the image feature extraction module is as follows: Figure 4 .

[0081] Wherein, the values ​​of m and n are 4 and 8 respectively. It should be noted that the present invention does not limit the values ​​of m and n, and m and n can be set to any positive integers according to the computing power of the device used. Optionally, m and n can be set to k and 2*k respectively, where k is a natural number.

[0082] Use Figure 4 The image feature extraction module with the structure shown can achieve the expected effect.

[0083] After the training process is completed, the image feature extraction module can be used to extract features from the input real scene image. Based on the operation mode of this image feature extraction module, a real scene image I can be input to obtain a high-dimensional feature ft. The process is:

[0084] F(I)→ft

[0085] Among them, F represents the image feature extraction module based on the deep learning algorithm model, I represents the input real scene picture, and ft represents the obtained high-dimensional features.

[0086] It should be noted that the present invention does not limit the format of the real scene image I, including but not limited to the RGB format.

[0087] Step S2, based on the multi-layer perceptron and the high-dimensional features obtained in the previous step, a series of illumination parameters required for the next step can be obtained. The multi-layer perceptron is designed as follows Figure 5 .

[0088] In practical applications, a multilayer perceptron can contain three layers of network blocks, and each block shares parameters.

[0089] Among them, the structure of each block is as follows Figure 6 shown.

[0090] Through this multi-layer perceptron, multiple lighting parameters can be finally obtained. The process is:

[0091] M(ft)→l_positon, l_intensity, l_pose, l_size

[0092] Among them, M represents Figure 5 The multi-layer perceptron module consists of multiple blocks. ft represents the high-dimensional features obtained in the previous step. l_position, l_intensity, l_pose, and l_size represent the obtained lighting parameters position, intensity (color), direction, and size. The specific parameters vary depending on the type of lighting. For example, for point light sources, the direction and size are not within the expected range of lighting parameters.

[0093] Step S3: To obtain a stable and usable image feature extraction module F and a multi-layer perceptron module M, it is necessary to further train the parameters in the module through iterative training.

[0094] The overall training process is as follows: Based on the illumination parameters l_position, l_intensity, l_pose, l_size obtained in the previous step and the corresponding differentiable rendering module, path tracing rendering is performed on a specific object scene to obtain an output image O that is similar to the illumination state of the real scene image I of the input image. The process is:

[0095] renderer(l_positon, l_intensity, l_pose, l_size)→O

[0096] Then, the loss is calculated based on the output image O and the label data O′. It should be noted that the present invention does not limit the calculation method of the loss, for example, a general method can be used. The corresponding gradient map is further calculated, and the reverse rendering process of the differentiable rendering module of step S3 is combined with the image feature extraction module and the multi-layer perceptron module of the aforementioned steps S1 and S2 to jointly update the back-propagation weights (also called model parameters or neural network parameters).

[0097] It should be noted that the present invention does not limit the specific neural network structure used in the aforementioned step S1 and step S2; for example, the number of layers of the neural network is not limited, but the number of layers can be adjusted according to actual conditions. In practice, the number of network layers can be regarded as a hyperparameter, wherein a hyperparameter refers to a parameter whose value can be arbitrarily set before the training process begins, and the hyperparameter is not obtained through training.

[0098] It should be noted that the training process proposed in the present invention performs overall back propagation of gradient descent. The entire step S1 to step S3 is an overall training process. The differentiable rendering module performs gradient calculation and back propagation in the reverse rendering process, which will also affect and update the network parameters of the image feature extraction module in step S1.

[0099] After the training is completed, an overall model (a complete link) can be obtained that can estimate lighting parameters and render highly realistic result images, including the corresponding first machine learning module F, the second machine learning module M, and the differentiable rendering module renderer.

[0100] It should be noted that since the differentiable rendering module generally involves a huge amount of computing, large computing resources such as servers and cloud computing can generally be used during the implementation process, and GPUs can be introduced to accelerate the computing process.

[0101] When in use, input any real-scene RGB picture or RGB video frame, and adjust the position, size and other properties of the object or scene to be rendered as the input of the differentiable rendering module renderer, which can produce a real lighting effect similar to the input picture or video frame.

[0102] It should be noted that the aforementioned illumination parameters l_position, l_intensity, l_pose, and l_size are intermediate results of the entire link and have clear and specific physical meanings.

[0103] When using the trained model, combined with the object to be rendered obj in the scene to be rendered, the complete process is as follows:

[0104] I→F(I)→M(ft)→renderer(obj, l_positon, l_intensity, l_pose, l_size)→O

[0105] It should be noted that the expressions of renderer(l_positon, l_intensity, l_pose, l_size) mentioned in the model training process and renderer(obj, l_positon, l_intensity, l_pose, l_size) mentioned in the model use process are different, but the specific process is the same. The object obj to be rendered is mentioned in the formula of the model use process because the user can change the object to be rendered at any time during the specific implementation process; while the object obj to be rendered is not mentioned in the formula of the model training process because the object to be rendered is generally pre-set during the training process, so the object to be rendered in the formula is simplified.

[0106] The rendering method proposed in the present invention can automatically extract lighting information, perform lighting fusion and obtain rendering effects. It can be applied to pure virtual scenes, augmented reality (virtual + reality), or real scenes and their corresponding reconstructions to obtain realistic rendering results.

[0107] It should be noted that the existing rendering method does not include step S3 of the present invention. After obtaining the lighting parameters, the existing rendering method only renders the object in combination with the lighting parameters through a traditional renderer.

[0108] The rendering model training method and rendering method of the present invention are advantageous for accurately extracting lighting parameters, and can make the lighting of the entire image obtained by rendering consistent, thereby obtaining a result image with a high sense of reality.

[0109] An embodiment of the present invention also provides a rendering model training device, which mainly includes: an acquisition module and a training module.

[0110] The acquisition module is used to acquire one or more training data, the training data including the to-be-rendered data of the first scene, the image of the second scene with illumination, and the label image of the first scene. The training module is used to:

[0111] Inputting the image of the second scene into the first machine learning module, the first machine learning module is used to extract high-dimensional features corresponding to the image of the second scene, wherein the high-dimensional features are used to characterize information of objects and / or environments contained in the image of the second scene;

[0112] Inputting the high-dimensional features into a second machine learning module, the second machine learning module is used to obtain lighting parameters corresponding to the second scene according to the high-dimensional features;

[0113] Inputting the illumination parameters and the data to be rendered of the first scene into a differentiable rendering module, wherein the differentiable rendering module is used to render the first scene according to the illumination parameters in a differentiable rendering manner to obtain an illumination-fused rendering image;

[0114] Calculate the loss according to the illumination-fused rendered image and the label image, calculate the gradient map of the loss, and perform backpropagation on the differentiable rendering module, the first machine learning module, and the second machine learning module to update the weights of the first machine learning module and the second machine learning module by using a gradient descent method;

[0115] After multiple rounds of training iterations, when the target conditions are met, a rendering model including a first machine learning module, a second machine learning module and a differentiable rendering module is obtained.

[0116] In addition, the rendering model training device shown in the embodiment of the present invention may also include modules and units corresponding to the rendering model training methods described in the aforementioned embodiments, and its detailed description and technical effects can refer to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0117] An embodiment of the present invention further provides a rendering device, which mainly includes: an inference module. The inference module is used to: input the to-be-rendered data of the third scene into the differentiable rendering module in the trained rendering model obtained by the aforementioned rendering model training device, input the image of the fourth scene into the first machine learning module in the rendering model, extract the illumination parameters of the image of the fourth scene, and render the third scene according to the illumination parameters, so as to obtain an illumination-fused rendering image of the third scene.

[0118] In addition, the rendering device shown in the embodiment of the present invention may also include modules and units for executing the rendering methods described in the aforementioned embodiments, and their detailed descriptions and technical effects can refer to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0119] Figure 7 is a schematic block diagram illustrating a rendering model training device according to an embodiment of the present invention. Figure 7 As shown, the rendering model training device 100 according to an embodiment of the present application includes a memory 101 and a processor 102.

[0120] The memory 101 is used to store non-temporary computer-readable instructions. Specifically, the memory 101 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc.

[0121] The processor 102 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the rendering model training device 100 to perform desired functions. In one embodiment of the present application, the processor 102 is used to run the computer-readable instructions stored in the memory 101, so that the rendering model training 100 performs all or part of the steps of the rendering model training method of each embodiment of the present application described above.

[0122] For detailed description and technical effects of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0123] Figure 8 1 is a schematic block diagram illustrating a rendering device according to an embodiment of the present invention. Figure 8 As shown, the rendering device 200 according to the embodiment of the present application includes a memory 201 and a processor 202 .

[0124] The memory 201 is used to store non-temporary computer-readable instructions. Specifically, the memory 201 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc.

[0125] The processor 202 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the rendering device 200 to perform desired functions. In one embodiment of the present application, the processor 202 is used to execute the computer-readable instructions stored in the memory 201, so that the rendering device 200 performs all or part of the steps of the rendering method of each embodiment of the present application described above.

[0126] For detailed description and technical effects of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0127] An embodiment of the present invention further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on a device, the device executes the above-mentioned related method steps to implement the rendering method in the above-mentioned embodiment.

[0128] An embodiment of the present invention further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the rendering method in the above-mentioned embodiment.

[0129] In addition, an embodiment of the present invention further provides a device, which may specifically be a chip, a component or a module, and the device may include a connected processor and a memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the rendering method in the above-mentioned method embodiments.

[0130] Among them, the device, computer storage medium, computer program product or chip provided by the present invention are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0131] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A rendering model training method, characterized in that: The method comprises the following steps: Acquire one or more training data, wherein the training data includes to-be-rendered data of a first scene, an image of a second scene with illumination, and a labeled image of the first scene; Inputting the image of the second scene into a first machine learning module, wherein the first machine learning module is used to extract high-dimensional features corresponding to the image of the second scene, wherein the high-dimensional features are used to characterize information of objects and / or environments contained in the image of the second scene; Inputting the high-dimensional features into a second machine learning module, where the second machine learning module is used to obtain illumination parameters corresponding to the second scene according to the high-dimensional features; Inputting the illumination parameters and the to-be-rendered data of the first scene into a differentiable rendering module, wherein the differentiable rendering module is used to render the first scene according to the illumination parameters in a differentiable rendering manner to obtain an illumination-fused rendering image; Calculating a loss based on the illumination-fused rendered image and the label image, calculating a gradient map for the loss, and performing backpropagation on the differentiable rendering module, the first machine learning module, and the second machine learning module to update the weights of the first machine learning module and the second machine learning module using a gradient descent method; After multiple rounds of training iterations, after the target conditions are met, a rendering model including the first machine learning module, the second machine learning module and the differentiable rendering module is obtained.

2. The rendering model training method according to claim 1, characterized in that: The rendering of the first scene according to the illumination parameters by using a differentiable rendering method includes: when calculating an objective function, calculating an integral relative to an input parameter to obtain an output result; The back-propagation of the differentiable rendering module, the first machine learning module and the second machine learning module includes: when back-propagating the objective function, differentiating the loss corresponding to the output result relative to the input parameter.

3. The rendering model training method according to claim 1, characterized in that: The lighting parameters include one or more of the number of light sources, the lighting type, position, lighting intensity, color, direction, and size of each light source.

4. The rendering model training method according to claim 1, characterized in that: The first scene includes virtual objects, or includes both virtual objects and real objects in the real scene; the second scene includes real objects in the real scene.

5. The rendering model training method according to claim 1, characterized in that: The first machine learning module is a deep learning module, including k downsampling layers and 2k circular convolution layers, where k is a natural number.

6. The rendering model training method according to claim 1, characterized in that: The second machine learning module is a multi-layer perceptron module, including three neural network blocks, each of which shares parameters; wherein each neural network block includes a first hidden layer, a first activation layer, a second hidden layer, a second activation layer and a third hidden layer arranged in sequence.

7. The rendering model training method according to claim 1, characterized in that: The rendering of the first scene according to the illumination parameters includes: Perform global illumination rendering on the first scene according to the illumination parameters.

8. A rendering method, characterized in that: The method comprises the following steps: The data to be rendered of the third scene is input into the differentiable rendering module in the trained rendering model obtained by the rendering model training method according to any one of claims 1 to 7, and the image of the fourth scene is input into the first machine learning module in the rendering model to extract lighting parameters from the image of the fourth scene and render the third scene according to the lighting parameters to obtain a lighting-fused rendering image of the third scene.

9. A rendering model training device, characterized in that: The device comprises: an acquisition module, configured to acquire one or more training data, wherein the training data includes data to be rendered of a first scene, an image of a second scene with illumination, and a label image of the first scene; and A training module is used to: input the image of the second scene into a first machine learning module, the first machine learning module is used to extract high-dimensional features corresponding to the image of the second scene, wherein the high-dimensional features are used to characterize information about objects and / or environment contained in the image of the second scene; input the high-dimensional features into a second machine learning module, the second machine learning module is used to obtain lighting parameters corresponding to the second scene according to the high-dimensional features; input the lighting parameters and the data to be rendered of the first scene into a differentiable rendering module, the differentiable rendering module is used to render the first scene according to the lighting parameters using a differentiable rendering method to obtain a lighting-fused rendered image; calculate the loss based on the lighting-fused rendered image and the label image, calculate a gradient map for the loss, and perform backpropagation on the differentiable rendering module, the first machine learning module and the second machine learning module to update the weights of the first machine learning module and the second machine learning module using a gradient descent method; after multiple rounds of training iterations, after reaching the target condition, a rendering model including the first machine learning module, the second machine learning module and the differentiable rendering module is obtained.

10. A rendering device, characterized in that: The device comprises: An inference module is used to: input the data to be rendered of the third scene into the differentiable rendering module in the trained rendering model obtained by the rendering model training device according to claim 9, and input the image of the fourth scene into the first machine learning module in the rendering model to extract lighting parameters from the image of the fourth scene and render the third scene according to the lighting parameters to obtain a lighting-fused rendering image of the third scene.

11. A rendering model training device, comprising: a memory for storing non-transitory computer-readable instructions; as well as A processor is used to run the computer-readable instructions so that when the computer-readable instructions are executed by the processor, the rendering model training method described in any one of claims 1 to 7 is implemented.

12. A rendering device, comprising: a memory for storing non-transitory computer-readable instructions; as well as A processor is used to run the computer-readable instructions, so that when the computer-readable instructions are executed by the processor, the rendering method of claim 8 is implemented.

13. A computer storage medium, characterized in that: It includes computer instructions, which, when executed on a device, enable the device to execute the rendering model training method as described in any one of claims 1 to 7, or the rendering method as described in claim 8.

Images