Micro-photon mapping method and system

Abstract

The invention provides a method and a system for micro photon mapping, and the method comprises the steps: obtaining an initial scene parameter based on a target picture, and generating a rendering result and an extended light path through random progressive photon mapping based on the initial scene parameter; calculating a loss function with a preset target picture based on the rendering result, and solving gradients of extended light path geometry and extended light path contribution about scene parameters at the same time to obtain an extended light path geometry gradient and an extended light path contribution gradient; based on the extended light path geometric gradient and the extended light path contribution gradient, a scene parameter gradient value of the loss function about scene parameters is calculated through a chain rule; the scene parameters are updated based on the loss function about the scene parameter gradient value, and primary optimization is completed; and through multiple times of optimization, the loss function is minimized, and the micro-photon mapping is completed. According to the method, the problem that gradient calculation cannot be carried out on a photon mapping algorithm in the prior art is solved, and the universality of micro rendering is improved.

Description

Technical Field

[0001] The present invention relates to the field of differentiable rendering technology, and in particular to a differentiable photon mapping method and system. Background Art

[0002] Rendering has been an important research issue in the field of computer graphics for many years. Its purpose is to transform the abstract description of the scene into a concrete picture. The theory and methods of this process have become increasingly mature. In recent years, people have begun to pay attention to the gradient solution of this process, that is, differentiable rendering.

[0003] Specifically, if rendering is regarded as a function, then the input of the function is scene parameters such as geometry, material, camera, etc., and the output is generally a two-dimensional image. Differentiable rendering is to solve the gradient of the output result of the rendering function with respect to the input parameters.

[0004] At present, differentiable rendering can be divided into two categories according to the rendering pipeline it uses: differentiable rasterization and differentiable ray tracing. The research focus of differentiable rasterization is to transform the rasterization process that is not differentiable itself, and provide the gradient of pixel color with respect to object geometry through approximate gradient, approximate rendering or analytical anti-aliasing. Differentiable ray tracing requires solving the gradient of the output of the rendering equation with respect to the input parameter. Since the rendering equation involves calculating integrals, and the integral is relatively complex, the Monte Carlo path tracing method is usually used for estimation. However, in the process of calculating the gradient, since the integrand may be discontinuous with respect to the scene parameters (such as visibility and specular reflection), this leads to the need to specially handle this part of the discontinuity when estimating the gradient using the same Monte Carlo method. Researchers define it as boundary integral. In recent years, researchers have proposed solutions such as boundary sampling and reparameterization for the discontinuities caused by geometry. There are relatively few studies on discontinuities caused by materials.

[0005] In addition, there are many specific solutions to the rendering equations in ray tracing, such as unidirectional path tracing, bidirectional path tracing, photon mapping, etc. Different methods have their own advantages and disadvantages in terms of efficiency and accuracy. For example, the photon mapping method has great advantages over other methods when rendering specular-diffuse-specular type light paths and caustics. For differentiable rendering, gradient calculation is also required for each rendering method. However, existing work only focuses on the calculation of unidirectional path tracing and bidirectional path tracing, and no gradient calculation method for photon mapping is proposed. This is because photon mapping is usually used to process scenes with light paths containing discontinuous materials such as complex specular reflections and refractions. As mentioned above, there is currently no theory on how to correctly calculate the gradient of this type of light path.

[0006] An important application of differentiable rendering is to solve the problem of reverse rendering, that is, to infer scene parameters from the target image. The general process of using differentiable rendering for reverse rendering is: first obtain an initial estimate of the scene parameters, and then iterate the following steps: perform differentiable rendering on the scene to obtain the rendering result, solve the loss function between the rendering result and the target image, calculate the gradient of the loss function with respect to the scene parameters, and optimize the scene parameters through a gradient descent-based optimization method to minimize the loss function. There are many rendering methods, and different methods are suitable for rendering different scenes. The same is true for reverse rendering. Currently, there is no method to complete the reverse rendering task using differentiable photon mapping, which limits the versatility of differentiable rendering. Summary of the invention

[0007] The present invention provides a differentiable photon mapping method and system, which are used to solve the problem in the prior art that the photon mapping algorithm cannot be gradient calculated, and improve the versatility of differentiable rendering.

[0008] The present invention provides a differentiable photon mapping method, characterized by comprising: Get initial scene parameters based on the target image; Generating a rendering result and an extended light path through random progressive photon mapping based on the initial scene parameters; Based on the loss function between the rendering result calculation and the preset target image, the gradients of the extended light path geometry and the extended light path contribution with respect to the scene parameters are solved to obtain the extended light path geometry gradient and the extended light path contribution gradient; Calculate the scene parameter gradient value of the loss function with respect to the scene parameter by the chain rule based on the extended light path geometric gradient and the extended light path contribution gradient; The scene parameters are updated based on the gradient value of the loss function with respect to the scene parameters to complete an optimization. After multiple optimizations, the loss function is minimized and the differentiable photon mapping is completed.

[0009] According to a differentiable photon mapping method provided by the present invention, the initial scene parameters are obtained based on the target map, including: Initial scene parameters are predicted from the target image through neural networks, COLMAP or manual design.

[0010] According to a differentiable photon mapping method provided by the present invention, the initial scene parameters are obtained, and a rendering result and an extended light path are generated by random progressive photon mapping based on the scene parameters, specifically including: Based on the scene parameters, starting from the light source and the viewpoint through random progressive photon mapping, a photon light path and a viewpoint light path are obtained through ray tracing; Connecting the photon optical path and the viewpoint optical path in the middle to form an extended optical path from the light source to the viewpoint; The contribution of the extended light path is calculated to generate a rendering result.

[0011] According to a differentiable photon mapping method provided by the present invention, the loss function between the rendering result calculation and a preset target image specifically includes: Find the target pixel for each pixel in the rendering result, compare the color and position between pixels, and generate a comparison result; A difference value is determined according to the comparison result, and the loss function is determined based on the difference value.

[0012] According to a differentiable photon mapping method provided by the present invention, the simultaneous solution of the extended light path geometry and the extended light path contribution gradients with respect to the scene parameters to obtain the extended light path geometry gradient and the extended light path contribution gradient specifically includes: Introducing a first constraint between the extended light path geometry and the scene parameter, constructing a functional relationship between the extended light path geometry and the scene parameter based on the first constraint, performing gradient solving on the functional relationship, and obtaining an extended light path geometry gradient of the extended light path geometry with respect to the scene parameter; The discontinuous parts of the geometric terms and weight terms in the extended light path contribution are ignored, and a second constraint is introduced on the material term in the extended light path contribution. A continuous function of the extended light path contribution with respect to the scene parameters is constructed, and the gradient of the function is solved to obtain the extended light path contribution gradient with respect to the scene parameters.

[0013] According to a differentiable photon mapping method provided by the present invention, a first constraint is introduced between the extended light path geometry and the scene parameter, a functional relationship between the extended light path geometry and the scene parameter is constructed based on the first constraint, and a gradient solution is performed on the functional relationship to obtain the extended light path geometry gradient of the extended light path geometry with respect to the scene parameter, specifically including: Introduce half-vector constraints, fixed position constraints, fixed direction constraints and fixed relative position constraints between the extended light path geometry and the scene parameters; Constructing a specific functional relationship of an extended optical path manifold based on vector constraints, fixed position constraints, fixed direction constraints and fixed relative position constraints; A gradient solution is performed based on the functional relationship of the extended light path manifold to obtain an extended light path geometry gradient of the extended light path geometry with respect to the scene parameter.

[0014] According to a differentiable photon mapping method provided by the present invention, the scene parameter gradient value of the loss function with respect to the scene parameter is calculated by the chain rule based on the extended light path geometric gradient and the extended light path contribution gradient, specifically comprising: Calculate the loss function based on the rendering result and the target image, and solve the gradient of the loss function with respect to the color and position of each pixel in the rendering result to obtain the loss function gradient; The pixel position is regarded as the mean of the intersection of the extended light path within the pixel and the imaging plane, so that the pixel position becomes a function of the light path geometry, and the gradient of the function is solved to obtain the pixel position gradient; Based on the light path geometric gradient and the extended light path contribution gradient, the pixel position gradient and the loss function gradient, the scene parameter gradient value of the loss function with respect to the scene parameters is solved by the chain rule.

[0015] According to a differentiable photon mapping method provided by the present invention, scene parameters are updated based on the gradient value of the scene parameters of the loss function to complete an optimization, which specifically includes: Based on the scene parameter gradient value, the scene parameter is updated through an optimization method based on gradient descent to complete a round of optimization process.

[0016] The present invention also provides a differentiable photon mapping system, the system comprising: A scene parameter acquisition module is used to obtain initial scene parameters based on the target image; An extended light path generation module, used to obtain scene parameters, and generate rendering results and an extended light path through random progressive photon mapping based on the scene parameters; A gradient calculation module, used to calculate the loss function between the rendering result and a preset target image, and to solve the gradients of the extended light path geometry and the extended light path contribution with respect to the scene parameters to obtain the extended light path geometry gradient and the extended light path contribution gradient; A gradient back-propagation module, used for calculating the scene parameter gradient value of the loss function with respect to the scene parameter by the chain rule based on the extended light path geometric gradient and the extended light path contribution gradient; The scene parameter update module is used to update the scene parameters based on the loss function with respect to the scene parameter gradient value, complete an optimization, and after multiple optimizations, minimize the loss function and complete the differentiable photon mapping.

[0017] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any one of the above-described differentiable photon mapping methods is implemented.

[0018] Based on a differentiable photon mapping method and system provided by the present invention, a rendering result and a gradient of the rendering result with respect to scene parameters are obtained through a differentiable photon mapping rendering method, so that the photon mapping rendering method can be integrated into the optimization task based on gradient descent, thereby improving the versatility of differentiable rendering. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a schematic flow chart of the differentiable photon mapping method provided by the present invention.

[0021] Figure 2 It is a schematic diagram of the execution process of the differentiable photon mapping method provided by the present invention.

[0022] Figure 3 It is a schematic diagram of module connection of the differentiable photon mapping system provided by the present invention.

[0023] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention.

[0024] Reference numerals: 110: scene parameter acquisition module; 120: extended light path generation module; 130: gradient calculation module; 140: gradient back-propagation module; 150: scene parameter update module; 410: processor; 420: communication interface; 430: memory; 440: communication bus. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Combine the following Figure 1 and Figure 2 , describing the differentiable photon mapping method of the present invention, including: step 100, obtaining initial scene parameters based on the target image.

[0027] Specifically, it includes: predicting the initial scene parameters from the target image through neural network, COLMAP or manual design.

[0028] Steps 200 to 500 are repeatedly executed until the loss function is minimized.

[0029] Step 200: Generate rendering results and extended light paths through random progressive photon mapping based on the initial scene parameters.

[0030] In the present invention, based on the scene parameters, random progressive photon mapping is used to start from the light source and the viewpoint, and the photon light path and the viewpoint light path are obtained by ray tracing; the photon light path and the viewpoint light path are connected in the middle to form an extended light path from the light source to the viewpoint; the contribution of the extended light path is calculated to generate a rendering result.

[0031] Specifically, the core idea of ​​the photon mapping method is to divide the sampling process of a light path into two parts: photon light path sampling starting from the light source and viewpoint light path sampling starting from the viewpoint, and connect the obtained photon light path and viewpoint light path somewhere in the middle to form an extended light path.

[0032] For a photon light path, each intersection point on the light path can be expressed in two-dimensional coordinates: The optical path can be formally expressed as ,in is the light source sampling point, is the intersection point that the light path passes through when it bounces in the scene. The contribution of the light path can be expressed as follows.

[0033] (1).

[0034] in is the intensity of light emitted by the light source, and They are the geometric term and scattering function term in the light transmission process. It can be further divided into the rough scattering function and the smooth scattering function , the rough scattering function It can be considered as a continuous function, corresponding to diffuse or rough reflective materials, and smooth scattering function It can be expressed as the following formula.

[0035] .

[0036] in is a continuous function, and h is a discontinuous function or a function close to being discontinuous.

[0037] On the other hand, for a viewpoint light path, it can be similarly expressed as ,in For viewpoint, is the intersection point that the viewpoint light path passes through when it bounces in the scene. The contribution of this light path can be expressed as follows.

[0038] .

[0039] in The contribution weight of the ray to the viewpoint.

[0040] The photon mapping method connects the photon optical path with the viewpoint optical path to form a complete extended optical path from the light source to the viewpoint. and a viewpoint path Can be combined into an extended optical path , the contribution of the extended optical path is as follows.

[0041] .

[0042] in, Indicates connection The non-negative weights of It is greater than 0 when the distance is very small, and the contribution of the extended light path is considered greater than 0 only when the photon light path is close to the end point of the viewpoint light path.

[0043] Step 300: Calculate the loss function between the rendering result and a preset target image, and simultaneously solve the gradients of the extended light path geometry and the extended light path contribution with respect to the scene parameters to obtain the extended light path geometry gradient and the extended light path contribution gradient.

[0044] In the present invention, the loss function between the rendering result and the preset target image is calculated based on the rendering result, specifically including: Find the target pixel for each pixel in the rendering result, compare the color and position between pixels, and generate a comparison result; A difference value is determined according to the comparison result, and the loss function is determined based on the difference value.

[0045] At the same time, the gradients of the extended light path geometry and the extended light path contribution with respect to the scene parameters are solved to obtain the extended light path geometry gradient and the extended light path contribution gradient, which specifically include: Introducing a first constraint between the extended light path geometry and the scene parameter, constructing a functional relationship between the extended light path geometry and the scene parameter based on the first constraint, performing gradient solving on the functional relationship, and obtaining an extended light path geometry gradient of the extended light path geometry with respect to the scene parameter; The discontinuous parts of the geometric terms and weight terms in the extended light path contribution are ignored, and a second constraint is introduced on the material term in the extended light path contribution. A continuous function of the extended light path contribution with respect to the scene parameters is constructed, and the gradient of the function is solved to obtain the extended light path contribution gradient with respect to the scene parameters.

[0046] Specifically, solve the extended optical path geometry About scene parameters Gradient And the extended optical path contribution About scene parameters Gradient ,in is the scene parameter vector, and k is the scene parameter dimension. Note that although and There is no explicit inclusion of , but in fact the light path geometry and light path contribution are related to the scene parameters, that is, .

[0047] For the gradient calculation of the extended light path geometry with respect to the scene parameters, since both are vectors, the actual solution target is a Jacobi matrix.

[0048] .

[0049] Solving this matrix is ​​a relatively complex problem. The core lies in constructing the functional relationship between the light path vertices and the scene parameters. On this issue, previous work is referenced and extended, which will be presented in detail next.

[0050] Firstly, a series of simple constraints on the extended light path and scene parameters are introduced. Then, based on these constraints, a complete functional relationship between the two is constructed, and finally, the gradient is solved.

[0051] There are four specific constraints, including: Half-vector constraint: Half-vector constraint is for three adjacent vertices on the light path , requiring that for the vertex , in its local coordinate system Down, the direction of the emitted light and the incident light direction The half vector remains unchanged.

[0052] .

[0053] in, is a 2x3 matrix representing the local tangent vector of the vertex, Represents a half vector.

[0054] Fixed position constraint: A fixed position constraint requires the position of a vertex in the path In one embodiment, the scene is expressed in the form of a triangular patch, the triangle where the vertex is located and the barycentric coordinates on the triangle are recorded, and the constraint is expressed as constraining the barycentric coordinates to remain unchanged.

[0055] .

[0056] in( ) represent the three vertices of a triangle, Calculation Point In the triangle Note that the barycentric coordinates are actually 2-dimensional.

[0057] Fixed direction constraint: The fixed direction constraint requires that the outgoing light from a certain vertex in the path The orientation of is maintained in the local coordinate system of this vertex.

[0058] .

[0059] in, It is a 2x3 matrix representing the local tangent vector of the vertex.

[0060] Fixed relative position constraint: The relative position between two adjacent vertices on the fixed optical path remains unchanged.

[0061] .

[0062] in, It is a 2x3 matrix representing the local tangent vector of the vertex.

[0063] These four constraints are all 2D vector constraints and can be uniformly expressed as .

[0064] Based on the constrained path space manifold, a specific extended light path space manifold can be constructed for gradient solution. Specifically, three ways to construct the extended light path space manifold are designed: Fixed position manifold,Construct the fixed position manifold through the following constraints: For the starting point of the expansion path and end point (Light source and camera) Apply fixed position constraints; On the optical path Apply half-vector constraints to the smooth material vertices outside; right and Use fixed direction constraints; On the optical path The emission direction of the non-smooth material vertices outside is constrained by a fixed direction.

[0065] Diffuse manifold, constructed by the following constraints: For the starting point of the expansion path and end point (Light source and camera) Apply fixed position constraints; On the optical path Apply half-vector constraints to the smooth material vertices outside; right Use fixed direction constraints; On the optical path The emission direction of the non-smooth material vertices outside is constrained by a fixed direction; right Use fixed position constraints.

[0066] Photon manifold,constructed by the following constraints: For the starting point of the expansion path and end point (Light source and camera) Apply fixed position constraints; On the optical path Apply half-vector constraints to the smooth material vertices outside; right Use fixed direction constraints; On the optical path The emission direction of the non-smooth material vertices outside is constrained by a fixed direction; right Adopt fixed relative position constraints.

[0067] All three manifold construction methods are used to extend the optical path geometry. n+m+2 constraints are applied, and each constraint can be written as in the form of.

[0068] .

[0069] At this time, the extended path (light path) space manifold can be expressed as a set of paths that satisfy the following conditions.

[0070] .

[0071] The light path gradient solution on the extended light path space manifold, for each extended light path on each manifold, is subject to n+m+2 2D constraints, which is exactly the same as the degree of freedom of the path, and because the constraints are independent of each other, so given the scene parameters In case of of is unique, based on which we can get the functional relationship between the two . And for the gradient of this function The solution can be obtained according to the implicit function derivative theory. .

[0072] Expand the light path contribution to the gradient of the scene parameters and consider solving , usually f is a three-dimensional function, but for the sake of simplicity of description, it can be regarded as a one-dimensional function.

[0073] .

[0074] in This is the geometric gradient of the optical path required in the previous chapter.

[0075] From formula (4), we can get the function The specific expression, however, about may be discontinuous, which makes and It cannot be solved directly. Specifically, The geometry items contained in , Material Item , weight item There are many methods to deal with the discontinuities in the geometric terms, which can be used to ignore the discontinuities. , under the condition that it is a kernel function, it can be set to any continuous function. In one embodiment, it can be defined as the following formula.

[0076] .

[0077] in , where r is a fixed radius.

[0078] So now only the material item is left Maybe about parameters or Not continuous.

[0079] According to the previous text, According to the material type, it can be divided into rough material and smooth material, among which only smooth material may be a discontinuous function, see formula (2). However, in the rendering process, the discontinuous function is actually a characteristic function.

[0080] .

[0081] When Satisfy the conditions hour is a constant Intuitively, smooth materials satisfy the constraints. The space formed by is continuous. Therefore, we only need to ensure Constraints are always satisfied , formula (2) can be simplified to the following formula.

[0082] .

[0083] becomes a continuous function. After the above discussion, Eventually it became about It is a continuous function that can be solved for gradient.

[0084] In the three path space manifolds defined in the present invention, half-vector constraints are used for smooth materials. This constraint is actually equivalent to the constraint in formula (14): , so when calculating the gradient on the given three manifolds, it can be guaranteed that about is continuous.

[0085] Step 400: Calculate the scene parameter gradient value of the loss function with respect to the scene parameter by using the chain rule based on the extended light path geometric gradient and the extended light path contribution gradient.

[0086] Specifically, it includes: calculating the loss function based on the rendering result and the target image, and solving the gradient of the loss function with respect to the color and position of each pixel in the rendering result to obtain the gradient of the loss function; The pixel position is regarded as the mean of the intersection of the extended light path within the pixel and the imaging plane, so that the pixel position becomes a function of the light path geometry, and the gradient of the function is solved to obtain the pixel position gradient; Based on the light path geometric gradient and the extended light path contribution gradient, the pixel position gradient and the loss function gradient, the scene parameter gradient value of the loss function with respect to the scene parameters is solved by the chain rule.

[0087] Step 500: Update the scene parameters based on the loss function with respect to the gradient value of the scene parameters to complete an optimization.

[0088] Specifically, it includes: based on the scene parameter gradient value, updating the scene parameter through an optimization method based on gradient descent, and completing a round of optimization process.

[0089] Based on a differentiable photon mapping method provided by the present invention, a rendering result and a gradient of the rendering result with respect to a scene parameter are obtained through the differentiable photon mapping rendering method, so that the photon mapping rendering method can be integrated into the optimization task based on gradient descent, thereby improving the versatility of differentiable rendering.

[0090] In a specific embodiment, the rendering result and the extended light path are obtained based on the random progressive photon mapping method, the loss function between the rendering result and the target image is calculated, and the gradient of the light path geometry and contribution with respect to the scene parameters is obtained. The gradient of the extended light path is solved by the method described in the previous chapter, and the gradient of the loss function with respect to the scene parameters is obtained by the chain rule. This gradient can be used to optimize the scene parameters.

[0091] Random progressive photon mapping,The random progressive photon mapping method is an improvement on the ordinary photon mapping rendering method.,This method is an iterative method. In each iteration, the photon light path and the viewpoint light path are obtained by ray tracing from the light source and the viewpoint, the extended light path is constructed, and the light path contribution is calculated, and finally the rendering result is obtained.

[0092] Initializing the algorithm requires specifying the number of viewpoint light paths emitted in each iteration , the number of photon paths And the initial collection radius , then start iteration, in each round of iteration: The first step is to construct the viewpoint light path, which is usually done by emitting rays from the viewpoint to several random positions in each pixel (the total number is ), and then use the ray tracing method to intersect the rays, calculate the direct lighting and scattered light. This process can be combined with direct light sampling, importance sampling and other methods to improve efficiency. For each ray, when it intersects with the diffuse material surface or high roughness surface, stop further scattering and indirect lighting calculations, and define the viewpoint light path obtained at this time as ,in For viewpoint, is the intersection of the light path and the scene, especially the last intersection is the collection point, and the set of all collection points is called the collection point set. For each viewpoint light path, save its light path geometry Contribution .

[0093] The second is random sampling from the light source positions, and emit photon light paths from these positions. The light path intersection and scattered light calculation are also performed using the ray tracing method. When the light path length is greater than one and the light path intersects with the diffuse reflective material surface or the high roughness surface, the collection point concentration is found to be located near the intersection point. Specifically, the photon light path at this time can be expressed as ,in is the light source sampling point, For the intersection points that the light path passes through when it bounces in the scene, find all the points that satisfy point , and get the corresponding viewpoint optical path and extended optical path , and then calculate the contribution of the extended light path using formula (4): .

[0094] After obtaining each extended light path and its contribution, each pixel will perform color calculations based on the corresponding extended light paths and update the radius applied in the next iteration. Specifically, assume that the weighted contribution of the extended light path collected by pixel p before the i+1th iteration is , the total number of weighted extended light paths is , the collection radius is , the total number of collected in the i+1th round is , the total contribution of the extended optical path is ,in For sampling The probability of can be obtained from the ray tracing process. Then the total extended light path contribution after the i+1th round is , total weighted extended optical path number , the collection radius is Respectively.

[0095] .

[0096] .

[0097] .

[0098] Assume that there are a total of rounds of iterations, the final pixel color is .

[0099] The loss function of the rendering result and the target image defines a rendered image With target picture The loss function between ,in , is the color of each pixel (h, w are the number of vertical and horizontal pixels of the image), is the position of each pixel in screen space. In one embodiment, (loss function) It can be simply defined as the pixel-by-pixel color difference between the rendered image and the target image.

[0100] .

[0101] Such as the widely used L1 or L2 loss functions.

[0102] In another embodiment, pixel position information can also be incorporated into the loss function. For example, a one-to-one match between pixels of a rendered image and a target image can be found. And define the loss function.

[0103] .

[0104] in , represents the weight of color and position in the loss function. This method is conducive to capturing the global correspondence between images. The specific matching method can use bipartite graph matching, optical flow method or optimal transmission method.

[0105] Gradient calculation and back propagation, the ultimate goal is to solve the loss function About scene parameters The gradient of , according to the chain rule.

[0106] .

[0107] in, The pixel color can be calculated according to the formula (18) and the light path contribution gradient calculation method described in the previous section. The solution can be obtained as follows: for each pixel, the extended light path corresponding to the image plane is defined as , find the mean of all intersection points (These two processes are directly differentiable) and let the following equation hold.

[0108] .

[0109] The present invention is not limited to calculating the gradient under a specific manifold. In actual use, one or more manifold constraints can be selected according to actual needs for gradient calculation. A more recommended method is to use a fixed position manifold to calculate the gradient when using a simple loss function such as the L1 / L2 loss function. When using a complex loss function such as a loss function that contains position information, using the diffuse reflection manifold and the photon manifold to calculate the gradient can obtain a non-zero value. To guide optimization.

[0110] Based on a differentiable photon mapping method provided by the present invention, a rendering result and a gradient of the rendering result with respect to a scene parameter are obtained through the differentiable photon mapping rendering method, so that the photon mapping rendering method can be integrated into the optimization task based on gradient descent, thereby improving the versatility of differentiable rendering.

[0111] refer to Figure 3 The present invention also discloses a differentiable photon mapping system, the system comprising: The scene parameter acquisition module 110 is used to acquire the initial scene parameters based on the target image.

[0112] An extended light path generation module 120, used to obtain scene parameters, and generate rendering results and an extended light path based on the scene parameters through random progressive photon mapping; A gradient calculation module 130, configured to calculate a loss function between the rendering result and a preset target image, and to simultaneously solve the gradients of the extended light path geometry and the extended light path contribution with respect to the scene parameters, to obtain an extended light path geometry gradient and an extended light path contribution gradient; A gradient back propagation module 140, configured to calculate a scene parameter gradient value of the loss function with respect to the scene parameter by using the chain rule based on the extended light path geometric gradient and the extended light path contribution gradient; The scene parameter updating module 150 is used to update the scene parameters based on the loss function with respect to the scene parameter gradient value, and complete an optimization. After multiple optimizations, the loss function is minimized, and the differentiable photon mapping is completed.

[0113] The initial scene parameters are obtained based on the target map, including: Initial scene parameters are predicted from the target image through neural networks, COLMAP or manual design.

[0114] Generating rendering results and extending light paths through random progressive photon mapping based on the initial scene parameters, specifically including: Based on the scene parameters, starting from the light source and the viewpoint through random progressive photon mapping, a photon light path and a viewpoint light path are obtained through ray tracing; Connecting the photon optical path and the viewpoint optical path in the middle to form an extended optical path from the light source to the viewpoint; The contribution of the extended light path is calculated to generate a rendering result.

[0115] The loss function between the rendering result and the preset target image is calculated based on the rendering result, specifically including: Find the target pixel for each pixel in the rendering result, compare the color and position between pixels, and generate a comparison result; A difference value is determined according to the comparison result, and the loss function is determined based on the difference value.

[0116] The step of simultaneously solving the gradients of the extended light path geometry and the extended light path contribution with respect to the scene parameters to obtain the extended light path geometry gradient and the extended light path contribution gradient specifically includes: Introducing a first constraint between the extended light path geometry and the scene parameter, constructing a functional relationship between the extended light path geometry and the scene parameter based on the first constraint, performing gradient solving on the functional relationship, and obtaining an extended light path geometry gradient of the extended light path geometry with respect to the scene parameter; The discontinuous parts of the geometric terms and weight terms in the extended light path contribution are ignored, and a second constraint is introduced on the material term in the extended light path contribution. A continuous function of the extended light path contribution with respect to the scene parameters is constructed, and the gradient of the function is solved to obtain the extended light path contribution gradient with respect to the scene parameters.

[0117] Introducing a first constraint between the extended light path geometry and the scene parameter, constructing a functional relationship between the extended light path geometry and the scene parameter based on the first constraint, performing gradient solving on the functional relationship, and obtaining the extended light path geometry gradient of the extended light path geometry with respect to the scene parameter, specifically comprising: Introduce half-vector constraints, fixed position constraints, fixed direction constraints and fixed relative position constraints between the extended light path geometry and the scene parameters; Constructing a specific functional relationship of an extended optical path manifold based on vector constraints, fixed position constraints, fixed direction constraints and fixed relative position constraints; A gradient solution is performed based on the functional relationship of the extended light path manifold to obtain an extended light path geometry gradient of the extended light path geometry with respect to the scene parameter.

[0118] The scene parameter gradient value of the loss function with respect to the scene parameter is calculated by the chain rule based on the extended light path geometric gradient and the extended light path contribution gradient, specifically including: Calculate the loss function based on the rendering result and the target image, and solve the gradient of the loss function with respect to the color and position of each pixel in the rendering result to obtain the loss function gradient; The pixel position is regarded as the mean of the intersection of the corresponding extended light path within the pixel and the imaging plane, so that the pixel position becomes a function of the light path geometry, and the gradient of the function is solved to obtain the pixel position gradient; Based on the light path geometric gradient and the extended light path contribution gradient, the pixel position gradient and the loss function gradient, the scene parameter gradient value of the loss function with respect to the scene parameters is solved by the chain rule.

[0119] The scene parameters are updated based on the gradient value of the loss function with respect to the scene parameters to complete an optimization, which includes: Based on the scene parameter gradient value, the scene parameter is updated through an optimization method based on gradient descent to complete a round of optimization process.

[0120] Based on a differentiable photon mapping system provided by the present invention, a rendering result and a gradient of the rendering result with respect to scene parameters are obtained through a differentiable photon mapping rendering method, so that the photon mapping rendering method can be integrated into the optimization task based on gradient descent, thereby improving the versatility of differentiable rendering.

[0121] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the differentiable photon mapping method, which includes: obtaining initial scene parameters based on the target image, generating rendering results and extended light paths through random progressive photon mapping based on the initial scene parameters; calculating the loss function between the rendering result and the preset target image, and solving the extended light path geometry and the extended light path contribution gradients with respect to the scene parameters to obtain the extended light path geometry gradient and the extended light path contribution gradient; calculating the scene parameter gradient value of the loss function with respect to the scene parameter through the chain rule based on the extended light path geometry gradient and the extended light path contribution gradient; updating the scene parameter based on the loss function with respect to the scene parameter gradient value, completing an optimization, and minimizing the loss function after multiple optimizations, completing the differentiable photon mapping.

[0122] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0123] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the differentiable photon mapping method provided by the above methods, which includes: obtaining initial scene parameters based on a target image; generating rendering results and extended light paths through random progressive photon mapping based on the initial scene parameters; calculating a loss function between the rendering results and a preset target image, and simultaneously solving the gradients of the extended light path geometry and the extended light path contribution with respect to the scene parameters to obtain the extended light path geometry gradient and the extended light path contribution gradient; calculating the scene parameter gradient value of the loss function with respect to the scene parameters through the chain rule based on the extended light path geometry gradient and the extended light path contribution gradient; updating the scene parameters based on the loss function with respect to the scene parameter gradient value to complete an optimization. After multiple optimizations, the loss function is minimized to complete the differentiable photon mapping.

[0124] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the differentiable photon mapping method provided by the above-mentioned methods, the method comprising: obtaining initial scene parameters based on a target image; generating rendering results and extended light paths through random progressive photon mapping based on the initial scene parameters; calculating a loss function between the rendering results and a preset target image, and simultaneously solving the gradients of the extended light path geometry and the extended light path contribution with respect to the scene parameters to obtain the extended light path geometry gradient and the extended light path contribution gradient; calculating the scene parameter gradient value of the loss function with respect to the scene parameters through the chain rule based on the extended light path geometry gradient and the extended light path contribution gradient; updating the scene parameters based on the loss function with respect to the scene parameter gradient value to complete an optimization, and after multiple optimizations, minimizing the loss function to complete the differentiable photon mapping.

[0125] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0126] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A differentiable photon mapping method, characterized in that: include: Get initial scene parameters based on the target image; Generating a rendering result and an extended light path through random progressive photon mapping based on the initial scene parameters; Based on the loss function between the rendering result calculation and the preset target image, the gradients of the extended light path geometry and the extended light path contribution with respect to the scene parameters are solved to obtain the extended light path geometry gradient and the extended light path contribution gradient; Calculate the scene parameter gradient value of the loss function with respect to the scene parameter by the chain rule based on the extended light path geometric gradient and the extended light path contribution gradient; The scene parameters are updated based on the gradient value of the loss function with respect to the scene parameters to complete an optimization. After multiple optimizations, the loss function is minimized and the differentiable photon mapping is completed.

2. The differentiable photon mapping method according to claim 1, characterized in that: The obtaining of initial scene parameters based on the target atlas includes: Initial scene parameters are predicted from the target image through neural networks, COLMAP or manual design.

3. The differentiable photon mapping method according to claim 1, characterized in that: Generating rendering results and extending light paths through random progressive photon mapping based on the initial scene parameters, specifically including: Based on the scene parameters, starting from the light source and the viewpoint through random progressive photon mapping, a photon light path and a viewpoint light path are obtained through ray tracing; Connecting the photon optical path and the viewpoint optical path in the middle to form an extended optical path from the light source to the viewpoint; The contribution of the extended light path is calculated to generate a rendering result.

4. The differentiable photon mapping method according to claim 1, characterized in that: The loss function between the rendering result and the preset target image is calculated based on the rendering result, specifically including: Find the target pixel for each pixel in the rendering result, compare the color and position between pixels, and generate a comparison result; A difference value is determined according to the comparison result, and the loss function is determined based on the difference value.

5. The differentiable photon mapping method according to claim 1, characterized in that: The step of simultaneously solving the gradients of the extended light path geometry and the extended light path contribution with respect to the scene parameters to obtain the extended light path geometry gradient and the extended light path contribution gradient specifically includes: Introducing a first constraint between the extended light path geometry and the scene parameter, constructing a functional relationship between the extended light path geometry and the scene parameter based on the first constraint, performing gradient solving on the functional relationship, and obtaining an extended light path geometry gradient of the extended light path geometry with respect to the scene parameter; The discontinuous parts of the geometric terms and weight terms in the extended light path contribution are ignored, and a second constraint is introduced on the material term in the extended light path contribution. A continuous function of the extended light path contribution with respect to the scene parameters is constructed, and the gradient of the function is solved to obtain the extended light path contribution gradient with respect to the scene parameters.

6. The differentiable photon mapping method according to claim 5, characterized in that: The step of introducing a first constraint between the extended light path geometry and the scene parameter, constructing a functional relationship between the extended light path geometry and the scene parameter based on the first constraint, and performing gradient solving on the functional relationship to obtain the extended light path geometry gradient with respect to the scene parameter specifically includes: Introduce half-vector constraints, fixed position constraints, fixed direction constraints and fixed relative position constraints between the extended light path geometry and the scene parameters; Constructing a specific functional relationship of an extended optical path manifold based on vector constraints, fixed position constraints, fixed direction constraints and fixed relative position constraints; A gradient solution is performed based on the functional relationship of the extended light path manifold to obtain an extended light path geometry gradient of the extended light path geometry with respect to the scene parameter.

7. The differentiable photon mapping method according to claim 1, characterized in that: The step of calculating the scene parameter gradient value of the loss function with respect to the scene parameter by the chain rule based on the extended light path geometric gradient and the extended light path contribution gradient specifically includes: Calculate the loss function based on the rendering result and the target image, and solve the gradient of the loss function with respect to the color and position of each pixel in the rendering result to obtain the loss function gradient; The pixel position is regarded as the mean of the intersection of the extended light path within the pixel and the imaging plane, so that the pixel position becomes a function of the light path geometry, and the gradient of the function is solved to obtain the pixel position gradient; Based on the light path geometric gradient and the extended light path contribution gradient, the pixel position gradient and the loss function gradient, the scene parameter gradient value of the loss function with respect to the scene parameters is solved by the chain rule.

8. The differentiable photon mapping method according to claim 1, characterized in that: The scene parameters are updated based on the gradient value of the loss function with respect to the scene parameters to complete an optimization, which includes: Based on the scene parameter gradient value, the scene parameter is updated through an optimization method based on gradient descent to complete a round of optimization process.

9. A differentiable photon mapping system, characterized in that: The system comprises: A scene parameter acquisition module is used to obtain initial scene parameters based on the target image; An extended light path generation module, used to obtain scene parameters, and generate rendering results and an extended light path through random progressive photon mapping based on the scene parameters; A gradient calculation module, used to calculate the loss function between the rendering result and a preset target image, and to solve the gradients of the extended light path geometry and the extended light path contribution with respect to the scene parameters to obtain the extended light path geometry gradient and the extended light path contribution gradient; A gradient back-propagation module, used for calculating the scene parameter gradient value of the loss function with respect to the scene parameter by the chain rule based on the extended light path geometric gradient and the extended light path contribution gradient; The scene parameter update module is used to update the scene parameters based on the loss function with respect to the scene parameter gradient value, complete an optimization, and after multiple optimizations, minimize the loss function and complete the differentiable photon mapping.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the differentiable photon mapping method according to any one of claims 1 to 6 is implemented.

Images