A microlithographic mapping method and system

Abstract

This invention provides a differentiable photon mapping method and system, comprising: obtaining initial scene parameters based on a target image; generating a rendering result and an extended optical path based on the initial scene parameters through stochastic progressive photon mapping; calculating a loss function between the rendering result and a preset target image, and simultaneously solving for the gradients of the extended optical path geometry and the extended optical path contribution with respect to the scene parameters, obtaining the extended optical path geometry gradient and the extended optical path contribution gradient; calculating the scene parameter gradient value of the loss function with respect to the scene parameters using the chain rule based on the extended optical path geometry gradient and the extended optical path contribution gradient; updating the scene parameters based on the scene parameter gradient value of the loss function, completing one optimization; and minimizing the loss function through multiple optimizations, thus completing the differentiable photon mapping. This invention solves the problem of the inability to perform gradient calculation on photon mapping algorithms in the prior art, improving the versatility of differentiable rendering.

Description

Technical Field

[0001] This invention relates to the field of differentiable rendering technology, and in particular to a differentiable photon mapping method and system. Background Technology

[0002] Rendering has been a crucial research problem in computer graphics for many years, aiming to transform abstract descriptions of scenes into concrete images. The theories and methods for this process have become increasingly mature. In recent years, attention has been paid to gradient solving in this process, also known as differentiable rendering.

[0003] Specifically, if rendering is viewed as a function, then the input of the function is scene parameters such as geometry, materials, camera, etc., and the output is generally a two-dimensional image. Differentiable rendering is to solve for the gradient of the output of the rendering function with respect to the input parameters.

[0004] Currently, differentiable rendering can be divided into two categories based on the rendering pipeline it employs: differentiable rasterization and differentiable ray tracing. Research on differentiable rasterization focuses on modifying the inherently non-differentiable rasterization process, providing pixel color gradients with respect to object geometry through approximate gradients, approximate rendering, or anti-aliasing analysis. Differentiable ray tracing, on the other hand, requires solving the rendering equation to obtain the gradient of the output with respect to the input parameters. Since the rendering equation involves calculating integrals, which are quite complex, Monte Carlo path tracing methods are typically used for estimation. However, during gradient calculation, the integrand may be discontinuous with respect to scene parameters (such as visibility and specular reflection). This necessitates special handling of these discontinuities when estimating gradients using the same Monte Carlo method; researchers define this as boundary integrals. In recent years, researchers have proposed solutions such as boundary sampling and reparameterization to address geometric discontinuities. Research on discontinuities caused by materials is relatively limited.

[0005] Furthermore, there are many specific methods for solving the rendering equations in ray tracing, such as unidirectional path tracing, bidirectional path tracing, and photon mapping. Each method has its own advantages and disadvantages in terms of efficiency and accuracy. For example, photon mapping has a significant advantage over other methods when rendering specular-diffuse-specular light paths and caustics. For differentiable rendering, gradient calculation is also required for each rendering method. However, existing work only focuses on the calculation of unidirectional and bidirectional path tracing, and has not proposed a gradient calculation method for photon mapping. This is because photon mapping is typically used to handle scenes with light paths containing complex specular reflections, refractions, and other discontinuous materials, and as mentioned earlier, there is currently no theory on the correct calculation of gradients for this type of light path.

[0006] One important application of differentiable rendering is solving the inverse rendering problem, that is, inferring scene parameters from a target image. The general process of inverse rendering using differentiable rendering is as follows: First, obtain an initial estimate of the scene parameters, then iteratively perform the following steps: perform differentiable rendering on the scene to obtain the rendering result, solve for the loss function between the rendering result and the target image, calculate the gradient of the loss function with respect to the scene parameters, and optimize the scene parameters using a gradient descent-based optimization method to minimize the loss function. There are many rendering methods, and different methods are suitable for rendering different scenes; the same applies to inverse rendering. Currently, there is no method using differentiable photon mapping to complete the inverse rendering task, which limits the versatility of differentiable rendering. Summary of the Invention

[0007] This invention provides a differentiable photon mapping method and system to solve the problem that the existing technology cannot perform gradient calculation on photon mapping algorithms, thereby improving the versatility of differentiable rendering.

[0008] This invention provides a differentiable photon mapping method, characterized in that it includes:

[0009] Obtain initial scene parameters based on the target image;

[0010] Based on the initial scene parameters, a rendering result and an extended optical path are generated through random progressive photon mapping;

[0011] Based on the rendering results, the loss function between the rendering results and the preset target image is calculated. At the same time, the gradients of the extended optical path geometry and the extended optical path contribution with respect to the scene parameters are solved to obtain the extended optical path geometry gradient and the extended optical path contribution gradient.

[0012] The scene parameter gradient value of the loss function with respect to the scene parameters is calculated using the chain rule based on the extended optical path geometric gradient and the extended optical path contribution gradient.

[0013] The scene parameters are updated based on the gradient values ​​of the loss function with respect to the scene parameters, completing one optimization. After multiple optimizations, the loss function is minimized, and the differentiable photon mapping is completed.

[0014] According to a differentiable photon mapping method provided by the present invention, the step of obtaining initial scene parameters based on the target map includes:

[0015] Predict initial scene parameters from target images using neural networks, COLMAP, or manual design.

[0016] According to a differentiable photon mapping method provided by the present invention, the step of obtaining initial scene parameters and generating rendering results and extended optical paths based on the scene parameters through random progressive photon mapping specifically includes:

[0017] Based on the scene parameters, the photon path and viewpoint path are obtained by ray tracing starting from the light source and viewpoint through random progressive photon mapping;

[0018] The photonic optical path and the viewpoint optical path are connected in the middle to form an extended optical path from the light source to the viewpoint;

[0019] The contribution of the extended optical path is calculated to generate the rendering result.

[0020] According to a differentiable photon mapping method provided by the present invention, the step of calculating a loss function between the rendering result and a preset target image specifically includes:

[0021] For each pixel in the rendered result, find the target pixel, compare the color and position between pixels, and generate the comparison result;

[0022] The difference value is determined based on the comparison result, and the loss function is determined based on the difference value.

[0023] According to a differentiable photon mapping method provided by the present invention, the step of simultaneously solving the gradients of the extended optical path geometry and the extended optical path contribution with respect to scene parameters to obtain the extended optical path geometry gradient and the extended optical path contribution gradient specifically includes:

[0024] A first constraint is introduced between the extended optical path geometry and the scene parameters. Based on the first constraint, a functional relationship between the extended optical path geometry and the scene parameters is constructed. The gradient of the functional relationship is solved to obtain the extended optical path geometry gradient with respect to the scene parameters.

[0025] The discontinuous parts of the geometric and weight terms in the extended optical path contribution are ignored. A second constraint is introduced into the material term in the extended optical path contribution. A continuous function of the extended optical path contribution with respect to the scene parameters is constructed, and the gradient of the function is solved to obtain the extended optical path contribution gradient with respect to the scene parameters.

[0026] According to a differentiable photon mapping method provided by the present invention, the step of introducing a first constraint between the extended optical path geometry and scene parameters, constructing a functional relationship between the extended optical path geometry and scene parameters based on the first constraint, and solving the gradient of the functional relationship to obtain the extended optical path geometry gradient with respect to the scene parameters specifically includes:

[0027] Introduce half-vector constraints, fixed position constraints, fixed direction constraints, and fixed relative position constraints between the extended optical path geometry and scene parameters;

[0028] Functional relationships for constructing specific extended optical path manifolds based on vector constraints, fixed position constraints, fixed direction constraints, and fixed relative position constraints;

[0029] The gradient is solved based on the functional relationship of the extended optical path manifold to obtain the extended optical path geometry gradient with respect to the scene parameters.

[0030] According to a differentiable photon mapping method provided by the present invention, the step of calculating the scene parameter gradient value of the loss function with respect to scene parameters based on the extended optical path geometric gradient and the extended optical path contribution gradient using the chain rule specifically includes:

[0031] The loss function is calculated based on the rendering result and the target image, and the gradient of the loss function with respect to the color and position of each pixel in the rendering result is obtained.

[0032] The pixel position is considered as the average of the intersection points of the extended optical path within the pixel and the imaging plane, making the pixel position a function of the optical path geometry. The gradient of this function is then calculated to obtain the pixel position gradient.

[0033] Based on the optical path geometric gradient, extended optical path contribution gradient, pixel position gradient, and loss function gradient, the scene parameter gradient value of the loss function with respect to scene parameters is solved by the chain rule.

[0034] According to the differentiable photon mapping method provided by the present invention, scene parameters are updated based on the gradient values ​​of the loss function with respect to scene parameters to complete an optimization, specifically including:

[0035] Based on the gradient values ​​of the scene parameters, the scene parameters are updated using a gradient descent-based optimization method to complete one round of optimization.

[0036] The present invention also provides a differentiable photon mapping system, the system comprising:

[0037] The scene parameter acquisition module is used to obtain initial scene parameters based on the target image;

[0038] An extended optical path generation module is used to acquire scene parameters and generate rendering results and extended optical paths based on the scene parameters through random progressive photon mapping;

[0039] The gradient calculation module is used to calculate the loss function between the rendering result and the preset target image, and at the same time solve the gradients of the extended optical path geometry and the extended optical path contribution with respect to the scene parameters to obtain the extended optical path geometry gradient and the extended optical path contribution gradient.

[0040] The gradient backpropagation module is used to calculate the scene parameter gradient value of the loss function with respect to the scene parameters based on the extended optical path geometric gradient and the extended optical path contribution gradient using the chain rule;

[0041] The scene parameter update module is used to update the scene parameters based on the gradient value of the loss function with respect to the scene parameters, complete one optimization, and after multiple optimizations, minimize the loss function and complete the differentiable photon mapping.

[0042] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the differentiable photon mapping method as described above.

[0043] Based on the differentiable photon mapping method and system provided by this invention, the rendering result and the gradient of the rendering result with respect to scene parameters are obtained through the differentiable photon mapping rendering method, so that the photon mapping rendering method can be integrated into the optimization task based on gradient descent, thereby improving the versatility of differentiable rendering. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is a schematic flowchart of the differential photon mapping method provided by the present invention.

[0046] Figure 2 This is a schematic diagram illustrating the execution process of the differential photon mapping method provided by the present invention.

[0047] Figure 3 This is a schematic diagram of the module connection of the microphoton mapping system provided by the present invention.

[0048] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0049] Reference numerals: 110: Scene parameter acquisition module; 120: Extended optical path generation module; 130: Gradient calculation module; 140: Gradient backpropagation module; 150: Scene parameter update module; 410: Processor; 420: Communication interface; 430: Memory; 440: Communication bus. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] The following is combined with Figure 1 and Figure 2 The present invention describes a differentiable photon mapping method, including: step 100, obtaining initial scene parameters based on a target image.

[0052] Specifically, this includes predicting initial scene parameters from target images using neural networks, COLMAP, or manual design.

[0053] Repeat steps 200-500 until the loss function is minimized.

[0054] Step 200: Based on the initial scene parameters, generate rendering results and extended light paths through random progressive photon mapping.

[0055] In this invention, based on the scene parameters, random progressive photon mapping is used to start from the light source and viewpoint, and photon light paths and viewpoint light paths are obtained through ray tracing; the photon light paths and viewpoint light paths are connected in the middle to form an extended light path from the light source to the viewpoint; the contribution of the extended light path is calculated to generate the rendering result.

[0056] Specifically, the core idea of ​​the photon mapping method is to divide the sampling process of an optical path into two parts: photon optical path sampling starting from the light source and viewpoint optical path sampling starting from the viewpoint. The obtained photon optical path and viewpoint optical path are connected at some point in the middle to form an extended optical path.

[0057] For a photon beam path, each intersection point on the path can be represented by two-dimensional coordinates. The optical path can be formally represented as follows: ,in For the light source sampling point, Let be the intersection point that the light path passes through when it bounces in the scene. The contribution of the light path can be expressed as follows.

[0058] (1).

[0059] in The intensity of the light emitted by the light source. and These represent the geometric terms and the scattering function terms in the light transmission process, respectively. (Scattering function term) It can be further divided into rough scattering function terms. and smooth scattering function term Rough scattering function term It can be viewed as a continuous function, corresponding to diffuse or rough reflective materials, while a smooth scattering function... It can be expressed as the following formula.

[0060] .

[0061] in h is a continuous function, and h is a discontinuous function or a function that is close to discontinuous.

[0062] On the other hand, a viewpoint optical path can be similarly represented as ,in From the perspective of Let be the intersection point that the viewpoint's light path passes through when it bounces in the scene. The contribution of this light path can be expressed as follows.

[0063] .

[0064] in The weight of the contribution of this ray to the viewpoint.

[0065] The photon mapping method connects the photonic optical path with the viewpoint optical path, forming a complete extended optical path from the light source to the viewpoint. Specifically, a photonic optical path... and a viewpoint light path It can be combined into an extended optical path The contribution of this extended optical path is given by the following formula.

[0066] .

[0067] in, Indicates connection The non-negative weights, and usually only in When the distance is very small, the contribution of the extended optical path is considered to be greater than 0 only when the photon optical path is close to the end point of the viewpoint optical path.

[0068] Step 300: Calculate the loss function between the rendering result and the preset target image, and simultaneously solve the gradients of the extended optical path geometry and the extended optical path contribution with respect to the scene parameters to obtain the extended optical path geometry gradient and the extended optical path contribution gradient.

[0069] In this invention, the loss function calculated based on the rendering result and the preset target image specifically includes:

[0070] For each pixel in the rendered result, find the target pixel, compare the color and position between pixels, and generate the comparison result;

[0071] The difference value is determined based on the comparison result, and the loss function is determined based on the difference value.

[0072] Simultaneously, the gradients of the extended optical path geometry and the extended optical path contribution are solved with respect to the scene parameters to obtain the gradients of the extended optical path geometry and the extended optical path contribution, specifically including:

[0073] A first constraint is introduced between the extended optical path geometry and the scene parameters. Based on the first constraint, a functional relationship between the extended optical path geometry and the scene parameters is constructed. The gradient of the functional relationship is solved to obtain the extended optical path geometry gradient with respect to the scene parameters.

[0074] The discontinuous parts of the geometric and weight terms in the extended optical path contribution are ignored. A second constraint is introduced into the material term in the extended optical path contribution. A continuous function of the extended optical path contribution with respect to the scene parameters is constructed, and the gradient of the function is solved to obtain the extended optical path contribution gradient with respect to the scene parameters.

[0075] Specifically, solving the extended optical path geometry Regarding scene parameters gradient and contributions to extended optical paths Regarding scene parameters gradient ,in Let k be the scene parameter vector, and k be the dimension of the scene parameters. Note that although... and There is no explicit inclusion in However, in reality, both optical path geometry and optical path contribution are related to scene parameters, that is... .

[0076] For the gradient calculation of extended optical path geometry with respect to scene parameters, since both are vectors, the actual solution target is a Jacobi matrix.

[0077] .

[0078] Solving for this matrix is ​​a relatively complex problem, the core of which lies in constructing the functional relationship between the light path vertices and scene parameters. This section references and expands upon previous work on this issue, and will proceed with a detailed explanation.

[0079] First, a series of simple constraints on the extended optical path and scene parameters are introduced. Then, a complete functional relationship between the two is constructed based on these constraints, and finally, the gradient is solved.

[0080] There are four specific constraints, including: Half-vector constraints: Half-vector constraints apply to three adjacent vertices on the optical path. For vertices, the requirement is that... In its local coordinate system Down, direction of emitted light and the direction of incident light The half-vector remains unchanged.

[0081] .

[0082] in, Let be a 2x3 matrix representing the local tangent vector of a vertex. This represents a half-vector.

[0083] Fixed position constraint: A fixed position constraint requires the position of a vertex in the path. The local area remains unchanged. In one embodiment, the scene is represented as a triangular patch, recording the triangle where the vertex is located and the coordinates of the centroid on the triangle, and the constraint is represented as constraining the centroid coordinates to remain unchanged.

[0084] .

[0085] in( () represents the three vertices of a triangle. Calculation points In triangle The centroid coordinates are shown on the graph. Note that these centroid coordinates are actually 2-dimensional.

[0086] Fixed direction constraint: A fixed direction constraint requires that the outgoing light emanating from a vertex in the path... The direction remains unchanged in the local coordinate system of this vertex.

[0087] .

[0088] in, It is a 2x3 matrix representing the local tangent vector of a vertex.

[0089] Fixed relative position constraint: The relative position between two adjacent vertices on a fixed optical path remains unchanged.

[0090] .

[0091] in, It is a 2x3 matrix representing the local tangent vector of a vertex.

[0092] These four constraints are all 2-dimensional vector constraints and can be uniformly represented as follows: .

[0093] Based on constrained path space manifolds, specific extended optical path space manifolds can be constructed for gradient solving. Specifically, three methods for constructing extended optical path space manifolds are designed:

[0094] A fixed-position manifold is constructed using the following constraints:

[0095] For the starting point of the extended path and the end point (Light source and camera) Apply fixed position constraints;

[0096] In addition to the optical path Apply half-vector constraints to the smooth material vertices outside;

[0097] right and Use fixed direction constraints;

[0098] Except for the optical path The emission direction of the non-smooth material vertices outside is constrained by a fixed direction.

[0099] A diffuse manifold is constructed using the following constraints:

[0100] For the starting point of the extended path and the end point (Light source and camera) Apply fixed position constraints;

[0101] In addition to the optical path Apply half-vector constraints to the smooth material vertices outside;

[0102] right Use fixed direction constraints;

[0103] In addition to the optical path The emission direction of the non-smooth material vertices outside the surface is constrained by a fixed direction.

[0104] right Fixed position constraints are used.

[0105] Photon manifolds are constructed using the following constraints:

[0106] For the starting point of the extended path and the end point (Light source and camera) Apply fixed position constraints;

[0107] In addition to the optical path Apply half-vector constraints to the smooth material vertices outside;

[0108] right Use fixed direction constraints;

[0109] In addition to the optical path The emission direction of the non-smooth material vertices outside the surface is constrained by a fixed direction.

[0110] right A fixed relative position constraint is adopted.

[0111] All three manifold construction methods affect the extended optical path geometry. With n+m+2 constraints applied, each constraint can be written uniformly as... In the form of.

[0112] .

[0113] At this point, the extended path (optical path) space manifold can be expressed as a set of paths that satisfy the following conditions.

[0114] .

[0115] Solving for the optical path gradient on an extended optical path space manifold involves considering that any extended optical path on each manifold is subject to n+m+2 two-dimensional constraints, which are exactly the same as the degrees of freedom of the path. Since the constraints are independent of each other, given scene parameters... Under the circumstances, satisfy of It is unique; based on this, the functional relationship between the two can be obtained. As for the gradient of this function... The solution can be obtained from the implicit function differentiation theory. .

[0116] The gradient of the extended optical path contribution with respect to scene parameters is considered in the solution. f is usually a three-dimensional function, but for simplicity, it can be regarded as a one-dimensional function.

[0117] .

[0118] in This is the optical path geometric gradient obtained in the previous chapter.

[0119] From formula (4), the function can be obtained. The specific expression, however about They are likely all discontinuous, which makes... and It cannot be solved directly. Specifically, Geometric terms included Material items Weighting items Both may involve discontinuities in the parameters. For the geometric term, several methods exist to handle the discontinuities; these methods can be used, and the discontinuities can be ignored for now. For the weight term… Provided that it meets the requirement of being a kernel function, it can be set to any continuous function. In one embodiment, it can be defined as follows.

[0120] .

[0121] in , where r is a fixed radius.

[0122] Therefore, only the material item remains at this point. Possibly regarding parameters or Discontinuous.

[0123] Based on the preceding text, Based on material type, materials can be divided into rough materials and smooth materials. Only smooth materials may be discontinuous functions, as shown in formula (2). However, during the rendering process, this discontinuous function is actually an indicative function.

[0124] .

[0125] That is when Meet the conditions hour A constant Intuitively, a smooth material satisfies constraints. The space it constitutes is continuous. Therefore, it is only necessary to ensure that Always satisfy constraints Formula (2) can be simplified to the following formula.

[0126] .

[0127] It becomes a continuous function. Based on the above discussion... Ultimately, it became about For continuous functions, gradient calculation can be performed.

[0128] In the three path space manifolds defined in this invention, a half-vector constraint is used for smooth materials. This constraint is actually equivalent to the constraint in formula (14). Therefore, when calculating the gradient on the given three manifolds, it can be guaranteed that... about It is continuous.

[0129] Step 400: Calculate the scene parameter gradient value of the loss function with respect to the scene parameters using the chain rule based on the extended optical path geometric gradient and the extended optical path contribution gradient.

[0130] Specifically, this includes: calculating the loss function based on the rendering result and the target image, and solving for the gradient of the loss function with respect to the color and position of each pixel in the rendering result, thus obtaining the gradient of the loss function;

[0131] The pixel position is considered as the average of the intersection points of the extended optical path within the pixel and the imaging plane, making the pixel position a function of the optical path geometry. The gradient of this function is then calculated to obtain the pixel position gradient.

[0132] Based on the optical path geometric gradient, extended optical path contribution gradient, pixel position gradient, and loss function gradient, the scene parameter gradient value of the loss function with respect to scene parameters is solved by the chain rule.

[0133] Step 500: Update the scene parameters based on the gradient values ​​of the loss function with respect to the scene parameters to complete one optimization.

[0134] Specifically, this includes: updating the scene parameters based on the gradient values ​​of the scene parameters using a gradient descent-based optimization method to complete one round of optimization.

[0135] Based on the differentiable photon mapping method provided by this invention, the rendering result and the gradient of the rendering result with respect to scene parameters are obtained through the differentiable photon mapping rendering method, so that the photon mapping rendering method can be integrated into the optimization task based on gradient descent, thereby improving the versatility of differentiable rendering.

[0136] In one specific embodiment, the rendering result and extended optical path are obtained based on the stochastic progressive photon mapping method. The loss function between the rendering result and the target image is calculated to obtain the gradient of the optical path geometry and contribution with respect to scene parameters. The gradient of the extended optical path is solved by applying the method described in the previous chapter, and the gradient of the loss function with respect to scene parameters is obtained by the chain rule. This gradient can be used to optimize the scene parameters.

[0137] Randomized progressive photon mapping (RPPM) is an improvement on conventional photon mapping rendering. This method is iterative; in each iteration, it uses ray tracing to obtain the photon path and viewpoint path, respectively, starting from the light source and viewpoint, constructs the extended path, calculates the path contribution, and finally obtains the rendering result.

[0138] Initializing this algorithm requires specifying the number of viewpoint optical paths emitted in each iteration. Number of photon optical paths and initial collection radius Next, the iteration begins, and in each iteration:

[0139] The first step is to construct the viewpoint light path. A common approach is to emit rays from the viewpoint to several random locations within each pixel (the total number of rays is...). The ray path is calculated by first finding the intersection of rays with the diffuse or high-roughness surface, and then using ray tracing to calculate direct and scattered light. This process can be combined with direct sampling and importance sampling methods to improve efficiency. For each ray, further calculations of scattering and indirect lighting are stopped once it intersects the diffuse or high-roughness surface; the resulting viewpoint ray path is defined as follows. ,in From the perspective of This is the intersection of the light path and the scene, specifically, the last intersection. Let each point be a collection point, and the set of all collection points is called the collection point set. For each viewpoint ray path, its ray path geometry is stored. With contribution .

[0140] Secondly, random sampling from the light source The system identifies several locations from which photons are emitted, and uses ray tracing to find intersections and calculate scattered rays. When the path length is greater than one and intersects a diffuse reflective surface or a high-roughness surface, it identifies collection points located near the intersection. Specifically, the photon path can be represented as follows: ,in For the light source sampling point, For the intersections along the path of light as it bounces through the scene, find the collection points to gather all the points that satisfy the conditions. point And obtain the corresponding viewpoint optical path and extended optical path. Then, the contribution of the extended optical path is calculated using formula (4). .

[0141] After obtaining each extended optical path and its contribution, each pixel will perform color calculations based on its corresponding extended optical paths and update the radius applied in the next iteration. Specifically, suppose that the weighted contribution of the extended optical path collected by pixel p before the (i+1)th iteration is... The total number of weighted extended optical paths is Collection radius is The total number of items collected in the (i+1)th round is The total contribution of the extended optical path is ,in To sample The probability can be obtained from the ray tracing process. Then, the total extended ray path contribution after the (i+1)th round... Total weighted extended optical path count Collection radius is They are respectively.

[0142] .

[0143] .

[0144] .

[0145] Assuming there are a total of After one iteration, the final pixel color is .

[0146] The loss function between the rendered result and the target image defines a rendered image. With the target image loss function between ,in , The color of each pixel (h, w are the number of pixels vertically and horizontally in the image), The location of each pixel in screen space. In one embodiment, (loss function) It can be simply defined as the pixel-by-pixel color difference between the rendered image and the target image.

[0147] .

[0148] Such as the widely used L1 or L2 loss functions.

[0149] In another embodiment, pixel location information can also be incorporated into the loss function. For example, a one-to-one match can be found between pixels in the rendered image and the target image. And define the loss function.

[0150] .

[0151] in , representing the weights of color and position in the loss function, is beneficial for capturing global correspondences between images. Specific matching methods can include bipartite graph matching, optical flow, or optimal transport methods.

[0152] The ultimate goal of gradient calculation and backpropagation is to solve for the loss function. Regarding scene parameters The gradient is determined by the chain rule.

[0153] .

[0154] in, The pixel color calculation formula (18) and the optical path contribution gradient calculation method described above can be used. The solution can be found as follows: for the extended optical path corresponding to each pixel, define its intersection with the imaging plane as... Find the mean of all intersection points. (These two processes are directly differentiable) and let the following equation hold true.

[0155] .

[0156] This invention does not limit gradient calculation to a specific manifold. In practical applications, one or more manifold constraints can be selected for gradient calculation based on actual needs. A recommended approach is to use a fixed-position manifold to calculate the gradient when using a simple loss function such as the L1 / L2 loss function. When using a complex loss function, such as a loss function that includes positional information, simultaneously calculating the gradient using a diffuse manifold and a photon manifold can yield non-zero gradients. To guide optimization.

[0157] Based on the differentiable photon mapping method provided by this invention, the rendering result and the gradient of the rendering result with respect to scene parameters are obtained through the differentiable photon mapping rendering method, so that the photon mapping rendering method can be integrated into the optimization task based on gradient descent, thereby improving the versatility of differentiable rendering.

[0158] refer to Figure 3 The present invention also discloses a differentiable photon mapping system, the system comprising:

[0159] Scene parameter acquisition module 110 is used to acquire initial scene parameters based on the target image.

[0160] The extended optical path generation module 120 is used to acquire scene parameters and generate rendering results and extended optical paths based on the scene parameters through random progressive photon mapping.

[0161] The gradient calculation module 130 is used to calculate the loss function between the rendering result and the preset target image, and at the same time solve the gradients of the extended optical path geometry and the extended optical path contribution with respect to the scene parameters to obtain the extended optical path geometry gradient and the extended optical path contribution gradient.

[0162] The gradient backpropagation module 140 is used to calculate the scene parameter gradient value of the loss function with respect to the scene parameters based on the extended optical path geometric gradient and the extended optical path contribution gradient using the chain rule.

[0163] The scene parameter update module 150 is used to update the scene parameters based on the gradient value of the loss function with respect to the scene parameters, complete one optimization, and after multiple optimizations, minimize the loss function to complete the differentiable photon mapping.

[0164] The initial scene parameters obtained based on the target map include:

[0165] Predict initial scene parameters from target images using neural networks, COLMAP, or manual design.

[0166] Based on the initial scene parameters, a rendering result and an extended light path are generated through random progressive photon mapping, specifically including:

[0167] Based on the scene parameters, the photon path and viewpoint path are obtained by ray tracing starting from the light source and viewpoint through random progressive photon mapping;

[0168] The photonic optical path and the viewpoint optical path are connected in the middle to form an extended optical path from the light source to the viewpoint;

[0169] The contribution of the extended optical path is calculated to generate the rendering result.

[0170] The loss function between the rendered result and the preset target image is calculated, specifically including:

[0171] For each pixel in the rendered result, find the target pixel, compare the color and position between pixels, and generate the comparison result;

[0172] The difference value is determined based on the comparison result, and the loss function is determined based on the difference value.

[0173] The simultaneous solution of the gradients of the extended optical path geometry and the extended optical path contribution with respect to the scene parameters, to obtain the gradients of the extended optical path geometry and the extended optical path contribution, specifically includes:

[0174] A first constraint is introduced between the extended optical path geometry and the scene parameters. Based on the first constraint, a functional relationship between the extended optical path geometry and the scene parameters is constructed. The gradient of the functional relationship is solved to obtain the extended optical path geometry gradient with respect to the scene parameters.

[0175] The discontinuous parts of the geometric and weight terms in the extended optical path contribution are ignored. A second constraint is introduced into the material term in the extended optical path contribution. A continuous function of the extended optical path contribution with respect to the scene parameters is constructed, and the gradient of the function is solved to obtain the extended optical path contribution gradient with respect to the scene parameters.

[0176] A first constraint is introduced between the extended optical path geometry and the scene parameters. Based on the first constraint, a functional relationship between the extended optical path geometry and the scene parameters is constructed. The gradient of the functional relationship is solved to obtain the extended optical path geometry gradient with respect to the scene parameters, specifically including:

[0177] Introduce half-vector constraints, fixed position constraints, fixed direction constraints, and fixed relative position constraints between the extended optical path geometry and scene parameters;

[0178] Functional relationships for constructing specific extended optical path manifolds based on vector constraints, fixed position constraints, fixed direction constraints, and fixed relative position constraints;

[0179] The gradient is solved based on the functional relationship of the extended optical path manifold to obtain the extended optical path geometry gradient with respect to the scene parameters.

[0180] The scene parameter gradient values ​​of the loss function with respect to scene parameters are calculated using the chain rule based on the extended optical path geometric gradient and the extended optical path contribution gradient, specifically including:

[0181] The loss function is calculated based on the rendering result and the target image, and the gradient of the loss function with respect to the color and position of each pixel in the rendering result is obtained.

[0182] The pixel position is considered as the mean of the intersection points of the corresponding extended optical path within the pixel and the imaging plane, making the pixel position a function of the optical path geometry. The gradient of this function is then solved to obtain the pixel position gradient.

[0183] Based on the optical path geometric gradient, extended optical path contribution gradient, pixel position gradient, and loss function gradient, the scene parameter gradient value of the loss function with respect to scene parameters is solved by the chain rule.

[0184] The scene parameters are updated based on the gradient values ​​of the loss function with respect to the scene parameters, completing one optimization, specifically including:

[0185] Based on the gradient values ​​of the scene parameters, the scene parameters are updated using a gradient descent-based optimization method to complete one round of optimization.

[0186] Based on the differentiable photon mapping system provided by this invention, the rendering result and the gradient of the rendering result with respect to scene parameters are obtained through the differentiable photon mapping rendering method, so that the photon mapping rendering method can be integrated into the optimization task based on gradient descent, thereby improving the versatility of differentiable rendering.

[0187] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a differentiable photon mapping method, which includes: obtaining initial scene parameters based on a target image; generating a rendering result and an extended optical path based on the initial scene parameters through stochastic progressive photon mapping; calculating a loss function between the rendering result and a preset target image, and simultaneously solving for the gradients of the extended optical path geometry and the extended optical path contribution with respect to the scene parameters, obtaining the extended optical path geometry gradient and the extended optical path contribution gradient; calculating the scene parameter gradient value of the loss function with respect to the scene parameters using the chain rule based on the extended optical path geometry gradient and the extended optical path contribution gradient; updating the scene parameters based on the scene parameter gradient value of the loss function, completing one optimization; and after multiple optimizations, minimizing the loss function, thus completing the differentiable photon mapping.

[0188] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0189] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the differentiable photon mapping method provided by the above methods. The method includes: obtaining initial scene parameters based on a target image; generating a rendering result and an extended optical path based on the initial scene parameters through stochastic progressive photon mapping; calculating a loss function between the rendering result and a preset target image, and simultaneously solving for the gradients of the extended optical path geometry and the extended optical path contribution with respect to the scene parameters, to obtain the extended optical path geometry gradient and the extended optical path contribution gradient; calculating the scene parameter gradient value of the loss function with respect to the scene parameters based on the extended optical path geometry gradient and the extended optical path contribution gradient using a chain rule; updating the scene parameters based on the scene parameter gradient value of the loss function, completing one optimization, and after multiple optimizations, minimizing the loss function to complete the differentiable photon mapping.

[0190] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the differentiable photon mapping method provided by the above methods. The method includes: obtaining initial scene parameters based on a target image; generating a rendering result and an extended optical path based on the initial scene parameters through stochastic progressive photon mapping; calculating a loss function between the rendering result and a preset target image, and simultaneously solving for the gradients of the extended optical path geometry and the extended optical path contribution with respect to the scene parameters, obtaining the extended optical path geometry gradient and the extended optical path contribution gradient; calculating the scene parameter gradient value of the loss function with respect to the scene parameters using a chain rule based on the extended optical path geometry gradient and the extended optical path contribution gradient; updating the scene parameters based on the scene parameter gradient value of the loss function, completing one optimization, and after multiple optimizations, minimizing the loss function, thus completing the differentiable photon mapping.

[0191] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0192] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A differentiable photon mapping method, characterized in that, include: Obtain initial scene parameters based on the target image; Based on the initial scene parameters, a rendering result and an extended optical path are generated through random progressive photon mapping; Based on the rendering results, the loss function between the rendering results and the preset target image is calculated. At the same time, the gradients of the extended optical path geometry and the extended optical path contribution with respect to the scene parameters are solved to obtain the extended optical path geometry gradient and the extended optical path contribution gradient. The scene parameter gradient value of the loss function with respect to the scene parameters is calculated using the chain rule based on the extended optical path geometric gradient and the extended optical path contribution gradient. The scene parameters are updated based on the gradient values ​​of the loss function with respect to the scene parameters, completing one optimization. After multiple optimizations, the loss function is minimized, and the differentiable photon mapping is completed. Based on the initial scene parameters, a rendering result and an extended light path are generated through random progressive photon mapping, specifically including: Based on the scene parameters, the photon path and viewpoint path are obtained by ray tracing starting from the light source and viewpoint through random progressive photon mapping; The photonic optical path and the viewpoint optical path are connected in the middle to form an extended optical path from the light source to the viewpoint; The contribution of the extended optical path is calculated to generate the rendering result.

2. The differentiable photon mapping method according to claim 1, characterized in that, The process of obtaining initial scene parameters based on the target map includes: Predict initial scene parameters from target images using neural networks, COLMAP, or manual design.

3. The differentiable photon mapping method according to claim 1, characterized in that, The loss function calculated based on the rendering result and the preset target image specifically includes: For each pixel in the rendered result, find the target pixel, compare the color and position between pixels, and generate the comparison result; The difference value is determined based on the comparison result, and the loss function is determined based on the difference value.

4. The differentiable photon mapping method according to claim 1, characterized in that, The simultaneous solution of the gradients of the extended optical path geometry and the extended optical path contribution with respect to the scene parameters, to obtain the gradients of the extended optical path geometry and the extended optical path contribution, specifically includes: A first constraint is introduced between the extended optical path geometry and the scene parameters. Based on the first constraint, a functional relationship between the extended optical path geometry and the scene parameters is constructed. The gradient of the functional relationship is solved to obtain the extended optical path geometry gradient with respect to the scene parameters. The discontinuous parts of the geometric and weight terms in the extended optical path contribution are ignored. A second constraint is introduced into the material term in the extended optical path contribution. A continuous function of the extended optical path contribution with respect to the scene parameters is constructed, and the gradient of the function is solved to obtain the extended optical path contribution gradient with respect to the scene parameters.

5. The differentiable photon mapping method according to claim 4, characterized in that, The step of introducing a first constraint between the extended optical path geometry and scene parameters, constructing a functional relationship between the extended optical path geometry and scene parameters based on the first constraint, and solving the gradient of the functional relationship to obtain the extended optical path geometry gradient with respect to the scene parameters specifically includes: Introduce half-vector constraints, fixed position constraints, fixed direction constraints, and fixed relative position constraints between the extended optical path geometry and scene parameters; Functional relationships for constructing specific extended optical path manifolds based on vector constraints, fixed position constraints, fixed direction constraints, and fixed relative position constraints; The gradient is solved based on the functional relationship of the extended optical path manifold to obtain the extended optical path geometry gradient with respect to the scene parameters.

6. The differentiable photon mapping method according to claim 1, characterized in that, The calculation of the scene parameter gradient value of the loss function with respect to scene parameters based on the extended optical path geometric gradient and the extended optical path contribution gradient using the chain rule specifically includes: The loss function is calculated based on the rendering result and the target image, and the gradient of the loss function with respect to the color and position of each pixel in the rendering result is obtained. The pixel position is considered as the average of the intersection points of the extended optical path within the pixel and the imaging plane, making the pixel position a function of the optical path geometry. The gradient of this function is then calculated to obtain the pixel position gradient. Based on the optical path geometric gradient, extended optical path contribution gradient, pixel position gradient, and loss function gradient, the scene parameter gradient value of the loss function with respect to scene parameters is solved by the chain rule.

7. The differentiable photon mapping method according to claim 1, characterized in that, The scene parameters are updated based on the gradient values ​​of the loss function with respect to the scene parameters, completing one optimization, specifically including: Based on the gradient values ​​of the scene parameters, the scene parameters are updated using a gradient descent-based optimization method to complete one round of optimization.

8. A differentiable photon mapping system, characterized in that, The system includes: The scene parameter acquisition module is used to obtain initial scene parameters based on the target image; An extended optical path generation module is used to acquire scene parameters and generate rendering results and extended optical paths based on the scene parameters through random progressive photon mapping; The gradient calculation module is used to calculate the loss function between the rendering result and the preset target image, and at the same time solve the gradients of the extended optical path geometry and the extended optical path contribution with respect to the scene parameters to obtain the extended optical path geometry gradient and the extended optical path contribution gradient. The gradient backpropagation module is used to calculate the scene parameter gradient value of the loss function with respect to the scene parameters based on the extended optical path geometric gradient and the extended optical path contribution gradient using the chain rule; The scene parameter update module is used to update the scene parameters based on the gradient value of the loss function with respect to the scene parameters, complete one optimization, and after multiple optimizations, minimize the loss function and complete the differentiable photon mapping. Specifically, the generation of rendering results and extended optical paths based on the initial scene parameters through random progressive photon mapping includes: Based on the scene parameters, the photon path and viewpoint path are obtained by ray tracing starting from the light source and viewpoint through random progressive photon mapping; The photonic optical path and the viewpoint optical path are connected in the middle to form an extended optical path from the light source to the viewpoint; The contribution of the extended optical path is calculated to generate the rendering result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the differentiable photon mapping method as described in any one of claims 1 to 7.

Images