Adaptive representation and dynamic drawing method and device based on deformable implicit optical transmission function

Abstract

The invention discloses an adaptive representation and dynamic drawing method and device based on a deformable implicit optical transmission function, and the method comprises the following steps: carrying out the object group division of a scene, calculating the implicit influence between object groups, and calculating the scene representation facing each object group based on the implicit influence, and each object group at least comprises one object; calculating a deformable implicit optical transmission function of each object group according to the scene representation of each object group; and adaptively combining the deformable implicit optical transmission functions of the object groups and performing dynamic drawing. And implicit combination is carried out on the optical transmission fields of a plurality of objects in the scene through adaptive representation, the contribution of each optical transmission field to a final drawing result is automatically considered, and high-quality drawing of the three-dimensional dynamic scene is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of rendering, and in particular relates to an adaptive representation and dynamic rendering method and device based on a deformable implicit light transfer function. Background Art

[0002] Rendering technology has long been an important research topic in computer graphics and is widely used in fields such as video games, virtual reality, and film and television special effects. The main goal of rendering technology is to generate realistic images like the real world through reasonable data modeling and simulation calculations. Traditional rendering technology, especially realistic rendering technology, requires massive calculations to obtain high-quality rendering results. Taking the most widely used path tracing technology as an example, it usually takes minutes or even hours to generate a high-quality single frame, which greatly increases the cost of use and limits the application scenarios. For example, in game applications, developers are usually required to render at a rate of at least 60 frames per second to provide players with a low-latency, high-sensitivity gaming experience, which means that the average rendering time per frame must not exceed 16.7 milliseconds, which is difficult to meet the requirements of high-quality rendering, and the rendering process and quality have to be simplified and compromised.

[0003] In recent years, neural network algorithms and hardware have developed rapidly. More and more researchers have begun to apply neural networks to the field of computer graphics, and have made breakthrough progress in complex lighting representation, fine material representation, and realistic appearance reconstruction. Neural networks have also shown great potential in high-quality and efficient realistic rendering. Although existing neural network-based rendering methods can quickly and high-quality render static scenes and simple dynamic scenes, it is still difficult to achieve satisfactory rendering quality for industrial-grade dynamic scenes containing complex geometric details and a large number of fine materials. In addition, existing methods are difficult to support the reconstruction of complex high-frequency rendering effects, such as caustics and indirect highlights. Summary of the invention

[0004] In view of the above, the purpose of the present invention is to provide an adaptive representation and dynamic rendering method and device based on a deformable implicit light transfer function, construct a deformable light transfer field through a deformable implicit light transfer function, implicitly merge the light transfer fields of multiple objects in the scene through adaptive representation, and automatically consider the contribution of each light transfer field to the final rendering result, so as to achieve high-quality rendering of three-dimensional dynamic scenes.

[0005] To achieve the above-mentioned object of the invention, an embodiment provides an adaptive representation and dynamic rendering method based on a deformable implicit light transfer function, comprising the following steps:

[0006] Divide the scene into object groups, calculate the implicit influence between the object groups, and calculate the scene representation for each object group based on the implicit influence, where each object group contains at least one object;

[0007] calculating a deformable implicit light transfer function for each object group based on the scene representation of each object group;

[0008] Adaptively merge deformable implicit light transfer functions for groups of objects and render them dynamically.

[0009] Compute implicit influences between groups of objects, including:

[0010] The first neural network is used to calculate the implicit influence between objects according to the relative positions, relative postures and material parameters between each object group.

[0011] Preferably, calculating a scene representation for each object group based on implicit influence includes:

[0012] For each object group in the scene, the implicit influence of other object groups on it is aggregated to obtain the scene representation of each object group.

[0013] Preferably, calculating the deformable implicit light transfer function of each object group based on the scene representation of each object group includes:

[0014] According to the scene representation of each object group, a second neural network is used to construct a deformable implicit light transfer function for each object group to model the influence of each object group on the rendering result, wherein the deformable implicit light transfer function is represented as a feature field, which calculates features according to the scene representation, viewing direction and visible surface position facing the corresponding object group.

[0015] Preferably, according to the scene representation of each object group, a deformable implicit light transfer function is constructed for each object group using a second neural network, including:

[0016] A third neural network is used to predict the offset based on the scene representation of each object group, and the offset position feature is obtained based on the offset position. Then, a second neural network constructs a deformable implicit light transfer function for each object group based on the offset position feature, the viewing direction and the scene representation of each object group.

[0017] Preferably, the deformable implicit light transfer functions of each object group are adaptively merged and dynamically rendered, including:

[0018] The deformable implicit light transfer functions of multiple object groups are implicitly merged, and the merged light transfer functions are sent to the fourth neural network to predict the rendering result in combination with the target camera position and the observation direction.

[0019] Preferably, the method of implicitly merging the deformable implicit light transfer functions of multiple object groups comprises:

[0020] Addition operation, multiplication operation, fusion based on attention mechanism, and fusion based on neural network.

[0021] To achieve the above-mentioned purpose of the invention, the embodiment further provides an adaptive representation and dynamic rendering device based on a deformable implicit light transfer function, comprising:

[0022] A scene representation module, which is used to divide the scene into object groups, calculate the implicit influence between the object groups, and calculate the scene representation for each object group based on the implicit influence, where each object group contains at least one object;

[0023] A light transfer function calculation module, which is used to calculate a deformable implicit light transfer function of each object group based on the scene representation of each object group;

[0024] The merging and rendering module is used to adaptively merge the deformable implicit light transfer functions of each object group and perform dynamic rendering.

[0025] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned adaptive representation and dynamic rendering method based on deformable implicit light transfer function.

[0026] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned adaptive representation and dynamic rendering method based on deformable implicit light transfer function is implemented.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] By constructing a deformable implicit light transfer function that changes according to the scene layout, we can better model complex rendering effects effectively, and further adaptively consider the impact of each object on the rendering results by implicitly merging the light transfer functions of multiple object groups. This can generate higher quality rendering results and support more complex rendering effects. In addition, faster rendering speeds can be achieved by appropriately reducing the size of the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 is a flow chart of an adaptive representation and dynamic rendering method based on a deformable implicit light transfer function provided by an embodiment;

[0031] Figure 2 is a schematic diagram of an implicit merging method of deformable implicit light transfer functions corresponding to multiple objects provided in an embodiment;

[0032] Figure 3 It is a structural schematic diagram of an adaptive representation and dynamic rendering device based on a deformable implicit light transfer function provided by an embodiment. DETAILED DESCRIPTION

[0033] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0034] like Figure 1 As shown, an adaptive representation and dynamic rendering method based on a deformable implicit light transfer function provided by an embodiment includes the following steps:

[0035] S1, divide the scene into object groups, calculate the implicit influence between object groups, and calculate the scene representation for each object group based on the implicit influence.

[0036] In the embodiment, the three-dimensional scene is divided into a plurality of object groups, each object group includes at least one object, that is, each object group is one object, or a combination of multiple objects. Generally, objects divided into the same object group are required to remain relatively still, that is, the relative position and posture between two objects remain unchanged, but the entire object group can be dynamic. In practice, for efficiency considerations, as many relatively still objects as possible are placed in the same object group.

[0037] In the embodiment, when calculating the implicit influence between object groups, the first neural network is used to calculate the implicit influence between the objects according to the relative positions, relative postures and material parameters between the object groups. Here is a feasible implementation scheme but is not limited to this implementation scheme: the implicit influence r of object group i on object group j ij can be expressed as:

[0038]

[0039] Among them, M ij Represents the transformation matrix from the object space of object group i to the object space of object group j, which contains the relative position and relative posture information between object group i and object group j. i represents the material parameters of object group i, represents the neural network used to calculate the influence of object i.

[0040] In the embodiment, after obtaining the implicit influence between each object group, the implicit influence of other object groups on each object group in the scene is aggregated to obtain the object-oriented scene representation. Specifically, for each object group, the scene representation oriented to the object is obtained by aggregating the influence of other object groups on it. Here, a feasible implementation scheme is provided but not limited to this implementation scheme: the scene representation r oriented to object group i i It can be calculated by summing up the implicit influence of all scene objects on it, that is:

[0041] r i =∑ j r ji

[0042] S2, calculating a deformable implicit light transfer function of each object group according to the scene representation of each object group.

[0043] In an embodiment, according to the object-oriented scene representation, a second neural network with optimizable parameters is used to represent the deformable implicit light transfer function of each object. The deformable implicit light transfer function can be deformed according to the scene layout change to model complex rendering effects with higher quality. Specifically, the deformable implicit light transfer function of the object group i can be represented as a feature field:

[0044]

[0045] Among them, the characteristic field returns the light source observed from the direction ω Lighting scene The characteristics of the sampling point x in . and Represents the scene. The scene representation is actually an abstract representation of the scene. and The abstract result.

[0046] In order to make the implicit light transfer function better able to model complex rendering effects, a deformable feature is further introduced. Here is a feasible implementation scheme including offset deformation, but more deformation features can be further added, such as rotation and scaling. Specifically, the third neural network is used According to the object-oriented scene representation r i Prediction drift And according to the offset position Get offset position feature Then the second neural network Hybrid encoding offset position feature The viewing direction ω and the scene representation r for each object groupi To construct a deformable implicit light transfer function for each object group:

[0047]

[0048] S3, adaptively merges the deformable implicit light transfer functions of each object group and performs dynamic rendering.

[0049] In the embodiment, each scene includes multiple object groups corresponding to multiple deformable implicit light transfer functions, and the deformable implicit light transfer functions of the multiple object groups are implicitly merged, specifically in an order-independent merging manner for predicting the result, and the merged feature field can be expressed as:

[0050]

[0051] in In this embodiment, the addition operation (such as Figure 2 As shown), but including attention mechanism, multiplication operation and order-independent neural network structure can be used as an implementation scheme.

[0052] In the embodiment, the merged light transfer function is combined with the target camera position and the viewing direction and sent to the fourth neural network to predict the rendering result. Specifically, the target viewing angle is first generated according to the target camera position and the viewing direction, and the geometric features and material features of the visible part of the three-dimensional scene under the target viewing angle are generated, and then these features are sent together with the merged light transfer function to the fourth neural network for predicting the rendering result.

[0053] The fourth neural network is supervised by using the true value of the rendering result of the three-dimensional scene, and the influence of the light transfer function of each object on the rendering result is automatically considered in a data-driven manner.

[0054] In order to train the neural network involved in the embodiment, it is necessary to obtain the true value of the rendering result, and use the image difference between it and the predicted rendering result as the loss function to jointly optimize all neural network parameters. By summing the implicit features of multiple object groups, it automatically optimizes the implicit contribution of the feature field of each object group to the final result according to the training data, and then decodes it through the fourth neural network.

[0055] Corresponding to the implicit merging of the present invention is the explicit merging, for example, predicting the color contribution of each object and then combining the colors of each object. This method has a clear true value of the color contribution, and can be supervised by the true value for training, but this may add additional constraints and worsen the effect. Compared with the explicit merging, the implicit merging method of the present invention does not add additional constraints and has a better drawing effect.

[0056] like Figure 3 As shown, the embodiment also provides an adaptive representation and dynamic rendering device 30 based on a deformable implicit light transfer function, including a scene representation module 31, a light transfer function calculation module 32, and a merging and rendering module 33, wherein the scene representation module 31 is used to divide the scene into object groups, and calculate the implicit influence between the object groups, and calculate the scene representation for each object group based on the implicit influence; the light transfer function calculation module 32 is used to calculate the deformable implicit light transfer function of each object group according to the scene representation of each object group; the merging and rendering module 33 is used to adaptively merge the deformable implicit light transfer functions of each object group and perform dynamic rendering.

[0057] It should be noted that the adaptive representation and dynamic rendering device based on deformable implicit light transfer function provided in the above embodiment should be illustrated by the division of the above functional modules when performing adaptive representation and dynamic rendering. The above functional distribution can be completed by different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the adaptive representation and dynamic rendering device based on deformable implicit light transfer function provided in the above embodiment and the adaptive representation and dynamic rendering method embodiment based on deformable implicit light transfer function belong to the same concept. The specific implementation process is detailed in the adaptive representation and dynamic rendering method embodiment based on deformable implicit light transfer function, which will not be repeated here.

[0058] Based on the same inventive concept, an embodiment further provides a computing device, including a memory and one or more processors, wherein executable codes are stored in the memory, and when the one or more processors execute the executable codes, the method is used to implement the above-mentioned adaptive representation and dynamic rendering method based on a deformable implicit light transfer function, specifically including the following steps:

[0059] S1, divide the scene into object groups, calculate the implicit influence between object groups, and calculate the scene representation for each object group based on the implicit influence;

[0060] S2, calculating a deformable implicit light transfer function for each object group based on the scene representation of each object group;

[0061] S3, adaptively merges the deformable implicit light transfer functions of each object group and performs dynamic rendering.

[0062] The computing device provided in the embodiment, in addition to the processor and memory, also includes hardware required for other services such as internal bus, network interface, memory, etc. at the hardware level. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the adaptive representation and dynamic rendering method based on the deformable implicit light transfer function described in S1-S3 above. Of course, in addition to the software implementation method, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0063] Based on the same inventive concept, an embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned adaptive representation and dynamic rendering method based on a deformable implicit light transfer function is implemented, which specifically includes the following steps:

[0064] S1, divide the scene into object groups, calculate the implicit influence between object groups, and calculate the scene representation for each object group based on the implicit influence;

[0065] S2, calculating a deformable implicit light transfer function for each object group based on the scene representation of each object group;

[0066] S3, adaptively merges the deformable implicit light transfer functions of each object group and performs dynamic rendering.

[0067] In the embodiment, computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0068] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An adaptive representation and dynamic rendering method based on a deformable implicit light transfer function, characterized in that: The following steps are involved: Divide the scene into object groups, calculate the implicit influence between the object groups, and calculate the scene representation for each object group based on the implicit influence, where each object group contains at least one object; calculating a deformable implicit light transfer function for each object group based on the scene representation of each object group; Adaptively merge deformable implicit light transfer functions for groups of objects and render them dynamically.

2. The adaptive representation and dynamic rendering method based on deformable implicit light transfer function according to claim 1, characterized in that: Compute implicit influences between groups of objects, including: The first neural network is used to calculate the implicit influence between objects according to the relative positions, relative postures and material parameters between each object group.

3. The adaptive representation and dynamic rendering method based on deformable implicit light transfer function according to claim 1, characterized in that: Compute scene representation for each object group based on implicit influences, including: For each object group in the scene, the implicit influence of other object groups on it is aggregated to obtain the scene representation of each object group.

4. The adaptive representation and dynamic rendering method based on deformable implicit light transfer function according to claim 1, characterized in that: The deformable implicit light transfer function of each object group is calculated based on the scene representation of each object group, including: According to the scene representation of each object group, a second neural network is used to construct a deformable implicit light transfer function for each object group to model the influence of each object group on the rendering result, wherein the deformable implicit light transfer function is represented as a feature field, which calculates features according to the scene representation, viewing direction and visible surface position facing the corresponding object group.

5. The adaptive representation and dynamic rendering method based on deformable implicit light transfer function according to claim 4, characterized in that: According to the scene representation of each object group, a second neural network is used to construct a deformable implicit light transfer function for each object group, including: A third neural network is used to predict the offset based on the scene representation of each object group, and the offset position feature is obtained based on the offset position. Then, a second neural network constructs a deformable implicit light transfer function for each object group based on the offset position feature, the viewing direction and the scene representation of each object group.

6. The adaptive representation and dynamic rendering method based on deformable implicit light transfer function according to claim 1, characterized in that: Adaptively merge deformable implicit light transfer functions for groups of objects and render them dynamically, including: The deformable implicit light transfer functions of multiple object groups are implicitly merged, and the merged light transfer functions are sent to the fourth neural network to predict the rendering result in combination with the target camera position and the observation direction.

7. The adaptive representation and dynamic rendering method based on deformable implicit light transfer function according to claim 6, characterized in that: Ways to implicitly combine deformable implicit light transfer functions for multiple groups of objects include: Addition operation, multiplication operation, fusion based on attention mechanism, and fusion based on neural network.

8. An adaptive representation and dynamic rendering device based on a deformable implicit light transfer function, characterized in that: include: A scene representation module, which is used to divide the scene into object groups, calculate the implicit influence between the object groups, and calculate the scene representation for each object group based on the implicit influence, where each object group contains at least one object; A light transfer function calculation module, which is used to calculate a deformable implicit light transfer function of each object group based on the scene representation of each object group; The merging and rendering module is used to adaptively merge the deformable implicit light transfer functions of each object group and perform dynamic rendering.

9. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the one or more processors execute the executable code, they are used to implement the adaptive representation and dynamic rendering method based on deformable implicit light transfer function described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the adaptive representation and dynamic rendering method based on a deformable implicit light transfer function as described in any one of claims 1 to 7 is implemented.

Images