Neural radiation field efficient rendering method based on Unity engine

Abstract

The invention discloses a neural radiation field efficient rendering method based on a Unity engine. The method comprises the following steps: constructing a 3D model of each element in the same scene, inputting the 3D model into a Unity engine for processing, and outputting a scene image; performing adaptive sampling processing on the scene image to obtain a scene image comprising adaptive sampling points with different densities, and inputting the scene image into a neural radiation field for training; and packaging the trained neural radiation field into a plug-in, integrating the plug-in into a Unity engine, and performing efficient rendering of an image or a 3D model. According to the method, the neural radiation field technology is combined with the Unity engine, the rendering efficiency and speed of the high-precision three-dimensional model are remarkably improved by combining the efficient rendering capability of the neural radiation field obtained through the image information of the Unity engine, the rendering process of the three-dimensional model in the Unity engine is accelerated, the overall processing efficiency is also greatly improved, and the method is suitable for large-scale popularization and application. The method provides solid technical support for efficient rendering and processing of the three-dimensional model, and has important research significance and application value.

Description

Technical Field

[0001] The present invention relates to a method for efficiently rendering a radiation field, and to the technical field of three-dimensional scene rendering, and in particular to a method for efficiently rendering a neural radiation field based on a Unity engine. Background Art

[0002] Neural Radiance Fields (NeRF) is an emerging 3D scene representation and rendering technology that has attracted much attention due to its ability to generate high-quality rendering results. Neural Radiance Fields technology combines the principles of deep learning and computer graphics to model and render 3D scenes through neural networks. However, traditional neural radiance field rendering methods have certain challenges in computational complexity and rendering efficiency. Due to the complexity of neural networks and the large number of parameters, the rendering process requires a lot of computing resources and time.

[0003] As a widely used real-time 3D content creation and operation platform, Unity engine has a wide range of applications in game development, architectural design, virtual simulation and other fields. Unity engine provides rich rendering functions and flexible rendering pipelines, allowing developers to easily create high-quality 3D scenes and interactive experiences. However, the built-in rendering method of Unity engine may have certain limitations for complex scenes and high-quality rendering requirements. Summary of the invention

[0004] In order to solve the problems existing in the background technology, the present invention provides a method for efficiently rendering a neural radiation field based on the Unity engine.

[0005] The technical solution adopted by the present invention is: The present invention provides a method for efficiently rendering a neural radiation field based on a Unity engine, comprising: 1) Build a 3D model of each element in the same scene that changes over time and input it into the Unity engine. After processing, the Unity engine outputs several scene images of the current scene that change over time under different perspectives and lighting conditions. The Unity engine can easily generate a large amount of image data under different perspectives and lighting conditions.

[0006] 2) After adaptive sampling of each scene image, each scene image including adaptive sampling points of different densities is obtained, and the scene images including adaptive sampling points of different densities are input into the neural radiation field for training. During the training process, the neural radiation field generates a new perspective image through network forward propagation, and the generated new perspective image is compared with the scene image input into the neural radiation field to obtain the loss until the loss converges to obtain the trained neural radiation field.

[0007] 3) The trained neural radiation field is encapsulated into a plug-in form and integrated into the Unity engine. The image or 3D model to be rendered is input into the Unity engine encapsulated with the trained neural radiation field for processing. After processing, the Unity engine outputs the rendered image or 3D model and displays it on the monitor to achieve efficient rendering.

[0008] In the step 1), several cameras are added to the virtual environment in the Unity engine, each camera captures different perspectives of the scene, and each camera adopts different positions, rotations and focal lengths to obtain a variety of perspectives; different types of light sources are added to the virtual environment, such as point light sources, direct light or ambient light, etc., and the lighting parameters of each light source are adjusted differently, such as changing the position, color, intensity and other properties of the light source to simulate different lighting conditions; the Unity engine uses each camera to obtain scene images of different perspectives under different lighting conditions and performs automatic batch rendering, and finally outputs several scene images in a preset format, and the output image data is exported to a format suitable for neural radiation field model training, such as PNG or JPEG sequence; all exported scene images are constructed as a training data set, including metadata of viewing angles and lighting conditions.

[0009] The Unity engine needs to install necessary plug-ins and tools for scene construction, lighting settings, camera control and image rendering, including Cinemachin and magic light probes. The elements in the scene include terrain, buildings, vegetation, etc. The Unity engine can apply appropriate materials and textures to the model to enhance the realism of the scene. The rendering parameters in the Unity engine need to be configured to ensure image quality, including resolution, anti-aliasing, etc.

[0010] In the step 2), for each scene image, the scene image is processed using an image processing algorithm to obtain important areas and unimportant areas in the scene image, and high resolution and more sampling points are used for the important areas, and low resolution and fewer sampling points are used for the unimportant areas, thereby obtaining a scene image including adaptive sampling points of different densities.

[0011] Adaptive sampling dynamically adjusts the resolution or sampling density of the input image based on the importance and complexity of the scene. For example, a higher resolution is used in areas with rich details, while a lower resolution is used in flat or insignificant areas. First, the scene is pre-analyzed and evaluated, and the original scene is quickly scanned using image processing algorithms such as gradient calculation and texture analysis. The information density and visual saliency of different regions are evaluated to determine which regions contain rich details or key features. Then an importance map is constructed. Based on the evaluation results, an importance map is constructed in which the value of each pixel or voxel represents the importance of the position. Areas with high importance usually correspond to edges, corners, or other significant features in the scene. Then adaptive sampling is performed, and the sampling strategy is dynamically adjusted based on the importance map. The number of sampling points is increased in areas with high importance. In flat areas or areas with less visual changes, the sampling density is reduced to reduce the computational burden. Finally, sampling point allocation and adjustment are performed. According to the adaptive sampling method, sampling points are allocated in the scene. During the training process, the distribution of sampling points is dynamically adjusted based on real-time feedback (such as rendering error, convergence speed, etc.). Finally, the scene images including adaptive sampling points of different densities can be used for network training and rendering, and the sampling points obtained by adaptive sampling can be used to train the neural radiation field network.

[0012] In the step 3), the trained neural radiation field is exported to a preset standard format, such as TensorFlow's SavedModel, ONNX or other deep learning model exchange formats, a new plug-in project is created in the integrated development environment IDE (Integrated Development Environment) and interacts with the Unity engine, the dependencies and libraries required for interaction with the Unity engine are configured, and then the trained neural radiation field in the preset standard format is integrated into the new plug-in project to ensure that the model can be loaded and run correctly, and finally the new plug-in project is encapsulated as a plug-in of the Unity engine using the encapsulation API, and the trained neural radiation field is called by the Unity engine for rendering; the interface for interacting with the plug-in is defined in the Unity engine, including data transfer, function call, etc. Scripts are written in the Unity engine, and the API provided by the plug-in is called through these scripts to realize the rendering of the neural radiation field model.

[0013] When developing a plug-in framework, you need to select programming languages ​​and tools. According to the existing development technology and the compatibility of the Unity engine, select the appropriate programming language (such as C# or C++) and development tools. The Unity engine supports the extension of its functions through plug-ins. Therefore, the trained neural radiation field is encapsulated in the form of a plug-in, which can be easily integrated into the Unity engine to enhance its rendering capabilities. By utilizing the parallel computing capabilities supported by the Unity engine, such as multi-threaded rendering or GPU acceleration, the rendering speed of the neural radiation field model can be significantly improved. By assigning rendering tasks to multiple processing units, different ray casting and color prediction calculations can be processed in parallel to ensure the real-time and quality of rendering.

[0014] The method for efficient rendering of neural radiation field based on Unity engine of the present invention first uses Unity engine to generate a large amount of image data under different viewing angles and lighting conditions. These image data are put into the neural radiation field algorithm model for training. Then the trained neural radiation field algorithm model is encapsulated into a plug-in. Finally, the plug-in is added to the Unity engine through a function or interface call. The plug-in realizes automatic rendering in the Unity engine and improves the rendering capability of the Unity engine.

[0015] The electronic device of the present invention comprises: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method as described above.

[0016] The computer-readable storage medium of the present invention stores program data thereon, and when the program data is executed by a processor, the method described above is implemented.

[0017] The beneficial effects of the present invention are: The method of the invention combines the neural radiation field technology with the Unity engine. The method significantly improves the rendering efficiency and speed of high-precision three-dimensional models through a carefully designed Unity engine image information acquisition strategy combined with the efficient rendering capability of the neural radiation field. It not only speeds up the rendering process of the three-dimensional model in the Unity engine, but also greatly improves the overall processing efficiency, providing solid technical support for the efficient rendering and processing of the three-dimensional model, and has important research significance and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The present invention is a flowchart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0020] The efficient rendering method of neural radiation field based on Unity engine of the present invention is as follows: 1) Build a 3D model of each element in the same scene that changes over time and input it into the Unity engine. After processing, the Unity engine outputs several scene images of the current scene that change over time under different perspectives and lighting conditions. The Unity engine can easily generate a large amount of image data under different perspectives and lighting conditions.

[0021] Several cameras are added to the virtual environment in the Unity engine, each camera captures a different perspective of the scene, and each camera uses a different position, rotation and focal length to obtain a variety of perspectives; different types of light sources are added to the virtual environment, such as point light sources, direct light or ambient light, and the lighting parameters of each light source are adjusted differently, such as changing the position, color, intensity and other properties of the light source to simulate different lighting conditions; the Unity engine uses each camera to obtain scene images of different perspectives under different lighting conditions and performs automatic batch rendering, and finally outputs several scene images in a preset format. The output image data is exported to a format suitable for neural radiation field model training, such as PNG or JPEG sequence; all exported scene images are constructed as a training data set, including metadata of viewing angles and lighting conditions.

[0022] The Unity engine needs to install necessary plug-ins and tools for scene construction, lighting settings, camera control and image rendering, including Cinemachin and magic light probes. The elements in the scene include terrain, buildings, vegetation, etc. The Unity engine can apply appropriate materials and textures to the model to enhance the realism of the scene. The rendering parameters in the Unity engine need to be configured to ensure image quality, including resolution, anti-aliasing, etc.

[0023] 2) After adaptive sampling of each scene image, each scene image including adaptive sampling points of different densities is obtained, and the scene images including adaptive sampling points of different densities are input into the neural radiation field for training. During the training process, the neural radiation field generates a new perspective image through network forward propagation, and the generated new perspective image is compared with the scene image input into the neural radiation field to obtain the loss until the loss converges to obtain the trained neural radiation field.

[0024] For each scene image, the scene image is processed using an image processing algorithm to obtain important areas and unimportant areas in the scene image. High resolution and more sampling points are used for the important areas, and low resolution and fewer sampling points are used for the unimportant areas, thereby obtaining a scene image including adaptive sampling points of different densities.

[0025] Adaptive sampling dynamically adjusts the resolution or sampling density of the input image based on the importance and complexity of the scene. For example, a higher resolution is used in areas with rich details, while a lower resolution is used in flat or insignificant areas. First, the scene is pre-analyzed and evaluated, and the original scene is quickly scanned using image processing algorithms such as gradient calculation and texture analysis. The information density and visual saliency of different regions are evaluated to determine which regions contain rich details or key features. Then an importance map is constructed. Based on the evaluation results, an importance map is constructed in which the value of each pixel or voxel represents the importance of the position. Areas with high importance usually correspond to edges, corners, or other significant features in the scene. Then adaptive sampling is performed, and the sampling strategy is dynamically adjusted based on the importance map. The number of sampling points is increased in areas with high importance. In flat areas or areas with less visual changes, the sampling density is reduced to reduce the computational burden. Finally, sampling point allocation and adjustment are performed. According to the adaptive sampling method, sampling points are allocated in the scene. During the training process, the distribution of sampling points is dynamically adjusted based on real-time feedback (such as rendering error, convergence speed, etc.). Finally, the scene images including adaptive sampling points of different densities can be used for network training and rendering, and the sampling points obtained by adaptive sampling can be used to train the neural radiation field network.

[0026] 3) The trained neural radiation field is encapsulated into a plug-in form and integrated into the Unity engine. The image or 3D model to be rendered is input into the Unity engine encapsulated with the trained neural radiation field for processing. After processing, the Unity engine outputs the rendered image or 3D model and displays it on the monitor to achieve efficient rendering.

[0027] Export the trained neural radiation field to a preset standard format, such as TensorFlow's SavedModel, ONNX, or other deep learning model exchange formats. Create a new plug-in project in the integrated development environment IDE (Integrated Development Environment) and interact with the Unity engine. Configure the dependencies and libraries required for interaction with the Unity engine. Then integrate the trained neural radiation field in the preset standard format into the new plug-in project to ensure that the model can be loaded and run correctly. Finally, use the encapsulation API to encapsulate the new plug-in project as a plug-in for the Unity engine, and call the trained neural radiation field through the Unity engine for rendering; define the interface for interacting with the plug-in in the Unity engine, including data transfer, function calls, etc. Write scripts in the Unity engine, and use these scripts to call the API provided by the plug-in to achieve the rendering of the neural radiation field model.

[0028] When developing a plug-in framework, you need to select programming languages ​​and tools. According to the existing development technology and the compatibility of the Unity engine, select the appropriate programming language (such as C# or C++) and development tools. The Unity engine supports the extension of its functions through plug-ins. Therefore, the trained neural radiation field is encapsulated in the form of a plug-in, which can be easily integrated into the Unity engine to enhance its rendering capabilities. By utilizing the parallel computing capabilities supported by the Unity engine, such as multi-threaded rendering or GPU acceleration, the rendering speed of the neural radiation field model can be significantly improved. By assigning rendering tasks to multiple processing units, different ray casting and color prediction calculations can be processed in parallel to ensure the real-time and quality of rendering.

[0029] The method for efficient rendering of neural radiation field based on Unity engine of the present invention first uses Unity engine to generate a large amount of image data under different viewing angles and lighting conditions. These image data are put into the neural radiation field algorithm model for training. Then the trained neural radiation field algorithm model is encapsulated into a plug-in. Finally, the plug-in is added to the Unity engine through a function or interface call. The plug-in realizes automatic rendering in the Unity engine and improves the rendering capability of the Unity engine.

[0030] The specific implementation of the present invention is as follows: like Figure 1 As shown, the method of the present invention first uses the Unity engine to construct image data for the Unity rendering environment, then performs adaptive sampling processing on the obtained image data, uses the processed data to train the neural radiation field model, encapsulates the trained model into a plug-in and calls the plug-in using a function or interface, and after calling the plug-in in the Unity engine, the plug-in realizes automatic rendering and improves the rendering capability of the Unity engine. Example

[0031] S1: Select a representative complex indoor scene, such as a living room with intricate decoration or a multi-functional office. Make sure the selected scene contains rich objects, materials, and lighting conditions to demonstrate the powerful expressiveness of the neural radiation field model. Accurately build the scene in the Unity engine, including the placement of objects, the assignment of materials, and the setting of lights. Set up multiple virtual cameras to capture image data of the scene at different angles and positions. These images will be used to train the neural radiation field model to learn the three-dimensional structure and lighting information of the scene. Collect at least 100 high-quality scene images from different perspectives to ensure the diversity and comprehensiveness of the data and provide a solid foundation for model training.

[0032] S2: Preprocess the collected image data, including format conversion, size adjustment, etc., to meet the input requirements of the neural radiation field model. Design a neural radiation field model with a multi-layer perceptron MLP structure, which should contain 8 hidden layers, each equipped with 256 neurons, to fully learn the complex features of the scene. Determine the output of the model as the color and opacity of the scene, which will be used in the subsequent volume rendering process.

[0033] S3: Use the Adam optimizer for model training and set the learning rate to 0.001 to ensure the stability and convergence speed of the training process. Set the training rounds to 100,000 times to ensure that the model can fully learn the characteristics of the scene and achieve high rendering quality. Optimize the model parameters by minimizing the difference between the rendered image and the real image (such as the mean square error loss), thereby continuously improving the rendering accuracy of the model.

[0034] S4: Convert the trained neural radiance field model to a format compatible with the Unity engine, such as ONNX or TensorFlow Lite. Create a custom rendering plug-in in the Unity engine to load and run the neural radiance field model. The plug-in should be able to seamlessly integrate with the rendering pipeline of the Unity engine to achieve efficient real-time rendering.

[0035] S5: Load the custom neural radiance field rendering plug-in in the Unity engine. Set rendering parameters such as resolution, sampling rate, etc. according to actual needs. These parameters will directly affect the quality and performance of the rendering. Make sure the plug-in works with other parts of the Unity engine (such as cameras, lights, etc.) to obtain the best rendering effect.

[0036] S6: Perform real-time rendering of the selected indoor scene to observe the rendering effect and performance. Compare with traditional rendering methods (such as rasterization or ray tracing) to evaluate the performance improvement that the neural radiation field rendering method can bring while maintaining high-quality rendering effects. Adjust the neural radiation field model or rendering parameters based on the evaluation results to further optimize the rendering effect and performance.

[0037] S7: Performance test on a computer with Intel Core i7 CPU and NVIDIA GTX 1080 GPU. Record the time required to render a frame of 1920x1080 resolution image and compare it with other rendering methods. It is expected that the time using the neural radiation field rendering method will be significantly reduced, for example, only about 0.1 seconds. Display the rendering results, including scene renderings at different viewing angles and comparisons with other methods, to intuitively show the advantages and effects of the neural radiation field rendering method. Example

[0038] S1: First, create a new project in the Unity engine and set up the basic environment, including skybox, terrain, etc. Import the required 3D models, textures, sound effects and other resources. These resources will be used to build static and dynamic elements in the scene. Use 3D models to create dynamic elements such as trees and water. Use the animation system of the Unity engine to animate these elements, such as the fluttering of leaves and the fluctuation of water. Use the physics engine to simulate natural phenomena, such as wind blowing leaves and water collision.

[0039] S2: Place multiple virtual cameras in the Unity engine to capture different perspectives of the scene. Capture image sequences of multiple perspectives of the scene through the virtual cameras. These images will be used as training data. Preprocess the captured images, including format conversion, size adjustment, normalization, etc., to adapt to the input requirements of the neural radiation field model.

[0040] S4: Choose a suitable deep learning framework (such as TensorFlow or PyTorch). Build the network structure of the neural radiation field model, including components such as MLP (Multi-layer Perceptron). Train the model using the captured image sequence, paying special attention to the accurate rendering of dynamic elements. Optimize the training process by adjusting model parameters and learning rate, etc.

[0041] S5: Convert the trained neural radiance field model to a format compatible with the Unity engine, such as ONNX. Create a custom rendering plugin in the Unity engine to load and run the converted neural radiance field model. Integrate the neural radiance field model into the rendering pipeline of the Unity engine. This may require writing some custom Unity engine scripts or plugins to implement. In the rendering pipeline of the Unity engine, use the neural radiance field model as a new rendering channel or effect. When the user interacts with the scene (such as changing the state of dynamic elements), the neural radiance field model updates the rendering results in real time based on the interaction.

[0042] S6: Analyze and optimize the rendering pipeline of the Unity engine to ensure that the integration with the neural radiance field model does not significantly degrade performance. Adjust the parameters and settings of the rendering pipeline to find the best balance between performance and rendering quality. Perform performance tests on a high-performance computer and record the time to render a frame of dynamic scenes. Optimize performance by adjusting the parameters of the neural radiance field model and the rendering settings of the Unity engine. Ensure that the time to render a frame of dynamic scenes is controlled within 0.05 seconds to meet the needs of real-time interaction.

Claims

1. A method for efficiently rendering neural radiation fields based on the Unity engine, characterized in that: include: 1) Build a 3D model of each element in the same scene that changes over time and input it into the Unity engine. After processing, the Unity engine outputs several scene images of the current scene that change over time under different viewing angles and different lighting conditions; 2) After adaptive sampling of each scene image, each scene image including adaptive sampling points of different densities is obtained, and the scene images including adaptive sampling points of different densities are input into the neural radiation field for training. During the training process, the neural radiation field generates a new perspective image through network forward propagation, and the generated new perspective image is compared with the scene image input into the neural radiation field to obtain the loss, until the loss converges, and the trained neural radiation field is obtained; 3) The trained neural radiation field is encapsulated into a plug-in form and integrated into the Unity engine. The image or 3D model to be rendered is input into the Unity engine encapsulated with the trained neural radiation field for processing. After processing, the Unity engine outputs the rendered image or 3D model and displays it on the monitor to achieve efficient rendering.

2. The method for efficiently rendering neural radiation fields based on the Unity engine according to claim 1, characterized in that: In the step 1), several cameras are added to the virtual environment in the Unity engine, each camera captures a different perspective of the scene, and each camera adopts a different position, rotation and focal length; different types of light sources are added to the virtual environment, and the lighting parameters of each light source are adjusted differently; The Unity engine uses each camera to obtain scene images from different perspectives under different lighting conditions and performs automatic batch rendering, ultimately outputting several scene images in preset formats.

3. The method for efficiently rendering neural radiation fields based on the Unity engine according to claim 1, characterized in that: In the step 2), for each scene image, the scene image is processed using an image processing algorithm to obtain important areas and unimportant areas in the scene image, and high resolution and more sampling points are used for the important areas, and low resolution and fewer sampling points are used for the unimportant areas, thereby obtaining a scene image including adaptive sampling points of different densities.

4. The method for efficiently rendering neural radiation fields based on the Unity engine according to claim 1, characterized in that: In the step 3), the trained neural radiation field is exported to a preset standard format, a new plug-in project is created in the integrated development environment IDE and interacts with the Unity engine, and then the trained neural radiation field in the preset standard format is integrated into the new plug-in project. Finally, the new plug-in project is encapsulated as a plug-in of the Unity engine, and the trained neural radiation field is called by the Unity engine for rendering.

5. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method according to any one of claims 1 to 4.

6. A computer-readable storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images