An optimized rendering interaction method for the fusion of neural radiation field and three-dimensional system

Abstract

The present invention provides an optimized rendering interaction method for the fusion of a neural radiation field and a three-dimensional system, the optimized rendering interaction method comprising: in the rendering of a three-dimensional scene, using a trained NeRF to render the real-scene three-dimensional data in the three-dimensional scene; in the interactive operation and analysis, obtaining the screen coordinates, depth and camera posture; converting the screen coordinate information into geographic space information, and transmitting the geographic space information and the non-spatial information required for the operation to the background, and measuring, querying and analyzing the real-scene three-dimensional data based on the traditional three-dimensional engine in the background; in the fusion of rendering and interaction, rendering the result after the operation and calculating the depth map, analyzing the occlusion relationship between the three-dimensional scene rendered by NeRF and the rendering map of the operation result, and fusing the rendered result. The fusion of NeRF and the real-scene three-dimensional system is realized, which not only improves the rendering effect and efficiency of the three-dimensional scene, but also has high-precision measurement, query and analysis results.

Description

Technical Field

[0001] The present invention relates to the field of real-scene three-dimensional (3D) and in particular to an optimized rendering interaction method for fusing a neural radiation field with a 3D system. Background Art

[0002] 3DReal Scene (3DRS) is a digital virtual space that reflects and expresses human production, life and ecological space in a real, three-dimensional and time-sequential manner. It is an important foundation for the construction of a modern national governance system and governance capacity. In the 3DReal Scene system, it is necessary to display the 3D scene, and also to perform interactive operations and analysis such as annotation, query and measurement to assist in completing urban planning, traffic management, emergency rescue, environmental protection and other related tasks.

[0003] Commonly used real-life 3D systems are generally developed based on WebGL or game engines, such as Cesium and Unreal Engine (UE). If you want to truly display a 3D scene, you need a refined manual model or dense and high-quality triangular mesh data, especially for complex entities such as irregular buildings and trees, which are expensive to model, have poor display effects, and low rendering performance. When processing a large amount of 3D data, in order to optimize the loading speed and effect, multi-level LOD is generally generated for the data, but the processed data is fragmented and large in quantity, and there is a delay when loading.

[0004] Neural Radiance Fields (NeRF) is a 3D reconstruction technology based on neural networks. Unlike traditional 3D reconstruction methods that represent scenes as explicit expressions such as point clouds, grids, and voxels, NeRF models scenes as a continuous 5D radiation field and stores it implicitly in a neural network, which enables the prediction of the geometric structure and surface reflection properties of the 3D scene, thereby rendering a 3D scene from any observation angle. It has realistic rendering effects, high rendering efficiency, and low cost. It can well display trees and buildings with complex shapes, and has attracted widespread attention. An existing invention (publication number: CN117422804A) uses it in the rendering of 3D scenes, but the existing invention only realizes the rendering of 3D scenes based on NeRF, and cannot realize interactive operations and 3D analysis such as arbitrary annotation, query, and measurement in 3D scenes, and cannot build a complete real-life 3D system to assist in completing urban planning, traffic management, emergency rescue, environmental protection and other related tasks. Summary of the invention

[0005] In view of the above problems, the present invention is proposed to provide an optimized rendering interaction method for fusing a neural radiation field with a three-dimensional system, which overcomes the above problems or at least partially solves the above problems.

[0006] According to one aspect of the present invention, an optimized rendering interaction method for integrating a neural radiation field with a three-dimensional system is provided, and the optimized rendering interaction method comprises:

[0007] In 3D scene rendering, the trained NeRF is used to render the real 3D data in the 3D scene.

[0008] In interactive operations and analysis, screen coordinates, depth and camera posture are obtained.

[0009] The screen coordinate information is converted into geographic space information, and the geographic space information is transmitted to the background together with the non-spatial information required for the operation.

[0010] The real-scene three-dimensional data is measured, queried and analyzed based on a traditional three-dimensional engine in the background.

[0011] The result after the operation is rendered and the depth map is calculated, and the occlusion relationship between the three-dimensional scene rendered by NeRF and the rendered image of the operation result is analyzed.

[0012] The rendering results are fused and the fused image is provided to the front end.

[0013] Optionally, the interactive operation and analysis specifically include: annotation, query and measurement interactive operations and three-dimensional analysis.

[0014] Optionally, the interactive operation includes labeling, querying, and measuring.

[0015] Optionally, the screen coordinate information specifically includes: screen coordinate information that needs to be measured, queried, and analyzed for interactive operations.

[0016] Optionally, rendering the result after the operation and calculating the depth map, and analyzing the occlusion relationship between the three-dimensional scene rendered by NeRF and the rendered image of the operation result specifically includes: rendering the result after the operation and calculating the depth map; comparing the depth map of the rendered result of the operation result with the depth map generated by NeRF; analyzing the occlusion relationship between the three-dimensional scene rendered by NeRF and the rendered image of the operation result.

[0017] Optionally, fusing the rendering results and providing the fused image to the front end specifically includes: fusing the rendering results and providing the fused image to the front end to realize interactive operations of real-scene three-dimensional rendering and annotation, query, measurement and three-dimensional analysis.

[0018] Optionally, the training of the NeRF model specifically includes: according to oblique photogrammetry aerial photographs and aerial triangulation information, or based on an existing real-scene three-dimensional model, by writing a program to obtain pictures, depth maps and camera parameters of the three-dimensional scene from various angles in a three-dimensional virtual scene, to form a data set, and divide it into training, verification and test data sets in proportion; using pytorch or other deep learning libraries to build a NeRF model; training the model with the three-dimensional scene pictures and depth maps of the training data as the target output of the model, using the verification data set to adjust the hyperparameters in the training process, saving the model when the loss function converges, and performing accuracy testing with the test data set.

[0019] Optionally, the operation result rendering specifically includes: converting the screen coordinate information, depth information and camera posture of the interactive operation and analysis of the current frame transmitted from the control and fusion module into geographic space information; based on the real-scene three-dimensional data loaded in the background, calculating and analyzing the geographic space information and non-spatial information transmitted by the control fusion module to obtain the results of the interactive operation and analysis; using the current frame camera posture to render the interactive operation and analysis results and calculate the depth map; transmitting the analysis result rendering, the calculated depth map, the current frame time and the rendering completion time back to the control and fusion module.

[0020] The present invention provides an optimized rendering interaction method for the fusion of a neural radiation field and a three-dimensional system, the optimized rendering interaction method comprising: in three-dimensional scene rendering, using a trained NeRF to render real-life three-dimensional data in a three-dimensional scene; in interactive operations and analysis, obtaining screen coordinates, depth, and camera posture; converting screen coordinate information into geographic spatial information, and transmitting the geographic spatial information and non-spatial information required for the operation to the background; and measuring, querying, and analyzing the real-life three-dimensional data based on a traditional three-dimensional engine in the background.

[0021] The result of the operation is rendered and the depth map is calculated. The occlusion relationship between the 3D scene rendered by NeRF and the rendered image of the operation result is analyzed. The rendered results are fused and the fused image is provided to the front end. The fusion of NeRF and the real-life 3D system is realized, which not only improves the rendering effect and efficiency of the 3D scene, but also realizes interactive operations and 3D analysis such as annotation, query, and measurement.

[0022] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0024] Figure 1 A flowchart of an optimized rendering interaction method for fusing a neural radiation field with a three-dimensional system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0026] The terms "comprises" and "having" and any variations thereof in the description embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.

[0027] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0028] like Figure 1 As shown in the figure, NeRF and traditional real-life 3D system rendering methods are integrated to optimize the rendering effect and efficiency of 3D scenes, and to realize interactive operations and analysis such as annotation, query, and measurement. Specifically, in 3D scene rendering, the trained NeRF is used to render the real-life 3D data in the 3D scene to improve the rendering effect and efficiency; in interactive operations and analysis such as annotation, query, and measurement, the screen coordinates, depth, and camera posture are obtained to convert the screen coordinate information required for interactive operations such as measurement, query, and analysis into geographic spatial information, and the information is transmitted to the background together with the non-spatial information required for the operation. In the background, the real-life 3D data is measured, queried, and analyzed based on the traditional 3D engine, and the results after the operation are rendered and the depth map is calculated. By comparing the depth map of the rendered result of the operation with the depth map generated by NeRF, the occlusion relationship between the 3D scene rendered by NeRF and the rendered image of the operation result is analyzed, and then the rendered results are fused, and the fused image is provided to the front end to realize real-life 3D rendering and interactive operations such as annotation, query, and measurement, and 3D analysis.

[0029] 1. Training the NeRF model includes: 1.1 Oblique photogrammetry of aerial photographs and aerial triangulation information, or based on existing real-life 3D models, by writing a program to obtain pictures, depth maps and camera parameters of 3D scenes from various angles in a 3D virtual scene, forming a data set, and dividing it into training, verification and test data sets in a certain proportion.

[0030] 1.2 Use pytorch or other deep learning libraries to build NeRF models.

[0031] 1.3 Train the model using the 3D scene images and depth maps of the training data as the target output of the model, and use the validation data set to adjust the hyperparameters during the training process. When the loss function converges, save the model and use the test data set for accuracy testing.

[0032] 2. Obtain the screen coordinates, depth, current frame camera posture, parameters, time and non-temporal and non-spatial information of the interactive operation through the front-end rendering program, and monitor their changes.

[0033] 3 When the screen coordinates, depth, current frame camera posture and non-spatiotemporal information in 2 change, their information will be combined with the time information of the current frame and passed to the control and fusion module, which will be synchronously transmitted to the NeRF-based 3D scene rendering module and the operation analysis result rendering module for rendering.

[0034] 4. 3D scene rendering.

[0035] 4.1 Based on the current frame camera posture transmitted by the control and fusion modules in 3, the trained neural radiation field is used to render the three-dimensional scene from the perspective of the current frame camera and generate a depth map.

[0036] 4.2 The three-dimensional scene graph, depth map, current frame time and rendering completion time generated by the neural radiation field under the current frame camera perspective are transmitted back to the control and fusion module in 3.

[0037] 5. Render the analysis results.

[0038] 5.1 Convert the interactive operation and analyzed screen coordinate information, depth information and camera posture of the current frame transmitted by the control and fusion module 3 into geospatial information.

[0039] 5.2 Based on the real-life 3D data loaded in the background, the geographic spatial information and non-spatial information transmitted by the control fusion module are calculated and analyzed to obtain the results of the interactive operation and analysis.

[0040] 5.3 Use the current frame camera posture to render the interactive operations and analysis results and calculate the depth map.

[0041] 5.4 passes the analysis result rendering, calculated depth map, current frame time and rendering completion time back to the control and fusion module in 3.

[0042] 6 Through the control and fusion module in 3, the rendering result of the three-dimensional scene and the operation analysis result is integrated.

[0043] 6.1 determines whether the current frame times returned by 4.2 and 5.4 are simultaneous frames. In the case of the same frame, the rendering completion time difference returned by 4.2 and 5.4 is determined to meet the delay requirements (e.g., less than 30 milliseconds is considered to meet the requirements). The returned results that do not meet the requirements are discarded.

[0044] 6.2 obtains the occlusion relationship between the three-dimensional scene graph returned by 4.2 and the analysis result rendering graph returned by 5.4 based on the depth maps returned by 4.2 and 5.4.

[0045] 6.3 fuses the three-dimensional scene graph returned by 4.2 under the camera perspective of the current frame and the analysis result rendering graph returned by 5.4 according to the occlusion relationship.

[0046] 6.4 Obtain the final three-dimensional scene and the rendering of the interactive operation and analysis results in the current frame.

[0047] Beneficial effects: The present invention realizes the integration of NeRF and real-life 3D system, which not only improves the rendering effect and efficiency of 3D scenes, but also realizes interactive operations and 3D analysis such as annotation, query, and measurement. It can build a real-life 3D system with good rendering effect, high efficiency, and relatively complete functions to assist in completing urban planning, traffic management, emergency rescue, environmental protection and other related tasks.

[0048] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An optimized rendering interaction method for integrating neural radiation field with three-dimensional system, characterized in that: The optimized rendering interaction method comprises: In 3D scene rendering, the trained NeRF is used to render the real 3D data in the 3D scene; In interactive operations and analysis, screen coordinates, depth and camera posture are obtained; Convert the screen coordinate information into geographic space information, and transmit the geographic space information and non-spatial information required for the operation to the backend; In the background, the real-scene three-dimensional data is measured, queried and analyzed based on a traditional three-dimensional engine; Render the result after the operation and calculate the depth map, and analyze the occlusion relationship between the three-dimensional scene rendered by NeRF and the rendering of the operation result; The rendering results are fused and the fused image is provided to the front end.

2. According to claim 1, the optimized rendering interaction method for integrating neural radiation field with three-dimensional system is characterized in that: The interactive operation and analysis specifically include: annotation, query and measurement interactive operations and three-dimensional analysis.

3. The optimized rendering interaction method for integrating neural radiation field with three-dimensional system according to claim 1, characterized in that: The interactive operations include labeling, querying, and measuring.

4. The optimized rendering interaction method for integrating neural radiation field with three-dimensional system according to claim 1, characterized in that: The screen coordinate information specifically includes: screen coordinate information that needs to be measured, queried, and analyzed for interactive operations.

5. The optimized rendering interaction method for integrating neural radiation field with three-dimensional system according to claim 1, characterized in that: The rendering of the result after the operation and calculation of the depth map, and analysis of the occlusion relationship between the three-dimensional scene rendered by NeRF and the rendering map of the operation result specifically include: Render the result of the operation and calculate the depth map; By comparing the depth map of the rendered result with the depth map generated by NeRF; Analyze the occlusion relationship between the three-dimensional scene rendered by NeRF and the rendering of the operation result.

6. The optimized rendering interaction method for integrating neural radiation field with three-dimensional system according to claim 1, characterized in that: The step of fusing the rendered results and providing the fused image to the front end specifically includes: The rendering results are fused and the fused image is provided to the front end to realize the interactive operation of real-scene 3D rendering and annotation, query, measurement and 3D analysis.

7. The optimized rendering interaction method for integrating neural radiation field with three-dimensional system according to claim 1, characterized in that: The training of the NeRF model specifically includes: According to the oblique photogrammetry aerial photographs and aerial triangulation information, or based on the existing real-life 3D model, a program is written to obtain the pictures, depth maps and camera parameters of the 3D scene from various angles in the 3D virtual scene to form a data set, which is then divided into training, verification and test data sets in proportion. Use pytorch or other deep learning libraries to build NeRF models; The model is trained using the 3D scene images and depth maps of the training data as the target output of the model, and the validation dataset is used to adjust the hyperparameters during the training process. When the loss function converges, the model is saved and the accuracy is tested using the test dataset.

8. The optimized rendering interaction method for integrating neural radiation field with three-dimensional system according to claim 1, characterized in that: The operation result rendering specifically includes: Convert the screen coordinate information, depth information and camera posture of the interactive operation and analysis of the current frame transmitted by the control and fusion module into geospatial information; Based on the real-life 3D data loaded in the background, the geospatial information and non-spatial information transmitted by the control fusion module are calculated and analyzed to obtain the results of interactive operation and analysis; Use the current frame camera pose to render the interactive operations and analysis results and calculate the depth map; The analysis result rendering, calculated depth map, current frame time and rendering completion time are returned to the control and fusion module.

Images