An automatic driving scene simulation method and system based on neural point rendering

Abstract

The application discloses a kind of based on point's neural rendering autonomous driving scene simulation method and system, method is through the reconstruction and extension of scene, can generate the real test data of autonomous driving system under different visual angle, using the real point cloud data collected to provide accurate depth and position guidance, avoid the geometric error that point cloud possibly contains obtained by multi-view reconstruction, additional features related to observation visual angle and sampling frequency are assigned to each point cloud to estimate its color information, extend neural descriptor, to better represent the appearance characteristics of scene under different observation visual angle and distance, use multi-scale feature fusion and progressive rendering etc. method fills the missing area of point cloud, and point cloud completion algorithm is proposed to increase point cloud density at close and far. Finally, edit dynamic obstacles by neural descriptor, to expand the richness of simulation, to simulate the different states of traffic participants, enrich and expand visual simulation scene.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving simulation, specifically relating to an autonomous driving scene simulation method and system based on neural point rendering. Background Technology

[0002] Simulation testing offers advantages in efficiency and safety, compensating for the inability of road testing to reproduce extreme traffic scenarios. It is crucial for addressing the challenges of long-distance, multi-scenario testing in autonomous driving. To enable autonomous driving systems to achieve realistic driving in simulation environments, constructing diverse and rich driving scenarios and simulating more realistic sensor data has become an important research focus. While graphics-based simulation software can simulate scenes from a 360-degree free-viewpoint, its reliance on hand-designed 3D models results in a sim-to-real gap in richness and realism compared to actual data. Neural network-based methods are data-driven and can capture the complexity of the real world, but their operation entirely in 2D space makes 3D control difficult. Neural rendering combines the advantages of both methods, offering high controllability and the ability to synthesize highly realistic image data, achieving significant results in scene reconstruction and new perspective synthesis.

[0003] Commonly used neural rendering methods are typically based on multi-view image data, focusing primarily on single objects and local scenes, or large outdoor scene datasets with complete viewpoints. Autonomous driving datasets are usually designed for tasks such as object detection and tracking, with limited and sparsely distributed viewpoints, making it difficult to achieve comprehensive multi-angle coverage of the entire scene. Therefore, without additional guiding information, it is difficult to generate models with good reconstruction results. Commonly used neural radiation field methods are time-consuming to fit multiple sampling points on a single ray, making efficient rendering difficult on autonomous driving scenes with wide coverage and large data volumes. Point-based neural rendering methods for single targets assign a learnable neural descriptor to each point and project it onto a two-dimensional plane for rendering; the same point features from different viewpoints are difficult to fit the rich appearance of large outdoor scenes. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides an autonomous driving scene simulation method based on neural point rendering. By reconstructing and expanding the scene, it can generate realistic test data for autonomous driving systems from different perspectives. It uses the collected real point cloud data to provide accurate depth and position guidance, avoiding geometric errors that may be present in point clouds obtained from multi-view reconstruction. Dynamic obstacles are edited using neural descriptors to enrich the simulation and simulate different states of traffic participants, thus enriching and expanding the visual simulation scene.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: an autonomous driving visual simulation method based on neural point rendering, comprising the following steps:

[0006] A 3D point cloud map is constructed based on real point cloud data to represent the geometric features of autonomous driving scenarios.

[0007] The constructed 3D point cloud map is completed by filling point cloud data in areas with low point cloud density and large holes to obtain a completed 3D point cloud map.

[0008] We introduce feature-enhanced neural descriptors related to the viewing direction, and construct a 3D neural point cloud by encoding each point cloud in the completed 3D point cloud map, which is used to represent the appearance features of the scene.

[0009] The 3D neural point cloud is projected onto the rendering plane using rasterization and z-buffer methods to obtain the original 2D feature map.

[0010] A neural rendering network is constructed to progressively render multi-resolution feature images to obtain the rendered images;

[0011] Based on the rendered image, features of the same and different resolutions are fused, and the neural descriptors of the point cloud are modified to generate an autonomous driving scene from a new perspective.

[0012] Furthermore, a 3D point cloud map is constructed based on real point cloud data to represent the geometric features of autonomous driving scenarios, specifically as follows:

[0013] The scene is scanned using LiDAR to obtain multiple frames of point cloud data. By aligning, filtering, and fusing the point cloud data of each frame, a dense point cloud map is constructed to initialize the 3D scene, resulting in a 3D point cloud description that represents the geometric features and location information of the scene.

[0014] Furthermore, the constructed 3D point cloud map is completed by filling in point cloud data in areas with low point cloud density and large holes, resulting in a completed 3D point cloud map including:

[0015] The original image is converted from RGB space to YUV space to minimize the weighted depth difference between pixel r and other pixels in its neighborhood. The completed depth information is then converted to a 3D point cloud in the world coordinate system, and the completed 3D point cloud is uniformly filtered to eliminate redundancy caused by different frames completing the same area of ​​the point cloud, resulting in a completed 3D point cloud map.

[0016] Furthermore, feature-enhanced neural descriptors related to the observation direction are introduced. A 3D neural point cloud is constructed by encoding each point cloud in the completed 3D point cloud map, as detailed below:

[0017] Using neural descriptor D = {d1, d2, ... d N Encoded point cloud P = {p1, p2, ... p} N The appearance characteristics of}, where d i It is an M-dimensional vector that describes the local features and illumination properties of the point cloud;

[0018] In addition to containing fixed pseudo-color features c, each point cloud also contains a direction vector q between the camera and each point cloud. v Used to characterize the effect of the observation angle, while also adding the camera ray direction q. r To characterize the impact of image resolution, positional encoding is used to encode directional features into a high-dimensional space, and the extended descriptor D is:

[0019] D = {(c1,q} v1 ,q r1 ),(c2,q v2 ,q r2 ),,(c N ,q vN ,q rN )}

[0020] Furthermore, rasterization and z-buffering methods are used to project the 3D neural point cloud onto the rendering plane to obtain the original 2D feature map, including:

[0021] By projecting the neural descriptor D onto canvases of different sizes, an image pyramid was constructed. Among them, S t The resolution is Because autonomous driving scenarios cover a wide range and involve a large number of point clouds, when multiple point clouds have the same pixel coordinates, the z-buffer algorithm is used to retain only the point closest to the camera on each pixel, and the neural descriptor of the point cloud with the smallest depth is used to fill the point cloud with the same pixel.

[0022] Furthermore, the image pyramid is used as the rendering network R θ The input is expected to reduce the rendering image I by simultaneously optimizing the network parameters θ and the neural descriptor. θ With real data I GT The perceptual loss between them is used to learn an accurate representation of the scene; the rendering result is:

[0023]

[0024] The loss function is expressed as:

[0025]

[0026] Where, Δ l Indicates I in VGG network θ with IGT The MSE loss between the features of the l-th layer.

[0027] Furthermore, the rendering network is based on the UNet network and employs an encoder-decoder architecture for progressive rendering of input images at different resolutions. For the original input feature image S... i Use a fully connected network to extract view-related features Then with pseudo color c i Connecting them gives the description D i Feature F i Introducing the ResNet module, the feature F i The result is added to the convolutional result, the output is restricted to between 0 and 1 using the Sigmoid activation function, and then multiplied element-wise with the feature map activated by ELU to obtain the synthesized image.

[0028] Furthermore, based on the rendered image, features of the same and different resolutions are fused, and the neural descriptors of the point cloud are modified to generate autonomous driving scenes from new perspectives.

[0029] Based on the constructed feature pyramid, multi-scale features of different resolutions are fused, and average pooling layers are used to average the features. The result F encoded by the encoder is then used. Ei On the one hand, the original features F of the corresponding resolution i-1 The signal is connected as the input to the next level encoder, and simultaneously connected with the features of the same size obtained after upsampling in the decoding stage, which are also used as the input to the next level decoder.

[0030] This invention also provides an autonomous driving visual simulation system based on neural point rendering, including a 3D point cloud map construction module, a map completion module, a 3D neural point cloud construction module, a 2D original feature map acquisition module, a rendering module, and a fusion module;

[0031] The 3D point cloud map building module constructs a 3D point cloud map based on real point cloud data, which is used to represent the geometric features of autonomous driving scenarios.

[0032] The map completion module is used to complete the constructed 3D point cloud map by filling point cloud data in areas with low point cloud density and large holes, thus obtaining a completed 3D point cloud map.

[0033] The 3D neural point cloud construction module is used to introduce feature-enhanced neural descriptors related to the viewing direction. It constructs a 3D neural point cloud by encoding each point cloud in the completed 3D point cloud map, which is used to characterize the appearance features of the scene.

[0034] The 2D original feature map acquisition module uses rasterization and z-buffer methods to project the 3D neural point cloud onto the rendering plane to obtain the 2D original feature map.

[0035] The rendering module constructs a neural rendering network to progressively render multi-resolution feature images and obtain the rendered image.

[0036] The fusion module fuses features of the same and different resolutions based on the rendered image, modifies the neural descriptors of the point cloud, and generates an autonomous driving scene from a new perspective.

[0037] Another computer device is provided, including a processor and a memory. The memory is used to store a computer-executable program. The processor reads the computer-executable program from the memory and executes it. When the processor executes the program, it can realize the autonomous driving visual simulation method based on neural point rendering described in this invention.

[0038] Compared with the prior art, the present invention has at least the following beneficial effects:

[0039] This invention proposes a data-driven method for simulating autonomous driving scenarios. It reconstructs 3D scenes using real-world point cloud and image data, providing multi-view visual image data for autonomous driving systems. A point cloud completion method is employed, based on RGB images and depth maps, to repair and fill sparse and empty areas in the point cloud map, thereby improving the quality of the rendering results. This invention enhances the representation capability of large-scale outdoor scenes with sparse point clouds and rich details through multi-scale rendering and view-related feature expansion, demonstrating excellent rendering results on public datasets. Based on the method described in this invention, controllable editing of the scene is supported. While ensuring the realism of the simulation, it allows for the addition, movement, and deletion of traffic participants such as vehicles, thereby synthesizing richer and more flexible simulation test scenarios. Attached Figure Description

[0040] Figure 1 This is a flowchart of an autonomous driving scenario simulation based on point neural network rendering.

[0041] Figure 2 This is a flowchart illustrating the point cloud completion method.

[0042] Figure 3 This is a partial example of the depth map and rendering results before and after using the completion method.

[0043] Figure 4 This relates to encoder structure and feature fusion methods.

[0044] Figure 5 A schematic flowchart for inserting vehicles into a scene.

[0045] Figure 6Example results of scene editing functionality Detailed Implementation

[0046] The exemplary embodiments of this application are described in detail below with reference to the accompanying drawings and specific implementations, including various details of the embodiments of this application to aid understanding.

[0047] Figure 1 This invention presents a flowchart of an autonomous driving scene simulation based on point neural rendering. This invention is a real-data-driven visual simulation method that uses self-constructed or publicly available autonomous driving datasets to reconstruct, expand, and render 3D scenes based on real-world point cloud and image data. First, multiple frames of point clouds are aligned, registered, and filtered. A point cloud representation of the 3D scene is obtained through fusion, and holes in the point cloud are filled using RGB images and depth maps. Second, rasterization is used to project the neural point cloud onto image planes of multiple resolutions, and encoded viewpoint features are appended to the neural descriptors of each pixel. Finally, progressive rendering and feature fusion methods are used to synthesize the autonomous driving scene from a new viewpoint.

[0048] Step 1: Construct a 3D point cloud map based on real point cloud data to represent the geometric features of the autonomous driving scenario.

[0049] This method uses LiDAR to scan the scene and obtain multiple frames of point cloud data. By aligning, filtering, and fusing the point clouds in each frame, a dense point cloud map is constructed to initialize the 3D scene, representing its geometric features and location information. This avoids the geometric errors that may be introduced when reconstructing scenes from multi-view image sets using 3D reconstruction methods such as SFM and MVS, as well as the point cloud holes caused by some areas being less covered by the camera. These errors affect the accuracy and effect of 3D reconstruction, thus reducing rendering quality and structural similarity.

[0050] Step 2: Complete the constructed 3D point cloud map by filling in areas with low point cloud density and large holes. The specific method is as follows:

[0051] Because real-world point cloud data suffers from uneven density distribution and limited scanning range, feature neural descriptors are densely distributed in the center of the image, while sparsely distributed in areas closer to and farther from the camera, resulting in significant blurring and gaps at the top and bottom edges of the image. Therefore, this invention completes the low-density parts of the point cloud, as illustrated in the flowchart below. Figure 2 As shown, drawing on color completion methods, it is assumed that regions with similar colors within the same time and local area have similar depths. The original image is converted from RGB space to YUV space, aiming to minimize the weighted depth difference between pixel r and the other pixels in its neighborhood.

[0052]

[0053] Where N(r) is the neighborhood of pixel r, typically taken as 3*3, w rs It represents the weights of points surrounding r, reflecting the similarity of each pixel to other pixels in its neighborhood on the Y channel (luminance channel). The greater the similarity, the higher the weight. rs The calculation formula is:

[0054] w rs =exp(-(Y(r)-Y(s)) 2 / 2σ 2 )

[0055] Furthermore, the completed depth information is converted into a 3D point cloud in the world coordinate system, and the completed point cloud is uniformly filtered to eliminate redundancy caused by different frames completing the same area of ​​the point cloud, thus obtaining a completed 3D point cloud map.

[0056] Figure 3 The images show depth maps and rendering results before and after point cloud completion. It can be seen that point cloud completion of sparse areas can alleviate blurring and shadows at image edges and provide detailed representation of scenes at the end of roads, such as signs, leaves, and building edges. Furthermore, the method proposed in this invention can better define regions and describe colors in the sky.

[0057] Step 3: Introduce feature-enhanced neural descriptors related to the viewing direction. By encoding each point cloud in the completed 3D point cloud map to construct a 3D neural point cloud, the appearance features of the scene are characterized.

[0058] Using neural descriptor D = {d1, d2, ... d N Encoded point cloud P = {p1, p2, ... p} N The appearance characteristics of}, where d i As an M-dimensional vector, it describes the local features and lighting attributes of the point cloud. Unlike new perspective generation work for indoor scenes or single objects, autonomous driving datasets are typically collected by onboard cameras traveling in a straight line. The observation of a certain area will undergo continuous changes from far to near, and the sampling frequency of the images varies greatly. In this case, assigning a fixed neural descriptor to each point cloud will cause aliasing in the rendered image at a distance and blurring at a closer distance. To solve this problem, this invention extends the neural descriptor, considering that each point cloud, in addition to containing a fixed feature c, also contains a direction vector q between the camera and each point cloud. v To characterize the effect of the observation perspective, and also to add the camera ray direction q rTo characterize the impact of image resolution, since deep neural networks tend to learn low-frequency information of features, positional encoding is used to encode directional features into a high-dimensional space to describe finer details of the scene. The descriptor D can be represented as:

[0059]

[0060] Among them, c i For point p i False color features; For point p i The encoding result of the direction vector between the camera and C; Represents point cloud p i The high-bit encoding result of the direction vector between the center point of the projected pixel and the camera C.

[0061] Step 4: Project the 3D neural point cloud onto the rendering plane using rasterization and z-buffer methods to obtain the original 2D feature map. Specific details are as follows:

[0062] For a point cloud, given the camera's intrinsic and extrinsic parameters C, the perspective projection will be d. i Mapped to pixel space coordinates [x, y]. This invention projects the neural descriptor D onto canvases of different sizes to construct an image pyramid. Among them, S t The resolution is Due to the wide coverage and large number of point clouds in autonomous driving scenarios, when multiple point clouds have the same pixel coordinates, the z-buffer algorithm is used to retain only the point closest to the camera on each pixel, thereby effectively reducing the number of neural descriptors and accelerating training and rendering. Therefore, the pixel is filled with the neural descriptor of the point cloud with the smallest depth, i.e., S. t (P, D, C)[x, y] = di, where di is used as a pseudo-color to describe the color and brightness information of the point, and is subsequently learned by a neural network to be converted into the color of the pixel.

[0063] Step 5: Construct a neural rendering network to progressively render multi-resolution feature images, specifically as follows: Use the image pyramid as the rendering network R. θ The input is expected to reduce the rendering image I by simultaneously optimizing the network parameters θ and the neural descriptor. θ With real data I GT The perceptual loss between them is used to learn an accurate representation of the scene. The rendering result is represented as:

[0064]

[0065] The loss function is expressed as:

[0066]

[0067] Where, Δ l Indicates I in VGG network θ with I GT The MSE loss between the features of the l-th layer.

[0068] The rendering network constructed in this invention is an extension of the UNet network, employing an encoder-decoder architecture to progressively render input images at different resolutions, thereby better handling features in the scene from coarse to fine. The encoder module is as follows: Figure 4 As shown. For the original input feature image S i Using a fully connected network to extract view-related features (q vi ,q ri ), and then with pseudo color c i Connecting them gives the description D i Feature F i Secondly, the ResNet module is introduced to incorporate feature F. i The initial features are added to the convolutional result to enhance their guiding role in network optimization. Gated convolutions are used to fill in gaps in the image not covered by neural descriptors in a learnable manner, thus synthesizing features such as sky and gaps in leaves. The Sigmoid activation function restricts the output to between 0 and 1 and multiplies it element-wise with the ELU-activated feature map to obtain the rendering result, which indicates the effectiveness of that region in the feature map, allowing the network to autonomously learn which parts need to be patched.

[0069] Step 6: Fuse features of the same and different resolutions to improve rendering results. Specific details are as follows:

[0070] The sparsity of point clouds can cause holes and blurriness in rendering results. Higher resolution images can better describe the details in a scene, but point clouds have low coverage and large holes. While lower resolution images are less affected by holes, they can only provide a coarse description of the original scene and are unable to depict rich appearance features. Based on feature pyramids, this invention fuses multi-scale features to compensate for the deficiencies of images at different scales in these two aspects. Images of different sizes learn features from each other, thus achieving better results in both detail and accuracy.

[0071] A neural descriptor for a point cloud encodes features within its local area. However, describing a pixel using only a single point cloud feature descriptor can lead to jagged edges and artifacts due to an excessively low sampling frequency, especially in high-resolution images. Fitting the color of a pixel using only a single-point feature neural descriptor can also result in jagged edges and artifacts due to the low sampling frequency. Therefore, using an average pooling layer to average the features effectively performs multiple samplings of a single pixel, thereby increasing the sampling rate and eliminating artifacts. Figure 4 As shown, the result F encoded by the encoder Ei On the one hand, the original features F of the corresponding resolution i-1 The concatenation serves as input to the next level encoder, compensating for the insufficient ability of a single point cloud to describe a local scene. Simultaneously, it is concatenated with features of the same size obtained after upsampling in the decoding stage, serving as input to the next level decoder to compensate for the loss of original features and fine details after multiple convolutions and upsampling.

[0072] By modifying the neural descriptor of point clouds, this invention supports the movement, insertion, and deletion of various objects in a scene, thereby enabling the simulation of specified scenes and even dangerous working conditions. Specific details are as follows:

[0073] A vehicle dataset was constructed using the simulation software AirSim to obtain a complete description of the vehicles, thereby enabling the insertion of vehicles into the scene from different perspectives. Figure 5 The method for vehicle insertion is demonstrated. Specifically, depth maps and semantic segmentation maps from five perspectives are acquired in a simulation platform. The depth information corresponding to the vehicle is transformed into 3D space and fused to obtain a 3D point cloud model of the vehicle. Next, multiple images are taken around the vehicle model in the simulation scene to acquire images at different distances and angles. The rendering network trained on the scene dataset is frozen, and the vehicle neural descriptor is trained. Finally, the vehicle point cloud is scaled, translated, and rotated, and inserted into the background point cloud. Simultaneously, the descriptors from the scene and vehicle data are fused, and the network is used for rendering.

[0074] For traffic participants that need to be moved or removed, this invention filters out the ground using the RANSAC method and performs clustering using DBSCAN, thereby determining the area where the target is located in the dense point cloud map and realizing the deletion or movement of point cloud blocks. Figure 6 Partial results are shown. The first two lines demonstrate the effect of vehicle removal; this invention can remove dynamic obstacles from the scene to synthesize a driving scene without dynamic vehicles. The last two lines show the rendering results after inserting the vehicle into the scene, simulating a scenario where a vehicle suddenly appears in front of the driver's view, to test the robustness and safety performance of the autonomous vehicle in dangerous scenarios.

[0075] The effectiveness of the proposed method was tested on the KITTI and Brno-Urban datasets, and its performance was compared with that of the classic neural point rendering method NPBG. Resident and Road scenes from the KITTI dataset and City scenes from the Brno-Urban dataset were selected, and the visual quality of the synthesized images was evaluated using perceptual loss (VGG loss), peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and learned perceptual image patch similarity (LPIPS). The test results are shown in the table below:

[0076]

[0077]

[0078] In summary, this invention provides an autonomous driving scene simulation method based on neural point rendering. By reconstructing and expanding the scene, it can generate realistic test data for autonomous driving systems from different perspectives. Accurate depth and position guidance is provided using collected real point cloud data, avoiding geometric errors that may be present in point clouds obtained from multi-view reconstruction. In addition to assigning latent variables to each point cloud to estimate its color information, features related to the viewing angle and sampling frequency are added, expanding the neural descriptor to better characterize the appearance features of the scene at different viewing angles and distances. Secondly, methods such as multi-scale feature fusion and progressive rendering are used to fill in missing regions of the point cloud, and a point cloud completion algorithm is proposed to increase point cloud density at closer and farther distances. Finally, dynamic obstacles are edited using neural descriptors to enrich the simulation and simulate different states of traffic participants, thus enriching and expanding the visual simulation scene.

[0079] With the same technical concept as the method described above, this invention provides an autonomous driving visual simulation system based on neural point rendering, including a 3D point cloud map construction module, a map completion module, a 3D neural point cloud construction module, a 2D original feature map acquisition module, a rendering module, and a fusion module;

[0080] The 3D point cloud map building module constructs a 3D point cloud map based on real point cloud data, which is used to represent the geometric features of autonomous driving scenarios.

[0081] The map completion module is used to complete the constructed 3D point cloud map by filling point cloud data in areas with low point cloud density and large holes, thus obtaining a completed 3D point cloud map.

[0082] The 3D neural point cloud construction module is used to introduce feature-enhanced neural descriptors related to the viewing direction. It constructs a 3D neural point cloud by encoding each point cloud in the completed 3D point cloud map, which is used to characterize the appearance features of the scene.

[0083] The 2D original feature map acquisition module uses rasterization and z-buffer methods to project the 3D neural point cloud onto the rendering plane to obtain the 2D original feature map.

[0084] The rendering module constructs a neural rendering network to progressively render multi-resolution feature images and obtain the rendered image.

[0085] The fusion module fuses features of the same and different resolutions based on the rendered image, modifies the neural descriptors of the point cloud, and generates an autonomous driving scene from a new perspective.

[0086] The present invention can also provide a computer device, including a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads the computer executable program from the memory and executes it, and the processor can implement the autonomous driving visual simulation method based on neural point rendering described in the present invention when executing the computer executable program.

[0087] The computer equipment may be a laptop, desktop computer, vehicle-mounted computer, or workstation.

[0088] The processor described in this invention may be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or an off-the-shelf programmable gate array (FPGA).

[0089] The memory described in this invention can be an internal storage unit of a laptop, desktop computer, or workstation, such as memory or hard disk; or it can be an external storage unit, such as a portable hard disk or flash memory card.

Claims

1. A visual simulation method for autonomous driving based on neural point rendering, characterized in that, Includes the following steps: A 3D point cloud map is constructed based on real point cloud data to represent the geometric features of autonomous driving scenarios. The constructed 3D point cloud map is completed by filling point cloud data in areas with low point cloud density and large holes to obtain a completed 3D point cloud map. A feature-enhanced neural descriptor related to the viewing direction is introduced. A 3D neural point cloud is constructed by encoding each point cloud in the completed 3D point cloud map, which is used to represent the appearance features of the scene; specifically as follows: Using neural descriptors Encoded point cloud Appearance features, among which It is an M-dimensional vector that describes the local features and illumination properties of the point cloud; Each point cloud contains fixed pseudo-color features. In addition, it also includes the direction vectors between the camera and each point cloud. Used to characterize the effect of the observation angle, while also adding the camera's light direction. To characterize the impact of image resolution, positional encoding is used to encode directional features into a high-dimensional space, and the extended descriptor D is: ； The 3D neural point cloud is projected onto the rendering plane using rasterization and z-buffering methods to obtain the original 2D feature map; this includes projecting the neural descriptor D onto canvases of different sizes to construct an image pyramid. ,in, The resolution is This allows for the acquisition of multi-resolution feature images; A neural rendering network is constructed to progressively render multi-resolution feature images to obtain the rendered images; Based on the rendered image, features of the same and different resolutions are fused, and the neural descriptors of the point cloud are modified to generate an autonomous driving scene from a new perspective.

2. The autonomous driving visual simulation method based on neural point rendering according to claim 1, characterized in that, A 3D point cloud map is constructed based on real point cloud data to represent the geometric features of autonomous driving scenarios, specifically as follows: The scene is scanned using LiDAR to obtain multiple frames of point cloud data. By aligning, filtering, and fusing the point cloud data of each frame, a dense point cloud map is constructed to initialize the 3D scene, resulting in a 3D point cloud description that represents the geometric features and location information of the scene.

3. The autonomous driving visual simulation method based on neural point rendering according to claim 1, characterized in that, The constructed 3D point cloud map is completed by filling in point cloud data in areas with low point cloud density and large holes, resulting in a completed 3D point cloud map including: The original image is converted from RGB space to YUV space to minimize the weighted depth difference between pixel r and other pixels in its neighborhood. The completed depth information is then converted to a 3D point cloud in the world coordinate system, and the completed 3D point cloud is uniformly filtered to eliminate redundancy caused by different frames completing the same area of ​​the point cloud, resulting in a completed 3D point cloud map.

4. The autonomous driving visual simulation method based on neural point rendering according to claim 1, characterized in that, Using the image pyramid as a rendering network The input is expected to optimize network parameters simultaneously. And neural descriptors, reduce the rendering image size With real data Perceptual loss between learning scenarios, and accurate representation of the learning scene; The rendering result is: The loss function is expressed as: in, Indicating VGG network and The l MSE loss between layer features.

5. The autonomous driving visual simulation method based on neural point rendering according to claim 1, characterized in that, The rendering network is based on the UNet network and employs an encoder-decoder architecture to progressively render input images at different resolutions, for the original input feature image. Use a fully connected network to extract view-related features Then with pseudo colors Connect to obtain the description Features Introducing the ResNet module to incorporate features The result is added to the convolutional result, the output is restricted to between 0 and 1 using the Sigmoid activation function, and then multiplied element-wise with the feature map activated by ELU to obtain the synthesized image.

6. The autonomous driving visual simulation method based on neural point rendering according to claim 1, characterized in that, Based on the rendered image, features of the same and different resolutions are fused, and the neural descriptors of the point cloud are modified to generate autonomous driving scenes from new perspectives. Based on the constructed feature pyramid, multi-scale features of different resolutions are fused. An average pooling layer is used to average the features, and the result is encoded by the encoder. On the one hand, the original features of the corresponding resolution The signal is connected as the input to the next level encoder, and simultaneously connected with the features of the same size obtained after upsampling in the decoding stage, which are also used as the input to the next level decoder.

7. An autonomous driving visual simulation system based on neural point rendering, characterized in that, It includes a 3D point cloud map construction module, a map completion module, a 3D neural point cloud construction module, a 2D original feature map acquisition module, a rendering module, and a fusion module; The 3D point cloud map building module constructs a 3D point cloud map based on real point cloud data, which is used to represent the geometric features of autonomous driving scenarios. The map completion module is used to complete the constructed 3D point cloud map by filling point cloud data in areas with low point cloud density and large holes, thus obtaining a completed 3D point cloud map. The 3D neural point cloud construction module is used to introduce feature-enhanced neural descriptors related to the viewing direction. It constructs a 3D neural point cloud by encoding each point cloud in the completed 3D point cloud map, which is used to represent the appearance features of the scene; specifically as follows: Using neural descriptors Encoded point cloud Appearance features, among which It is an M-dimensional vector that describes the local features and illumination properties of the point cloud; Each point cloud contains fixed pseudo-color features. In addition, it also includes the direction vectors between the camera and each point cloud. Used to characterize the effect of the observation angle, while also adding the camera's light direction. To characterize the impact of image resolution, positional encoding is used to encode directional features into a high-dimensional space, and the extended descriptor D is: ； The 2D raw feature map acquisition module uses rasterization and z-buffering methods to project the 3D neural point cloud onto the rendering plane to obtain the 2D raw feature map; this includes: projecting the neural descriptor D onto canvases of different sizes to construct an image pyramid. ,in, The resolution is This allows for the acquisition of multi-resolution feature images; The rendering module constructs a neural rendering network to progressively render multi-resolution feature images and obtain the rendered image. The fusion module fuses features of the same and different resolutions based on the rendered image, modifies the neural descriptors of the point cloud, and generates an autonomous driving scene from a new perspective.

8. A computer device, characterized in that, It includes a processor and a memory, the memory being used to store a computer-executable program, the processor reading part or all of the computer-executable program from the memory and executing it, and the processor executing part or all of the computer-executable program is able to implement the autonomous driving visual simulation method based on neural point rendering as described in any one of claims 1 to 6.

Images