A 3D reconstruction method for traffic accident scenes based on NeRF

Abstract

The present invention discloses a method for 3D reconstruction of traffic accident scenes based on Neural Radiance Field (NeRF), which relates to the technical field of traffic accident investigation and scene reconstruction. The method utilizes NeRF technology to quickly and accurately reconstruct the 3D scene of the accident scene from 2D images and dashcam data of the accident scene, providing strong technical support for the investigation, analysis, and liability determination of traffic accidents. The method mainly includes the following steps: preparing the original dashcam data and collecting the corresponding camera parameters; then, building a new network structure based on the original NeRF structure and training it; finally, using the trained network for verification. If the verification is passed, the trained and tested network is used to restore the scene; if it fails, the method returns to the extraction step and repeats the cycle.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a NeRF-based three-dimensional reconstruction method for traffic accident scenes. Background Art

[0002] In traffic accident investigations, accident scene reconstruction plays a crucial role in analyzing the cause and determining liability. Traditional methods for reconstructing traffic accident scenes typically rely on on-site measurements and record-keeping by on-site investigators. This is not only labor-intensive and time-consuming, but also subject to multiple factors, including on-site conditions and human factors, resulting in inaccurate reconstruction results.

[0003] In recent years, with the rapid development of computer vision and deep learning technologies, image-based 3D reconstruction techniques have gradually demonstrated their unique advantages. In particular, the emergence of Neural Radiance Field (NeRF) technology has made it possible to recover high-quality 3D scenes from a small number of 2D images. By learning the Neural Radiance Field function for each point in the scene, NeRF technology can predict and generate 3D scene images from any perspective, thereby achieving high-precision 3D reconstruction. Therefore, this paper designs a NeRF-based 3D reconstruction method for traffic accident scenes. Summary of the Invention

[0004] The purpose of the present invention is to solve the problems in the prior art and to propose a three-dimensional reconstruction method for traffic accident scenes based on NeRF.

[0005] A NeRF-based 3D reconstruction method for traffic accident scenes includes the following steps:

[0006] S1: Data collection: 2-camera group scene, 3-camera group scene and 4-camera group scene are provided for different scenes during data collection;

[0007] S2: 3D reconstruction model and training, building a new network structure based on the original NeRF structure and training;

[0008] S3: 3D reconstruction quality verification, verifies the quality of the trained model, and uses the trained and tested model to restore the scene. If it fails, returns to step S1 and repeats the cycle.

[0009] In the above-mentioned NeRF-based three-dimensional reconstruction method for traffic accident scenes, data collection includes collecting driving recorder data at the traffic accident scene.

[0010] In the above-mentioned NeRF-based 3D reconstruction method for traffic accident scenes, data collection specifically includes the following steps:

[0011] S1: Set up a camera group around the accident scene. At the same time, if the accident scene provides dashcam data, collect and record the dashcam's internal parameter matrix;

[0012] S2: Use the built camera group to collect 2D image data of the traffic accident scene.

[0013] In the above-mentioned NeRF-based 3D reconstruction method for traffic accident scenes, the camera group adopts a transposition cross-collection method to reduce errors, wherein the transposition cross-collection method includes the following steps:

[0014] S1: Centered on the traffic accident scene, select four locations to place cameras. The locations are numbered A to D, and the cameras are numbered 1 to 4. Ensure that the lines connecting cameras 1 and 3, and 2 and 4, are perpendicular to each other. That is, the initial positions are: camera 1 corresponds to position A, and so on.

[0015] S2: After the cameras are positioned, image data is collected. Each camera collects five images of the same scene. The image naming format is "camera number - position number - image number". For example, "1-A-2" means the second image taken by camera No. 1 at position A.

[0016] S3: Perform sequence swapping. Based on the initial position, camera 1 is moved to position D, camera 2 is moved to position A, camera 3 is moved to position B, and finally camera 4 is moved to position C. Each camera collects 5 images again.

[0017] S4: Repeat S1 until the final position is that camera No. 1 corresponds to position B, camera No. 2 corresponds to position C, position No. 3 corresponds to position D, and camera No. 4 corresponds to position A. At this time, each camera has collected images of positions AD, and each position has 5 images, for a total of 20 images. Accordingly, when the camera group is 3 cameras, 15 images are finally collected for each position. Similarly, when the camera group is 2 cameras, 10 images are finally collected for each position.

[0018] In the NeRF-based 3D reconstruction method for traffic accident scenes, the 3D reconstruction model and training are based on and improved upon the original NeRF model. The Encoder portion of the Transformer architecture replaces the two fully connected layers in the original model. The training method for the 3D reconstruction model includes the following steps:

[0019] S1: Perform unsupervised partial weight training;

[0020] S2: Load the weights trained in the first step and then train the entire model.

[0021] In the above-mentioned NeRF-based 3D reconstruction method for traffic accident scenes, the 3D reconstruction quality verification includes two cases:

[0022] In case 1, if the collected data includes data from a driving recorder, the data from the driving recorder is used for training;

[0023] In the second case, if the collected data does not include the data from the driving recorder, the multi-view verification method is used.

[0024] In the above-mentioned NeRF-based 3D reconstruction method for traffic accident scenes, the steps of the multi-view verification method include:

[0025] S1: Select a target point and perform inference from the perspective of four cameras to obtain the color and transparency of the target point. At the same time, the color and transparency of the corresponding target point are calculated using the intrinsic parameter matrix of the four cameras.

[0026] S2: Calculate the average error. Assume that RGBα(i) is the result of four-camera inference (i represents the camera number), RGBα(Ti) is the color and transparency of the corresponding target point calculated by the camera intrinsic parameter matrix, A(RGBα(Ti)) is the color and transparency of the corresponding target point calculated by the average camera intrinsic parameter matrix, and E is the average error. The specific calculation method is as follows:

[0027]

[0028] S3: Set the error threshold to determine whether E can pass the detection standard and verify it through a large number of scenarios.

[0029] In the above-mentioned NeRF-based 3D reconstruction method for traffic accident scenes, the error threshold in the multi-view verification method is set to 6% by default.

[0030] Compared with the existing technology, the advantages of the present invention are:

[0031] 1. Applying NeRF technology to traffic accident scene reconstruction can not only greatly improve the accuracy and efficiency of reconstruction, but also reduce the interference of human factors, providing more accurate and reliable technical support for traffic accident investigations.

[0032] 2. The network structure proposed in this invention replaces the fully connected layer in the original NeRF, reduces the number of parameters, and reduces the training time of the model while ensuring accuracy as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is an overall flow chart of the NeRF-based 3D reconstruction method for traffic accident scenes proposed in this invention.

[0034] Figure 2This is a schematic diagram of the camera positions of the two cameras of the present invention.

[0035] Figure 3 This is a schematic diagram of the three-camera position of the present invention.

[0036] Figure 4 This is a schematic diagram of the positions of the four cameras of the present invention.

[0037] Figure 5 Schematic diagram of the 4-camera cross-collection method of the present invention

[0038] Figure 6 This is a model structure diagram of the present invention. DETAILED DESCRIPTION

[0039] Reference Figure 1-6 A NeRF-based 3D reconstruction method for traffic accident scenes includes the following steps performed in sequence:

[0040] Step 1: Collect on-site data. According to the situation at the traffic accident scene, select a matching camera group. At the same time, if there is a dashcam data stored at the scene, collect it as well. Figure 4 The four-camera group and Figure 5 The cross-collection method is described as follows:

[0041] The first step is to initialize the physical scene construction, select 4 appropriate locations, and place cameras around the accident scene, such as Figure 4 As shown in the case of , the locations are numbered AD, and the camera numbers are 1-4, which respectively collect the intrinsic parameter matrix of the camera and the intrinsic parameter matrix of the on-site driving recorder.

[0042] The second step is to combine Figure 5 After the initialization of the physical scene is completed, data collection begins. Figure 5 The rotation method shown in FIG1 collects scene data in sequence and finally obtains image data.

[0043] Step 2: Data information extraction. First, establish a world coordinate system and map the image data to points in the world coordinate system. The final image data is converted into a 5-dimensional vector (x, y, z, θ, φ), where (x, y, z) is the position of the sampling point and (θ, φ) is the observation angle corresponding to the sampling point. Assuming that the camera origin is O, the sampling distance is d, and the camera observation direction is τ, the calculation method of the sampling point can be expressed as:

[0044] (x, y, z) = O + dτ

[0045] Step 3: Build a network model. The present invention is based on the original NeRF structure and is optimized. The optimized structure is as follows: Figure 6As shown in the figure, the present invention introduces the Encoder part of the latest Transformer structure to replace the 2-layer fully connected layer, with the aim of reducing the training time of the model while ensuring accuracy as much as possible.

[0046] Step 4: training the network model, which is divided into two processes: unsupervised training and overall training.

[0047] The first step is to perform unsupervised learning and save only Figure 6 The weights of the network layers before the 4th fully connected layer are used. The input data is the masked image, and the label value is the original image data, thereby obtaining more advanced features.

[0048] The second step is to load the weights trained in the first step, and then use the data processed in step 2 to train the entire model.

[0049] Step 5: Verify the trained model and set the error rate. If it passes the verification, proceed to subsequent deployment. If it fails the verification, it needs to return to the network training stage for secondary training.

[0050] The first step is to load the model trained in step 4

[0051] In the second step, the point to be verified is selected and the color and transparency of the verification point are calculated using the camera intrinsic parameter matrix.

[0052] The third step is to calculate the error and compare it with the set threshold. The default threshold is 6%. If the error is higher than the set value, the model needs to be retrained. If it passes the verification, proceed to the next step. If RGBα(i) is the result of four camera inference (i represents the camera number), RGBα(Ti) is the color and transparency of the corresponding target point calculated by the camera intrinsic parameter matrix, A(RGBα(Ti)) is the color and transparency of the corresponding target point calculated by the average camera intrinsic parameter matrix, and E is the average error. The specific calculation method is as follows:

[0053]

[0054] In step 6, if the model passes the verification, the model is converted into an ONNX file, and targeted optimization is performed using Nvidia hardware. Finally, TensorRT is used to deploy the model.

[0055] It is understood from common technical knowledge that the present invention may be implemented by other embodiments that do not depart from its spirit or essential features. Therefore, the embodiments disclosed above are, in all respects, merely illustrative and not exclusive. All modifications within the scope of the present invention or equivalent to the scope of the present invention are intended to be encompassed by the present invention.

Claims

1. A NeRF-based 3D reconstruction method for traffic accident scenes, characterized in that: The following steps are involved: Step 1: Data collection; 2-camera group scene, 3-camera group scene and 4-camera group scene are provided for different scenes; Data collection involves collecting dashcam data from traffic accident scenes, and specifically includes the following steps: S11: Set up a camera group around the accident scene. At the same time, if the accident scene provides dashcam data, collect and record the dashcam's internal parameter matrix; S12: Using the built camera group, collect 2D image data of the traffic accident scene; The camera group adopts a transposition cross-collection method to reduce errors, wherein the transposition cross-collection method includes the following steps: S21: With the traffic accident scene as the center, select four locations to place cameras. The locations are numbered A to D, and the cameras are numbered 1 to 4. Ensure that the lines connecting cameras 1 and 3, and 2 and 4 are perpendicular to each other. That is, the initial positions are: camera 1 corresponds to position A, and so on. S22: After the cameras are positioned, image data is collected. Each camera collects 5 images of the same scene. The image naming format is: camera number - position number - image number. S23: Perform sequence swapping. Based on the initialization position, camera No. 1 is swapped to position D, camera No. 2 is swapped to position A, camera No. 3 is swapped to position B, and finally camera No. 4 is swapped to position C. Each camera collects 5 images again. S24: Repeat S21 until the final position is that camera No. 1 corresponds to position B, camera No. 2 corresponds to position C, position No. 3 corresponds to position D, and camera No. 4 corresponds to position A. At this time, each camera has collected images of positions AD, and each position has 5 images, for a total of 20 images. Accordingly, when the camera group consists of 3 cameras, 15 images are finally collected for each position. Similarly, when the camera group consists of 2 cameras, 10 images are finally collected for each position. Step 2: Data information extraction: First, establish a world coordinate system and map the image data to points in the world coordinate system. The final image data is converted into a 5-dimensional vector (x, y, z, θ, φ), where (x, y, z) is the position of the sampling point and (θ, φ) is the observation angle corresponding to the sampling point. Step 3: Build a new network structure based on the original NeRF structure and perform network model training; The 3D reconstruction model and training are based on and improved upon the original NeRF model. The Encoder portion of the Transformer structure replaces the two fully connected layers in the original model structure. The training method for the 3D reconstruction model includes the following steps: S31: Perform unsupervised partial weight training, and only save the weights of the network layers before the fourth fully connected layer in the 3D reconstruction model. The input data is the masked image, and the label value is the original image data, thereby obtaining more advanced features. S32: Load the weights trained in step S31, and then use the data processed in step 2 to train the entire model; Step 4: 3D reconstruction quality verification. Verify the quality of the trained model. If the verification fails, return to step 3 to train the model, and repeat the cycle. If the verification passes, use the trained and tested model to restore the scene.

2. The NeRF-based 3D reconstruction method for traffic accident scenes according to claim 1, characterized in that: There are two cases in 3D reconstruction quality verification: In case 1, if the collected data includes data from a driving recorder, the data from the driving recorder is used for training; In the second case, if the collected data does not include the data from the driving recorder, the multi-view verification method is used.

3. The NeRF-based 3D reconstruction method for traffic accident scenes according to claim 2, characterized in that: The steps of the multi-view verification method include: S1: Select a target point and perform inference from the perspective of four cameras to obtain the color and transparency of the target point. At the same time, the color and transparency of the corresponding target point are calculated using the intrinsic parameter matrix of the four cameras. S2: Calculate the average error. Assume that RGBα(i) is the result of four-camera inference, i represents the camera number, RGBα(Ti) is the color and transparency of the corresponding target point calculated by the camera intrinsic parameter matrix, A(RGBα(Ti)) is the color and transparency of the corresponding target point calculated by the average camera intrinsic parameter matrix, and E is the average error. The specific calculation method is as follows: S3: Set the error threshold to determine whether E can pass the detection standard and verify it through a large number of scenarios.

4. The NeRF-based 3D reconstruction method for traffic accident scenes according to claim 3, characterized in that: The error threshold in the multi-view verification method is set to 6% by default.

Images