A method for three-dimensional reconstruction of traffic accident scene combining air-ground visual angle

Abstract

The application provides a traffic accident scene three-dimensional reconstruction method combining air-ground visual angles, and relates to the technical field of three-dimensional reconstruction.The method comprises the following steps: collecting traffic accident scene images by using cameras of two platforms of a UAV and an unmanned vehicle, performing semantic segmentation on the images, and generating a pixel-level mask of a key target; extracting an ROI from the images according to the pixel-level mask of the key target, performing image enhancement and denoising processing on the ROI, and obtaining a processed ROI image; optimizing the internal and external parameters of the cameras of the two platforms of the UAV and the unmanned vehicle, and restoring absolute scale, based on the pixel-level mask of the key target and the processed ROI image, reconstructing a three-dimensional surface model by combining an SfM method of a joint camera optimization target function, an MVS method, and a surface reconstruction method, and performing digital quantitative analysis on the accident scene.The application can solve the problems of insufficient camera calibration optimization and lack of absolute scale information in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional reconstruction, in particular to a traffic accident scene three-dimensional reconstruction method combining ground and aerial views. BACKGROUND

[0002] Visual three-dimensional reconstruction technology aims to automatically reconstruct the three-dimensional geometry and texture information of a scene by analyzing multi-view images. In the field of traffic accident handling, this technology is of great significance for the fixation of the accident scene, the determination of responsibility, the restoration of traffic, and the post-analysis.

[0003] Traditional accident scene investigation mainly relies on manual measurement and photography, which has the problems of low efficiency, poor precision, high risk (especially when there is dangerous goods leakage), and incomplete information (there are a lot of occlusions and blind areas). In recent years, single-platform UAV aerial reconstruction has been introduced, which can provide a macro overview of the scene, but it is powerless for key information such as vehicle bottom, side collision details, and occluded evidence. Similarly, using only ground equipment (such as handheld cameras or unmanned vehicles) can capture local details, but it lacks an overall understanding of the accident scene and is difficult to associate with global coordinates. Whether from the air or the ground, there are inherent information blind spots. UAVs cannot effectively capture the bottom and vertical sides of vehicles, while ground views cannot obtain the overall distribution of roof deformations and scattered objects. Traditional methods cannot automatically identify and separate accident vehicles from the background environment, resulting in reconstructed models containing a large amount of irrelevant information, affecting the accuracy of subsequent analysis. When fusing data from multiple platforms, there is a lack of accurate joint camera calibration optimization algorithms, resulting in inconsistent scales and registration errors in the reconstructed model. The reconstructed model often lacks absolute scale references in the real world, making it impossible to perform accurate measurement analysis. Accident scene investigation must be completed in the shortest time to restore traffic as soon as possible and ensure the safety of investigators. The existing process often cannot balance efficiency, comprehensiveness, and safety. SUMMARY

[0004] The purpose of the present application is to solve the problems of insufficient camera calibration optimization and lack of absolute scale information in existing three-dimensional reconstruction, and to propose a traffic accident scene three-dimensional reconstruction method combining ground and aerial views, comprising the following steps:

[0005] S1, using the cameras of the two platforms of the UAV and the unmanned vehicle to collect traffic accident scene images, performing semantic segmentation on the images, and generating pixel-level masks of key targets;

[0006] S2, extracting ROIs (Region of Interest) from the images according to the pixel-level masks of the key targets, performing image enhancement and denoising processing on the ROIs, and obtaining processed ROI images;

[0007] S3, optimize the camera internal and external parameters of the two platforms of UAV and unmanned vehicle, and restore the absolute scale, based on the pixel-level mask of key targets and the processed ROI image, reconstruct the three-dimensional surface model by combining the SfM (Structure from Motion) method, MVS (Multi-View Stereo) method and surface reconstruction method of the joint camera optimization target function;

[0008] S4, carry out digital quantitative analysis of the accident scene through the three-dimensional surface model.

[0009] Further, the image is subjected to semantic segmentation by using a segmentation network of a SAM architecture.

[0010] Further, the UAV camera internal parameter matrix is , the unmanned vehicle camera internal parameter matrix is , and the expression is:

[0011]

[0012]

[0013] wherein, represents the value of the equivalent focal length of the UAV camera in the x-axis under the pixel coordinates, represents the value of the equivalent focal length of the UAV camera in the y-axis under the pixel coordinates, represents the value of the imaging center of the UAV camera in the x-axis, represents the value of the imaging center of the UAV camera in the y-axis, represents the value of the equivalent focal length of the unmanned vehicle camera in the x-axis under the pixel coordinates, represents the value of the equivalent focal length of the unmanned vehicle camera in the y-axis under the pixel coordinates, represents the value of the imaging center of the unmanned vehicle camera in the x-axis, represents the value of the imaging center of the unmanned vehicle camera in the y-axis.

[0014] Further, the calculation formula of the joint camera optimization target function is as follows:

[0015]

[0016] wherein, represents the optimized parameter, M represents the total number of three-dimensional points, is the UAV camera internal parameter matrix, is the unmanned vehicle camera internal parameter matrix, represents the UAV camera rotation matrix corresponding to the i-th frame of UAV camera image, represents the UAV camera translation vector corresponding to the i-th frame of UAV camera image, RUGV k represents the UGV camera rotation matrix corresponding to the k-th frame of UGV camera image, tUGV k represents the UGV camera translation vector corresponding to the k-th frame of UGV camera image, NUGV represents the number of UGV images / poses participating in optimization, NUGV i represents the number of valid observations in the i-th frame of UGV image, zUGV i j represents the pixel observation of the j-th point in the i-th frame of UGV image, XW j represents the position of the j-th three-dimensional point in the world coordinate system, the three-dimensional point coordinates are obtained through the co-visibility pixel points of multiple two-dimensional images, NUAV represents the number of UAV images / poses participating in optimization, NUAV i represents the number of valid observations in the i-th frame of UAV image, zUAV i j represents the pixel observation of the j-th point in the i-th frame of UAV image, XW j represents the position of the j-th three-dimensional point in the world coordinate system, , , , , respectively represent the weights of the same point constraint, the scale constraint and the geometric consistency constraint, Nc represents the number of co-visibility same points of the two platforms, zUAV c represents the pixel observation of the c-th same point in the UAV image, zUGV c represents the pixel observation of the c-th same point in the UGV image, XW a represents the position of the a-th three-dimensional point in the distance prior, XW b represents the position of the b-th three-dimensional point in the distance prior, D represents the known physical distance between the point and , I represents the index set of the point pairs with known physical distance, R represents the relative rotation between the two cameras, t represents the relative translation between the two cameras, π represents the camera projection function.

[0017] Further, the absolute scale recovery method is as follows:

[0018] distance measurement is performed by using the depth camera carried by the unmanned vehicle; a calibration board with a known size is set within the visual range of the unmanned aerial vehicle; a scale proportion relationship is established:

[0019]

[0020] wherein, denotes a scale ratio for scaling the reconstructed model from reconstruction units to real metric, denotes a real distance measured by the depth camera, denotes a relative distance in the reconstructed model, denotes a three-dimensional coordinate of the first point in the real space, denotes a three-dimensional coordinate of the second point in the real space, denotes a three-dimensional point coordinate corresponding to the first point in the reconstructed model, denotes a three-dimensional point coordinate corresponding to the second point in the reconstructed model. denotes a three-dimensional point coordinate corresponding to the first point in the reconstructed model. denotes a three-dimensional point coordinate corresponding to the second point in the reconstructed model.

[0021] The application also provides a traffic accident scene three-dimensional reconstruction system combining ground and air perspectives, comprising:

[0022] a semantic segmentation module, configured to collect traffic accident scene images using cameras of two platforms of a UAV and an unmanned vehicle, perform semantic segmentation on the images, and generate pixel-level masks of key targets;

[0023] an ROI image acquisition module, configured to extract ROIs from the images according to the pixel-level masks of the key targets, perform image enhancement and denoising processing on the ROIs, and obtain processed ROI images;

[0024] a three-dimensional reconstruction module, configured to optimize internal and external parameters of the cameras of the two platforms of the UAV and the unmanned vehicle, and perform absolute scale recovery, and based on the pixel-level masks of the key targets and the processed ROI images, reconstruct a three-dimensional surface model by combining an SfM method, an MVS method, and a surface reconstruction method of a joint camera optimization target function;

[0025] an accident scene analysis module, configured to perform digital quantitative analysis of the accident scene based on the three-dimensional surface model.

[0026] The application also provides a computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the traffic accident scene three-dimensional reconstruction method combining ground and air perspectives.

[0027] The application also provides an electronic device comprising a processor and a memory, wherein the processor and the memory are connected to each other, the memory is configured to store a computer program, the computer program comprises computer readable instructions, and the processor is configured to invoke the computer readable instructions to implement the traffic accident scene three-dimensional reconstruction method combining ground and air perspectives.

[0028] The application also provides a computer program product comprising computer programs / instructions, wherein the computer programs / instructions are executed by a processor to implement the steps of the traffic accident scene three-dimensional reconstruction method combining ground and air perspectives.

[0029] The technical scheme provided by the present application has the beneficial effects that:

[0030] The present application combines the macroscopic perspective of the unmanned aerial vehicle and the microscopic perspective of the unmanned vehicle, effectively eliminates the observation blind area of a single perspective, realizes the all-around coverage of the accident scene, reconstructs the three-dimensional surface model by combining the SfM method, the MVS method and the surface reconstruction method of the joint camera optimization target function, realizes the high-precision fusion of the multi-perspective data of the unmanned aerial vehicle and the unmanned vehicle, and ensures the consistency of the internal and external parameters of the reconstructed model. The dual constraint mechanism of the depth camera and the high-precision calibration plate provides absolute scale guarantee. Through the depth information assisted scale calibration and the combination of the calibration plate geometric correction, it is ensured that the three-dimensional reconstruction result has the absolute scale of the real world. The present application can solve the problems of insufficient camera calibration optimization and lack of absolute scale information in the existing three-dimensional reconstruction method. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is a flow chart of a traffic accident scene three-dimensional reconstruction method combining ground and air perspectives of an embodiment of the present application;

[0032] Figure 2 is a schematic diagram of a surrounding aerial photography planning of an unmanned aerial vehicle and an unmanned vehicle combined by an embodiment of the present application;

[0033] Figure 3 is a schematic diagram of the vertical aerial photography height of an unmanned aerial vehicle and the horizontal photography planning of an unmanned vehicle of an embodiment of the present application;

[0034] Figure 4 is a schematic diagram of the aerial photography angle of an unmanned aerial vehicle and the photography distance planning of an unmanned vehicle of an embodiment of the present application;

[0035] Figure 5 is a schematic diagram of a sparse reconstruction result of the head of an oil tank truck of an embodiment of the present application;

[0036] Figure 6 is a schematic diagram of a high-precision dense three-dimensional model of the head of an oil tank truck of an embodiment of the present application;

[0037] Figure 7 is a quantitative analysis result of the head of an oil tank truck of an embodiment of the present application;

[0038] Figure 8 is a block diagram of an electronic device in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described below with reference to the drawings.

[0040] The flow chart of the traffic accident scene three-dimensional reconstruction method combining ground and air perspectives of an embodiment of the present application is as follows:Figure 1 , specifically comprising the following steps:

[0041] S1, multi-platform collaborative deployment is carried out at the traffic accident scene, including deploying unmanned aerial vehicles (UAVs), unmanned ground vehicles (UGVs), and placing calibration objects for scale recovery. Specifically, the unmanned aerial vehicle usually performs a fly-around flight to obtain global overhead views and multi-angle oblique images of the accident scene from the air, which are used to grasp the macro layout and overall situation of the scene. The unmanned ground vehicle then approaches the accident vehicle and key evidence on the ground, focusing on shooting areas such as the vehicle bottom, side collision details, and tire marks that cannot be effectively covered by the unmanned aerial vehicle perspective, to make up for the blind area of the macro perspective. The absolute scale calibration board placed on the scene has a pre-known physical size.

[0042] Reference Figure 2 , Figure 3 and Figure 4 , Figure 2 is a schematic diagram of the fly-around flight planning of the unmanned aerial vehicle and unmanned ground vehicle in the embodiment of the present application; Figure 3 is a schematic diagram of the vertical flight height of the unmanned aerial vehicle and the horizontal shooting planning of the unmanned ground vehicle in the embodiment of the present application; Figure 4 is a schematic diagram of the flight angle of the unmanned aerial vehicle and the shooting distance of the unmanned ground vehicle in the embodiment of the present application. (1) Unmanned aerial vehicle (UAV) aerial data acquisition setting: First, the UAV acquisition path is divided into a regular fly-around path and a layered flight path. In the regular fly-around path, as shown in Figure 2 , the unmanned aerial vehicle flies around the accident vehicle. Preferably, the flight radius is about 10-12 m, and the included angle of the flight points in one circle can be set to about 15°, 22.5°, 45°. The unmanned aerial vehicle can hover at each flight point for 2-5 seconds, and the RGB camera carried by the unmanned aerial vehicle can shoot the accident vehicle at different pitch angles. Preferably, one or more pitch angle shootings can be performed at one or more flight points, such as about 45°, 50°, 55°, 60°, to obtain vehicle facade information at different angles. In particular, for the key areas of vehicle collision damage, encrypted collection can be performed, i.e., increasing the flight point density or the number of pitch angles. In addition, as shown in Figure 8 , the unmanned aerial vehicle performs layered flight with the entire tanker traffic accident scene as the object. Preferably, it can be divided into two height levels, an "air layer" with a height of about 5-8 m and a "ground layer" with a height of about 0-5 m. The shooting data of the "air layer" is mainly used to obtain the macro layout and overall situation of the scene; the shooting data of the "ground layer" is used to capture the details of the upper half of the vehicle and the surrounding scattered objects more clearly. (2) Unmanned ground vehicle (UGV) ground data acquisition setting: At the same time or after the UAV performs aerial data acquisition, as shown in Figure 2 , the unmanned ground vehicle carrying an RGB camera and a depth camera is deployed to start shooting at a close-in radius of about 3-4 m. As shown in Figure 3The unmanned vehicle is responsible for collecting the areas where the unmanned aerial vehicle perspective exists occlusion and blind area, such as the side, bottom and low-angle collision details of the vehicle. (3) Dense collection of key areas: In order to obtain the most detailed data of the key collision damaged parts of large vehicles such as "two passengers and one dangerous", a dense collection strategy of air-ground joint is adopted, such as Figure 4 At this time, the unmanned aerial vehicle can take key supplementary pictures of the damaged area at different altitudes and with different camera pitch angles (difference of about 5-10°). At the same time, the unmanned vehicle can take detailed pictures by further moving forward 1m and using its closer distance to the damaged location. Through this air-ground cooperation, combination of near and far, and multi-angle complementation, the key evidence information is collected without dead angle.

[0043] The unmanned aerial vehicle and unmanned vehicle of the multi-platform cooperative deployment perform data collection tasks on the scene, and transmit the acquired image data to the on-site computing unit in real time. The computing unit performs semantic segmentation on the image through a deep learning model to generate a pixel-level mask of the key target. Specifically, in the process of generating a key target mask by deep learning semantic segmentation, the system inputs the collected RGB image into a pre-trained segmentation network, such as a segmentation network based on the Segment Anything Model (SAM) architecture. The segmentation network automatically identifies and accurately segments the accident vehicle, road markings, and key scattered objects that are crucial to accident analysis, and generates an independent pixel-level binary mask for each target.

[0044] S2, according to the pixel-level mask of the key target, the region of interest (ROI) is extracted from the image, the ROI is subjected to image enhancement and denoising processing, the target-image correspondence relationship is constructed, and the processed ROI image is obtained. The significance of this is that it can focus the reconstruction on the effective target, greatly reducing the interference of irrelevant information such as the background environment, providing high-quality ROI for subsequent feature point extraction and matching, and significantly improving the accuracy and efficiency of reconstruction.

[0045] S3, the camera internal and external parameters of the unmanned aerial vehicle and unmanned vehicle platforms are optimized, and the absolute scale is restored, based on the pixel-level mask of the key target and the processed ROI image, the three-dimensional surface model is reconstructed by combining the SfM method, MVS method and surface reconstruction method of the joint camera optimization target function.

[0046] The specific steps are as follows:

[0047] The camera internal parameter matrix of the unmanned aerial vehicle is The camera internal parameter matrix of the unmanned vehicle is The expression is:

[0048]

[0049]

[0050] wherein, represents the value of the equivalent focal length of the UAV camera in the pixel coordinate in the x-axis, represents the value of the equivalent focal length of the UAV camera in the pixel coordinate in the y-axis, represents the value of the imaging center of the UAV camera in the x-axis, represents the value of the imaging center of the UAV camera in the y-axis, represents the value of the equivalent focal length of the UGV camera in the pixel coordinate in the x-axis, represents the value of the equivalent focal length of the UGV camera in the pixel coordinate in the y-axis, represents the value of the imaging center of the UGV camera in the x-axis, represents the value of the imaging center of the UGV camera in the y-axis.

[0051] Since the cameras of the two platforms of the UAV and the UGV both adopt square pixels and non-orthogonal readout, the tilt coefficient is set to 0. If there is a single camera calibration, the result can be directly used; otherwise, a heuristic initial value can be given by using a checkerboard method or using the image resolution center and the number of diagonal pixels to ensure the convergence of subsequent optimization.

[0052] In the preferred embodiment of the application, a joint camera optimization objective function is constructed to optimize the camera intrinsic parameters of the two platforms of the UAV and the UGV and , the extrinsic parameters , and the three-dimensional point coordinates, and unify the camera intrinsic and extrinsic parameters of the two platforms of the UAV and the UGV. The following total residual least squares problem is defined, wherein the constraint terms include the same point constraint, the scale constraint and the geometric consistency constraint, and the calculation formula is as follows:

[0053]

[0054] wherein, represents the optimized parameters, and M represents the total number of three-dimensional points, is the UAV camera intrinsic parameter matrix, is the UGV camera intrinsic parameter matrix, represents the UAV camera rotation matrix corresponding to the i-th frame of the UAV camera image, and the camera rotation matrix indicates that the image is rotated by different angles around different coordinate axes from the world coordinate system to the camera coordinate system, represents the UAV camera translation vector corresponding to the i-th frame of the UAV camera image, and the camera translation vector represents the translation transformation of the image from the world coordinate system to the camera coordinate system, represents the UGV camera rotation matrix corresponding to the k-th frame of the UGV camera image, denotes the UGV camera translation vector corresponding to the k-th frame of UGV camera image, denotes the number of optimized UAV images / poses, denotes the number of valid observations in the i-th frame of UAV image, denotes the pixel observation of the j-th point in the i-th frame of UAV image, denotes the position of the j-th 3D point in the world coordinate system, which is obtained by the co-visible pixel points of multiple 2D images, denotes the number of optimized UGV images / poses, denotes the number of valid observations in the k-th frame of UGV image, denotes the pixel observation of the j-th point in the k-th frame of UGV image, denotes the position of the j-th 3D point in the world coordinate system, , , , , denote the weights of the same point constraint, scale constraint and geometric consistency constraint respectively, denotes the number of co-visible same points of two platforms, denotes the pixel observation of the c-th same point in the UAV image, denotes the pixel observation of the c-th same point in the UGV image, denotes the position of the a-th 3D point in the distance prior, denotes the position of the b-th 3D point in the distance prior, denotes the known physical distance between the point and , denotes the index set of the point pairs with known physical distance, denotes the relative rotation matrix (UGV→UAV) between two cameras, denotes the relative translation (UGV→UAV) between two cameras, denotes the set of UAV / UGV pose pairs used for geometric consistency constraint, and π denotes the camera projection function .

[0055] The least squares problem can be solved iteratively by the sparse Levenberg–Marquardt (LM) method of Ceres Solver. The solving strategy is: first, initialize all K, R, T with single-camera calibration (or heuristic) results. Then use sparse LM to solve The iteration minimization is performed. The damping factor is automatically adjusted in the subsequent iteration until the pixel re-projection error and the constraint residual are simultaneously converged. By adding the three types of priori terms of "corresponding points, scale, geometry" to the camera re-projection error, the joint optimization can simultaneously complete the cross-platform calibration, scale alignment and pose refinement, and lay a geometrically consistent foundation for subsequent multi-source fusion. The image feature points, camera internal and external parameters and scale constraint information from the unmanned aerial vehicle and the unmanned vehicle are bundled and adjusted (Bundle Adjustment, BA), the re-projection errors of all views are minimized, and the perfect alignment of the two heterogeneous platforms in space and scale is realized.

[0056] Then, a sparse three-dimensional point cloud with absolute scale information is generated through a structure from motion (SfM) process based on a joint camera optimization objective function. The absolute scale recovery method is as follows:

[0057] The distance measurement is performed by using the depth camera carried by the unmanned vehicle; the calibration board with a known size is set within the visual range of the unmanned aerial vehicle; and the scale proportion relationship is established.

[0058]

[0059] wherein, represents the scale proportion, which is used to scale the reconstruction model from the reconstruction unit to the real meter, represents the real distance measured by the depth camera, represents the relative distance in the reconstruction model, represents the three-dimensional coordinates of the first point in the actual space, represents the three-dimensional coordinates of the second point in the actual space, represents the three-dimensional point coordinates in the reconstruction model corresponding to the first point, represents the three-dimensional point coordinates in the reconstruction model corresponding to the second point. represents the three-dimensional point coordinates in the reconstruction model corresponding to the second point.

[0060] Finally, the sparse point cloud is densified to generate a high-precision three-dimensional surface model through a universal stereo reconstruction process. The stereo reconstruction process is as follows: first, the feature point matching is performed on the entire image library (including the unmanned aerial vehicle and unmanned vehicle images). The feature matching supports various feature description operators, including but not limited to SIFT, SURF, ORB, SuperPoint and other mature schemes. The false matching can be filtered through geometric constraints, and mature schemes including but not limited to RANSAC and its variants can be used, and the key is to successfully match the corresponding points between the unmanned aerial vehicle image and the unmanned vehicle image, and effective feature screening is performed based on the target mask. The incremental sparse reconstruction is designed as a universal SfM architecture, and mature schemes including but not limited to OpenMVG, COLMAP, VisualSfM can be used, and the unified input and output interfaces are provided.​

[0061] The subsequent stereoscopic reconstruction process is designed as a general MVS reconstruction architecture. After obtaining the sparse three-dimensional structure, mature schemes such as PMVS, OpenMVS, MVE, etc. can be used to generate a dense three-dimensional structure. A complete quality control and optimization mechanism is established. This mechanism includes dense reconstruction quality evaluation based on target masks, multi-scale geometric consistency verification, and adaptive parameter adjustment strategies. It supports multiple surface reconstruction methods, including but not limited to Poisson, Delaunay, etc. It supports multiple texture mapping techniques, including but not limited to multi-view texture fusion, seam elimination optimization, and texture atlas generation, etc. mature schemes. This design ensures that even when dealing with complex scenes and heterogeneous data, through strict geometric filtering and surface optimization, a complete and accurate three-dimensional model with high-fidelity texture can be generated.

[0062] S4, digital quantitative analysis of the accident scene by three-dimensional surface model. For example, non-contact accurate measurement of vehicle damage length, distribution range of scattered objects, collision analysis, etc. Finally, according to the analysis results, a digital investigation report that can be used for accident liability identification and post-analysis is output. Specifically, the final model generated is a 1:1, measurable digital accident scene reconstruction of the real world. Rescue personnel or accident analysis experts can observe and measure the model from any angle in a safe office environment through the back-end analysis module, including but not limited to MeshLab, CloudCompare, etc. For example, the degree of deformation of the vehicle, the size of the collision contact surface, the position of the scattered objects relative to the vehicle, etc. can be accurately measured, which is difficult to quickly and accurately obtain through traditional investigation methods.

[0063] More specifically, taking a traffic accident caused by a tanker rear-end collision at a highway intersection as an example, the accident caused serious damage to the front of the tanker, and the scene was deployed with unmanned aerial vehicles, unmanned vehicles, and calibration objects. The accident scene and vehicle have been preliminarily identified, and high-precision three-dimensional reconstruction of the key parts of the vehicle head is required. The three-dimensional reconstruction steps of the traffic accident scene by the joint unmanned aerial vehicle and unmanned vehicle air-ground perspective of the invention for this accident scene are as follows:

[0064] (1) Data Acquisition: First, a quadcopter UAV was controlled to fly around the damaged front of the tanker truck. The flight radius was set to approximately 10m, flying at two altitude levels: 5m and 8m. 16 main waypoints were set around the front of the truck at each level, with the angle between waypoints ranging from 22.5° to 45°. At each waypoint, the onboard camera took photos at an angle of approximately 45°, for a total of 32 photos. 31 more photos were taken of the area with the most severe frontal collision. A total of 63 aerial images were acquired during this process. At the same time, a Mecanum wheel UAV was deployed, circling around the side of the tanker truck's front at a radius of approximately 4m. Every 20°, its onboard RGB camera and depth camera continuously photographed the damaged front, tires, and chassis details, acquiring a total of 8 close-up images of the UAV.

[0065] (2) Reconstruction and Result Display: A total of 71 images (63 from UAVs and 8 from unmanned vehicles) were preprocessed and input into the reconstruction system of this invention. After joint calibration and optimization processes and sparse reconstruction processes, the following results can be obtained: Figure 5 The image shows the sparse reconstruction result of the tanker truck's front end. This result, presented in Colmap, accurately displays the camera poses of the 71 photos used in the reconstruction (clearly showing the aerial, horizontally layered UAV camera poses and the ground-based, linearly surrounding UAV camera poses) and the sparse 3D point cloud structure of the tanker truck's front end and surrounding environment, composed of matched feature points. This image visually verifies the effectiveness and high accuracy of the air-to-ground data fusion method of this invention. Subsequently, after a denser reconstruction process, a high-precision 3D model of the tanker truck's front end and surrounding environment is finally generated, which can be a standard file format such as PLY. (Reference) Figure 6 A high-precision, dense 3D model of the front of an oil tanker truck.

[0066] (3) Quantitative analysis presentation: such as Figure 7 As shown, Figure 7 This is a quantitative analysis of the tanker truck's cab. Using Meshlab's "Pick Point Measure" function, extreme points on both sides of the cab can be directly selected on a 3D point cloud to obtain the spatial distance of the cab's width. Through three independent measurements of the same large tanker truck, the average width read from each measurement was approximately 2.94m, while the actual measured value was 3.00m, with an absolute error controlled within approximately 6cm. This result verifies the model's usability in traffic accident reconstruction—achieving absolute dimensional accuracy within 10cm and relative error accuracy within 5% for large passenger and hazardous materials vehicles without the need for additional scales.

[0067] In one exemplary embodiment, a three-dimensional reconstruction system for a traffic accident scene from a combined ground-to-air perspective is included, comprising:

[0068] The semantic segmentation module is used to collect images of traffic accident scenes using cameras on both drone and unmanned vehicle platforms, perform semantic segmentation on the images, and generate pixel-level masks of key targets.

[0069] The ROI image acquisition module is used to extract the ROI from the image based on the pixel-level mask of the key target, perform image enhancement and noise reduction on the ROI, and obtain the processed ROI image.

[0070] The 3D reconstruction module is used to optimize the camera intrinsic and extrinsic parameters of both UAV and unmanned vehicle platforms and perform absolute scale recovery. Based on the pixel-level mask of key targets and the processed ROI image, it reconstructs the 3D surface model by combining the SfM method, MVS method and surface reconstruction method with joint camera optimization objective function.

[0071] The accident scene analysis module is used for digital and quantitative analysis of accident scenes through three-dimensional surface models.

[0072] In one exemplary embodiment, a computer-readable storage medium is included, which stores a computer program that, when executed by a processor, implements the above-described method for three-dimensional reconstruction of traffic accident scenes from a combined ground-to-air perspective.

[0073] Please see Figure 8 In one exemplary embodiment, the device further includes an electronic device including at least one processor, at least one memory, and at least one communication bus.

[0074] The memory stores a computer program, which includes computer-readable instructions. The processor calls the computer-readable instructions stored in the memory through the communication bus to execute the aforementioned three-dimensional reconstruction method of traffic accident scene from a combined ground and air perspective.

[0075] In one exemplary embodiment, a computer program product is proposed, including a computer program / instruction that, when executed by a processor, implements the steps of the above-described method for three-dimensional reconstruction of traffic accident scenes from a combined ground-air perspective.

[0076] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for three-dimensional reconstruction of traffic accident scenes from a combined ground-air perspective, characterized in that, Includes the following steps: S1. Use cameras from both drones and unmanned vehicles to collect images of traffic accident scenes, perform semantic segmentation on the images, and generate pixel-level masks of key targets. S2. Extract the ROI from the image based on the pixel-level mask of the key target, perform image enhancement and noise reduction on the ROI, and obtain the processed ROI image. S3. Optimize the camera intrinsic and extrinsic parameters of the two platforms, UAV and unmanned vehicle, and perform absolute scale recovery. Based on the pixel-level mask of the key target and the processed ROI image, reconstruct the 3D surface model by combining the SfM method, MVS method and surface reconstruction method of joint camera optimization objective function. S4. Conduct digital and quantitative analysis of the accident scene using a three-dimensional surface model; The formula for calculating the objective function of the joint camera optimization is as follows: in, The parameters to be optimized are represented by M, where M represents the total number of 3D points. This is the intrinsic parameter matrix of the drone camera. This is the intrinsic parameter matrix of the autonomous vehicle camera. This represents the UAV camera rotation matrix corresponding to the i-th frame of the UAV camera image. This represents the UAV camera translation vector corresponding to the i-th frame of the UAV camera image. This represents the UGV camera rotation matrix corresponding to the k-th frame UGV camera image. This represents the UGV camera translation vector corresponding to the k-th frame UGV camera image. This indicates the number of UAV images / poses involved in the optimization. This represents the number of valid observation points in the i-th frame of the UAV image. This represents the pixel observation at the j-th point in the i-th frame of the UAV image. This represents the position of the j-th 3D point in the world coordinate system. The coordinates of the 3D point are obtained by using the common view pixels of multiple 2D images. This indicates the number of UGV images / poses involved in the optimization. This represents the number of valid observation points in the k-th frame of the UGV image. Indicates the k-th frame of the UGV image. Pixel observations at each point Indicates the first The position of a 3D point in the world coordinate system , , These represent the weights of the corresponding point constraint, the scale constraint, and the geometric consistency constraint, respectively. This indicates the number of points with the same name that are viewed by both platforms. This represents the pixel observation of the c-th corresponding point in the UAV image. This represents the pixel observation of the c-th corresponding point in the UGV image. This represents the position of the a-th 3D point in the prior knowledge. This represents the position of the b-th 3D point in the prior knowledge. Point and The known physical distance This represents a set of indexes of point pairs with known physical distances. This indicates the relative rotation between the two cameras. This indicates the relative translation between the two cameras. represents the set of UAV / UGV pose pairs used for geometric consistency constraints, and π represents the camera projection function.

2. The method for three-dimensional reconstruction of a traffic accident scene from a combined ground-air perspective according to claim 1, characterized in that, A segmentation network with a SAM architecture is used to perform semantic segmentation on images.

3. The method for three-dimensional reconstruction of a traffic accident scene from a combined ground-air perspective according to claim 1, characterized in that, The drone camera intrinsic parameter matrix is The intrinsic parameter matrix of the autonomous vehicle camera is The expression is: in, This represents the equivalent focal length of the drone camera on the x-axis in pixel coordinates. This represents the equivalent focal length of the drone camera on the y-axis in pixel coordinates. This indicates the x-axis value representing the image center of the drone camera. This indicates the value on the y-axis representing the image center of the drone camera. This represents the equivalent focal length of the autonomous vehicle's camera on the x-axis in pixel coordinates. This represents the y-axis value of the equivalent focal length of the autonomous vehicle's camera in pixel coordinates. This represents the x-axis value indicating the imaging center of the autonomous vehicle's camera. This indicates the value on the y-axis representing the imaging center of the autonomous vehicle's camera.

4. The method for three-dimensional reconstruction of traffic accident scenes from a combined ground-air perspective according to claim 1, characterized in that, The absolute scale recovery method is as follows: Distance measurement is performed using a depth camera mounted on an unmanned vehicle; a calibration board of known dimensions is set up within the drone's field of vision; and scale relationships are established. in, Indicates the scale ratio, used to scale the reconstructed model from the reconstruction unit to the actual metric system. This represents the actual distance measured by the depth camera. Represents the relative distance in the reconstruction model. This represents the three-dimensional coordinates of the first point in actual space. This represents the three-dimensional coordinates of the second point in actual space. Indicating the relationship between the reconstruction model and The corresponding three-dimensional point coordinates Indicating the relationship between the reconstruction model and The corresponding three-dimensional point coordinates.

5. A three-dimensional reconstruction system for traffic accident scenes using a combined ground-air perspective, characterized in that, include: The semantic segmentation module is used to collect images of traffic accident scenes using cameras on both drone and unmanned vehicle platforms, perform semantic segmentation on the images, and generate pixel-level masks of key targets. The ROI image acquisition module is used to extract the ROI from the image based on the pixel-level mask of the key target, perform image enhancement and noise reduction on the ROI, and obtain the processed ROI image. The 3D reconstruction module is used to optimize the camera intrinsic and extrinsic parameters of both UAV and unmanned vehicle platforms and perform absolute scale recovery. Based on the pixel-level mask of key targets and the processed ROI image, it reconstructs the 3D surface model by combining the SfM method, MVS method and surface reconstruction method with joint camera optimization objective function. The accident scene analysis module is used for digital and quantitative analysis of accident scenes through three-dimensional surface models. The formula for calculating the objective function of the joint camera optimization is as follows: in, The parameters to be optimized are represented by M, where M represents the total number of 3D points. This is the intrinsic parameter matrix of the drone camera. This is the intrinsic parameter matrix of the autonomous vehicle camera. This represents the UAV camera rotation matrix corresponding to the i-th frame of the UAV camera image. This represents the UAV camera translation vector corresponding to the i-th frame of the UAV camera image. This represents the UGV camera rotation matrix corresponding to the k-th frame UGV camera image. This represents the UGV camera translation vector corresponding to the k-th frame UGV camera image. This indicates the number of UAV images / poses involved in the optimization. This represents the number of valid observation points in the i-th frame of the UAV image. This represents the pixel observation at the j-th point in the i-th frame of the UAV image. This represents the position of the j-th 3D point in the world coordinate system. The coordinates of the 3D point are obtained by using the common view pixels of multiple 2D images. This indicates the number of UGV images / poses involved in the optimization. This represents the number of valid observation points in the k-th frame of the UGV image. Indicates the k-th frame of the UGV image. Pixel observations at each point Indicates the first The position of a 3D point in the world coordinate system , , These represent the weights of the corresponding point constraint, the scale constraint, and the geometric consistency constraint, respectively. This indicates the number of points with the same name that are viewed by both platforms. This represents the pixel observation of the c-th corresponding point in the UAV image. This represents the pixel observation of the c-th corresponding point in the UGV image. This represents the position of the a-th 3D point in the prior knowledge. This represents the position of the b-th 3D point in the prior knowledge. Point and The known physical distance This represents a set of indexes of point pairs with known physical distances. This indicates the relative rotation between the two cameras. This indicates the relative translation between the two cameras. represents the set of UAV / UGV pose pairs used for geometric consistency constraints, and π represents the camera projection function.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method as described in any one of claims 1-4.

7. An electronic device, characterized in that, The device includes a processor and a memory interconnected thereto, wherein the memory is used to store a computer program, the computer program including computer-readable instructions, and the processor is configured to invoke the computer-readable instructions to perform the method as described in any one of claims 1-4.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-4.

Images