Traffic accident scene three-dimensional reconstruction method combined with ground-air view angle

Abstract

The invention provides a three-dimensional reconstruction method for a traffic accident scene in combination with a ground-air view angle, and relates to the technical field of three-dimensional reconstruction, and the method comprises the steps: carrying out the image collection of the traffic accident scene through employing cameras of two platforms, namely an unmanned plane and an unmanned vehicle, carrying out the semantic segmentation of the image, and generating a pixel-level mask of a key target; extracting an ROI from the image according to the pixel-level mask of the key target, and performing image enhancement and denoising processing on the ROI to obtain a processed ROI image; and optimizing internal and external parameters of cameras of the unmanned aerial vehicle and the unmanned vehicle, performing absolute scale recovery, reconstructing a three-dimensional surface model by combining an SfM method, an MVS method and a surface reconstruction method of a combined camera optimization objective function based on a pixel-level mask of a key target and a processed ROI image, and performing digital quantitative analysis of an accident scene. The method can solve the problem that the existing three-dimensional reconstruction method is insufficient in camera calibration optimization and lacks absolute scale information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional reconstruction, in particular to a traffic accident scene three-dimensional reconstruction method combining ground and aerial views. BACKGROUND

[0002] Visual three-dimensional reconstruction technology aims to automatically reconstruct the three-dimensional geometry and texture information of a scene by analyzing multi-view images. In the field of traffic accident handling, this technology has revolutionary significance for the fixation of the accident scene, the determination of responsibility, the restoration of traffic, and the later analysis.

[0003] Traditional accident scene investigation mainly relies on manual measurement and photography, which has low efficiency, poor accuracy, high risk (especially when there is dangerous goods leakage), and incomplete information (there are a lot of occlusions and blind areas). In recent years, single-platform UAV aerial reconstruction has been introduced, which can provide a macro overview of the scene, but it is powerless for key information such as vehicle bottom, side collision details, and occluded evidence. Similarly, using only ground equipment (such as handheld cameras or unmanned vehicles) can capture local details, but it lacks an overall understanding of the accident scene and is difficult to associate with global coordinates. Whether from the air or the ground, there are inherent information blind spots. UAVs cannot effectively capture the bottom and vertical sides of vehicles, while ground views cannot obtain the overall distribution of roof deformations and scattered objects. Traditional methods cannot automatically identify and separate accident vehicles from the background environment, resulting in reconstructed models containing a large amount of irrelevant information, affecting the accuracy of subsequent analysis. When fusing multi-platform data, there is a lack of accurate joint camera calibration optimization algorithms, resulting in inconsistent scales and registration errors in the reconstructed model. The reconstructed model often lacks absolute scale references in the real world, making it impossible to perform accurate measurement analysis. Accident scene investigation must be completed in the shortest time to restore traffic as soon as possible and ensure the safety of investigators. Existing processes often struggle to balance efficiency, comprehensiveness, and safety. SUMMARY

[0004] The present application aims to solve the problems of insufficient camera calibration optimization and lack of absolute scale information in existing three-dimensional reconstruction. A traffic accident scene three-dimensional reconstruction method combining ground and aerial views is proposed, including the following steps: S1, using the cameras of the two platforms of the UAV and the unmanned vehicle to collect traffic accident scene images, performing semantic segmentation on the images, and generating pixel-level masks of key targets; S2, extracting ROI (Region of Interest) from the images according to the pixel-level masks of the key targets, performing image enhancement and denoising on the ROI, and obtaining processed ROI images; S3, optimize the camera internal and external parameters of the two platforms of the UAV and the unmanned vehicle, and restore the absolute scale, based on the pixel-level mask of the key target and the processed ROI image, reconstruct the three-dimensional surface model by combining the SfM (Structure from Motion) method, the MVS (Multi-View Stereo) method and the surface reconstruction method of the joint camera optimization target function; S4, perform digital quantitative analysis of the accident site based on the three-dimensional surface model.

[0005] Further, the image is subjected to semantic segmentation by using a segmentation network of the SAM architecture.

[0006] Further, the UAV camera internal parameter matrix is The unmanned vehicle camera internal parameter matrix is The expression is:

[0007]

[0008] wherein, f_u represents the value of the equivalent focal length of the UAV camera in the x-axis under the pixel coordinates, f_u represents the value of the equivalent focal length of the UAV camera in the y-axis under the pixel coordinates, c_u represents the value of the imaging center of the UAV camera in the x-axis, c_u represents the value of the imaging center of the UAV camera in the y-axis, f_v represents the value of the equivalent focal length of the unmanned vehicle camera in the x-axis under the pixel coordinates, f_v represents the value of the equivalent focal length of the unmanned vehicle camera in the y-axis under the pixel coordinates, c_v represents the value of the imaging center of the unmanned vehicle camera in the x-axis, c_v represents the value of the imaging center of the unmanned vehicle camera in the y-axis.

[0009] Further, the calculation formula of the joint camera optimization target function is as follows:

[0010] wherein, P represents the optimized parameters, and M represents the total number of three-dimensional points, is the UAV camera internal parameter matrix, is the unmanned vehicle camera internal parameter matrix, R_u represents the UAV camera rotation matrix corresponding to the i-th frame of the UAV camera image, t_u represents the UAV camera translation vector corresponding to the i-th frame of the UAV camera image, R_v represents the UGV camera rotation matrix corresponding to the k-th frame of the UGV camera image, This represents the UGV camera translation vector corresponding to the k-th frame UGV camera image. This indicates the number of UAV images / poses involved in the optimization. This represents the number of valid observation points in the i-th frame of the UAV image. This represents the pixel observation at the j-th point in the i-th frame of the UAV image. This represents the position of the j-th 3D point in the world coordinate system. The coordinates of the 3D point are obtained by using the common view pixels of multiple 2D images. This indicates the number of UGV images / poses involved in the optimization. This represents the number of valid observation points in the k-th frame of the UGV image. Indicates the k-th frame of the UGV image. Pixel observations at each point Indicates the first The position of a 3D point in the world coordinate system , , These represent the weights of the corresponding point constraint, the scale constraint, and the geometric consistency constraint, respectively. This indicates the number of points with the same name that are viewed by both platforms. This represents the pixel observation of the c-th corresponding point in the UAV image. This represents the pixel observation of the c-th corresponding point in the UGV image. This represents the position of the a-th 3D point in the prior knowledge. This represents the position of the b-th 3D point in the prior knowledge. Point and The known physical distance This represents a set of indexes of point pairs with known physical distances. This indicates the relative rotation between the two cameras. This indicates the relative translation between the two cameras. represents the set of UAV / UGV pose pairs used for geometric consistency constraints, and π represents the camera projection function.

[0011] Furthermore, the absolute scale recovery method is as follows: Distance measurement is performed using a depth camera mounted on an unmanned vehicle; a calibration board of known dimensions is set up within the drone's field of vision; and scale relationships are established.

[0012] in, Indicates the scale ratio, used to scale the reconstructed model from the reconstruction unit to the actual metric system. This represents the actual distance measured by the depth camera. Represents the relative distance in the reconstruction model. represents three-dimensional coordinates of the first point in the actual space, represents three-dimensional coordinates of the second point in the actual space, represents three-dimensional point coordinates corresponding to the first point in the reconstructed model, represents three-dimensional point coordinates corresponding to the second point in the reconstructed model, represents three-dimensional point coordinates corresponding to the first point in the reconstructed model, represents three-dimensional point coordinates corresponding to the second point in the reconstructed model.

[0013] The application further provides a traffic accident scene three-dimensional reconstruction system combining ground and air perspectives, comprising: a semantic segmentation module, which is used for collecting traffic accident scene images by using cameras of two platforms of a UAV and an unmanned vehicle, performing semantic segmentation on the images, and generating pixel-level masks of key targets; an ROI image acquisition module, which is used for extracting ROIs from the images according to the pixel-level masks of the key targets, performing image enhancement and denoising processing on the ROIs, and obtaining processed ROI images; a three-dimensional reconstruction module, which is used for optimizing internal and external parameters of the cameras of the two platforms of the UAV and the unmanned vehicle, performing absolute scale recovery, and reconstructing a three-dimensional surface model by combining an SfM method, an MVS method and a surface reconstruction method of a joint camera optimization target function based on the pixel-level masks of the key targets and the processed ROI images; an accident scene analysis module, which is used for performing digital quantitative analysis of the accident scene by using the three-dimensional surface model.

[0014] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the traffic accident scene three-dimensional reconstruction method combining ground and air perspectives.

[0015] The application further provides an electronic device, which comprises a processor and a memory, the processor and the memory are connected to each other, the memory is used for storing a computer program, the computer program comprises computer readable instructions, the processor is configured to call the computer readable instructions, and the traffic accident scene three-dimensional reconstruction method combining ground and air perspectives is executed.

[0016] The application further provides a computer program product, which comprises computer programs / instructions, and the computer programs / instructions are executed by a processor to realize steps of the traffic accident scene three-dimensional reconstruction method combining ground and air perspectives.

[0017] The application provides the technical scheme, and the beneficial effects are as follows: The application combines the macroscopic perspective of the unmanned aerial vehicle and the microscopic perspective of the unmanned vehicle, effectively eliminates the observation blind area of a single perspective, realizes all-around coverage of the accident scene, reconstructs a three-dimensional surface model by combining the SfM method, the MVS method and the surface reconstruction method of the joint camera optimization target function, realizes high-precision fusion of the multi-perspective data of the unmanned aerial vehicle and the unmanned vehicle, and guarantees the consistency of the internal and external parameters of the reconstructed model. The dual constraint mechanism of the depth camera and the high-precision calibration board is adopted to provide absolute scale guarantee. The depth information is used for auxiliary scale calibration, and the calibration board is used for geometric correction, so that the three-dimensional reconstruction result has the absolute scale of the real world. The application can solve the problems of insufficient camera calibration optimization and lack of absolute scale information in the existing three-dimensional reconstruction method. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a flow chart of a traffic accident scene three-dimensional reconstruction method of joint air-ground perspective according to an embodiment of the application; Figure 2 is a schematic diagram of joint unmanned aerial vehicle and unmanned vehicle surrounding aerial photography planning according to an embodiment of the application; Figure 3 is a schematic diagram of unmanned aerial vehicle vertical direction aerial photography height and unmanned vehicle horizontal photography planning according to an embodiment of the application; Figure 4 is a schematic diagram of unmanned aerial vehicle aerial photography angle and unmanned vehicle photography distance planning according to an embodiment of the application; Figure 5 is a schematic diagram of sparse reconstruction result of the oil tanker head according to an embodiment of the application; Figure 6 is a schematic diagram of high-precision dense three-dimensional model of the oil tanker head according to an embodiment of the application; Figure 7 is a quantitative analysis result of the oil tanker head according to an embodiment of the application; Figure 8 is a block diagram of an electronic device in an example embodiment according to an embodiment of the application. DETAILED DESCRIPTION

[0019] To make the object, technical scheme and advantages of the application clearer, the embodiments of the application will be further described below with reference to the drawings.

[0020] The flow chart of the traffic accident scene three-dimensional reconstruction method of joint air-ground perspective according to an embodiment of the application is as shown in Figure 1 , and specifically includes the following steps: S1, multi-platform cooperative deployment is carried out at the traffic accident scene, including deployment of unmanned aerial vehicles (UAVs), unmanned ground vehicles (UGVs), and placement of calibration objects for scale recovery. Specifically, the unmanned aerial vehicle usually performs a fly-around, obtaining global overhead views and multi-angle oblique images of the accident scene from the air, for grasping the macro layout and overall situation of the scene. The unmanned ground vehicle then approaches the accident vehicle and key evidence on the ground, focusing on shooting areas such as the vehicle bottom, side collision details, and tire marks that cannot be effectively covered by the unmanned aerial vehicle perspective, to make up for the blind area of the macro perspective. The absolute scale calibration board placed on the scene has a pre-known physical size.

[0021] Reference Figure 2 , Figure 3 and Figure 4 , Figure 2 is a schematic diagram of the fly-around planning of the unmanned aerial vehicle and the unmanned ground vehicle in the embodiment of the present application; Figure 3 is a schematic diagram of the vertical flight height of the unmanned aerial vehicle and the horizontal shooting planning of the unmanned ground vehicle in the embodiment of the present application; Figure 4 is a schematic diagram of the flight angle of the unmanned aerial vehicle and the shooting distance of the unmanned ground vehicle in the embodiment of the present application. (1) Unmanned aerial vehicle (UAV) aerial data acquisition setting: first, the unmanned aerial vehicle acquisition path is divided into a conventional fly-around path and a layered flight path. In the conventional fly-around path, as shown in Figure 2 , the unmanned aerial vehicle flies around the accident vehicle. Preferably, the flight radius is about 10-12 m, and the included angle of the flight points in one circle can be set to about 15°, 22.5°, 45°. The unmanned aerial vehicle can hover at each flight point for 2-5 seconds, and the RGB camera carried by the unmanned aerial vehicle can shoot the accident vehicle at different pitch angles. Preferably, one or more pitch angle shootings can be performed at one or more flight points, such as about 45°, 50°, 55°, 60°, to obtain vehicle facade information at different angles. In particular, for the key areas of the vehicle collision damage, encrypted collection can be performed, i.e., increasing the flight point density or the number of pitch angles. In addition, as shown in Figure 8 , the unmanned aerial vehicle performs layered flight with the entire tanker traffic accident scene as the object. Preferably, it can be divided into two height levels, an "air layer" with a height of about 5-8 m and a "ground layer" with a height of about 0-5 m. The shooting data of the "air layer" is mainly used to obtain the macro layout and overall situation of the scene; the shooting data of the "ground layer" is used to capture the details of the upper half of the vehicle and the surrounding scattered objects more clearly. (2) Unmanned ground vehicle (UGV) ground data acquisition setting: at the same time or after the UAV performs aerial data acquisition, as shown in Figure 2 , the unmanned ground vehicle carrying an RGB camera and a depth camera is deployed to start shooting at a close-in radius of about 3-4 m. As shown in Figure 3The unmanned vehicle is responsible for collecting the areas where the unmanned aerial vehicle perspective exists occlusion and blind area, such as the side, bottom and low-angle collision details of the vehicle. Figure 4 At this time, the unmanned aerial vehicle can take key supplementary pictures of the damaged area at different altitudes and with different camera pitch angles (difference of about 5-10°). At the same time, the unmanned vehicle can take detailed pictures by using its closer distance to the damaged position. Through the air-ground cooperation, near-far combination and multi-angle complementary way, the key evidence information is collected without dead angle.

[0022] The unmanned aerial vehicle and the unmanned vehicle cooperatively deployed in multiple platforms perform the data collection task on the scene, and transmit the acquired image data to the on-site computing unit in real time. The computing unit performs semantic segmentation on the image through a deep learning model to generate a pixel-level mask of the key target. Specifically, in the process of generating the key target mask by deep learning semantic segmentation, the system inputs the collected RGB image into a pre-trained segmentation network, for example, based on the Segment Anything Model (SAM) architecture. The segmentation network automatically identifies and accurately segments the accident vehicle, road markings, key scattered objects and other targets that are crucial to accident analysis, and generates an independent pixel-level binary mask for each target.

[0023] S2, according to the pixel-level mask of the key target, the region of interest (ROI) is extracted from the image, the ROI is processed by image enhancement and denoising, the target-image correspondence relationship is constructed, and the processed ROI image is obtained. The significance of this step is that it can focus the reconstruction on the effective target, greatly reducing the interference of irrelevant information such as background environment, providing high-quality ROI for subsequent feature point extraction and matching, and significantly improving the reconstruction accuracy and efficiency.

[0024] S3, the camera internal and external parameters of the unmanned aerial vehicle and the unmanned vehicle are optimized, and the absolute scale is restored, based on the pixel-level mask of the key target and the processed ROI image, the three-dimensional surface model is reconstructed by combining the SfM method, MVS method and surface reconstruction method of the joint camera optimization target function.

[0025] The specific steps are as follows: The camera internal parameter matrix of the unmanned aerial vehicle is The camera internal parameter matrix of the unmanned vehicle is The expression is:

[0026]

[0027] wherein, represents the value of the equivalent focal length of the UAV camera in the pixel coordinate in the x-axis, represents the value of the equivalent focal length of the UAV camera in the pixel coordinate in the y-axis, represents the value of the imaging center of the UAV camera in the x-axis, represents the value of the imaging center of the UAV camera in the y-axis, represents the value of the equivalent focal length of the UGV camera in the pixel coordinate in the x-axis, represents the value of the equivalent focal length of the UGV camera in the pixel coordinate in the y-axis, represents the value of the imaging center of the UGV camera in the x-axis, represents the value of the imaging center of the UGV camera in the y-axis.

[0028] Since the cameras of the two platforms of the UAV and the UGV both adopt square pixels and non-orthogonal readout, the tilt coefficient is set to 0. If there is a single camera calibration, the result can be directly used; otherwise, a heuristic initial value can be given by using the chessboard method or using the image resolution center and the number of diagonal pixels to ensure the convergence of subsequent optimization.

[0029] In the preferred embodiment of the application, a joint camera optimization objective function is constructed to optimize the camera intrinsic parameters of the two platforms of the UAV and the UGV and , the camera extrinsic parameters , and the three-dimensional point coordinates, and unify the camera intrinsic and extrinsic parameters of the two platforms of the UAV and the UGV. The following total residual least squares problem is defined, wherein the constraint terms include the same point constraint, the scale constraint and the geometric consistency constraint, and the calculation formula is as follows:

[0030] wherein, represents the optimized parameters, and M represents the total number of three-dimensional points, is the UAV camera intrinsic parameter matrix, is the UGV camera intrinsic parameter matrix, represents the UAV camera rotation matrix corresponding to the i-th frame of the UAV camera image, and the camera rotation matrix indicates that the image is rotated by different angles around different coordinate axes from the world coordinate system to the camera coordinate system, represents the UAV camera translation vector corresponding to the i-th frame of the UAV camera image, and the camera translation vector indicates the translation transformation of the image from the world coordinate system to the camera coordinate system, represents the UGV camera rotation matrix corresponding to the k-th frame of the UGV camera image, represents the UGV camera translation vector corresponding to the k-th frame of the UGV camera image, This indicates the number of UAV images / poses involved in the optimization. This represents the number of valid observation points in the i-th frame of the UAV image. This represents the pixel observation at the j-th point in the i-th frame of the UAV image. This represents the position of the j-th 3D point in the world coordinate system. The coordinates of the 3D point are obtained by using the common view pixels of multiple 2D images. This indicates the number of UGV images / poses involved in the optimization. This represents the number of valid observation points in the k-th frame of the UGV image. Indicates the k-th frame of the UGV image. Pixel observations at each point Indicates the first The position of a 3D point in the world coordinate system , , These represent the weights of the corresponding point constraint, the scale constraint, and the geometric consistency constraint, respectively. This indicates the number of points with the same name that are viewed by both platforms. This represents the pixel observation of the c-th corresponding point in the UAV image. This represents the pixel observation of the c-th corresponding point in the UGV image. This represents the position of the a-th 3D point in the prior knowledge. This represents the position of the b-th 3D point in the prior knowledge. Point and The known physical distance This represents a set of indexes of point pairs with known physical distances. This represents the relative rotation matrix between the two cameras (UGV→UAV). This indicates the relative translation between the two cameras (UGV→UAV). Represents the set of UAV / UGV pose pairs used for geometric consistency constraints, where π represents the camera projection function. ).

[0031] This least squares problem can be solved iteratively using Ceres Solver's sparse Levenberg-Marquardt (LM) method. The solution strategy is as follows: first, initialize all K, R, and T with single-machine calibration (or heuristic) results. Then, use the sparse LM method to solve the problem. The iteration minimization is made. The damping factor is automatically adjusted in the subsequent iteration until the pixel re-projection error and the constraint residual are converged simultaneously. By adding the three types of priori terms of "corresponding points, scale, geometry" to the camera re-projection error, the joint optimization can complete the cross-platform calibration, scale alignment and pose refinement at the same time, and lay a geometrically consistent foundation for subsequent multi-source fusion. The image feature points, camera internal and external parameters and scale constraint information from the unmanned aerial vehicle and the unmanned vehicle are bundled and adjusted (Bundle Adjustment, BA), the re-projection errors of all views are minimized, and the perfect alignment of the two heterogeneous platforms in space and scale is realized.

[0032] Then, a sparse three-dimensional point cloud with absolute scale information is generated through a structure from motion (SfM) process based on a joint camera optimization objective function. The absolute scale recovery method is as follows: The distance measurement is made by using the depth camera carried by the unmanned vehicle; the calibration board with a known size is set within the visual range of the unmanned aerial vehicle; and the scale proportion relationship is established:

[0033] wherein, represents the scale proportion, which is used to scale the reconstruction model from the reconstruction unit to the real meter, represents the real distance measured by the depth camera, represents the relative distance in the reconstruction model, represents the three-dimensional coordinates of the first point in the actual space, represents the three-dimensional coordinates of the second point in the actual space, represents the three-dimensional point coordinates in the reconstruction model corresponding to the first point, represents the three-dimensional point coordinates in the reconstruction model corresponding to the second point. represents the three-dimensional point coordinates in the reconstruction model corresponding to the first point. represents the three-dimensional point coordinates in the reconstruction model corresponding to the second point.

[0034] Finally, the sparse point cloud is densified to generate a high-precision three-dimensional surface model through a general stereo reconstruction process. The stereo reconstruction process is as follows: first, the feature point matching is made on the entire image library (including the unmanned aerial vehicle and unmanned vehicle images). The feature matching supports various feature description operators, including but not limited to SIFT, SURF, ORB, SuperPoint and other mature schemes. The false matching can be filtered through geometric constraints, and mature schemes including but not limited to RANSAC and its variants can be used, and the key is to successfully match the corresponding points between the unmanned aerial vehicle image and the unmanned vehicle image, and effective feature screening is made based on the target mask. The incremental sparse reconstruction is designed as a general SfM architecture, which can use schemes including but not limited to OpenMVG, COLMAP, VisualSfM, etc., and has a unified input and output interface.

[0035] The subsequent stereoscopic reconstruction process is designed as a general MVS reconstruction architecture. After obtaining the sparse three-dimensional structure, mature schemes such as PMVS, OpenMVS, MVE, etc. can be used to generate a dense three-dimensional structure. A complete quality control and optimization mechanism is established. This mechanism includes dense reconstruction quality evaluation based on target masks, multi-scale geometric consistency verification, and adaptive parameter adjustment strategies. It supports multiple surface reconstruction methods, including but not limited to Poisson, Delaunay, etc. It supports multiple texture mapping techniques, including but not limited to multi-view texture fusion, seam elimination optimization, and texture atlas generation, etc. mature schemes. This design ensures that even when dealing with complex scenes and heterogeneous data, through strict geometric filtering and surface optimization, a complete and accurate three-dimensional model with high-fidelity texture can be generated.

[0036] S4, digital quantitative analysis of the accident scene is carried out through the three-dimensional surface model. For example, non-contact accurate measurement of vehicle damage length, distribution range of scattered objects, collision analysis, etc. Finally, according to the analysis results, a digital investigation report that can be used for accident liability identification and post-analysis is output. Specifically, the final model generated is a 1:1 real-world, measurable digital accident scene reconstruction. Rescue personnel or accident analysis experts can observe and measure the model from any angle in a safe office environment through the back-end analysis module, including but not limited to MeshLab, CloudCompare, etc. For example, the degree of deformation of the vehicle, the size of the collision contact surface, the position of the scattered objects relative to the vehicle, etc. can be accurately measured, which is difficult to quickly and accurately obtain through traditional investigation methods.

[0037] More specifically, taking a traffic accident caused by a tanker rear-end collision at a highway intersection as an example, the accident caused serious damage to the front of the tanker, and the scene was deployed with drones, unmanned vehicles, and calibration objects. The accident scene and vehicle have been preliminarily identified, and high-precision three-dimensional reconstruction of the key parts of the vehicle head is required. The three-dimensional reconstruction steps of the traffic accident scene by the joint unmanned aerial vehicle and unmanned vehicle air-ground perspective of the invention for this accident scene are as follows: (1) Data collection: First, a quadrotor UAV was controlled to fly around the damaged front of the tanker truck as the center. The flight radius was set to be about 10 m, and the flight was performed at two height levels of 5 m and 8 m. At each level, 16 main waypoints were set around the front of the truck with an angle of 22.5°-45° between the waypoints. At each waypoint, the on-board camera was used to take photos with an angle of about 45° pitch, and a total of 32 photos were taken. For the most severely damaged area of the front of the truck, 31 photos were taken for encryption collection. In this process, a total of 63 UAV aerial images were obtained. At the same time, a Mecanum wheel unmanned vehicle was deployed to surround the front of the tanker truck at a close radius of about 4 m, and its on-board RGB camera and depth camera were used to continuously take photos of the damaged front of the truck, tires and chassis details every 20°, and a total of 8 close-up photos were obtained.

[0038] (2) Reconstruction and result display: After the total of 71 images (63 UAV images + 8 unmanned vehicle images) were pre-processed, they were input into the reconstruction system of the present application, and after joint calibration and optimization process and sparse reconstruction process, the sparse reconstruction result of the front of the tanker truck was obtained as shown in Figure 5 . The result is a view presented in Colmap, which accurately shows the camera poses of the 71 photos involved in the reconstruction (clearly showing the aerial horizontal layered UAV camera poses and the ground linearly surrounding unmanned vehicle camera poses) and the sparse three-dimensional point cloud structure of the front of the tanker truck and the surrounding environment composed of matching feature points. This figure directly verifies the effectiveness and high precision of the air-ground data fusion of the present application. Subsequently, after the densification reconstruction process, a high-precision three-dimensional model of the front of the tanker truck and the surrounding environment was finally generated, which can be a file in standard format such as PLY. Figure 6 .

[0039] (3) Quantitative analysis display: as shown in Figure 7 , Figure 7 is the quantitative analysis result of the front of the tanker truck. With the help of the "Pick Point Measure" function of Meshlab, the extreme points on both sides of the front can be directly selected on the three-dimensional point cloud to obtain the spatial distance of the width of the front. Through three independent measurements of the same large tanker tractor, the average width of a single reading is about 2.94 m, and the actual measured value is 3.00 m, with an absolute error of about 6 cm. This result verifies the usability of the model in traffic accident reconstruction - without additional scales, it can achieve an absolute size accuracy of 10 cm or less and a relative error accuracy of 5% or less for large two-passenger and one-hazard vehicles.

[0040] In an exemplary embodiment, a joint air-ground view three-dimensional reconstruction system for a traffic accident scene is provided, comprising: The semantic segmentation module is configured to collect images of the traffic accident scene by using cameras of the two platforms of the UAV and the unmanned vehicle, perform semantic segmentation on the images, and generate a pixel-level mask of a key target. The ROI image acquisition module is configured to extract an ROI from the images according to the pixel-level mask of the key target, perform image enhancement and denoising processing on the ROI, and obtain a processed ROI image. The three-dimensional reconstruction module is configured to optimize internal and external parameters of the cameras of the two platforms of the UAV and the unmanned vehicle, perform absolute scale recovery, and reconstruct a three-dimensional surface model based on the pixel-level mask of the key target and the processed ROI image by combining an SfM method, an MVS method, and a surface reconstruction method of a joint camera optimization target function. The accident scene analysis module is configured to perform digital quantitative analysis on the accident scene by using the three-dimensional surface model.

[0041] In an example embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the joint air-ground view three-dimensional reconstruction method of a traffic accident scene.

[0042] Please refer to Figure 8 In an example embodiment, an electronic device is also provided, which includes at least one processor, at least one memory, and at least one communication bus.

[0043] The memory stores a computer program, which includes computer readable instructions. The processor invokes the computer readable instructions stored in the memory through the communication bus to execute the joint air-ground view three-dimensional reconstruction method of a traffic accident scene.

[0044] In an example embodiment, a computer program product is provided, which includes computer program / instructions. The computer program / instructions are executed by a processor to implement the steps of the joint air-ground view three-dimensional reconstruction method of a traffic accident scene.

[0045] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for three-dimensional reconstruction of a traffic accident scene in conjunction with an air-ground perspective, characterized in that, The method comprises the following steps: S1, collecting traffic accident scene images using cameras of two platforms of a UAV and an unmanned vehicle, performing semantic segmentation on the images, and generating a pixel-level mask of a key target; S2, extracting an ROI from the images according to the pixel-level mask of the key target, performing image enhancement and denoising processing on the ROI, and obtaining a processed ROI image; S3, optimizing internal and external parameters of cameras of the two platforms of the UAV and the unmanned vehicle, performing absolute scale recovery, and reconstructing a three-dimensional surface model based on the pixel-level mask of the key target and the processed ROI image by combining an SfM method, an MVS method, and a surface reconstruction method of a joint camera optimization target function; S4, performing digital quantitative analysis of the accident scene by using the three-dimensional surface model.

2. The method according to claim 1, wherein, The image is subjected to semantic segmentation by using a segmentation network of a SAM architecture.

3. The method of claim 1, wherein, The camera intrinsic matrix of the unmanned aerial vehicle is The camera intrinsic matrix of the unmanned vehicle is The expression is: wherein, represents a value of the equivalent focal length of the drone camera in the pixel coordinates in the x-axis, represents a value of the equivalent focal length of the drone camera in the pixel coordinates in the y-axis, represents a value of the imaging center of the drone camera in the x-axis, represents a value of the imaging center of the drone camera in the y-axis, represents a value of the equivalent focal length of the drone camera in the pixel coordinates in the x-axis, represents a value of the equivalent focal length of the drone camera in the pixel coordinates in the y-axis, represents a value of the imaging center of the drone camera in the x-axis, represents a value of the imaging center of the drone camera in the y-axis.

4. The method of claim 1, wherein, A calculation formula of the joint camera optimization target function is as follows: in, The parameters to be optimized are represented by M, where M represents the total number of 3D points. This is the intrinsic parameter matrix of the drone camera. This is the intrinsic parameter matrix of the autonomous vehicle camera. This represents the UAV camera rotation matrix corresponding to the i-th frame of the UAV camera image. This represents the UAV camera translation vector corresponding to the i-th frame of the UAV camera image. This represents the UGV camera rotation matrix corresponding to the k-th frame UGV camera image. This represents the UGV camera translation vector corresponding to the k-th frame UGV camera image. This indicates the number of UAV images / poses involved in the optimization. This represents the number of valid observation points in the i-th frame of the UAV image. This represents the pixel observation at the j-th point in the i-th frame of the UAV image. This represents the position of the j-th 3D point in the world coordinate system. The coordinates of the 3D point are obtained by using the common view pixels of multiple 2D images. This indicates the number of UGV images / poses involved in the optimization. This represents the number of valid observation points in the k-th frame of the UGV image. Indicates the k-th frame of the UGV image. Pixel observations at each point Indicates the first The position of a 3D point in the world coordinate system , , These represent the weights of the corresponding point constraint, the scale constraint, and the geometric consistency constraint, respectively. This indicates the number of points with the same name that are viewed by both platforms. This represents the pixel observation of the c-th corresponding point in the UAV image. This represents the pixel observation of the c-th corresponding point in the UGV image. This represents the position of the a-th 3D point in the prior knowledge. This represents the position of the b-th 3D point in the prior knowledge. Point and The known physical distance This represents a set of indexes of point pairs with known physical distances. This indicates the relative rotation between the two cameras. This indicates the relative translation between the two cameras. represents the set of UAV / UGV pose pairs used for geometric consistency constraints, and π represents the camera projection function.

5. The method of claim 1, wherein, An absolute scale recovery method is as follows: A depth camera carried by the unmanned vehicle is used to measure distances, a calibration board with a known size is arranged within a visible range of the UAV, and a scale ratio relationship is established: in, Indicates the scale ratio, used to scale the reconstructed model from the reconstruction unit to the actual metric system. This represents the actual distance measured by the depth camera. Represents the relative distance in the reconstruction model. This represents the three-dimensional coordinates of the first point in actual space. This represents the three-dimensional coordinates of the second point in actual space. Indicating the relationship between the reconstruction model and The corresponding three-dimensional point coordinates Indicating the relationship between the reconstruction model and The corresponding three-dimensional point coordinates.

6. A traffic accident scene 3D reconstruction system combining air-to-ground view, characterized in that, The method comprises the following steps: A semantic segmentation module is configured to collect traffic accident scene images using cameras of two platforms of a UAV and an unmanned vehicle, perform semantic segmentation on the images, and generate a pixel-level mask of a key target; An ROI image acquisition module is configured to extract an ROI from the images according to the pixel-level mask of the key target, perform image enhancement and denoising processing on the ROI, and obtain a processed ROI image; A three-dimensional reconstruction module is configured to optimize internal and external parameters of cameras of the two platforms of the UAV and the unmanned vehicle, perform absolute scale recovery, and reconstruct a three-dimensional surface model based on the pixel-level mask of the key target and the processed ROI image by combining an SfM method, an MVS method, and a surface reconstruction method of a joint camera optimization target function; An accident scene analysis module is configured to perform digital quantitative analysis of the accident scene by using the three-dimensional surface model.

7. A computer readable storage medium storing a computer program, characterized in that: The computer program is executed by a processor to implement the method of any one of claims 1-5.

8. An electronic device, comprising: The computer program is executed by a processor to implement the method of any one of claims 1-5.

9. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by a processor to implement the steps of the method of any one of claims 1-5.

Images