A vehicle-end perspective accident scene restoration method based on a vehicle event data recorder video

Abstract

The application belongs to the technical field of automatic driving cars, and particularly relates to a vehicle-end visual angle accident scene restoration method based on a vehicle record video, which comprises the following steps: step one, pre-processing the accident video; step two, performing semantic understanding on the scene; step three, extracting the pixel track of a traffic participant; step four, jointly calibrating the inverse perspective mapping parameters; step five, performing inverse perspective transformation; and step six, post-processing the road contour and track data. In view of the camera parameters being unknown, different and lacking of multi-source synchronous information, the application proposes a vehicle-end visual angle accident scene restoration framework with a camera inverse perspective mapping parameter joint calibration strategy as the core, can extract the scene space-time information including the road contour and the traffic participant track from the vehicle record video without any prior information support, establish the accident scene data for automatic driving car testing, and effectively improve the testing process of the automatic driving car.

Description

Technical Field

[0001] This invention belongs to the field of autonomous vehicle technology, specifically a method for reconstructing accident scenes from the vehicle's perspective based on dashcam video. Background Technology

[0002] In recent years, autonomous vehicles have developed rapidly, and their ability to cope with complex scenarios has been significantly enhanced. However, the driving scenarios in the real world are endlessly diverse, and complex, extreme, long-tail scenario data with extremely low probability of occurrence has a significant impact on the safe operation of autonomous vehicles. To meet the public's increasingly urgent demand for safe and reliable autonomous vehicles, it is necessary to conduct thorough and efficient testing of autonomous vehicles to improve their ability to handle complex traffic environments, especially long-tail scenarios. Accident scenarios, as an important component of long-tail scenarios, are of paramount importance to autonomous vehicle testing. However, the process of collecting accident scenario data is extremely dangerous, costly, and inefficient. Existing open-source natural driving datasets that can be directly used for testing, such as HighD, rarely have accidents. Therefore, there is an increasing demand for accident data. How to obtain high-quality accident scenario data at low cost and high efficiency for autonomous vehicle testing is an important problem that urgently needs to be solved.

[0003] Dashcams record the dynamic evolution of a scene within a certain range in front of a vehicle using a single-lens camera on the vehicle. This video footage allows for in-depth analysis of accident scenes to determine liability. Therefore, some accident datasets exist stored in single-lens video format. Furthermore, numerous similar accident scenes can be searched on online platforms, making it possible to acquire high-value accident scene data at low cost. This data source significantly reduces data acquisition costs and possesses highly realistic interactive characteristics, greatly enhancing the algorithm's adaptability in real-world open-world scenarios. Moreover, this source of accident data covers complex and varied scene types, effectively ensuring scene diversity.

[0004] However, the limited video data is insufficient for direct application in the testing of autonomous vehicles. Accident reconstruction based on dashcam video is necessary to extract key scene information. Traditional accident reconstruction based on dashcam video relies heavily on auxiliary data. Limited by geographical distribution and actual costs, on-site surveys of accident scenes are not feasible, resulting in a lack of multi-source synchronous information in the accident videos stored by dashcams. This makes it difficult to achieve accurate accident reconstruction and effective extraction of scene information. AI-powered computer vision technology extracts features from scene images visually, effectively reducing the reliance on auxiliary information. However, when different dashcams store accident videos, the camera parameters are completely unknown, and different camera configurations lead to significant differences in camera parameters. The lack of synchronous multi-source information makes it impossible to directly obtain the transformation matrix used for camera mapping. Although this effectively reduces the reliance on auxiliary information, the aforementioned phenomena mean that the existing research framework still cannot complete the task of extracting key information from the vehicle-side accident scene from dashcam video.

[0005] For accident video data with unknown and different camera parameters and lack of multi-source synchronization information, this invention proposes a vehicle-side perspective accident scene reconstruction framework with a joint calibration strategy of camera inverse perspective mapping parameters as the core. It can extract scene spatiotemporal information, including road contours and traffic participant trajectories, from dashcam videos without any prior information support, and establish accident scene data for use in autonomous vehicle testing. Summary of the Invention

[0006] To address the aforementioned issues, this invention provides a method for reconstructing accident scenes from a vehicle's perspective based on dashcam video. First, the monocular accident video captured by the dashcam is preprocessed to optimize video quality. Then, a scene semantic understanding module obtains semantic masks for pixel road boundaries and dynamic image regions. Next, a robust traffic participant identification and tracking algorithm is designed to extract pixel motion trajectories. Subsequently, a joint calibration algorithm for vehicle-side dynamic scene parameters based on GeometricallySalient Feature Bundle Adjustment (GSFBA) is used to calibrate inverse perspective mapping parameters with high precision. Then, inverse perspective transformation is performed on pixel coordinates using camera parameters. Finally, a post-processing module for road contour and trajectory information is designed to extract the actual road contour based on the pixel road boundary mapping intensity, and a systematic trajectory processing module is used to fit and optimize the trajectory information. This completes the accident scene reconstruction task based on dashcam video, establishing accident scene data for use in autonomous vehicle testing.

[0007] The technical solution of this invention is described below in conjunction with the accompanying drawings:

[0008] A method for reconstructing accident scenes from a vehicle's perspective based on dashcam video includes the following steps:

[0009] Step 1: Preprocess the accident video;

[0010] The video is denoised using an efficient multi-level noise reduction algorithm, and the video is enhanced by frame interpolation using a real-time intermediate stream estimation algorithm.

[0011] Step 2: Perform semantic understanding of the scene;

[0012] Using the DeepLabV3Plus algorithm as a semantic feature extractor, the video is processed frame by frame to obtain multi-frame semantic images of the scene. The multi-frame pixel road boundaries are extracted through the edge detection algorithm, and based on this, the main focus area of ​​the target detection algorithm and the background area are divided. Based on the semantic information, the motion consistency check algorithm is used to obtain the dynamic region and dynamic semantic mask.

[0013] Step 3: Extract the pixel trajectories of traffic participants;

[0014] By combining the improved YOLOV8 algorithm with the DeepSORT algorithm, key traffic participants are robustly identified and correlated frame by frame, enabling precise extraction of their pixel trajectories.

[0015] Step 4: Perform joint calibration of the inverse perspective mapping parameters;

[0016] The algorithm for joint calibration of inverse perspective mapping parameters in vehicle-side dynamic scenes based on geometrically salient feature bundle adjustment is designed. First, the feature information of the image is extracted. Then, optical flow correction prediction and confidence update are performed to obtain key image frames for front-end factor map construction and intermediate quantities for back-end factor map optimization. Then, the factor map is initialized and constructed in the front-end process. Finally, the joint iterative calibration of parameters is achieved in the back-end process using geometrically salient feature bundle adjustment (GSFBA).

[0017] Step 5: Perform inverse perspective transformation;

[0018] Map the pixel trajectories of key traffic participants and the pixel contours of roads to the world coordinate system for subsequent scene data processing;

[0019] Step 6: Post-process the road outline and trajectory data;

[0020] The design incorporates a road contour extraction method based on pixel road boundary mapping intensity and a systematic trajectory data post-processing workflow, including missing trajectory prediction, fluctuating trajectory smoothing, and abnormal trajectory constraint, to further improve data quality and ensure the accuracy and applicability of key scene information extraction.

[0021] Furthermore, the specific method for step one is as follows:

[0022] 11) The video is processed using an efficient multistage video denoising (EMVD) algorithm, which reduces noise in the input video through three processing stages: temporal fusion, spatial denoising, and spatiotemporal refinement.

[0023] 12) The video is enhanced by using the Real-time Intermediate Flow Estimation (RIFE) algorithm. After outputting an approximate intermediate flow, backward deformation and fusion are performed to achieve accurate prediction of the intermediate frame image.

[0024] Furthermore, the specific method for step two is as follows:

[0025] 21) The DeepLabV3Plus algorithm, which incorporates dilated convolution structures, is selected as the semantic feature extractor;

[0026] 22) The video is processed frame by frame by a feature extractor to obtain multi-frame semantic images of the scene. Multi-frame pixel road boundaries are extracted by an edge detection algorithm. The pixel road boundaries are used as the boundary to divide the main focus area of ​​the target detection algorithm and the background area.

[0027] 23) The semantic mask of the dynamic region of the image is obtained by using the motion consistency check algorithm in conjunction with semantic information. The process is as follows: Different traffic participants will be assigned different colors in the semantic information. The outline of the traffic participants is regarded as the candidate region to be checked for motion. First, the feature points in the candidate region to be checked for motion in the first frame image are detected. The optical flow pyramid is calculated to obtain the feature points in the region to be checked for motion in the second frame that match the first frame. The fundamental matrix and epipolar line between the feature points are calculated using the Random Sample Consensus (RANSAC) algorithm. The distance between the matching feature points and the epipolar line is calculated. If it is greater than the set threshold, it indicates that the feature point is moving. If the moving feature points in a certain region to be checked for motion are greater than a certain percentage threshold of the total number of feature points, it is regarded as a moving region. Thus, the task of obtaining the semantic mask of the dynamic region in the first frame image is completed until all frames of the video are processed.

[0028] The fundamental matrix is ​​used to map the feature points in the first frame to the corresponding search domain, i.e., the epipolar line, in the second frame. Assuming that p1 and p2 are the matching feature points in the first and second frames, respectively, they are expressed in homogeneous coordinate form as shown in equation (1):

[0029] p1=(u1,v1,1),p2=(u2,v2,1) (1)

[0030] In the formula, u and v are pixel values ​​in the image;

[0031] Then, the polar line I is calculated using equation (2). m :

[0032] I m =(x m ,y m ,z m ) T =F m p1 (2)

[0033] In the formula, x m y m z m F is a line vector; m The basic matrix;

[0034] Then, the distance D from the matching feature point to the corresponding epipolar line is calculated using equation (3). m :

[0035]

[0036] In the formula, D m To match the distance from the feature point to the corresponding epipolar line.

[0037] Furthermore, the specific method for step three is as follows:

[0038] 31) Based on the YOLOV8 algorithm, a global attention mechanism (GAM) is introduced to optimize the YOLOV8 algorithm backbone network, and the ASPP (Atrous Spatial Pyramid Pooling) module is used to optimize the algorithm feature pyramid, thereby obtaining the target detector for the video.

[0039] Using the target detector constructed above, prediction and generation are performed based on feature extraction to obtain traffic participant detection results. At the same time, the main regions of interest obtained from scene semantic understanding are incorporated into the algorithm detection process to eliminate invalid background interference in the scene.

[0040] 32) Based on the target detection algorithm, the DeepSORT algorithm is used to generate trajectories after estimating the motion of traffic participants, thereby obtaining data association results. That is, the cross-frame detection results of the same vehicle are associated and assigned a unique ID, thus forming continuous pixel trajectory data of traffic participants.

[0041] Furthermore, the specific method for step four is as follows:

[0042] 41) Obtain the dynamic semantic mask of the image through scene semantic understanding, and use the masked image to represent P = (I i ,I j As input, the feature encoder obtains the feature information of adjacent frames and performs correlation calculation, and obtains the initial optical flow. The correlation encoder and optical flow encoder are used to encode the correlation and optical flow respectively. At the same time, the context encoder obtains the context information for subsequent optical flow correction prediction and pixel confidence update.

[0043] 42) During the parameter iterative optimization process, it is necessary to continuously and correctly update the optical flow change and pixel confidence weights to ensure that the backend parameter joint calibration algorithm continuously solves for the optimal parameter values ​​based on the camera intrinsic parameters g. k Camera pose p k Depth d k Calculate image I using equation (4) i medium pixel u i To image I j medium pixel u ij :

[0044] u ij =π(p ij ·π -1 (u i ,d i ,g),g) (4)

[0045] In the formula, p ij =p j ·p i -1 The relative pose of the camera between two frames; π(·) represents the mapping relationship between the camera's 3D world coordinates and 2D pixel coordinates; π -1 (·) represents the inverse projection process from 2D pixel coordinates to 3D world coordinates. The dashcam is modeled as a unified camera model and a distortion factor α is introduced, so that the algorithm can be applied to monocular videos with different camera types and different degrees of distortion during the parameter calibration process, as shown in Equation (5):

[0046]

[0047] In the formula, X(x,y,z)∈R 3 α represents 3D world coordinates; f represents distortion factor; f represents 3D world coordinates; α represents distortion factor; f represents 3D world coordinates ... x f y Let be the camera focal length; u0 and v0 be the camera principal points, and

[0048] In each iteration, context information, optical flow, and image I... j medium pixel u ij The relevant information is then fed into a spatiotemporal convolutional network, which integrates the information temporally and uses different convolutional layers to predict the change in optical flow Δu. ij and the corresponding confidence weight w ij The optical flow variation values ​​are used to select keyframes with sufficient motion in the video as the input for the front-end factor map construction. Simultaneously, the optical flow variation values ​​and pixel confidence weights are used in the back-end factor map optimization process. Furthermore, in each iteration, the spatiotemporal convolutional network converts the input hidden state h... ij k Updated to h ij k+1 ;

[0049] 43) The joint parameter calibration process is transformed into a graph optimization problem by constructing a graph model. The optimal parameter values ​​are continuously solved using the iterative optimization process of the factor graph. The factor graph is a directed acyclic graph containing a series of nodes and factors. Nodes represent the variable values ​​of the parameters to be calibrated, and factors f... ij Represents a pair of images (I i ,I j The error correlation term, i.e. the constraint relationship between calibration parameters;

[0050] The factor graph construction process in the front-end process is as follows: use the feature information of key image frames to add edges of adjacent frames and distance-based edges to the factor graph and update the factor graph. Then remove redundant edges in the factor graph and update the factor graph again to obtain the factor graph for back-end optimization.

[0051] 44) Integrate Geometrically Salient Feature Bundle Adjustment (GSFBA) into the back-end factor graph optimization process. Use the saliency of feature points to represent the geometric gaze reality and improve the cost function of reprojection error during iterative optimization, as detailed below:

[0052] Extract edge feature points and feature lines from geometric feature regions in an image, and perform mask filling at the locations of feature points and feature lines to obtain a single-frame image I. iThe corresponding initial saliency image SI i Equation (6) is used to calculate the pixel depth d. i Corrected saliency image S i :

[0053]

[0054] In the formula, a and b are constant correction factors set to avoid insufficient feature information due to small scene saliency areas, which would lead to failure of the back-end parameter optimization and calibration process.

[0055] Factor graph constructed using the front-end process, and flow correction prediction value Δu ij Pixel confidence weight w ij and the parameter g from the previous iteration k p k d k As input, the backend process obtains the updated optimal parameter value g by minimizing the cost function E(g,p,d). k+1 p k+1 d k+1 The cost function represents the sum of the reprojection errors of all keyframe pixels, and the saliency image S i The saliency of the pixels obtained after normalization i It is incorporated into the cost function, as shown in equation (7):

[0056]

[0057] In the formula, i and j are the indices of the image keyframes; ∑ ij =diag(wij); f ij The error correlation term of the factor is calculated by equation (8):

[0058] f ij =u ij -π(p ij ·π -1 (u i ,d i ,g),g)(8)

[0059] In the formula, u ij =u ij +Δu ij These are the pixel coordinates predicted after optical flow correction;

[0060] The optimization objective of the backend process is shown in equation (9), which is to obtain the parameter values ​​to be calibrated through a joint calibration process of the parameters:

[0061]

[0062] In the formula, g, p, and d are the optimized parameter values ​​in the joint calibration process;

[0063] The backend process uses the classic Gauss-Newton steps as the optimization method, as shown in equation (10):

[0064] J T WJΔξ=J T Wf(10)

[0065] In the formula, f is the optimized residual from equation (8); J is the Jacobian matrix of the residual; W is the diagonal matrix composed of the confidence weights of all images; Δξ=(Δg,Δp,Δd) is the update amount of the calibration parameters.

[0066] Furthermore, the specific method for step five is as follows:

[0067] 51) Using equation (11) to set the pixel coordinates P i (u,v) is mapped to world coordinate point P. w (X w ,Y w Z w This involves mapping the pixel trajectories of key traffic participants and the pixel contours of roads to the world coordinate system.

[0068]

[0069] In the formula, R and T are the camera's rotation matrix and translation vector, respectively; d i For the corresponding pixel point P i The depth value.

[0070] Furthermore, the specific method for step six is ​​as follows:

[0071] 61) Design a road contour extraction method that integrates multi-frame information and is based on pixel road boundary mapping intensity, i.e., after passing through the inverse perspective transformation module, it can obtain the coordinate points w of the multi-frame road contour in world coordinates. i (i = 1, 2, ..., n), after excluding outliers, the w of all frames i The pixel road boundary mapping intensity distribution is constructed by mapping the information of multiple frames to the same coordinate system. The intensity value represents the probability of the pixel road boundary being mapped to different coordinates. The coordinate with the larger intensity value has a higher confidence in the actual road contour. The mapping intensity distribution map is then decomposed into several independent sub-regions as candidate regions for discrete road contour extraction. For each candidate region, the density of coordinate points is calculated along the x and y directions. Based on the density curves in the two directions, the road contour boundary line with the highest confidence in the candidate region is extracted. The SG (Savitzky-Golay) algorithm is used to fit the extracted contour to obtain a complete and accurate road contour.

[0072] 62) Design a systematic trajectory data post-processing workflow, the first step of which is missing trajectory prediction and fluctuating trajectory smoothing, using discrete world coordinate points w t Based on this, Fourier series fitting is performed, and the KD-Tree algorithm is used to globally search and match the trajectories of the preceding and following breakpoints. The Kalman algorithm is used to predict the missing trajectories. The algorithm includes two parts: prediction and update. For the trajectories with observations, the algorithm performs the complete process of prediction and update. For the missing trajectories without observations, the algorithm only predicts the state without performing the update process. The predicted value is used as the optimal trajectory point at that moment. After obtaining the complete time-series trajectory data of traffic participants, the SG algorithm is used to fit the time-series trajectory to further enhance its ability to represent the time-series trend of the real trajectory.

[0073] Next is the abnormal trajectory constraint process. Based on the basic fact that the vehicle is driving within the road contour constraint area, the road contour is used to correct and constrain the vehicle's driving trajectory. A rectangular bounding box with a width of w and a length of l is added along the tangent direction of the trajectory with the vehicle's time-series trajectory point as the center to represent the vehicle size. The width w and length l can be estimated according to different vehicle types. In actual driving, the rectangular bounding box should not exceed the road boundary. The trajectory points that exceed the boundary of the rectangular box are corrected along the original tangent direction towards the inside of the road boundary.

[0074] This completes the entire process of reconstructing the accident scene, obtaining spatiotemporal information of the scene, including the trajectories of traffic participants and the road outline.

[0075] The beneficial effects of this invention are as follows:

[0076] The vehicle-side accident scene reconstruction method based on dashcam video provided by this invention takes the joint calibration strategy of inverse perspective mapping parameters as its core. It is designed for vehicle-side accident videos with unknown and different camera parameters and lack of multi-source synchronization information. It can extract scene spatiotemporal information, including road contours and the trajectories of traffic participants, without any prior information support. This allows the establishment of diverse accident scene data with significant long-tail characteristics and the ability to show real interaction features. It significantly reduces the cost of accident scene acquisition while significantly improving the efficiency of accident scene acquisition, and can effectively accelerate the testing process of autonomous vehicles. Attached Figure Description

[0077] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 This is a flowchart of the present invention;

[0079] Figure 2 The flowchart shows the joint calibration algorithm for vehicle-side dynamic scene parameters based on GSFBA.

[0080] Figure 3 Factor plot for the joint calibration process of parameters;

[0081] Figure 4 Flowchart for post-processing road outline and trajectory data;

[0082] Figure 5 A schematic diagram of temporal trajectory correction considering road contour constraints;

[0083] Figure 6 This is a global information display diagram for a simulated accident scenario;

[0084] Figure 7a This is a pixel road boundary mapping intensity distribution map of sub-region 1 in the world coordinate system;

[0085] Figure 7b This is a pixel road boundary mapping intensity distribution map of sub-region 2 in the world coordinate system;

[0086] Figure 7c This is a pixel road boundary mapping intensity distribution map of sub-region 3 in the world coordinate system;

[0087] Figure 7d This is a pixel road boundary mapping intensity distribution map of sub-region 4 in the world coordinate system;

[0088] Figure 8 Extracting the rendering of the road outline in the scene;

[0089] Figure 9 Image showing the effect of target detection and tracking for traffic participants;

[0090] Figure 10a Extract a comparison image of the trajectory and the actual trajectory for Vehicle1;

[0091] Figure 10b Extract a comparison image of the trajectory and the actual trajectory for Vehicle2;

[0092] Figure 10c Extract a comparison image of the trajectory and the actual trajectory for Vehicle3;

[0093] Figure 11 Extract a comparison image of the camera's trajectory and the actual trajectory;

[0094] Figure 12a Image showing the result of extracting speed information for Vehicle 0;

[0095] Figure 12b Image showing the result of extracting speed information for Vehicle1;

[0096] Figure 12c Image showing the result of extracting speed information for Vehicle2;

[0097] Figure 12d Image showing the result of extracting speed information for Vehicle3;

[0098] Figure 13a This is a comparison chart of information extracted by the algorithm and real information in a real vehicle scenario;

[0099] Figure 13b A comparison chart of the extracted speed information for Vehicle 1 and the actual speed information;

[0100] Figure 13c A comparison chart of extracted speed information for Vehicle2 and actual speed information;

[0101] Figure 14a This is a schematic diagram of accident scenario 1;

[0102] Figure 14b This is a schematic diagram of accident scenario 2;

[0103] Figure 15a This is a keyframe image of accident scenario 1;

[0104] Figure 15b This is a keyframe detection result image for accident scenario 1;

[0105] Figure 15c This is a result image showing the global road contour information and key trajectory points of traffic participants for accident scenario 1.

[0106] Figure 16a Keyframes for accident scenario 2;

[0107] Figure 16b This is a keyframe detection result image for accident scenario 2;

[0108] Figure 16c This is a result diagram showing the global road contour information and key trajectory points of traffic participants for accident scenario 2. Detailed Implementation

[0109] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0110] Example 1

[0111] See Figure 1 This embodiment provides a method for reconstructing accident scenes from a vehicle's perspective based on dashcam video, including the following steps:

[0112] Step 1: Preprocess the accident video;

[0113] The video is denoised using an efficient multi-stage video denoising (EMVD) algorithm, and enhanced by frame interpolation using a real-time intermediate flow estimation (RIFE) algorithm, as detailed below:

[0114] 11) The video is processed using an efficient multi-stage video denoising (EMVD) algorithm. The input video is denoised in a high-quality manner through three processing stages: temporal fusion, spatial denoising, and spatiotemporal refinement.

[0115] 12) The video is enhanced by using the Real-time Intermediate Flow Estimation (RIFE) algorithm. By outputting an approximate intermediate flow and then performing backward deformation and fusion, the intermediate frame image is accurately predicted, thereby improving the smoothness of the input video.

[0116] Step 2: Perform semantic understanding of the scene;

[0117] Using the DeepLabV3Plus algorithm as the semantic feature extractor, the video is processed frame by frame to obtain multi-frame semantic images of the scene. Multi-frame pixel road boundaries are extracted using an edge detection algorithm, and based on this, the target detection algorithm's main focus area and background area are divided. Furthermore, based on the semantic information, a motion consistency check algorithm is used to obtain dynamic regions and dynamic semantic masks, as detailed below:

[0118] 21) The DeepLabV3Plus algorithm, which incorporates dilated convolution structures, is selected as the semantic feature extractor;

[0119] 22) The video is processed frame by frame by a feature extractor to obtain multi-frame semantic images of the scene. Multi-frame pixel road boundaries are extracted by an edge detection algorithm. The pixel road boundaries are used as the boundary to divide the main focus area of ​​the target detection algorithm and the background area.

[0120] 23) A motion consistency check algorithm is used in conjunction with semantic information to obtain the semantic mask of the dynamic region of the image. Different traffic participants will be assigned different colors in the semantic information. The outline of the traffic participants is regarded as the region to be checked for motion. First, the feature points in the region to be checked for motion in the first frame image are detected. The optical flow pyramid is calculated to obtain the feature points in the region to be checked for motion in the second frame that match the first frame. The Random Sample Consensus (RANSAC) algorithm is used to calculate the fundamental matrix and epipolar line between the feature points, and the distance of the matching feature point from the epipolar line is calculated. If it is greater than the set threshold, it indicates that the feature point is moving. If the moving feature points in a certain region to be checked for motion are greater than a certain percentage threshold of the total number of feature points, it is regarded as a moving region.

[0121] The fundamental matrix is ​​used to map the feature points in the first frame to the corresponding search domain, i.e., the epipolar line, in the second frame. Assuming that p1 and p2 represent the matching feature points in the first and second frames, respectively, they are expressed in homogeneous coordinate form as shown in equation (1):

[0122] p1=(u1,v1,1),p2=(u2,v2,1) (1)

[0123] In the formula, u and v are pixel values ​​in the image;

[0124] Then, the polar line I can be calculated using equation (2). m :

[0125] I m =(x m ,y m ,z m ) T =F m p1 (2)

[0126] In the formula, x m y m z m F is a line vector; m The basic matrix;

[0127] Then, the distance D from the matching feature point to the corresponding epipolar line can be calculated using equation (3). m :

[0128]

[0129] In the formula, D m To match the distance from the feature point to the corresponding epipolar line.

[0130] Step 3: Extract the pixel trajectories of traffic participants;

[0131] By combining the improved YOLOv8 algorithm with the DeepSORT algorithm, and robustly identifying key traffic participants, frame-by-frame correlation is performed to achieve accurate extraction of traffic participant pixel trajectories, as detailed below:

[0132] 31) Based on the YOLOV8 algorithm, a global attention mechanism (GAM) is introduced to optimize the backbone network of the YOLOV8 algorithm, reduce information loss and enhance the global interactive representation of information. The ASPP (Atrous Spatial Pyramid Pooling) module is used to optimize the feature pyramid of the algorithm, and the model's generalization ability for multi-scale features of the target is enhanced by using convolutional kernels with different receptive fields.

[0133] Using this as the target detector, prediction and generation are performed based on feature extraction to obtain the traffic participant detection results. At the same time, the main regions of interest obtained from scene semantic understanding are incorporated into the algorithm detection process to eliminate invalid background interference in the scene.

[0134] 32) Based on the target detection algorithm, the DeepSORT algorithm is used to generate trajectories after estimating the motion of traffic participants, thereby obtaining data association results. That is, the cross-frame detection results of the same vehicle are associated and assigned a unique ID, thus forming continuous pixel trajectory data of traffic participants.

[0135] Step 4: Perform joint calibration of the inverse perspective mapping parameters;

[0136] See Figure 2 This invention designs a joint calibration algorithm for inverse perspective mapping parameters in vehicle-side dynamic scenes based on Geometrically Salient Feature Bundle Adjustment (GSFBA). First, feature information is extracted from the image. Then, optical flow correction prediction and confidence updates are performed to obtain key image frames for front-end factor map construction and intermediate quantities for back-end factor map optimization. Subsequently, the factor map is initialized and constructed in the front-end process. Finally, joint iterative calibration of parameters is achieved in the back-end process using Geometrically Salient Feature Bundle Adjustment, as detailed below:

[0137] 41) Obtain the dynamic semantic mask of the image through scene semantic understanding, and use the masked image to represent P = (I i ,I jAs input, the feature encoder obtains the feature information of adjacent frames and performs correlation calculation. High correlation represents similar regions in the image and obtains the initial optical flow. The correlation encoder and optical flow encoder encode the two, and at the same time, the context encoder obtains the context information for subsequent optical flow correction prediction and pixel confidence update.

[0138] 42) During the parameter iterative optimization process, it is necessary to continuously and correctly update the optical flow change and pixel confidence weights to ensure that the backend parameter joint calibration algorithm continuously solves for the optimal parameter values ​​based on the camera intrinsic parameters g obtained from the previous iteration. k Camera pose p k Depth d k Calculate image I using equation (4) i medium pixel u i To image I j medium pixel u ij :

[0139] u ij =π(p ij ·π -1 (u i ,d i ,g),g) (4)

[0140] In the formula, p ij =p j ·p i -1 The relative pose of the camera between two frames; π(·) represents the mapping relationship between the camera's 3D world coordinates and 2D pixel coordinates; π -1 (·) represents the inverse projection process from 2D pixel coordinates to 3D world coordinates. This invention models the dashcam as a unified camera model and introduces a distortion factor α, so that the algorithm can be applied to monocular videos with different camera types and different degrees of distortion during parameter calibration, as shown in equation (5):

[0141]

[0142] In the formula, X(x,y,z)∈R 3 α represents 3D world coordinates; f represents distortion factor; f represents 3D world coordinates; α represents distortion factor; f represents 3D world coordinates ... x f y Let be the camera focal length; u0 and v0 be the camera principal points, and

[0143] In each iteration, context information, optical flow, and image I... j medium pixel u ij The relevant information is fed into a spatiotemporal convolutional network, which integrates the information temporally and uses different convolutional layers to predict the change in optical flow Δu. ijand the corresponding confidence weight w ij The optical flow variation values ​​are used to select keyframes with sufficient motion in the video as the input for the front-end factor map construction. Simultaneously, the optical flow variation values ​​and pixel confidence weights are used in the back-end factor map optimization process. Furthermore, in each iteration, the spatiotemporal convolutional network converts the input hidden state h... ij k Updated to h ij k+1 ;

[0144] 43) This invention transforms the joint parameter calibration process into a graph optimization problem by constructing a graph model. It continuously solves for the optimal parameter values ​​using an iterative optimization process of the factor graph. The factor graph is a directed acyclic graph containing a series of nodes and factors. Nodes represent the variable values ​​of the parameters to be calibrated, and factors f... ij Represents a pair of images (I i ,I j The error correlation terms, i.e., the constraint relationships between calibration parameters, can be represented by a factor plot. Figure 3 ;

[0145] The factor graph construction process in the front-end process is as follows: the feature information of key image frames is used to add edges of adjacent frames and distance-based edges to the factor graph and update the factor graph. Then, redundant edges in the factor graph are removed and the factor graph is updated again to obtain the factor graph for back-end optimization.

[0146] 44) This invention integrates geometric saliency feature bundle adjustment into the back-end factor map optimization process, using the saliency of feature points to express the geometric gaze reality and improve the cost function of reprojection error in the optimization process. Specifically, it mines edge feature points and feature lines of geometric feature regions in the image and performs mask filling in the vicinity to obtain a single-frame image I. i The corresponding initial saliency image SI i Considering the problems of decreased sharpness and loss of geometric feature details due to increased noise in distant image parts, equation (6) is used to calculate the pixel depth d. i Corrected saliency image S i :

[0147]

[0148] In the formula, a and b are constant correction factors set to avoid insufficient feature information due to small scene saliency areas, which would lead to failure of the back-end parameter optimization and calibration process.

[0149] Factor graph constructed using the front-end process, and flow correction prediction value Δu ij Pixel confidence weight w ij and the parameter g from the previous iteration k pk d k As input, the backend process obtains the updated optimal parameter value g by minimizing the cost function E(g,p,d). k+1 p k+1 d k+1 The cost function represents the sum of reprojection errors for all keyframe pixels. This invention uses the saliency image S... i The saliency of the pixels obtained after normalization i It is incorporated into the cost function, as shown in equation (7):

[0150]

[0151] In the formula, i and j are the indices of the image keyframes; ∑ ij =diag(w ij ), f ij The error correlation term of the factor is calculated by equation (8):

[0152] f ij =u ij -π(p ij ·π -1 (u i ,d i ,g),g) (8)

[0153] In the formula, u ij =u ij +Δu ij These are the pixel coordinates predicted after optical flow correction;

[0154] The optimization objective of the backend process is shown in equation (9), which is to obtain the parameter values ​​to be calibrated through a joint calibration process of the parameters:

[0155]

[0156] In the formula, g, p, and d are the optimized parameter values ​​in the joint calibration process;

[0157] The backend process uses the classic Gauss-Newton steps as the optimization method, so that the optimization process can be carried out in a differentiable and efficient manner, as shown in equation (10):

[0158] J T WJΔξ=J T Wf(10)

[0159] In the formula, f is the optimized residual from equation (8); J is the Jacobian matrix of the residual; W is the diagonal matrix composed of the confidence weights of all images; Δξ=(Δg,Δp,Δd) is the update amount of the calibration parameters.

[0160] Step 5: Perform inverse perspective transformation;

[0161] The pixel trajectories of key traffic participants and the pixel contours of roads are mapped to the world coordinate system for subsequent scene data processing, as follows:

[0162] 51) Using equation (11) to set the pixel coordinates P i (u,v) is mapped to world coordinate point P. w (X w ,Y w Z w This involves mapping the pixel trajectories of key traffic participants and the pixel contours of roads to the world coordinate system.

[0163]

[0164] In the formula, R and T are the camera's rotation matrix and translation vector, respectively, and d i This represents the pixel P. i The depth value.

[0165] Step 6: Post-process the road outline and trajectory data;

[0166] The design of a road contour extraction method based on pixel road boundary mapping intensity and a systematic trajectory data post-processing workflow, including missing trajectory prediction, fluctuating trajectory smoothing, and abnormal trajectory constraint, further improves data quality. (See also...) Figure 4 To ensure the accuracy and applicability of key information extracted from the scene, the following measures are taken:

[0167] 61) This invention designs a road contour extraction method that integrates multi-frame information and is based on pixel road boundary mapping intensity. Specifically, after passing through the inverse perspective transformation module, it is possible to obtain the coordinate points w of the multi-frame road contour in world coordinates. i (i = 1, 2, ..., n), after excluding outliers, the w of all frames i The pixel road boundary mapping intensity distribution is constructed by mapping the information of multiple frames to the same coordinate system. The intensity value represents the probability of the pixel road boundary being mapped to different coordinates. The coordinate with the larger intensity value has a higher confidence in the actual road contour. The mapping intensity distribution map is then decomposed into several independent sub-regions as candidate regions for discrete road contour extraction. For each candidate region, the density of coordinate points is calculated along the x and y directions. Based on the density curves in the two directions, the road contour boundary line with the highest confidence in the candidate region is extracted. The SG (Savitzky Golay) algorithm is used to fit the extracted contour to obtain a complete and accurate road contour.

[0168] 62) This invention designs a systematic trajectory data post-processing workflow, the first of which is the missing trajectory prediction and fluctuating trajectory smoothing process, using discrete world coordinate points wt Based on this, Fourier series fitting is performed, and the KD-Tree algorithm is used to globally search and match the trajectories of the preceding and following breakpoints. The Kalman algorithm is used to predict the missing trajectories. The algorithm includes two parts: prediction and update. For the trajectories with observations, the algorithm performs the complete process of prediction and update. For the missing trajectories without observations, the algorithm only predicts the state without performing the update process. The predicted value is used as the optimal trajectory point at that moment. After obtaining the complete time-series trajectory data of traffic participants, the SG algorithm is used to fit the time-series trajectory to further enhance its ability to represent the time-series trend of the real trajectory.

[0169] This leads to the abnormal trajectory constraint process, which can be found in [reference needed]. Figure 5 Based on the fundamental fact that vehicles travel within the road contour constraint area, the road contour is used to correct and constrain the vehicle's trajectory. A rectangular bounding box with a width of w and a length of l is added along the tangent direction of the trajectory, centered on the vehicle's time-series trajectory point, to represent the vehicle size. The width w and length l can be estimated according to different vehicle types. In actual driving, the rectangular bounding box should not exceed the road boundary. The trajectory points that exceed the boundary are corrected along the original tangent direction towards the inside of the road boundary.

[0170] This completes the entire process of reconstructing the accident scene, obtaining spatiotemporal information of the scene, including the trajectories of traffic participants and the road outline.

[0171] Example 2

[0172] This invention conducts comprehensive experimental verification of the method through a simulated accident scenario and a real vehicle platform data acquisition scenario, and applies the method to the reconstruction process of two actual accident scenarios.

[0173] I. Experimental verification preparation;

[0174] The DeepLabV3Plus model was trained on the CityScapes and VirtualKITTI datasets using a GeForce RTX 4090D GPU, and the joint parameter calibration algorithm was trained end-to-end on the TartanAir dataset. The depth loss shown in Equation (12) was added to the optical flow loss, pose loss, intrinsic parameter loss, and residual loss to supervise the algorithm training process.

[0175]

[0176] In the formula, For calibration depth, d is the actual depth, and γ is a fixed weighting coefficient set to 0.9;

[0177] II. Simulation scenario experimental verification;

[0178] This invention utilizes the Carla simulator to construct an accident scene that simulates real-world accident scenarios and is stored in the form of monocular video from a dashcam, while also possessing truth-based information. (See also...) Figure 6 The scene lasts for about 8 seconds, and the temporal dynamic process of the trajectory is as follows: Figure 6 The trajectory points are represented by their colors. This scenario occurs at an intersection and involves four vehicles. Vehicle 1, while turning near the intersection, collides with Vehicle 3, which suddenly appears, resulting in a traffic accident. Vehicle 0, equipped with a dashcam, records the dynamic process of the accident from a monocular camera perspective. Vehicle 2 continues to travel straight, with a speed between 0-50 km / h. Based on this dynamic scenario, the accuracy of road contour extraction and traffic participant trajectory extraction are experimentally verified. During the verification process, the monocular video recorded by the camera is used as input. Camera parameters and global information of the scene are only used for experimental accuracy comparison and do not participate in the verification process of the proposed method. It should also be noted that the accident scenario constructed in the Carla simulation environment is mainly used for accuracy verification of the proposed algorithm and does not focus on the basic rules that vehicles should follow for normal driving.

[0179] 1. Verification of road contour extraction accuracy;

[0180] For the pixel road boundary mapping intensity distribution in the world coordinate system, please refer to [reference needed]. Figures 7a-7d As shown in the figure, the intensity distribution of the road contour mapping in the world coordinate system is concentrated in four different sub-regions. Coordinates with darker colors and larger density curve values ​​have higher confidence in the actual road contour. The trends of the road contour curves in each sub-region are obvious, and the contour lines extracted based on the horizontal and vertical density curves are reasonable. To further verify the road contour extraction effect, the extracted road contour lines and the actual road contour lines can be compared to [reference needed]. Figure 8 ,Depend on Figure 8 It can be seen that the algorithm can extract the road contour information of each sub-region near the vehicle interaction area relatively accurately. It should be noted that the algorithm extracts the contours of sub-region 1 and sub-region 2 as a single curve. This is because the multi-level propagation of errors makes it difficult to decompose the mapping intensity distribution of this small area into discrete sub-regions. The pixel mapping area tends to concentrate, causing the contour extraction result to tend towards the median line. However, it can still reasonably describe the basic information of the road contour, proving the effectiveness of the method in extracting road contour information. It should be pointed out that... Figure 8The image shown is the true contour from a global perspective. This invention focuses only on the vehicle interaction area, and due to the limitations of the camera's field of view, it can only extract the road contour within the camera's field of view. This is not a limitation of the method. Therefore, the root mean square error (RMSE) is used to quantify and evaluate the accuracy of the road contour extraction. For each extracted road contour point, the RMSE is calculated based on the distance between it and the real road contour point with the smallest distance. The average RMSE near the vehicle interaction area is only 1.53m. The quantification results show that the extracted contour can accurately match the real contour, verifying the accuracy of the method of this invention in extracting road contour information from the vehicle's perspective.

[0181] 2. Accuracy of traffic participant trajectory extraction;

[0182] Accurate identification and stable tracking of traffic participants are necessary prerequisites for trajectory information extraction. Therefore, the effectiveness of the target detection and tracking algorithm is first tested. The target detection and tracking results of traffic participants in scene keyframes can be found in [reference needed]. Figure 9 ,Depend on Figure 9 It can be seen that the algorithm can accurately identify traffic participants and assign them IDs to achieve stable tracking. It performs well when facing multi-scale changes in traffic participants. Furthermore, the scene semantic understanding module eliminates the interference of image background areas on the algorithm, enabling the algorithm to focus more on vehicles on the road and effectively improve the ability to extract key scene information.

[0183] Then, the accuracy of trajectory information extraction was verified. The trajectory after inverse perspective transformation and Fourier series fitting, and the trajectory curve optimized by the systematic data post-processing workflow, can be referred to. Figures 10a-10c The movement trajectory of Vehicle 0, which carries a camera, can be viewed. Figure 11 ,Depend on Figures 10a-10c It can be seen that the trajectory obtained by inverse perspective transformation and Fourier series fitting can initially describe the temporal trend of the true trajectory. Figure 10c It can be seen that the KD-Tree algorithm can match the broken trajectory caused by ID switching well. Furthermore, the trajectory improved by the data post-processing workflow can achieve a good matching effect with the real trajectory. It can accurately repair trajectory missing points caused by occlusion, missed detection, etc. Trajectory fitting and trajectory constraints can further optimize trajectory quality. Figure 10b and Figure 10c Both trajectories contain partial missing parts, but the causes are different. Figure 10b The missing trajectory is due to the saturation value of the depth value at a distance during the parameter calibration process. This saturation, limited by the inaccuracy of the depth value, makes it difficult to accurately map the trajectory of distant pixels. Furthermore, the closer the depth value is to the desired depth... Figure 10b For long-distance trajectory segments in the medium range (-40m, -70m), the trajectory error increases significantly, while for... Figure 10c The missing portion of the trajectory was due to ID loss, which can be predicted and repaired using the Kalman algorithm. To further quantify the trajectory information, the RMSE, which is calculated using the same method as road contour quantization, was used to evaluate the accuracy of traffic participant trajectory extraction. The RMSE quantization results for Vehicle 0, Vehicle 1, Vehicle 2, and Vehicle 3 were 0.21m, 0.78m, 0.68m, and 0.83m, respectively, with an average RMSE of only 0.50m, verifying the effectiveness and accuracy of the method of the present invention in extracting traffic participant trajectory information.

[0184] Depend on Figures 10a-10c , Figure 11 The error comparison curves and quantization results show that the trajectory accuracy of the vehicle with the camera is much higher than that of other vehicles. This is because the two vehicles extract their trajectories in different ways. For the vehicle with the camera, the pose estimation of the vehicle in the world coordinate system can be achieved during the parameter joint calibration process. Its accuracy depends on the effect of the parameter joint calibration algorithm. The trajectory of other vehicles is obtained by mapping the pixel coordinates to the world coordinates using calibration parameters. Both parameter calibration error and pixel coordinate error will affect the trajectory accuracy.

[0185] This invention uses trajectory information as a basis to verify speed information, further validating the applicability of reconstructed scene information in autonomous vehicle testing. The speed information extraction results can be found in [reference needed]. Figures 12a-12d ,Depend on Figures 12a-12d It can be seen that the speed curves calculated from the reconstructed time-series trajectory points match the actual speed curves well. The RMSE-quantified speed error results for Vehicle0, Vehicle1, Vehicle2, and Vehicle3 are 0.17 m / s, 0.49 m / s, 0.37 m / s, and 0.77 m / s, respectively. The quantization results show that the reconstructed speed information is also highly accurate and sufficient for use in the testing process of autonomous vehicles.

[0186] III. Real-vehicle scenario experimental verification;

[0187] Based on simulation scenario verification, this invention further examines the practical application effect of the method by building a real vehicle platform to evaluate the performance of the method in a real vehicle scenario. This invention collects scene data in Changchun City, Jilin Province, and selects a dynamic scene of about 11 seconds. Vehicles 1 and 2 drive freely on a straight two-lane road at a speed between 10-30 km / h. Vehicle 1 records monocular video of Vehicle 2 driving ahead from the camera's perspective. The real vehicle platform can accurately acquire the real trajectories of the two vehicles and align and measure the road contour coordinates of the scene based on the vehicle positions, providing sufficient ground truth information for subsequent verification.

[0188] This invention converts WGS84 data acquired by RT devices into the UTM coordinate system and performs appropriate translation. For comparison of the road contours, traffic participant trajectories, and speed information extracted by this method in real-world vehicle scenarios with actual data, please refer to [link / reference]. Figures 13a-13c ;

[0189] Depend on Figures 13a-13c As can be seen, the method provided by this invention can also effectively extract key information in real vehicle scenarios. The RMSE trajectory error of vehicle 1 is 0.21m and the RMSE speed error is 0.13m / s. The RMSE trajectory error of vehicle 2 is 0.61m and the RMSE speed error is 0.32m / s. The RMSE error of the road contour is only 0.48m. The results show that it still has sufficient extraction accuracy in real vehicle scenarios. Thus, the extraction accuracy of key information in both simulation and real vehicle scenarios has been effectively verified.

[0190] IV. Reconstruction of actual accident scenarios;

[0191] This invention applies a vehicle-view accident scene reconstruction method in practice. Two accident scene videos are selected as input, and spatiotemporal information, including road contours and the trajectories of traffic participants, is extracted. An accident scene illustration can be found in the attached diagram. Figure 14a and 14b The scene information extraction results can be found separately. Figures 15a-15c and Figures 16a-16c ;

[0192] exist Figures 15a-15c , Figures 16a-16cThe keyframes before and after the accident, keyframe detection results, global road contour information, and key trajectory points of traffic participants were plotted. Due to the challenges of multi-scale changes in traffic participants and the limitations of the target detector's generalization ability, missed identification and misidentification are inevitable. The systematic trajectory post-processing process can effectively improve the above phenomena and achieve effective extraction of trajectory information. At the same time, the road contour extraction method can also depict the basic shape of the road, indicating that the key scene information extracted by the method of this invention has sufficient applicability.

[0193] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for restoring a vehicle-end perspective accident scene based on a vehicle event data recorder video, characterized in that, Comprise the following steps: Step one, pre-processing of accident video; Utilize high-efficiency multi-stage noise reduction algorithm to reduce noise of video, utilize real-time intermediate stream estimation algorithm to enhance video by frame interpolation; Step two, semantic understanding of scene; Take DeepLabV3Plus algorithm as semantic feature extractor, process video frame by frame to obtain multi-frame semantic image of scene, extract multi-frame pixel road boundary by edge detection algorithm, and divide main attention area and background area of target detection algorithm based on the pixel road boundary, and obtain dynamic area and dynamic semantic mask by motion consistency checking algorithm based on semantic information; Step three, extraction of pixel trajectory of traffic participants; Through the coordination between improved YOLOV8 algorithm and DeepSORT algorithm, the pixel trajectory of traffic participants is accurately extracted by frame-by-frame association based on the robust identification of key traffic participants; Step four, joint calibration of inverse perspective mapping parameters; Design a joint calibration algorithm of inverse perspective mapping parameters of dynamic scene based on geometric saliency feature bundle adjustment, first extract the feature information of the image, then perform optical flow correction prediction and confidence update to obtain the key image frames for front-end factor graph construction and the intermediate quantities for back-end factor graph optimization, then initialize and construct the factor graph in the front-end process, and finally realize the joint iterative calibration of parameters by geometric saliency feature bundle adjustment in the back-end process; Step five, inverse perspective transformation; Map the pixel trajectory of key traffic participants and the pixel contour of road to the world coordinate system for subsequent scene data processing; Step six, post-processing of road contour and trajectory data; Design a road contour extraction method based on pixel road boundary mapping intensity and a systematic trajectory data post-processing process including missing trajectory prediction, fluctuation trajectory smoothing, and abnormal trajectory constraint to further improve data quality and ensure the accuracy and applicability of key information extraction of the scene; The specific method of step four is as follows: 41) Obtain the dynamic semantic mask of the image by scene semantic understanding, and input the image masked by the mask P=(I i ,I j ) into the feature encoder to obtain the feature information of the adjacent frame and perform correlation calculation, and obtain the initial optical flow, encode the correlation and the optical flow by using the correlation encoder and the optical flow encoder, and obtain the context information by using the context encoder, which is used for subsequent optical flow correction prediction and pixel point confidence update; 42) The light flow change and pixel confidence weight need to be updated correctly during the parameter iteration optimization process to ensure that the backend parameter joint calibration algorithm is constantly solving the optimal parameter value, according to the camera intrinsic g k , camera pose p k , depth d k The pixel point u i in the image I i is calculated by formula (4) j The pixel point u ij = π(p ij · π -1 (u i , d i , g), g) (4) where p ij = p j · p i -1 is the relative pose of the camera between two frames; π(·) is the mapping from 3D world coordinates to 2D pixel coordinates; π -1 (·) is the inverse projection from 2D pixel coordinates to 3D world coordinates. The dash cam is modeled as a unified camera model with a distortion factor a, so that the algorithm can be applied to different camera types and different degrees of distortion in the parameter calibration process, as shown in equation (5): where X(x, y, z) e R 3 are 3D world coordinates; a is a distortion factor; f x , f y is the camera focal length; u0, v0are the camera principal points, and Context information, optical flow, image I j Middle pixel point and correlation are input into the spatio-temporal convolution network, which integrates information in time sequence and uses different convolution layers to predict the change value△u ij and the corresponding confidence weight w ij of the optical flow. The change value of the optical flow is used to select the key frame with sufficient motion in the video as the input part of the front-end factor graph construction. At the same time, the change value of the optical flow and the confidence weight of the pixel point are used for the back-end factor graph optimization process. In addition, the input hidden state h ij k of the spatio-temporal convolution network is updated to h ij k+1 in each iteration process. 43) converting the parameter joint calibration process into a graph optimization problem by means of constructing a graph model, solving the optimal parameter values iteratively by means of an iterative optimization process of a factor graph, the factor graph being a directed acyclic graph comprising a series of nodes and factors, the nodes representing variable values of the parameters to be calibrated, the factors f ij representing error correlation terms of a pair of images (I i ,I j ), i.e. constraint relations between the calibration parameters; The factor graph construction process in the front-end process is as follows: add the edges of adjacent frames and the distance-based edges to the factor graph using the feature information of the key image frames, and update the factor graph, then remove the redundant edges in the factor graph and update the factor graph again, to obtain the factor graph for back-end optimization; 44) Integrate geometric saliency feature bundle adjustment into the back-end factor graph optimization process, express geometric gaze live by the saliency of feature points, and improve the cost function of re-projection error in the iterative optimization process, as follows: Edge feature points and feature lines in a geometric feature region in the image are mined, and mask filling is performed at the positions of the feature points and the feature lines to obtain a single-frame image I i A corresponding initial saliency image SI i , and the pixel depth d i The modified saliency image S i : Where a and b are constant correction factors set to avoid the smallness of scene saliency area, which leads to insufficient feature information and thus invalidates the back-end parameter optimization and calibration process; Factor graph constructed by front-end process, flow correction prediction value △u ij , pixel point confidence weight w ij And last iteration parameter g k , p k , d k As input, the back-end process obtains updated optimal parameter values g k+1 , p k+1 , d k+1 By minimizing the cost function E(g, p, d), the cost function represents the sum of all key frame pixel point re-projection errors, and the pixel point saliency s i After normalization, the saliency image S i Is integrated into the cost function, as shown in equation (7): In the formula, i, j are indexes of image key frames; ∑ ij = diag(w ij ); f ij is an error correlation term of the factor, calculated by formula (8): f ij = u ij - π(p ij · π -1 (u i , d i , g), g) (8) In the formula, is the pixel coordinate value after the optical flow correction prediction; The optimization objective of the back-end process is as shown in equation (9), and the parameter value to be calibrated is obtained through the joint calibration process of parameters: In the formula, are optimized parameter values in the joint calibration process; The back-end process adopts the classic Gauss-Newton step as the optimization method, as shown in equation (10): J T WJ△ξ=J T Wf (10) In the formula, f is an optimized residual from formula (8); J is a Jacobian matrix of the residual; W is a diagonal matrix composed of all image confidence weights; and △ξ=(△g,△p,△d) is an update amount of the calibration parameter.

2. The method of claim 1, wherein the method further comprises: The specific method of step one is as follows: 11) A high-efficiency multi-stage noise reduction algorithm is used to process the video, and through three processing stages of time fusion, spatial denoising and temporal and spatial refinement, the input video is denoised; 12) A real-time intermediate flow estimation algorithm is used to enhance the video by frame interpolation, and the precise prediction of the intermediate frame image is realized through backward deformation and fusion after outputting the approximate intermediate flow.

3. The method of claim 1, wherein the method further comprises: The specific method of step two is as follows: 21) The DeepLabV3Plus algorithm with a hollow convolution structure is selected as a semantic feature extractor; 22) The video is processed frame by frame by the feature extractor to obtain multiple frames of semantic images of the scene, and the pixel road boundary is extracted by an edge detection algorithm, and the main attention area and the background area of the target detection algorithm are divided by taking the pixel road boundary as the boundary; 23) A motion consistency checking algorithm is used to obtain the semantic mask of the dynamic area of the image in cooperation with the semantic information, and the process is as follows: different colors are assigned to different traffic participants in the semantic information, and the outlines of the traffic participants are regarded as candidate regions to be checked for motion, first, the feature points in the candidate regions to be checked for motion in the first frame image are detected, a flow pyramid is calculated to obtain the feature points in the second frame that match the first frame in the candidate regions to be checked for motion, a basic matrix and an epipolar line between the feature points are calculated by using a random sample consensus algorithm, and the distance of the matching feature points from the epipolar line is calculated, if the distance is greater than a certain threshold value, it indicates that the feature point is in motion, if the number of motion feature points in a certain candidate region to be checked for motion is greater than a certain percentage threshold value of the total number of feature points, the candidate region is regarded as a motion region, thus completing the task of obtaining the semantic mask of the dynamic area in the first frame image, and all frames of the video are processed. The basic matrix is used to map the feature points in the first frame to the corresponding search domain, i.e. the epipolar line, assuming that p1 and p2 are matching feature points in the first frame and the second frame, respectively, which are expressed in homogeneous coordinates as shown in formula (1): P1=(u1,v1,1),P2=(u2,v2,1) (1) In the formula, u and v are pixel values in the image. Further, the polar line I is calculated from the equation (2) m : I m = (x m ,y m ,z m ) T = F m P1 (2) where x m , y m , z m are line vectors; F m is a base matrix; Further, the distance D of the matching feature point to the corresponding epipolar line is calculated by formula (3) m : In the formula, D m is the distance of the matching feature point to the corresponding epipolar line.

4. The method of claim 1, wherein the method further comprises: The specific method of step three is as follows: 31) Based on the YOLOV8 algorithm, a global attention mechanism is introduced to optimize the backbone network of the YOLOV8 algorithm, and an ASPP module is used to optimize the algorithm feature pyramid, so as to obtain a target detector for the video; The target detector constructed by the above method is used to predict and generate traffic participant detection results based on feature extraction, and at the same time, the main attention area obtained by scene semantic understanding is integrated into the algorithm detection process to exclude invalid background interference in the scene; 32) Based on the target detection algorithm, the DeepSORT algorithm is used to generate a trajectory after motion estimation of the traffic participant, and then a data association result is obtained, i.e. the cross-frame detection results of the same vehicle are associated and assigned a unique ID, so as to form continuous pixel trajectory data of the traffic participant.

5. The method of claim 1, wherein the method further comprises: The specific method of the step five is as follows: 51) Map pixel coordinate point P i (u,v) to world coordinate point P w (X w ,Y w ,Z w ) using equation (11), i.e. map key traffic actor pixel trajectories and road pixel outlines to the world coordinate system: In the formula, R and T are respectively a rotation matrix and a translation vector of the camera; d i is a depth value corresponding to the pixel point P i .

6. The method of claim 1, wherein, The specific method of the step six is as follows: 61) A method of fusing multi-frame information and extracting road profile based on pixel road boundary mapping intensity, that is, obtaining multi-frame road profile coordinate points w in world coordinates after the inverse perspective transformation module i (i = 1, 2, …, n), after excluding outliers, mapping all frames of w i to the same coordinate system to construct the pixel road boundary mapping intensity distribution after fusing multi-frame information. The intensity value size represents the probability of the pixel road boundary mapping to different coordinates. The coordinate with a larger intensity value has a higher actual road profile confidence. Then, the mapping intensity distribution map is disassembled into several independent sub-regions as candidate regions for discrete road profile extraction. For each candidate region, the coordinate point density is calculated along the x and y directions. Based on the density curves in the two directions, the highest confidence road profile boundary line in the candidate region is extracted, and the SG algorithm is used to fit the extracted profile to obtain a complete and accurate road profile. 62) Design a systematic trajectory data post-processing flow, first is the missing trajectory prediction and fluctuation trajectory smoothing process, to the discrete world coordinate point w t Fourier series fitting is performed based on the missing trajectory prediction and fluctuation trajectory smoothing process, global search and matching of the front and rear breakpoint trajectories are performed by using the KD-Tree algorithm, and the Kalman algorithm is used for missing trajectory prediction, the algorithm includes prediction and updating two parts, for the trajectory part with observation value, the algorithm performs the complete process of prediction and updating, for the missing trajectory part without observation value, the algorithm only predicts the state without performing the updating process, and the predicted value is taken as the optimal trajectory point at the moment, after obtaining the complete time sequence trajectory data of the traffic participants, the SG algorithm is used for fitting the time sequence trajectory, so as to further enhance the representation ability of the time sequence trajectory to the real trajectory time sequence trend; Further, the abnormal trajectory constraint process is based on the basic fact that the vehicle travels in the road contour constraint area, the road contour is used to correct and constrain the vehicle travel trajectory, a rectangular boundary box with a width of w and a length of l is added along the tangent direction of the trajectory centering on the vehicle time sequence trajectory point to represent the vehicle size, the width w and the length l can be estimated according to different vehicle types, and in the actual driving process, the rectangular boundary box should not exceed the road boundary, and the trajectory points of the rectangular box exceeding the boundary are corrected along the original tangent direction towards the inside of the road boundary. By then, the complete process of accident scene restoration is completed, and the scene space-time information including the trajectories of traffic participants and the road contour is obtained.

Images