Vehicle end visual angle accident scene restoration method based on automobile data recorder video

Abstract

The invention belongs to the technical field of autonomous vehicles, and particularly relates to a vehicle end visual angle accident scene restoration method based on an automobile data recorder video. Comprising the following steps: step 1, preprocessing an accident video; 2, performing semantic understanding on the scene; step 3, extracting pixel tracks of the traffic participants; step 4, carrying out joint calibration on the inverse perspective mapping parameters; 5, carrying out inverse perspective transformation; step 6, carrying out post-processing on the road contour and track data; for accident video data with unknown and different camera parameters and lack of multi-source synchronization information, the invention provides a vehicle-end visual angle accident scene restoration framework taking a camera inverse perspective mapping parameter joint calibration strategy as a core. Scene space-time information including a road contour and a traffic participant track can be extracted from an automobile data recorder video without any prior information support, accident scene data is established for testing an automatic driving automobile, and the testing process of the automatic driving automobile can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving vehicles, and specifically is a method for restoring accident scenes from a vehicle's perspective based on driving recorder videos. Background Art

[0002] In recent years, autonomous vehicles have developed rapidly, and their ability to cope with complex scenarios has been significantly enhanced. However, driving scenarios in the real world are endlessly rich, and complex, extreme, and long-tail scenario data with extremely low probability of occurrence have an important impact on the safe operation of autonomous vehicles. In order to meet the public's increasingly urgent demand for safe and reliable autonomous vehicles, autonomous vehicles need to be fully and efficiently tested to improve their ability to handle complex traffic environments, especially long-tail scenarios. As an important part of long-tail scenarios, accident scenarios are of self-evident importance to autonomous vehicle testing. However, the process of collecting accident scenes is extremely dangerous, costly, and inefficient. Existing open source natural driving datasets such as HighD that can be directly used in the testing process also rarely have accidents, so there is gradually an urgent need for accident data. How to obtain high-quality accident scene data at low cost and high efficiency for autonomous vehicle testing is an important issue that needs to be solved urgently.

[0003] The dashcam can record the dynamic evolution of the scene within a certain range in front of the vehicle from the perspective of the vehicle-side monocular camera, and conduct in-depth analysis of the accident scene through video to investigate the responsible party of the accident. Therefore, there are some accident data sets stored in the form of monocular videos. In addition, a large number of similar accident scenes can be searched on the network platform, making it possible to obtain high-value accident scene data at low cost. The accident scenes from this data source can greatly reduce the cost of data acquisition, and at the same time contain highly realistic interactive characteristics, which can significantly enhance the adaptability of the algorithm in real open world scenes. In addition, the accident data from this source can cover complex and changeable scene types, and the diversity of scenes can be effectively guaranteed.

[0004] However, it is difficult to directly apply the existing video data to the testing process of autonomous vehicles. Accident reconstruction based on dashcam videos is required to extract key scene information. Traditional accident reconstruction based on dashcam videos relies on a large amount of auxiliary data support. Due to geographical distribution and actual costs, it is impossible to conduct on-site surveys of the accident scene, which leads to the lack of multi-source synchronous information in the accident videos stored in the dashcam, making it difficult to accurately reconstruct the accident and effectively extract scene information. AI-enabled computer vision technology extracts features from scene images in a visual way, which can effectively reduce the dependence on auxiliary information. However, when different dashcams store accident videos, the camera parameters are completely unknown and different camera configurations will lead to large differences between camera parameters. The lack of synchronous multi-source information makes it impossible to directly obtain the transformation matrix used for camera mapping. Although it effectively reduces the dependence on auxiliary information, the existence of the above phenomenon makes it impossible for the existing research framework to complete the task of extracting key information from the vehicle-side perspective of the accident scene based on dashcam videos.

[0005] Aiming at accident video data with unknown and different camera parameters and lack of multi-source synchronous information, the present invention proposes a vehicle-side perspective accident scene restoration framework with a joint calibration strategy of camera inverse perspective mapping parameters as the core. It can extract scene spatiotemporal information including road contours and traffic participant trajectories from dash cam videos without any prior information support, and establish accident scene data for autonomous driving vehicle testing. Summary of the invention

[0006] To solve the above problems, the present invention provides a method for restoring accident scenes from a vehicle-side perspective based on a dashcam video. First, the monocular accident video collected by the dashcam is preprocessed to optimize the video quality. Then, the semantic masks of the pixel road boundaries and the dynamic areas of the image are obtained through a scene semantic understanding module. Then, a robust identification and tracking algorithm for traffic participants is designed to extract the pixel motion trajectory. Then, a vehicle-side dynamic scene parameter joint calibration algorithm based on geometrically saliency feature bundle adjustment (GSFBA) is used to calibrate the inverse perspective mapping parameters with high precision. Then, the pixel coordinates are inversely transformed using camera parameters. Finally, a road contour and trajectory information post-processing module is designed to extract the actual road contour based on the pixel road boundary mapping intensity, and a systematic trajectory processing module is used to fit and optimize the trajectory information, so as to complete the accident scene restoration task based on the dashcam video and establish accident scene data for autonomous driving vehicle testing.

[0007] The technical solution of the present invention is described as follows in conjunction with the accompanying drawings:

[0008] A method for restoring an accident scene from a vehicle's perspective based on a driving recorder video comprises the following steps:

[0009] Step 1: Preprocess the accident video;

[0010] Use an efficient multi-level noise reduction algorithm to reduce the noise of the video, and use a real-time intermediate flow estimation algorithm to enhance the video frame addition.

[0011] Step 2: Semantically understand the scene;

[0012] Using the DeepLabV3Plus algorithm as a semantic feature extractor, the video is processed frame by frame to obtain multi-frame semantic images of the scene. The multi-frame pixel road boundaries are extracted through the edge detection algorithm, and based on this, the target detection algorithm is divided into the main focus area and background area. Based on the semantic information, the motion consistency check algorithm is used to obtain the dynamic area and dynamic semantic mask.

[0013] Step 3: Extracting pixel trajectories of traffic participants;

[0014] Through the synergy between the improved YOLOV8 algorithm and the DeepSORT algorithm, key traffic participants are robustly identified and associated frame by frame, achieving the task of accurately extracting the pixel trajectories of traffic participants.

[0015] Step 4: Jointly calibrate the inverse perspective mapping parameters;

[0016] A joint calibration algorithm for inverse perspective mapping parameters of vehicle-side dynamic scenes based on geometrically salient feature bundle adjustment is designed. First, the feature information of the image is extracted, and then the optical flow correction prediction and confidence update are performed to obtain the key image frames for front-end factor graph construction and the intermediate quantities for back-end factor graph optimization. Then, the factor graph is initialized and constructed in the front-end process, and finally, the joint iterative calibration of parameters is realized by using geometrically salient feature bundle adjustment (GSFBA) in the back-end process.

[0017] Step 5: Perform inverse perspective transformation;

[0018] Map the pixel trajectories of key traffic participants and the pixel contours of the road to the world coordinate system for subsequent scene data processing;

[0019] Step 6: Post-process the road profile and trajectory data;

[0020] A road contour extraction method based on pixel road boundary mapping intensity and a systematic trajectory data post-processing process including missing trajectory prediction, fluctuating trajectory smoothing, and abnormal trajectory constraint are designed to further improve data quality and ensure the accuracy and applicability of scene key information extraction.

[0021] Furthermore, the specific method of step one is as follows:

[0022] 11) The video is processed using the Efficient Multistage Video Denoising (EMVD) algorithm, which reduces the noise of the input video through three processing stages: time fusion, spatial denoising, and spatiotemporal refinement.

[0023] 12) The real-time intermediate flow estimation (RIFE) algorithm is used to perform frame enhancement on the video, and the intermediate frame image is accurately predicted by outputting an approximate intermediate flow and then performing backward deformation and fusion.

[0024] Furthermore, the specific method of step 2 is as follows:

[0025] 21) The DeepLabV3Plus algorithm with a dilated convolution structure is selected as the semantic feature extractor;

[0026] 22) Process the video frame by frame through the feature extractor to obtain multi-frame semantic images of the scene, extract multi-frame pixel road boundaries through the edge detection algorithm, and divide the target detection algorithm's main focus area and background area based on the pixel road boundaries;

[0027] 23) The motion consistency check algorithm is used in conjunction with the semantic information to obtain the semantic mask of the dynamic area of ​​the image. The process is as follows: different traffic participants will be assigned different colors in the semantic information, and the outline of the traffic participant is regarded as the candidate area to be checked for motion. First, the feature points in the candidate area to be checked for motion in the first frame image are detected, and the optical flow pyramid is calculated to obtain the feature points in the area to be checked for motion in the second frame that match the first frame. The basic matrix and the polar line between the feature points are calculated using the Random Sample Consensus (RANSAC) algorithm, and the distance between the matching feature points and the polar line is calculated. If it is greater than the set threshold, it indicates that the feature point is moving. If the moving feature point in a certain area to be checked for motion is greater than a certain percentage threshold of the total number of feature points, it is regarded as a moving area. At this point, the task of obtaining the semantic mask of the dynamic area in the first frame image is completed until all frames of the video are processed;

[0028] The basic matrix is ​​used to map the feature points in the first frame to the search domain corresponding to the second frame, i.e., the epipolar lines. Assuming that p1 and p2 are the matching feature points in the first frame and the second frame, respectively, they are expressed in the form of homogeneous coordinates as shown in formula (1):

[0029] p1=(u1,v1,1),p2=(u2,v2,1) (1)

[0030] Where u and v are pixel values ​​in the image;

[0031] Then, the polar line I is calculated by formula (2): m :

[0032] I m =(x m ,y m ,z m ) T =F m p1 (2)

[0033] In the formula, x m ,y m 、z m is the line vector; F m is the basic matrix;

[0034] Then, the distance D from the matching feature point to the corresponding epipolar line is calculated by formula (3): m :

[0035]

[0036] Where D m is the distance from the matching feature point to the corresponding epipolar line.

[0037] Furthermore, the specific method of step three is as follows:

[0038] 31) Based on the YOLOV8 algorithm, the global attention mechanism (GAM) is introduced to optimize the YOLOV8 algorithm backbone network, and the ASPP (Atrous Spatial Pyramid Pooling) module is used to optimize the algorithm feature pyramid to obtain the object detector of the video;

[0039] Using the object detector constructed above, prediction is performed based on feature extraction to obtain traffic participant detection results. At the same time, the main focus areas obtained by scene semantic understanding are integrated into the algorithm detection process to eliminate invalid background interference in the scene.

[0040] 32) Based on the target detection algorithm, the DeepSORT algorithm is used to generate trajectories after estimating the motion of traffic participants and then obtain data association results. That is, the cross-frame detection results of the same vehicle are associated and a unique ID is assigned to it, thereby forming continuous pixel trajectory data of traffic participants.

[0041] Furthermore, the specific method of step 4 is as follows:

[0042] 41) The dynamic semantic mask of the image is obtained through scene semantic understanding, and the masked image pair P = (I i ,I j ) as input, the feature encoder obtains the feature information of adjacent frames and performs correlation calculation, and obtains the initial optical flow. The correlation encoder and the optical flow encoder are used to encode the correlation and optical flow respectively, and the context encoder is used to obtain the context information for subsequent optical flow correction prediction and pixel confidence update;

[0043] 42) During the parameter iteration optimization process, it is necessary to continuously and correctly update the optical flow change and pixel confidence weight to ensure that the back-end parameter joint calibration algorithm continuously solves the optimal parameter value according to the camera intrinsic parameter g k , camera pose p k , depth d k Calculate image I by formula (4) i Middle pixel u i To Image I j Middle pixel u ij :

[0044] u ij =π(p ij ·π -1 (u i ,d i ,g),g) (4)

[0045] In the formula, p ij =p j ·p i -1 is the relative position of the camera between the two frames; π(·) is the mapping relationship from the 3D world coordinates to the 2D pixel coordinates of the camera; π -1 (·) is the reverse projection process of 2D pixel coordinates to 3D world coordinates. The driving recorder is modeled as a unified camera model and the distortion factor α is introduced, so that the algorithm can be applied to monocular videos with different camera types and different degrees of distortion during the parameter calibration process, as shown in formula (5):

[0046]

[0047] Where X(x,y,z)∈R 3 is the 3D world coordinate; α is the distortion factor; f x 、f y is the focal length of the camera; u0 and v0 are the principal points of the camera, and

[0048] In each iteration, context information, optical flow, image I j Middle pixel u ij The information and correlation are passed to the spatiotemporal convolutional network, which integrates the information in time sequence and uses different convolutional layers to predict the change value Δu of the optical flow. ij and the corresponding confidence weight w ij , the key frames with sufficient motion in the video are selected as the input part of the front-end factor graph construction using the optical flow change value. At the same time, the optical flow change value and the confidence weight of the pixel point are used in the back-end factor graph optimization process. In addition, in each iteration, the spatiotemporal convolutional network converts the input hidden state h ij k Updated to h ij k+1 ;

[0049] 43) The parameter joint calibration process is converted into a graph optimization problem by constructing a graph model, and the optimal parameter value is continuously solved by the iterative optimization process of the factor graph. The factor graph is a directed acyclic graph containing a series of nodes and factors. The nodes represent the variable values ​​of the parameters to be calibrated, and the factors f ij Represents a pair of images (I i ,I j ), that is, the constraint relationship between calibration parameters;

[0050] The factor graph construction process in the front-end process is as follows: using the feature information of the key image frame to add the edges of adjacent frames and the edges based on distance to the factor graph and update the factor graph, then remove the redundant edges in the factor graph and update the factor graph again, so as to obtain the factor graph for back-end optimization;

[0051] 44) Incorporate Geometrically Salient Feature Bundle Adjustment (GSFBA) into the backend factor graph optimization process to express the geometric gaze reality with the salience of feature points and improve the cost function of the reprojection error in the iterative optimization process, as follows:

[0052] Mining the edge feature points and feature lines of the geometric feature area in the image, and performing mask filling at the feature point and feature line positions to obtain a single frame image I iThe corresponding initial saliency image SI i , using equation (6) to calculate the pixel depth d i Corrected saliency image S i :

[0053]

[0054] Where a and b are constant correction factors set to avoid the small salient area of ​​the scene resulting in insufficient feature information and thus causing the failure of the back-end parameter optimization and calibration process;

[0055] Factor graph constructed by front-end process, flow correction prediction value Δu ij , pixel confidence weight w ij and the last iteration parameter g k 、p k d k As input, the backend process obtains the updated optimal parameter value g by minimizing the cost function E(g,p,d) k+1 、p k+1 d k+1 , the cost function represents the sum of the reprojection errors of all key frame pixels, and the saliency image S i The pixel saliency obtained after normalization i Incorporate it into the cost function, as shown in formula (7):

[0056]

[0057] Where i, j are the indexes of the key frames of the image; ∑ ij =diag(wij);f ij is the error correlation term of the factor, which is calculated by formula (8):

[0058] f ij =u ij -π(p ij ·π -1 (u i ,d i ,g),g)(8)

[0059] In the formula, u ij =u ij +Δu ij is the pixel coordinate value predicted after optical flow correction;

[0060] The optimization goal of the backend process is as shown in formula (9), and the parameter values ​​to be calibrated are obtained through the joint calibration process of the parameters:

[0061]

[0062] Where g, p, and d are the optimized parameter values ​​in the joint calibration process;

[0063] The backend process uses the classic Gauss-Newton step as the optimization method, as shown in formula (10):

[0064] J T WJΔξ=J T Wf(10)

[0065] Where f is the optimized residual from equation (8); J is the Jacobian matrix of the residual; W is the diagonal matrix composed of all image confidence weights; Δξ = (Δg, Δp, Δd) is the update amount of the calibration parameters.

[0066] Furthermore, the specific method of step five is as follows:

[0067] 51) Use equation (11) to transform the pixel coordinate point P i (u,v) is mapped to the world coordinate point P w (X w ,Y w ,Z w ), that is, mapping the pixel trajectories of key traffic participants and the pixel contours of the road to the world coordinate system:

[0068]

[0069] Where R and T are the rotation matrix and translation vector of the camera respectively; d i corresponds to the pixel point P i The depth value of .

[0070] Furthermore, the specific method of step six is ​​as follows:

[0071] 61) Design a road contour extraction method that fuses multi-frame information and is based on pixel road boundary mapping intensity, that is, after the inverse perspective transformation module, the multi-frame road contour coordinate points w in the world coordinates can be obtained. i (i=1,2,…,n), after excluding outliers, w of all frames i Mapped to the same coordinate system, the intensity distribution of pixel road boundary mapping after multi-frame information fusion is constructed. The intensity value represents the probability of mapping the pixel road boundary to different coordinates. The coordinate with a large intensity value has a higher confidence in the actual road contour. Then the mapping intensity distribution map is decomposed into several independent sub-regions as candidate regions for discrete road contour extraction. For each candidate region, the density of coordinate points is calculated along the x and y directions respectively. The road contour boundary line with the highest confidence in the candidate region is extracted based on the density curves in the two directions, and the extracted contour is fitted using the SG (Savitzky-Golay) algorithm to obtain a complete and accurate road contour.

[0072] 62) Design a systematic trajectory data post-processing process, which starts with missing trajectory prediction and fluctuating trajectory smoothing, with discrete world coordinate points w t Fourier series fitting is performed based on the data, and the KD-Tree algorithm is used to globally search and match the front and rear breakpoint trajectories. The Kalman algorithm is used to predict the missing trajectories. The algorithm includes two parts: prediction and update. For the trajectory part with observation values, the algorithm performs the complete process of prediction and update. For the missing trajectory part without observation values, the algorithm only predicts the state without performing the update process. The predicted value is used as the optimal trajectory point at that moment. After obtaining the complete time series trajectory data of traffic participants, the SG algorithm is used to fit the time series trajectory to further enhance its ability to characterize the time series trend of the real trajectory.

[0073] Then comes the abnormal trajectory constraint process. Based on the basic fact that the vehicle is traveling within the road contour constraint area, the road contour is used to correct the vehicle's driving trajectory. A rectangular bounding box with a width of w and a length of l is added along the tangent direction of the trajectory with the vehicle's time-series trajectory point as the center to represent the size of the vehicle. The width w and length l can be estimated according to different vehicle types. In the actual driving process, the rectangular bounding box should not exceed the road boundary. The trajectory points of the rectangular box that exceed the boundary are corrected along the original tangent direction toward the inside of the road boundary.

[0074] At this point, the entire process of accident scene restoration has been completed, and the scene's spatiotemporal information, including the trajectories of traffic participants and road contours, has been obtained.

[0075] The beneficial effects of the present invention are:

[0076] The vehicle-side perspective accident scene restoration method based on driving recorder video provided by the present invention is centered on the inverse perspective mapping parameter joint calibration strategy. It is oriented to vehicle-side accident videos with unknown and different camera parameters and lack of multi-source synchronous information. It can extract scene spatiotemporal information including road contours and traffic participant trajectories without any prior information support, thereby establishing diversified accident scene data with significant long-tail characteristics and capable of showing real interactive features. It significantly improves the accident scene acquisition efficiency while greatly reducing the accident scene acquisition cost, and can effectively accelerate the testing process of autonomous driving vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0078] Figure 1 is a flow chart of the present invention;

[0079] Figure 2 This is the flow chart of the joint calibration algorithm of vehicle-side dynamic scene parameters based on GSFBA;

[0080] Figure 3 is the factor graph of the parameter joint calibration process;

[0081] Figure 4 This is a flowchart for post-processing of road profile and trajectory data;

[0082] Figure 5 Schematic diagram of timing trajectory correction considering road contour constraints;

[0083] Figure 6 This is a global information display diagram for the simulated accident scene;

[0084] Figure 7a The intensity distribution map of the pixel road boundary mapping of sub-area 1 in the world coordinate system;

[0085] Figure 7b Mapping intensity distribution map of pixel road boundary for sub-area 2 in world coordinate system;

[0086] Figure 7c The intensity distribution map of the pixel road boundary mapping of sub-area 3 in the world coordinate system;

[0087] Figure 7d A pixel road boundary mapping intensity distribution map for sub-region 4 in the world coordinate system;

[0088] Figure 8 Extract the renderings for the scene road contour;

[0089] Fig. 9 This is the effect diagram of traffic participant target detection and tracking;

[0090] Fig.10a Comparison chart of extracted trajectory and real trajectory for Vehicle1;

[0091] Fig.10b Comparison chart of extracted trajectory and real trajectory for Vehicle2;

[0092] Fig.10c Comparison chart between the extracted trajectory and the real trajectory of Vehicle3;

[0093] Fig.11 Compare the trajectory extracted by the camera with the actual trajectory;

[0094] Fig.12a This is the rendering of the speed information extraction of vehicle Vehicle0;

[0095] Figure 12b This is the rendering of the speed information extraction of vehicle Vehicle1;

[0096] Fig.12c This is the rendering of the speed information extraction of vehicle Vehicle2;

[0097] Fig.12d This is the rendering of the speed information extraction of vehicle Vehicle3;

[0098] Fig.13a This is a comparison chart between information extracted by the algorithm and real information in a real car scenario;

[0099] Fig.13b This is a comparison chart of the extracted speed information and the real information of vehicle Vehicle1;

[0100] Fig.13c This is a comparison chart of the extracted speed information and the real information of vehicle Vehicle2;

[0101] Fig.14a is a schematic diagram of accident scene 1;

[0102] Fig.14b is a schematic diagram of accident scene 2;

[0103] Fig.15a This is the key frame diagram of accident scene 1;

[0104] Fig.15b This is the key frame detection result diagram of accident scene 1;

[0105] Fig.15c The global road profile information and key trajectory points of traffic participants in accident scene 1 are shown in the result map;

[0106] Fig.16a This is the key frame diagram of accident scene 2;

[0107] Fig.16b This is the key frame detection result diagram of accident scene 2;

[0108] Fig.16c The global road contour information and key trajectory points of traffic participants in accident scene 2 are shown in the figure. DETAILED DESCRIPTION

[0109] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0110] Embodiment 1

[0111] See also Figure 1 This embodiment provides a method for restoring an accident scene from a vehicle perspective based on a driving recorder video, comprising the following steps:

[0112] Step 1: Preprocess the accident video;

[0113] The video is denoised using the Efficient Multi-stage Video Denoising (EMVD) algorithm, and the video is enhanced by adding frames using the Real-time Intermediate Flow Estimation (RIFE) algorithm. The details are as follows:

[0114] 11) The video is processed using an efficient multi-stage video denoising algorithm (EMVD). Through three processing stages: time fusion, spatial denoising, and spatiotemporal refinement, high-quality denoising is performed on the input video.

[0115] 12) The real-time intermediate flow estimation algorithm (RIFE) is used to perform frame enhancement on the video. By outputting the approximate intermediate flow and then performing backward deformation and fusion, the intermediate frame image can be accurately predicted, thereby improving the smoothness of the input video.

[0116] Step 2: Semantically understand the scene;

[0117] Using the DeepLabV3Plus algorithm as a semantic feature extractor, the video is processed frame by frame to obtain multi-frame semantic images of the scene. The multi-frame pixel road boundaries are extracted through the edge detection algorithm, and based on this, the target detection algorithm is divided into the main focus area and the background area. Based on the semantic information, the motion consistency check algorithm is used to obtain the dynamic area and dynamic semantic mask, as follows:

[0118] 21) The DeepLabV3Plus algorithm with a dilated convolution structure is selected as the semantic feature extractor;

[0119] 22) Process the video frame by frame through the feature extractor to obtain multi-frame semantic images of the scene, extract multi-frame pixel road boundaries through the edge detection algorithm, and divide the target detection algorithm's main focus area and background area based on the pixel road boundaries;

[0120] 23) The motion consistency check algorithm is used in combination with semantic information to obtain the semantic mask of the dynamic area of ​​the image. Different traffic participants will be assigned different colors in the semantic information. The outline of the traffic participant is regarded as the area to be checked for motion. First, the feature points in the area to be checked for motion in the first frame image are detected, and the optical flow pyramid is calculated to obtain the feature points in the area to be checked for motion in the second frame that match the first frame. The basic matrix and the polar line between the feature points are calculated using the Random Sample Consensus (RANSAC) algorithm, and the distance between the matching feature point and the polar line is calculated. If it is greater than the set threshold, it indicates that the feature point is moving. If the moving feature point in a certain area to be checked for motion is greater than a certain percentage threshold of the total number of feature points, it is regarded as a moving area.

[0121] The basic matrix is ​​used to map the feature points in the first frame to the search domain corresponding to the second frame, i.e., the epipolar lines. Assuming that p1 and p2 represent the matching feature points in the first frame and the second frame, respectively, they are expressed in the form of homogeneous coordinates as shown in formula (1):

[0122] p1=(u1,v1,1),p2=(u2,v2,1) (1)

[0123] Where u and v are pixel values ​​in the image;

[0124] Then the polar line I can be calculated by formula (2): m :

[0125] I m =(x m ,y m ,z m ) T =F m p1 (2)

[0126] In the formula, x m ,y m 、z m is the line vector; F m is the basic matrix;

[0127] Then, the distance D from the matching feature point to the corresponding epipolar line can be calculated by formula (3): m :

[0128]

[0129] Where D m is the distance from the matching feature point to the corresponding epipolar line.

[0130] Step 3: Extracting pixel trajectories of traffic participants;

[0131] Through the synergy between the improved YOLOV8 algorithm and the DeepSORT algorithm, the key traffic participants are robustly identified and associated frame by frame to achieve the task of accurately extracting the pixel trajectories of traffic participants, as follows:

[0132] 31) Based on the YOLOV8 algorithm, the global attention mechanism (GAM) is introduced to optimize the backbone network of the YOLOV8 algorithm, reduce information loss and enhance the ability to represent global information interactions. The ASPP (Atrous Spatial Pyramid Pooling) module is used to optimize the algorithm feature pyramid, and the convolution kernels with different receptive fields are used to enhance the generalization ability of the model for target multi-scale features.

[0133] Using this as a target detector, predictions are generated based on feature extraction to obtain traffic participant detection results. At the same time, the main areas of interest obtained by scene semantic understanding are integrated into the algorithm detection process to eliminate invalid background interference in the scene.

[0134] 32) Based on the target detection algorithm, the DeepSORT algorithm is used to generate trajectories after estimating the motion of traffic participants and then obtain data association results. That is, the cross-frame detection results of the same vehicle are associated and a unique ID is assigned to it, thereby forming continuous pixel trajectory data of traffic participants.

[0135] Step 4: Jointly calibrate the inverse perspective mapping parameters;

[0136] See also Figure 2 The present invention designs a joint calibration algorithm for inverse perspective mapping parameters of vehicle-side dynamic scenes based on Geometrically Salient Feature Bundle Adjustment (GSFBA). First, the feature information of the image is extracted, and then the optical flow correction prediction and confidence update are performed to obtain the key image frame for front-end factor graph construction and the intermediate quantity for back-end factor graph optimization. Then, the factor graph is initialized and constructed in the front-end process. Finally, the joint iterative calibration of parameters is realized by using the Geometrically Salient Feature Bundle Adjustment in the back-end process. The details are as follows:

[0137] 41) The dynamic semantic mask of the image is obtained through scene semantic understanding, and the masked image pair P = (I i ,I j) as input, the feature encoder obtains the feature information of adjacent frames and performs correlation calculation. High correlation represents similar areas in the image, and the initial optical flow is obtained. The correlation encoder and the optical flow encoder are used to encode the two. At the same time, the context encoder obtains the context information for subsequent optical flow correction prediction and pixel confidence update;

[0138] 42) During the parameter iteration optimization process, it is necessary to continuously and correctly update the optical flow change and pixel confidence weight to ensure that the back-end parameter joint calibration algorithm continuously solves the optimal parameter value. k , camera pose p k , depth d k Calculate image I by formula (4) i Middle pixel u i To Image I j Middle pixel u ij :

[0139] u ij =π(p ij ·π -1 (u i ,d i ,g),g) (4)

[0140] In the formula, p ij =p j ·p i -1 is the relative position of the camera between the two frames; π(·) is the mapping relationship from the 3D world coordinates to the 2D pixel coordinates of the camera; π -1 (·) is the reverse projection process of 2D pixel coordinates to 3D world coordinates. The present invention models the driving recorder as a unified camera model and introduces a distortion factor α, so that the algorithm can be applied to monocular videos of different camera types and different degrees of distortion during the parameter calibration process, as shown in formula (5):

[0141]

[0142] Where X(x,y,z)∈R 3 is the 3D world coordinate; α is the distortion factor; f x 、f y is the focal length of the camera; u0 and v0 are the principal points of the camera, and

[0143] In each iteration, context information, optical flow, image I j Middle pixel u ij The information and correlation are passed to the spatiotemporal convolutional network, which integrates the information in time sequence and uses different convolutional layers to predict the change value Δu of the optical flow. ijand the corresponding confidence weight w ij , the key frames with sufficient motion in the video are selected as the input part of the front-end factor graph construction using the optical flow change value. At the same time, the optical flow change value and the confidence weight of the pixel point are used in the back-end factor graph optimization process. In addition, in each iteration, the spatiotemporal convolutional network converts the input hidden state h ij k Updated to h ij k+1 ;

[0144] 43) The present invention converts the parameter joint calibration process into a graph optimization problem by constructing a graph model, and continuously solves the optimal parameter value by using the iterative optimization process of the factor graph. The factor graph is a directed acyclic graph containing a series of nodes and factors. The nodes represent the variable values ​​of the parameters to be calibrated, and the factors f ij Represents a pair of images (I i ,I j ), that is, the constraint relationship between the calibration parameters. The factor graph representation method can be found in Figure 3 ;

[0145] The factor graph construction process in the front-end process is as follows: the feature information of the key image frame is used to add the edges of adjacent frames and the edges based on distance to the factor graph and update the factor graph, and then the redundant edges in the factor graph are removed and the factor graph is updated again, so as to obtain the factor graph for back-end optimization;

[0146] 44) The present invention integrates geometric saliency feature bundle adjustment into the back-end factor graph optimization process, uses the saliency of feature points to express the geometric gaze reality and improves the cost function of the reprojection error in the optimization process. Specifically, the edge feature points and feature lines of the geometric feature area in the image are mined and masked nearby to obtain a single frame image I i The corresponding initial saliency image SI i Considering the problems of decreased clarity and loss of geometric feature details caused by increased noise in distant images, the pixel depth d is calculated using equation (6). i Corrected saliency image S i :

[0147]

[0148] Where a and b are constant correction factors set to avoid the small salient area of ​​the scene resulting in insufficient feature information and thus causing the failure of the back-end parameter optimization and calibration process;

[0149] Factor graph constructed by front-end process, flow correction prediction value Δu ij , pixel confidence weight w ij and the last iteration parameter g k 、pk d k As input, the backend process obtains the updated optimal parameter value g by minimizing the cost function E(g,p,d) k+1 、p k+1 d k+1 The cost function represents the sum of the reprojection errors of all key frame pixels. The present invention converts the saliency image S i The pixel saliency obtained after normalization i Incorporate it into the cost function, as shown in formula (7):

[0150]

[0151] Where i, j are the indexes of the key frames of the image; ∑ ij =diag(w ij ), f ij is the error correlation term of the factor, which is calculated by formula (8):

[0152] f ij =u ij -π(p ij ·π -1 (u i ,d i ,g),g) (8)

[0153] In the formula, u ij =u ij +Δu ij is the pixel coordinate value predicted after optical flow correction;

[0154] The optimization goal of the backend process is as shown in formula (9), and the parameter values ​​to be calibrated are obtained through the joint calibration process of the parameters:

[0155]

[0156] Where g, p, and d are the optimized parameter values ​​in the joint calibration process;

[0157] The backend process uses the classic Gauss-Newton step as the optimization method, so that the optimization process can be carried out in a differentiable and effective way, as shown in formula (10):

[0158] J T WJΔξ=J T Wf(10)

[0159] Where f is the optimized residual from equation (8); J is the Jacobian matrix of the residual; W is the diagonal matrix composed of all image confidence weights; Δξ = (Δg, Δp, Δd) is the update amount of the calibration parameters.

[0160] Step 5: Perform inverse perspective transformation;

[0161] The pixel trajectories of key traffic participants and the pixel contours of the road are mapped to the world coordinate system for subsequent scene data processing, as follows:

[0162] 51) Use equation (11) to transform the pixel coordinate point P i (u,v) is mapped to the world coordinate point P w (X w ,Y w ,Z w ), that is, mapping the pixel trajectories of key traffic participants and the pixel contours of the road to the world coordinate system:

[0163]

[0164] In the formula, R and T are the rotation matrix and translation vector of the camera respectively, d i Represents the pixel P i The depth value of .

[0165] Step 6: Post-process the road profile and trajectory data;

[0166] A road contour extraction method based on pixel road boundary mapping intensity and a systematic trajectory data post-processing process including missing trajectory prediction, fluctuating trajectory smoothing, and abnormal trajectory constraint are designed to further improve data quality. Figure 4 , to ensure the accuracy and applicability of scene key information extraction, as follows:

[0167] 61) The present invention designs a road contour extraction method based on pixel road boundary mapping intensity by fusing multi-frame information. Specifically, after the inverse perspective transformation module, the multi-frame road contour coordinate points w in the world coordinates can be obtained. i (i=1,2,…,n), after excluding outliers, w of all frames i Mapped to the same coordinate system, the intensity distribution of pixel road boundary mapping after multi-frame information fusion is constructed. The intensity value represents the probability of mapping the pixel road boundary to different coordinates. The coordinate with a large intensity value has a higher confidence in the actual road contour. Then the mapping intensity distribution map is decomposed into several independent sub-regions as candidate regions for discrete road contour extraction. For each candidate region, the density of coordinate points is calculated along the x and y directions respectively. The road contour boundary line with the highest confidence in the candidate region is extracted based on the density curves in the two directions, and the extracted contour is fitted using the SG (Savitzky Golay) algorithm to obtain a complete and accurate road contour.

[0168] 62) The present invention designs a systematic trajectory data post-processing process, which first includes missing trajectory prediction and fluctuating trajectory smoothing, with discrete world coordinate points wt Fourier series fitting is performed based on the data, and the KD-Tree algorithm is used to globally search and match the front and rear breakpoint trajectories. The Kalman algorithm is used to predict the missing trajectories. The algorithm includes two parts: prediction and update. For the trajectory part with observation values, the algorithm performs the complete process of prediction and update. For the missing trajectory part without observation values, the algorithm only predicts the state without performing the update process. The predicted value is used as the optimal trajectory point at that moment. After obtaining the complete time series trajectory data of traffic participants, the SG algorithm is used to fit the time series trajectory to further enhance its ability to characterize the time series trend of the real trajectory.

[0169] Then there is the abnormal trajectory constraint process, which can be seen in Figure 5 Based on the basic fact that the vehicle travels within the road contour constraint area, the road contour is used to correct the vehicle's driving trajectory. A rectangular bounding box with a width of w and a length of l is added along the tangent direction of the trajectory with the vehicle's time-series trajectory point as the center to represent the vehicle size. The width w and length l can be estimated according to different vehicle types. In the actual driving process, the rectangular bounding box should not exceed the road boundary. The trajectory points of the rectangular box that exceed the boundary are corrected along the original tangent direction toward the inside of the road boundary.

[0170] At this point, the entire process of accident scene restoration has been completed, and the scene's spatiotemporal information, including the trajectories of traffic participants and road contours, has been obtained.

[0171] Embodiment 2

[0172] The present invention conducts comprehensive experimental verification of the method through a simulated accident scene and a real vehicle platform acquisition scene, and applies the method to the restoration process of two actual accident scenes.

[0173] 1. Experimental verification preparation;

[0174] The DeepLabV3Plus model is trained on the CityScapes and VirtualKITTI datasets using the GeForce RTX 4090D GPU, and the parameter joint calibration algorithm is trained end-to-end on the TartanAir dataset. On the basis of optical flow loss, pose loss, intrinsic parameter loss, and residual loss, the deep loss shown in formula (12) is added to supervise the algorithm training process:

[0175]

[0176] In the formula, is the calibration depth, d is the actual depth, γ is a fixed weight coefficient and is set to 0.9;

[0177] 2. Simulation scenario experimental verification;

[0178] The present invention uses the Carla simulator to simulate the real accident scene to construct an accident scene stored in the form of a monocular video of a driving recorder and having true value information, which can be referred to Figure 6 The scene lasts about 8 seconds, and the temporal dynamic process of the trajectory is Figure 6 The color of the trajectory points in the figure is used to represent the scene. This scene occurs at an intersection and includes a total of four vehicles. Vehicle 1 collides with Vehicle 3 which suddenly rushes out while turning near the intersection, resulting in a traffic accident. Vehicle 0 carries a driving recorder, which records the dynamic process of the accident in front as a monocular camera perspective. Vehicle 2 keeps going straight, and the vehicle speed is between 0-50km / h. Based on this dynamic scene, the accuracy of road contour extraction and the accuracy of traffic participant trajectory extraction are experimentally verified. In the verification process, the monocular video recorded by the camera is used as input. The camera parameters and the global information of the scene are only used for experimental accuracy comparison and do not participate in the verification process of this method. At the same time, it should be pointed out that the accident scene constructed in the Carla simulation environment is mainly used for the accuracy verification of this algorithm, and does not focus on the basic rules that vehicles should follow for normal driving.

[0179] 1. Verification of road profile extraction accuracy;

[0180] The intensity distribution of pixel road boundary mapping in the world coordinate system can be found in Figure 7a-7d As can be seen from the figure, the intensity distribution of road contour mapping in the world coordinate system is concentrated in 4 different sub-areas. The coordinates with darker colors and larger density curve values ​​have higher actual road contour confidence. The road contour curve trends of each sub-area are obvious, and the contour line extracted according to the horizontal and vertical density curves is reasonable. To further verify the road contour extraction effect, the extracted road contour line and the real road contour line can be referred to Figure 8 ,Depend on Figure 8 It can be seen that the algorithm can extract the road contour information of each sub-area near the vehicle interaction area more accurately. It should be noted that the algorithm extracts the contours of sub-area 1 and sub-area 2 as a curve. This is because the multi-level transmission of errors makes it difficult to decompose the mapping intensity distribution of the small area into discrete sub-areas. The pixel mapping area tends to be concentrated, making the contour extraction result tend to the median line, but it can still reasonably describe the basic information of the road contour, proving the effectiveness of the method of the present invention in extracting road contour information. It should be pointed out that Figure 8The figure shows the true value contour under the global perspective. The present invention only focuses on the vehicle interaction area. Due to the limitation of the camera perspective, only the road contour extraction within the camera field of view can be realized, which is not a limitation of the method. The root mean square error (RMSE) is used to quantify the road contour extraction accuracy. For each extracted road contour point, the RMSE is calculated based on the distance between the real road contour point with the smallest distance. The average RMSE near the vehicle interaction area is only 1.53m. The quantitative results show that the extracted contour can achieve accurate matching with the real contour, which verifies the accuracy of the method of the present invention in extracting the road contour information from the vehicle perspective.

[0181] 2. Accuracy of traffic participant trajectory extraction;

[0182] The accurate identification and stable tracking of traffic participants by the algorithm are necessary prerequisites for trajectory information extraction. Therefore, the effect of target detection and tracking algorithm is first tested. The effect of traffic participant target detection and tracking in scene key frames can be found in Fig. 9 ,Depend on Fig. 9 It can be seen that the algorithm can accurately identify traffic participants and assign IDs to achieve stable tracking. It performs well when facing multi-scale changes of traffic participants. The scene semantic understanding module eliminates the interference of the image background area on the algorithm, allowing the algorithm to pay more attention to vehicles on the road, effectively improving the ability to extract key scene information.

[0183] Then the accuracy of trajectory information extraction is verified. The trajectory after inverse perspective transformation and Fourier series fitting and the trajectory curve after optimization by systematic data post-processing process can be found in Figure 10a-Figure 10c The motion trajectory of Vehicle 0 carrying the camera can be found in Fig.11 ,Depend on Figure 10a-Figure 10c It can be seen that the trajectory transformed by inverse perspective and fitted by Fourier series can preliminarily describe the time series trend of the real trajectory. Fig.10c It can be seen that the KD-Tree algorithm can well match the breakpoint trajectory caused by ID switching. At the same time, the trajectory improved by the data post-processing process can achieve a good matching effect with the real trajectory. The missing trajectory caused by occlusion, missed detection and other reasons can be accurately repaired. The trajectory fitting and trajectory constraint can further optimize the trajectory quality. Fig.10b and Fig.10c There are some missing trajectories in both, but the reasons for the two are different. Fig.10b The missing tracks in the figure are due to the saturation of the long-distance depth values ​​during the parameter calibration process. Due to the inaccuracy of the depth values, it is difficult to accurately map the long-distance pixel point tracks. Fig.10b In the long-distance trajectory segment (-40m, -70m), the trajectory error increases significantly. Fig.10c The missing part of the trajectory is caused by the loss of ID. The Kalman algorithm can be used to predict and repair this section of the trajectory. In order to further quantify the trajectory information, the RMSE, which is the same as the road profile quantification calculation method, is used to evaluate the extraction accuracy of the traffic participant trajectory. The RMSE quantification results of Vehicle 0, Vehicle 1, Vehicle 2, and Vehicle 3 are 0.21m, 0.78m, 0.68m, and 0.83m, respectively, and the average RMSE is only 0.50m, which verifies the effectiveness and accuracy of the method in extracting the trajectory information of traffic participants.

[0184] Depend on Figure 10a-Figure 10c , Fig.11 From the error comparison curve and quantification results, it can be found that the trajectory accuracy of the vehicle carrying the camera is much higher than that of other vehicles. This is because the extraction methods of the trajectories of the two are different. For the vehicle carrying the camera, the vehicle's pose estimation in the world coordinate system can be achieved during the parameter joint calibration process. Its accuracy depends on the effect of the parameter joint calibration algorithm. The trajectory of other vehicles is obtained by mapping pixel coordinates to world coordinates using calibration parameters. Both parameter calibration error and pixel coordinate error will affect the trajectory accuracy.

[0185] The present invention verifies the speed information based on the trajectory information, and further verifies the applicability of restoring scene information in the test of autonomous driving vehicles. The speed information extraction effect can be found in Figure 12a-12d ,Depend on Figure 12a-12d It can be seen that the speed curve calculated from the restored time-series trajectory points is consistent with the real speed curve. The speed error quantification results of Vehicle 0, Vehicle 1, Vehicle 2, and Vehicle 3 after RMSE are 0.17m / s, 0.49m / s, 0.37m / s, and 0.77m / s, respectively. It can be seen from the quantification results that the restored speed information is also highly accurate and sufficient for the test process of autonomous vehicles.

[0186] 3. Real vehicle scenario experimental verification;

[0187] The present invention further tests the actual application effect of the method of the present invention on the basis of the simulation scene verification, and evaluates the performance of the method in the real vehicle scene by building a real vehicle platform; the present invention collects scene data in Changchun City, Jilin Province and selects a dynamic scene of about 11 seconds, in which Vehicle 1 and Vehicle 2 travel freely on a straight two-lane road with a driving speed of 10-30km / h, wherein Vehicle 1 is used as a camera perspective to record a monocular video of Vehicle 2 driving in front, and the real vehicle platform can realize the accurate acquisition of the real trajectories of the two vehicles, and align and measure the road contour coordinates of the scene based on the vehicle positions, so as to provide sufficient truth information for subsequent verification;

[0188] The present invention converts the WGS84 data format collected by the RT device into the UTM coordinate system and performs appropriate translation. The road profile, traffic participant trajectory and speed information extracted by this method for the real vehicle scene are compared with the real information. Figure 13a-Figure 13c ;

[0189] Depend on Figure 13a-Figure 13c It can be seen that the method provided by the present invention can also achieve effective extraction of key information in real vehicle scenarios. The RMSE trajectory error of vehicle 1 is 0.21m, and the RMSE speed error is 0.13m / s. The RMSE trajectory error of vehicle 2 is 0.61m, and the RMSE speed error is 0.32m / s. The RMSE error of the road profile is only 0.48m. The results show that it still has sufficient extraction accuracy in real vehicle scenarios. So far, the extraction accuracy of key scene information has been effectively verified from simulation scenarios to real vehicle scenarios.

[0190] 4. Restoration of the actual accident scene;

[0191] The present invention puts the vehicle-side view accident scene restoration method into practical application, selects two accident scene videos as input, and extracts the scene spatiotemporal information including the road contour and the trajectory of traffic participants. The schematic diagram of the accident scene can be found in Fig.14a and 14b , the scene information extraction results can be found in Figure 15a-Figure 15c and Figure 16a-Figure 16c ;

[0192] exist Figure 15a-Figure 15c , Figure 16a-16cThe key frames before and after the accident process, key frame detection results, global road contour information and key trajectory points of traffic participants are plotted. Due to the challenges of multi-scale changes of traffic participants and the limitations of the generalization ability of target detectors, missed recognition and misidentification are inevitable. The systematic trajectory post-processing process can effectively improve the above phenomena to achieve effective extraction of trajectory information. At the same time, the road contour extraction method can also depict the basic shape of the road, indicating that the scene key information extracted by the method of the present invention has sufficient applicability.

[0193] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for restoring accident scenes from a vehicle's perspective based on a driving recorder video, characterized in that: The following steps are involved: Step 1: Preprocess the accident video; Use an efficient multi-level noise reduction algorithm to reduce the noise of the video, and use a real-time intermediate flow estimation algorithm to enhance the video frame addition. Step 2: Semantically understand the scene; Using the DeepLabV3Plus algorithm as a semantic feature extractor, the video is processed frame by frame to obtain multi-frame semantic images of the scene. The multi-frame pixel road boundaries are extracted through the edge detection algorithm, and based on this, the target detection algorithm is divided into the main focus area and background area. Based on the semantic information, the motion consistency check algorithm is used to obtain the dynamic area and dynamic semantic mask. Step 3: Extracting pixel trajectories of traffic participants; Through the synergy between the improved YOLOV8 algorithm and the DeepSORT algorithm, key traffic participants are robustly identified and associated frame by frame, achieving the task of accurately extracting the pixel trajectories of traffic participants. Step 4: Jointly calibrate the inverse perspective mapping parameters; A joint calibration algorithm for inverse perspective mapping parameters of vehicle-side dynamic scenes based on geometric saliency feature bundle adjustment is designed. First, the feature information of the image is extracted, and then the optical flow correction prediction and confidence update are performed to obtain the key image frames for front-end factor graph construction and the intermediate quantities for back-end factor graph optimization. Then, the factor graph is initialized and constructed in the front-end process, and finally the joint iterative calibration of parameters is realized by using geometric saliency feature bundle adjustment in the back-end process. Step 5: Perform inverse perspective transformation; Map the pixel trajectories of key traffic participants and the pixel contours of the road to the world coordinate system for subsequent scene data processing; Step 6: Post-process the road profile and trajectory data; A road contour extraction method based on pixel road boundary mapping intensity and a systematic trajectory data post-processing process including missing trajectory prediction, fluctuating trajectory smoothing, and abnormal trajectory constraint are designed to further improve data quality and ensure the accuracy and applicability of scene key information extraction.

2. The method for restoring an accident scene from a vehicle-side perspective based on a driving recorder video according to claim 1, characterized in that: The specific method of step one is as follows: 11) Use an efficient multi-level noise reduction algorithm to process the video, and reduce the noise of the input video through three processing stages: time fusion, spatial denoising, and spatiotemporal refinement; 12) Use the real-time intermediate flow estimation algorithm to perform frame enhancement on the video, and accurately predict the intermediate frame image by outputting the approximate intermediate flow and performing backward deformation and fusion.

3. The method for restoring an accident scene from a vehicle-side perspective based on a driving recorder video according to claim 1, characterized in that: The specific method of step 2 is as follows: 21) The DeepLabV3Plus algorithm with a dilated convolution structure is selected as the semantic feature extractor; 22) Process the video frame by frame through the feature extractor to obtain multi-frame semantic images of the scene, extract multi-frame pixel road boundaries through the edge detection algorithm, and divide the target detection algorithm's main focus area and background area based on the pixel road boundaries; 23) The motion consistency check algorithm is used in combination with the semantic information to obtain the semantic mask of the dynamic area of ​​the image. The process is as follows: different traffic participants will be assigned different colors in the semantic information, and the outline of the traffic participant is regarded as the candidate area to be checked for motion. First, the feature points in the candidate area to be checked for motion in the first frame image are detected, and the optical flow pyramid is calculated to obtain the feature points in the area to be checked for motion in the second frame that match the first frame. The basic matrix and the polar line between the feature points are calculated using the random sampling consistency algorithm, and the distance between the matching feature point and the polar line is calculated. If it is greater than the set threshold, it indicates that the feature point is moving. If the moving feature point in a certain area to be checked for motion is greater than a certain percentage threshold of the total number of feature points, it is regarded as a moving area. At this point, the task of obtaining the semantic mask of the dynamic area in the first frame image is completed until all frames of the video are processed; The basic matrix is ​​used to map the feature points in the first frame to the search domain corresponding to the second frame, i.e., the epipolar lines. Assuming that p1 and p2 are the matching feature points in the first frame and the second frame, respectively, they are expressed in the form of homogeneous coordinates as shown in formula (1): p1=(u1,v1,1),p2=(u2,v2,1) (1) Where u and v are pixel values ​​in the image; Then, the polar line I is calculated by formula (2): m : I m =(x m ,y m ,z m ) T =F m p1 (2) In the formula, x m ,y m 、z m is the line vector; F m is the basic matrix; Then, the distance D from the matching feature point to the corresponding epipolar line is calculated by formula (3): m : Where D m is the distance from the matching feature point to the corresponding epipolar line.

4. The method for restoring an accident scene from a vehicle-side perspective based on a driving recorder video according to claim 1, characterized in that: The specific method of step three is as follows: 31) Based on the YOLOV8 algorithm, the global attention mechanism is introduced to optimize the YOLOV8 algorithm backbone network, and the ASPP module is used to optimize the algorithm feature pyramid to obtain the object detector of the video; Using the object detector constructed above, prediction is performed based on feature extraction to obtain traffic participant detection results. At the same time, the main focus areas obtained by scene semantic understanding are integrated into the algorithm detection process to eliminate invalid background interference in the scene. 32) Based on the target detection algorithm, the DeepSORT algorithm is used to generate trajectories after estimating the motion of traffic participants and then obtain data association results. That is, the cross-frame detection results of the same vehicle are associated and a unique ID is assigned to it, thereby forming continuous pixel trajectory data of traffic participants.

5. The method for restoring an accident scene from a vehicle-side perspective based on a driving recorder video according to claim 1, characterized in that: The specific method of step 4 is as follows: 41) The dynamic semantic mask of the image is obtained through scene semantic understanding, and the masked image pair P = (I i ,I j ) as input, obtain feature information of adjacent frames through a feature encoder and perform correlation calculation, and obtain the initial optical flow at the same time, use the correlation encoder and the optical flow encoder to encode the correlation and optical flow respectively, and obtain context information through a context encoder for subsequent optical flow correction prediction and pixel confidence update; 42) During the parameter iteration optimization process, it is necessary to continuously and correctly update the optical flow change and pixel confidence weight to ensure that the back-end parameter joint calibration algorithm continuously solves the optimal parameter value according to the camera intrinsic parameter g k , camera pose p k , depth d k Calculate image I by formula (4) i Middle pixel u i To Image I j Middle pixel u ij : u ij π(p ij ·π -1 (u i ,d i ,g),g) (4) In the formula, p ij =p j ·p i -1 is the relative position of the camera between the two frames; π(·) is the mapping relationship from the 3D world coordinates to the 2D pixel coordinates of the camera; π -1 (·) is the reverse projection process of 2D pixel coordinates to 3D world coordinates. The driving recorder is modeled as a unified camera model and the distortion factor α is introduced, so that the algorithm can be applied to monocular videos with different camera types and different degrees of distortion during the parameter calibration process, as shown in formula (5): Where X(x,y,z)∈R 3 is the 3D world coordinate; α is the distortion factor; f x 、f y is the focal length of the camera; u0 and v0 are the principal points of the camera, and In each iteration, context information, optical flow, image I j Middle pixel u ij The information and correlation are passed to the spatiotemporal convolutional network, which integrates the information in time sequence and uses different convolutional layers to predict the change value Δu of the optical flow. ij and the corresponding confidence weight w ij , the key frames with sufficient motion in the video are selected as the input part of the front-end factor graph construction using the optical flow change value. At the same time, the optical flow change value and the confidence weight of the pixel point are used in the back-end factor graph optimization process. In addition, in each iteration, the spatiotemporal convolutional network converts the input hidden state h ij k Updated to h ij k+1 ; 43) The parameter joint calibration process is converted into a graph optimization problem by constructing a graph model, and the optimal parameter value is continuously solved by the iterative optimization process of the factor graph. The factor graph is a directed acyclic graph containing a series of nodes and factors. The nodes represent the variable values ​​of the parameters to be calibrated, and the factors f ij Represents a pair of images (I i ,I j ), that is, the constraint relationship between calibration parameters; The factor graph construction process in the front-end process is as follows: using the feature information of the key image frame to add the edges of adjacent frames and the edges based on distance to the factor graph and update the factor graph, then remove the redundant edges in the factor graph and update the factor graph again, so as to obtain the factor graph for back-end optimization; 44) Integrate geometric saliency feature bundle adjustment into the backend factor graph optimization process, use the saliency of feature points to express the geometric gaze reality and improve the cost function of the reprojection error in the iterative optimization process, as follows: Mining the edge feature points and feature lines of the geometric feature area in the image, and performing mask filling at the feature point and feature line positions to obtain a single frame image I i The corresponding initial saliency image SI i , using equation (6) to calculate the pixel depth d i Corrected saliency image S i : Where a and b are constant correction factors set to avoid the small salient area of ​​the scene resulting in insufficient feature information and thus causing the failure of the back-end parameter optimization and calibration process; Factor graph constructed by front-end process, flow correction prediction value Δu ij , pixel confidence weight w ij and the last iteration parameter g k 、p k ,d k As input, the backend process obtains the updated optimal parameter value g by minimizing the cost function E(g,p,d) k+1 、p k+1 ,d k+1 , the cost function represents the sum of the reprojection errors of all key frame pixels, and the saliency image S i The pixel saliency obtained after normalization i Incorporate it into the cost function, as shown in formula (7): Where i, j are the indexes of the key frames of the image; ∑ ij =diag(wij);f ij is the error correlation term of the factor, which is calculated by formula (8): f ij s ij -π(p ij ·π -1 (u i ,d i ,g),g)(8) In the formula, u ij =u ij +Δu ij is the pixel coordinate value predicted after optical flow correction; The optimization goal of the backend process is as shown in formula (9), and the parameter values ​​to be calibrated are obtained through the joint calibration process of the parameters: Where g, p, and d are the optimized parameter values ​​in the joint calibration process; The backend process uses the classic Gauss-Newton step as the optimization method, as shown in formula (10): J T WJΔξ=J T Wf(10) Where f is the optimized residual from equation (8); J is the Jacobian matrix of the residual; W is the diagonal matrix composed of all image confidence weights; Δξ = (Δg, Δp, Δd) is the update amount of the calibration parameters.

6. The method for restoring an accident scene from a vehicle-side perspective based on a driving recorder video according to claim 1, characterized in that: The specific method of step five is as follows: 51) Use equation (11) to transform the pixel coordinate point P i (u,v) is mapped to the world coordinate point P w (X w ,Y w ,Z w ), that is, mapping the pixel trajectories of key traffic participants and the pixel contours of the road to the world coordinate system: Where R and T are the rotation matrix and translation vector of the camera respectively; d i corresponds to the pixel point P i The depth value of .

7. The method for restoring an accident scene from a vehicle-side perspective based on a driving recorder video according to claim 1, characterized in that: The specific method of step six is ​​as follows: 61) Design a road contour extraction method that fuses multi-frame information and is based on pixel road boundary mapping intensity, that is, after the inverse perspective transformation module, the multi-frame road contour coordinate points w in the world coordinates can be obtained. i (i=1,2,…,n), after excluding outliers, w of all frames i Mapped to the same coordinate system, to construct the intensity distribution of pixel road boundary mapping after multi-frame information fusion. The intensity value represents the probability of mapping the pixel road boundary to different coordinates. The coordinate with a large intensity value has a higher confidence in the actual road contour. Then, the mapping intensity distribution map is decomposed into several independent sub-regions as candidate regions for discrete road contour extraction. For each candidate region, the density of coordinate points is calculated along the x and y directions respectively. The road contour boundary line with the highest confidence in the candidate region is extracted based on the density curves in the two directions, and the extracted contour is fitted using the SG algorithm to obtain a complete and accurate road contour. 62) Design a systematic trajectory data post-processing process, which starts with missing trajectory prediction and fluctuating trajectory smoothing, with discrete world coordinate points w t Fourier series fitting is performed based on the data, and the KD-Tree algorithm is used to globally search and match the front and rear breakpoint trajectories. The Kalman algorithm is used to predict the missing trajectories. The algorithm includes two parts: prediction and update. For the trajectory part with observation values, the algorithm performs the complete process of prediction and update. For the missing trajectory part without observation values, the algorithm only predicts the state without performing the update process. The predicted value is used as the optimal trajectory point at that moment. After obtaining the complete time series trajectory data of traffic participants, the SG algorithm is used to fit the time series trajectory to further enhance its ability to characterize the time series trend of the real trajectory. Then comes the abnormal trajectory constraint process. Based on the basic fact that the vehicle is traveling within the road contour constraint area, the road contour is used to correct the vehicle's driving trajectory. A rectangular bounding box with a width of w and a length of l is added along the tangent direction of the trajectory with the vehicle's time-series trajectory point as the center to represent the size of the vehicle. The width w and length l can be estimated according to different vehicle types. In the actual driving process, the rectangular bounding box should not exceed the road boundary. The trajectory points of the rectangular box that exceed the boundary are corrected along the original tangent direction toward the inside of the road boundary. At this point, the entire process of accident scene restoration has been completed, and the scene's spatiotemporal information, including the trajectories of traffic participants and road contours, has been obtained.

Images