A Large-Model Traffic Accident Analysis Method Integrating Blind Spot Completion and Physics-Driven Reasoning

Abstract

This invention discloses a large-scale traffic accident analysis method that integrates blind spot completion and physics-driven reasoning. In the video preprocessing stage, this method achieves spatiotemporal synchronous localization of the accident time window and the collision impact center, providing stable physical anchors for subsequent analysis. Simultaneously, it enables cross-temporal 3D perception and reconstruction of blind spot scenes. Furthermore, by introducing an active reconstruction mechanism driven by pixel-level depth uncertainty, collision impact zone-level uncertainty, and object-level depth uncertainty in 3D reconstruction, the quality of blind spot recovery is improved. In addition, physical laws are encoded into the reasoning process to achieve deep causal analysis. Finally, the reasoning results are cross-validated from three dimensions—time, space, and semantics—to quantify the reliability of the analysis results, thereby improving the engineering usability, credibility, and auditability of automated traffic accident analysis.

Description

Technical Field

[0001] This invention belongs to the field of traffic control systems and relates to the detection of traffic movements that need to be statistically analyzed or controlled. Specifically, it relates to a large-scale traffic accident analysis method that integrates blind spot completion and physical-driven reasoning. Background Technology

[0002] With the large-scale deployment of road surveillance cameras, vehicle dashcams, and autonomous driving sensing facilities, traffic management departments and road operators acquire massive amounts of traffic operation video data every day. How to use artificial intelligence technology to automatically extract, analyze, and reconstruct the evolution process of traffic accidents (such as collisions, scrapes, and dangerous approach) from these massive video streams has become a core research direction in the field of intelligent traffic safety analysis.

[0003] In recent years, Multimodal Large Language Models (MLLMs) have been increasingly incorporated into traffic video analysis due to their powerful visual understanding and natural language generalization capabilities. However, when faced with traffic accident scenarios that involve strict physical causal chains and legal liability requirements, existing technical solutions still suffer from serious technical deficiencies in areas such as spatiotemporal positioning accuracy, blind spot handling, physical constraints, and result reliability. These deficiencies are specifically manifested in the following aspects: First, most existing video preprocessing and candidate event screening methods can only perform coarse-grained discrimination of the time period of an accident, lacking the ability to simultaneously locate the spatial position of the collision, making it difficult to provide reliable physical anchors for subsequent 3D reconstruction and causal inference. Specifically, most existing video preprocessing methods only filter the timeline based on global motion energy, local threshold mutations, or target trajectory anomalies, usually only outputting the time window of a suspected accident, unable to further extract the collision impact center. Due to the lack of stable spatial priors, subsequent modules often need to search key areas within the entire frame, resulting in significant spatial redundancy in 3D reconstruction and difficulty in providing verifiable underlying physical anchors for subsequent inference. Although existing screening schemes relying on target detection and tracking can obtain the 2D bounding boxes and trajectories of road users, their focus is usually on the existence of targets and the continuity of trajectories, which is difficult to meet the requirements of blind spot reconstruction and causal analysis for prior spatial localization.

[0004] Second, existing video analytics technologies lack the ability to perceive and reconstruct 3D accidents in the camera's out-of-view (OOV) field of view. In real traffic scenarios, especially from the perspective of in-vehicle dashcams, many collisions occur in blind spots behind or at the edges of the frame, such as blind spot peeking out or side-lane collisions. Existing 2D analysis methods can only infer based on the visible pixels of the current frame. Once the key involved object moves out of the frame at the moment of collision, the causal chain of the accident is broken. Furthermore, simple multi-frame point cloud overlay is prone to coordinate drift due to vehicle bumps and self-motion. Most of the few existing temporal fusion techniques rely on 2D feature warping, making it difficult to truly recover the lost geometric structure and semantic state in the blind spot in 3D physical space. For example, Chinese patent CN121716697A proposes a Transformer-based blind spot trajectory prediction method for autonomous vehicles, but it only predicts the future trajectory of known targets and cannot perform cross-temporal 3D geometric mapping and scene completion for objects that have moved out of the field of view. Chinese patent CN121792702A proposes a multi-camera collaborative blind spot-free intelligent monitoring method, but it relies on multi-camera hardware deployment to eliminate blind spots and cannot achieve 3D reconstruction of blind spots under single-camera conditions. Therefore, existing technologies are insufficient to meet the needs of complete reconstruction and analysis of blind spot accidents in complex real-world scenarios.

[0005] Third, the quality of cross-temporal 3D reconstruction is severely affected by the uncertainty of depth estimation, lacking an uncertainty-driven active reconstruction mechanism. Traffic accidents often occur in adverse environments such as rain, fog, low-light conditions at night, or reflective, waterlogged roads, where the accuracy of monocular depth estimation drops drastically. Existing technologies typically blindly adopt all depth predictions, leading to low-confidence noise severely contaminating the 3D geometry of the accident's core area, subsequently causing a chain of errors in collision distance and angle calculations. In cross-temporal 3D reconstruction, the reliability of depth estimation varies significantly across different pixel regions and distance ranges: near, textured areas have higher estimation accuracy, while distant, sparsely textured or occluded areas have extremely large estimation errors. Existing 3D reconstruction methods usually employ a uniform back-projection strategy for all regions, failing to consider the spatial distribution differences in depth perception uncertainty. This results in reconstruction noise from low-confidence areas being equally weighted, severely impacting the quality and controllability of OOV blind zone recovery. Chinese patent CN121515871A proposes a blind spot warning method for heavy vehicles based on a PV-RCNN network. Although it introduces 3D point cloud features, it is only used for single-frame 3D target detection and warning, and does not address the uncertainty modeling problem in cross-temporal 3D scene fusion. Chinese patent CN114782782A proposes an uncertainty quantification method for the learning performance of a monocular depth estimation model, but this method only focuses on the performance evaluation and correction of the depth estimation model itself, and does not further transmit uncertainty information to the downstream 3D reconstruction process for weighted guidance and active reconstruction. The lack of uncertainty modeling mechanisms in the 3D reconstruction process leads to uncontrollable blind spot recovery quality, severely restricting the practical value of traffic accident analysis systems in complex real-world scenarios.

[0006] Fourth, existing technologies for assessing traffic accidents lack the ability to conduct in-depth causal analysis driven by physical laws. Current computer vision-based analysis solutions typically limit their core technical objective to assigning pre-defined, coarse-grained semantic labels such as "collision" or "rear-end collision" to specific video segments, oversimplifying the complex traffic accident process into a single classification mapping. This prevents them from deeply modeling the causal logic chain within the accident. Purely visual large-scale models have a blurred perception of pixel-level physical motion and struggle to perceive the changing patterns of underlying physical quantities such as speed, acceleration, and time to collision (TTC), leading to frequent inaccuracies in judgment due to video distortion or viewpoint bias. For example, Chinese patent CN121929194A proposes a blind spot risk prevention and control method based on neighboring vehicle behavior inference. Although it involves behavioral reasoning, its reasoning basis is only the macroscopic movement trend of neighboring vehicles, failing to encode quantitative physical quantities such as speed, acceleration, and TTC into the reasoning process, thus lacking the ability to conduct in-depth causal analysis driven by physical laws. Furthermore, purely visual large models have a blurred perception of pixel-level physical motion, making it difficult to accurately capture the changing patterns of underlying physical quantities from video frames, leading to frequent misjudgments.

[0007] Fifth, the inference output of multimodal large language models lacks objective confidence quantification and self-auditing mechanisms. In serious application scenarios such as traffic police liability determination and insurance claims, the conclusions output by technical tools must have high auditability. Existing deep learning models typically exhibit black-box characteristics, and their output analysis results lack objective and quantitative confidence assessment mechanisms. When the model misjudges or exhibits model illusions, the system struggles to self-audit or provide risk warnings to users. Chinese patent CN119964101A proposes an end-to-end traffic road state perception method based on a multimodal large model. Its technical approach involves associating traffic video frame features with traffic element annotations to train a large model for scene understanding. However, this method directly outputs perceived text results without any verification or quantitative evaluation of the consistency between the large model's inference output and the underlying physical features, thus making it difficult to identify inference biases and model illusions. In existing traffic accident analysis systems, regardless of whether a large model architecture is used, the output usually only includes accident classification labels or text descriptions. No self-evaluation system has been established that can cross-validate the consistency between the model text description and the underlying physical features and historical case truth values. This greatly limits its practical application in serious scenarios such as traffic law enforcement and insurance claims, where the reliability of results is strictly required. Summary of the Invention

[0008] In view of the shortcomings and deficiencies of existing technologies, the purpose of this invention is to provide a large-scale traffic accident analysis method that integrates blind spot completion and physics-driven reasoning. This method achieves spatiotemporal synchronous localization of the accident time window and the collision impact center during the video preprocessing stage, providing stable physical anchor points for subsequent analysis. Simultaneously, it enables cross-temporal 3D perception and reconstruction of blind spot scenes, and improves blind spot recovery quality by introducing an uncertainty-driven active reconstruction mechanism in 3D reconstruction. Furthermore, it encodes physical laws into the reasoning process to achieve deep causal analysis. Finally, it constructs a confidence evaluation system with multi-source cross-validation to quantify the reliability of the analysis results, thereby improving the engineering usability, credibility, and auditability of automatic traffic accident analysis.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: A large-scale traffic accident analysis method integrating blind spot completion and physics-driven inference, comprising the following steps: Step S1. Acquire a continuous video stream, extract global motion, object-level dynamics and multi-target interaction features in parallel based on the acquired data, filter candidate time slices based on risk scoring, and extract collision spatial localization information based on the global motion intensity within the candidate time slices. Step S2. Select candidate time slices Monocular depth estimation is performed on each frame of the image to obtain a depth map. The depth map is then combined with the pixel points to back-project to the 3D physical space in the camera coordinate system, and the object-level 3D position is extracted simultaneously. Then, the 3D points of each frame are uniformly mapped to the reference frame coordinate system, and the extracted object-level 3D positions are aligned to the reference frame coordinate system. Finally, a semantic completion network based on 3D sparse convolution and cross-attention mechanism is used to perform blind zone semantic completion and generate a global 3D accident snapshot. Step S3. Combine pixel-level depth uncertainty, collision-affected partition-level uncertainty and object-level depth uncertainty to perform differentiated quality control on 3D scene reconstruction, and at the same time combine global 3D accident snapshots and pixel-level uncertainty to generate the enhanced visual flow required in step S4. Step S4. Combining the outputs of steps S2 and S3, construct a dynamic description containing physical quantity encoding. Multimodal input feature set It utilizes a multimodal large model for reasoning, clearly defines reasoning constraint instructions and structured output instructions, and outputs the reasoning results; Step S5. Based on the output reasoning results and the physical reference information extracted in steps S1 and S2, perform consistency cross-validation on the reasoning results from three dimensions: time, space, and semantics.

[0010] As a preferred embodiment of the present invention, when screening candidate time slices in step S1, the optical flow field between adjacent frames is calculated, and the average modulus of the optical flow field is used as the first... Global motion intensity of a frame The target detection and tracking algorithm is run to detect and track targets in each frame, and calculates the velocity and acceleration of road users on the image plane based on the pixel coordinate sequence. For any pair of traffic participants, calculate their relative pixel distance and closing pixel velocity on the image plane to determine the collision time. And map it to proximity score Then, a time-axis risk score is generated using a linear weighting method based on global motion intensity, total average acceleration at the object level, and maximum proximity score. : ; in, For a moment The number of road users detected , and These are the weights for motion intensity, acceleration, and proximity, respectively. Finally, the continuous risk score sequence is compared with a preset threshold. Comparison, risk score at a certain moment When the threshold is exceeded, the moment is marked as a triggered state; subsequently, the consecutively triggered moments are merged into candidate time slices. For each candidate time slice Record its corresponding original visual fragment With the group of parties involved .

[0011] As a preferred embodiment of the present invention, in step S1, the amplitude-weighted centroid of the optical flow field is calculated based on the global motion intensity within the candidate time slice, and this centroid is used as the spatial location where the motion disturbance is most concentrated. ; in, and pixel coordinates Optical flow displacement components along the width and height directions of the image. and These are the image height and width, respectively. For the first Amplitude-weighted centroid of the frame light flow field.

[0012] As a preferred embodiment of the present invention, step S2 involves targeting the detected road users. , detect the box The covered pixel space and the global pixel set that satisfies the effective depth constraint The intersection is obtained, and the average of the three-dimensional coordinates of all pixels in the intersection is calculated to extract the object-level three-dimensional position, that is, the three-dimensional spatial coordinates of the road-using object in the camera coordinate system.

[0013] As a preferred embodiment of the present invention, step S2 is based on the continuous risk score in step S1. Determine the peak risk moment In a local time window Within the frame, the moment with the smallest centroid displacement between adjacent frames is selected as the reference frame. : ; in, To preset half window width, For the first Amplitude-weighted centroid of the frame-optical flow field; Then, the 3D points of each frame are uniformly mapped to the coordinate system of the reference frame to obtain a cross-temporal 3D scene representation. The extracted object-level 3D positions are aligned to the reference frame coordinate system, and the 3D physical velocity is calculated based on the aligned object-level 3D position sequence. With 3D physics acceleration .

[0014] As a preferred embodiment of the present invention, the semantic completion network in step S2 includes a 3D sparse convolutional encoder, a cross-attention completion module, and a motion pattern prior fusion layer; cross-temporal 3D scene representation. The voxelized 3D point cloud is input to the 3D sparse convolutional encoder. The 3D sparse convolutional encoder uses 3 layers of 3D sparse convolution to extract local 3D geometric features layer by layer. Its output historical frame 3D geometric features are used as key and value inputs to the cross-attention completion module. The cross-attention completion module uses the feature encoding of the blind zone spatial location in the current frame as the query and the 3D geometric features of historical frames as the key and value. Through a multi-head cross-attention mechanism, it transfers the 3D geometric features related to the blind zone location in historical frames to the blind zone voxels in the current frame, achieving cross-temporal and spatiotemporal completion of blind zone geometry and semantics, and obtaining preliminary 3D completion features. The blind zone spatial location in the current frame is based on the current field of view 3D scene representation. The difference between the observations and those observed across time series is determined to be that which was observed in the historical frame but not in the reference frame. Not appeared in The voxel positions within the blind zone are defined as the candidate voxel set. ; The motion pattern prior fusion layer is used to integrate motion pattern priors. The encoded vector, with the same dimensions as the 3D completion features, is fused into the output of the cross-attention completion module through element-wise addition and layer normalization, ultimately outputting a blind-spot-completed 3D scene representation. Wherein, the motion pattern prior It is by using candidate time slices Object-level 3D position after inner alignment 3D physics speed With 3D physics acceleration Obtained by performing time-series aggregation; The global 3D accident snapshot Then, the global 3D accident snapshot, the set of involved objects, and the object-level motion states are encoded into scene prior text. This allows step S4 to be invoked seamlessly.

[0015] As a preferred embodiment of the present invention, the method for determining the pixel-level depth uncertainty in step S3 is as follows: for each pixel depth value output by the depth estimation, its cognitive uncertainty is extracted simultaneously. The method for determining the partition-level uncertainty of collision impact is as follows: for each frame image within the candidate time slice... In its visual appearance features and depth map Adaptive oversegmentation is performed on the joint feature space to obtain initial scene fragments. For each initial scene fragment, a 3D traffic attribute feature vector is extracted, and traffic attribute consistency-driven hierarchical merging is performed on the traffic attribute feature vectors of all initial scene fragments to obtain the final accident semantic partition. Finally, all pixel-level depth uncertainties within each accident semantic partition are aggregated to obtain the partition-level depth uncertainty of each accident semantic partition. The method for determining object-level depth uncertainty is as follows: for each detected road use object, based on its detection bounding box... The covered pixel area calculates object-level depth uncertainty. That's all; If the partition-level depth uncertainty of an incident semantic partition exceeds a set threshold, then the incident semantic partition is considered a high-uncertainty region. For any valid pixel, if its region belongs to the set of high-uncertainty regions... Then, actively activate the cross-frame aggregation strategy: from candidate time slices Reliable observations were searched from historical frames, and 3D points were output after smoothing correction through multi-frame weighted averaging. Conversely, the original mapping points are directly retained; ultimately, an effective 3D point cloud is obtained after uncertainty suppression and active reconstruction enhancement. ; ; in, For pixels In the The collision geometry contribution weight in a frame is inversely proportional to the pixel-level depth uncertainty and the uncertainty of the region of the object involved. For the first Frame-time pixels Coordinates in the reference frame coordinate system.

[0016] As a preferred embodiment of the present invention, the enhanced fusion function is used in step S3. Original visual fragments With global 3D accident snapshot Pixel-level uncertainty two-dimensional graph and effective 3D point cloud Perform joint rendering to obtain the enhanced visual stream required for the multimodal large language model in step S4. : ; in, This is the original visual fragment. To enhance the fusion function, a linear superposition mechanism based on Alpha channel blending and 3D reprojection is adopted; t is the image of the t-th frame in the original visual segment.

[0017] As a preferred embodiment of the present invention, step S4 employs a multimodal large model based on the Transformer architecture, wherein the physical quantities encode dynamic descriptions. The construction method is as follows: for the set of involved objects output in step S1 The object in the middle, its 3D physical velocity 3D Physics Acceleration as well as The text is transformed into structured text according to the rules of physical kinematics; at the same time, the reference frame is... Corresponding global motion intensity Encode as scene-level motion perturbation indicators; The multimodal input feature set .

[0018] As a preferred embodiment of the present invention, in step S5, the reference frame used as the spatial alignment reference in step S2 is used. Defined as physical reference time Simultaneously, retrieve the proximity score at that moment. The two biggest key individuals involved And utilize its aligned three-dimensional position and Calculate the location of the accident impact center Simultaneously calculate the three-dimensional physical distance between key involved entities. The reference accident category is automatically determined based on the relationship between three-dimensional physical distance and safety threshold. Then compare the accident time in step S4 with the reasoning. With physical reference time A time consistency score is obtained; the three-dimensional coordinates of the center of the accident impact area are compared and inferred. Location of the accident's impact center A temporal and spatial consistency score is obtained; accident categories are compared and inferred. Compared with reference accident categories The type consistency score is obtained; finally, the harmonic average method is used to fuse the scores to obtain the final confidence score of the candidate time slice. When the final confidence score is lower than the preset confidence threshold, the risk review mechanism is automatically triggered and the corresponding inference result is marked as a low confidence event.

[0019] Advantages and beneficial effects of the present invention: (1) This invention provides a spatiotemporal synchronous localization and physical anchor point extraction mechanism based on cumulative optical flow features. Addressing the technical shortcomings of existing video preprocessing methods that rely solely on global motion features for temporal dimension filtering and cannot locate the spatial position of collisions, thus leading to a lack of prior knowledge in downstream reconstruction, this application innovatively introduces a spatiotemporal analysis mechanism based on cumulative optical flow. By calculating the spatial cumulative distribution of dense optical flow amplitude over time within the detected accident time window and further extracting the weighted centroid of this cumulative amplitude map, the scope of optical flow features is expanded from a single temporal boundary filtering to spatiotemporal synchronous localization. This mechanism simultaneously extracts the collision impact center while outputting the accident time window in the early stage, i.e., outputting a highly reliable two-dimensional collision impact center in one step, providing a collision spatial reference benchmark for subsequent cross-temporal 3D geometric mapping. This not only effectively reduces the spatial search redundancy of subsequent 3D reconstruction but also provides a powerful underlying objective physical anchor point for the subjective reasoning stage.

[0020] (2) This invention provides a cross-temporal 3D geometric mapping and scene completion mechanism for collision reconstruction in the field of view (OOV). Addressing the problem that large models cannot obtain evidence when key objects move out of the camera's field of view during an accident (such as side collisions or blind spot crossings), this application proposes a cross-temporal 3D geometric mapping and scene completion mechanism based on physical benchmarks. The system utilizes the risk score peak and optical flow centroid stability extracted in the first step to adaptively select a reference frame with the highest physical significance and spatial robustness. Then, depth estimation is used to back-project the 2D features of consecutive historical frames to a unified 3D metric space and complete rigid body alignment. Finally, combining 3D sparse convolution and a cross-attention network, the geometric structure and semantic state of the objects involved in the historical frames are transferred to the current frame's blind spot, achieving cross-temporal extraction and completion of the geometry and semantics of the objects involved in the current blind spot (blind spot accidents, such as ghost collisions or side scrapes). This mechanism overcomes the limitations of the physical field of view of a single camera, providing a global 3D accident snapshot containing the blind spot structure for multimodal large models.

[0021] (3) This invention provides a layered uncertainty-driven 3D reconstruction enhancement for OOV blind zone recovery. It addresses the problems of decreased monocular depth estimation accuracy and severe contamination of cross-temporal 3D reconstruction quality by low-confidence noise in complex traffic scenarios (such as rain, fog, nighttime, and reflective surfaces). This application, from the perspective of traffic scene attribute perception, jointly models depth perception uncertainty at three traffic semantic levels: road texture unit, collision impact zone, and involved traffic participants. Specifically, at the collision impact zone level, it innovatively proposes an adaptive accident semantic partitioning method based on traffic scene attribute perception. Through hierarchical merging driven by traffic attribute consistency, the partition boundaries naturally align with the collision contact area and the outline of the involved vehicle, ensuring that the partition-level uncertainty accurately reflects the traffic physical state of the area. Furthermore, it guides the quality of 3D backprojection through uncertainty weighting and actively activates cross-frame feature aggregation strategies in local areas identified as high-risk blind zones, effectively suppressing the propagation of underlying depth estimation errors. This mechanism realizes a paradigm shift from passive frame feature overlay (passive stitching) to active confidence reconstruction, significantly improving the physical fidelity of 3D geometric and semantic feature recovery in blind spot scenarios.

[0022] (4) This invention provides a multimodal input prompt construction mechanism driven by physical laws. Addressing the problem of blurred pixel-level physical motion perception in purely visual large models, this application constructs a three-in-one input prompt structure comprising enhanced visual flow, scene text priors, and quantitative dynamic data. Quantitative features (quantitative physical quantities) such as the velocity, acceleration, approximate collision time, and global motion intensity of the objects involved in the three-dimensional physical space are structured and encoded into text, and integrated as hard constraints into the forward inference of the large model. This mechanism forces the large language model to strictly follow the physical kinematics foundation for logical deduction, effectively overcoming the inaccuracies in judgments caused by the lack of factual basis in traditional end-to-end black-box large models.

[0023] (5) This invention constructs a Physically Anchored Spatiotemporal Semantic Cross-Validation (PA-SSC) confidence assessment system. To address the potential hallucinations and uncontrollable reasoning issues in multimodal large models, this invention innovatively proposes an online self-checking mechanism for PA-SSC. This application extracts the time, space, and category of traffic accidents as the minimum evaluation feature basis and reuses the objective kinematic parameters extracted from the underlying computation as the physical anchoring benchmark. Furthermore, an anisotropic Gaussian flexible penalty mechanism and a semantic soft matching algorithm based on hierarchical trees are introduced to accurately quantify the spatiotemporal and qualitative deviations of the model output. A harmonic average fusion strategy is used to construct a forced blocking mechanism for a single abnormal dimension, causing the overall confidence to drop rapidly when any dimension shows a significant deviation, thus assigning a quantitative confidence index to each analysis report. This system achieves deep cross-validation between natural language inference results and underlying objective physical laws, assigning reliable quantitative confidence indexes to analysis reports, improving the credibility and auditability of the system in serious scenarios such as law enforcement, and thoroughly meeting the strict engineering requirements for the credibility and interpretability of technical evidence in traffic law enforcement scenarios. Attached Figure Description

[0024] Figure 1 Flowchart of the large-model traffic accident analysis method that integrates blind spot completion and physics-driven reasoning provided by this invention; Figure 2 This is a flowchart of the data processing for a semantic completion network based on 3D sparse convolution and cross-attention mechanisms. Detailed Implementation

[0025] To enable those skilled in the art to clearly understand and reproduce the technical solutions proposed in this application, the application will be described in detail below with reference to the accompanying drawings and mathematical formulas.

[0026] like Figure 1 As shown, the core of the large-scale traffic accident analysis method that integrates blind spot completion and physics-driven inference provided in this embodiment lies in a cascaded processing architecture. The method includes the following steps: Step S1. Acquire a continuous video stream, extract global motion, object-level dynamics and multi-target interaction features in parallel based on the acquired data, filter candidate time slices based on risk scoring, and extract collision spatial localization information based on the global motion intensity within the candidate time slices. Step S2. Cross-temporal 3D geometric mapping and scene completion oriented towards the field of view (OOV); specifically, for candidate time slices... Images in each frame Monocular depth estimation is performed to obtain a depth map; the depth map is then combined with the pixel points to back-project to the 3D physical space in the camera coordinate system, and the object-level 3D position is extracted simultaneously; then the 3D points of each frame are uniformly mapped to the reference frame coordinate system, and the extracted object-level 3D positions are aligned to the reference frame coordinate system; finally, a semantic completion network based on 3D sparse convolution and cross-attention mechanism is used to perform blind zone semantic completion and generate a global 3D accident snapshot; Step S3. Combine pixel-level depth uncertainty, collision-affected partition-level uncertainty and object-level depth uncertainty to perform differentiated quality control on 3D scene reconstruction, and at the same time combine global 3D accident snapshots and pixel-level uncertainty to generate the enhanced visual flow required in step S4. Step S4. Combining the outputs of steps S2 and S3, construct a dynamic description containing physical quantity encoding. Multimodal input feature set It utilizes a multimodal large model for reasoning, clearly defines reasoning constraint instructions and structured output instructions, and outputs the reasoning results; Step S5. Based on the output reasoning results and the physical reference information extracted in steps S1 and S2, perform consistency cross-validation on the reasoning results from three dimensions: time, space, and semantics.

[0027] Furthermore, in this embodiment, step S1 specifically includes the following steps: Step S1.1 Data Acquisition: First, a continuous video stream is acquired from one or more video sources. These video sources can be fixed surveillance cameras installed on roads (such as intersections or highway sections) or dashcams installed inside vehicles. After acquiring the continuous video stream, the system decodes it to obtain a sequence of frames arranged in chronological order. ;in For the first Frame image, and These are the image height and width (in pixels), respectively.

[0028] Step S1.2. Candidate time slice selection based on time axis risk score; This step efficiently filters and locates short clips containing potential accident risks from massive amounts of video data. It involves rapidly scanning the timeline and using a lightweight preprocessing module to calculate a timeline risk score for each moment. This enables the location of triggers during periods of suspected risk.

[0029] In the specific feature extraction stage, this embodiment extracts global motion, object-level dynamics, and multi-target interaction features in parallel.

[0030] First, in order to characterize the overall motion disturbance of the scene and capture the sudden changes caused by the accident, the system calculates the optical flow field between adjacent frames. ,in and pixel coordinates The optical flow displacement components (unit: pixels / frame) along the image width direction (horizontal) and height direction (vertical) are defined, and the average modulus of the optical flow field is defined as the first... Global motion intensity of a frame :

[0031] In addition, to characterize the kinematic changes of road users, a lightweight object detection and tracking algorithm is run to perform object detection and tracking for each frame and obtain the time-series data. Road use objects and its detection frame The pixel coordinates of the center of the detection box in the image are: Based on the pixel coordinate sequence, the motion velocity of the object on the image plane is defined. With acceleration as follows:

[0032] in, The time interval between adjacent frames (unit: seconds). Pixel velocity and pixel acceleration (units: pixels / second, pixels / second) ).

[0033] Based on this, in order to estimate the potential collision approach risk, this embodiment considers any pair of traffic participants. Calculate its relative pixel distance on the image plane. With closing pixel speed :

[0034] in, It is a numerically stable term.

[0035] Subsequently, in order to convert the above physical quantities into a risk metric, the system defines an approximate collision time based on the closing velocity. When the closing velocity is positive (the two objects are approaching each other), A smaller value means a higher risk of collision; when the closing velocity is zero or negative (the two objects are moving away from each other or are stationary), there is no risk of collision. Defined as positive infinity. Furthermore, based on... Define proximity score :

[0036]

[0037] in, For proximity scale parameters (preferred value range is ) (seconds). When hour, In other words, objects that are far from the risk score do not contribute to the risk score.

[0038] To integrate the aforementioned multidimensional features, the system then performs risk score fusion, using a linear weighting method to generate a time-axis risk score based on global motion intensity, object-level average acceleration, and maximum proximity score.

[0039] in, For a moment The number of road users detected , and These are the motion intensity weight, acceleration weight, and proximity weight, all of which are parameters obtained through self-learning during the training process.

[0040] Finally, the continuous risk score sequence is compared with a preset threshold. Comparison, risk score at a certain moment When the threshold is exceeded, the moment is marked as a triggered state; subsequently, the consecutively triggered moments are merged into candidate time slices:

[0041] in, and The first The start and end times of each candidate time slice. This represents the number of candidate time slices. For each candidate time slice... The system defines its corresponding original visual fragments and the set of objects involved:

[0042] in, For candidate time slices The original image sequence within, If the proximity score exceeds the preset risk threshold within this time slice. The set of identifiers of the involved parties.

[0043] Step S1.3. Extraction of collision spatial positioning information; This step determines the center of motion disturbance in the collision-affected area based on the global motion intensity within the candidate time slice, thereby achieving accurate spatial positioning of the collision.

[0044] Furthermore, in this embodiment, in order to achieve global motion intensity Based on this, further extract the spatial positioning information of the collision and calculate the first... Amplitude-weighted centroid of the optical flow field This is taken as the spatial location where the motion perturbation is most concentrated in that frame:

[0045] In this embodiment, optical flow features are expanded from a simple temporal candidate screening tool to a spatiotemporal synchronous positioning tool, which not only answers "when the accident occurred", but also further answers "where the collision disturbances gather", providing a collision space reference benchmark for subsequent cross-temporal 3D geometric mapping and scene completion.

[0046] Furthermore, in this embodiment, step S2 includes the following steps: Step S2.1 2D-to-3D back projection based on depth estimation; Traffic accident analysis relies on three-dimensional geometric information, such as physical distance, relative speed, and collision angle, which cannot be directly obtained from two-dimensional pixels. Given that actual data primarily comes from monocular monitoring and dashcam equipment, and that multi-view solutions require additional hardware support and have low engineering feasibility, this embodiment uses a monocular depth estimation method to obtain scene depth.

[0047] Specifically, a deep estimation model based on the Transformer architecture (Depth Anything V2) is used to evaluate candidate time slices. Images in each frame Perform monocular depth estimation to obtain a depth map. ;in, Represents pixel coordinates The depth value of a scene point along the camera's optical axis, i.e., the depth of that scene point in the camera coordinate system. Physical distance along the axis, in meters. Combined with the intrinsic parameter matrix obtained from camera calibration. The pixels are back-projected into the 3D physical space of the camera coordinate system:

[0048] in, Represents the three-dimensional spatial coordinates of a pixel in the camera coordinate system.

[0049] Furthermore, regarding the road usage objects detected in step S1 To eliminate interference from background areas and invalid depth estimates within the detection box, the detection box is... The covered pixel space and the global pixel set that satisfies the effective depth constraint Take the intersection (i.e.) This serves as the effective pixel anchor point for depicting the object.

[0050] Based on this, the object-level 3D position representation is extracted by averaging the 3D coordinates of all pixels within the intersection:

[0051] in, For object-level 3D position, that is, the 3D spatial coordinates of the object in the camera coordinate system for the i-th road.

[0052] Through the above formula, the two-dimensional detection box in step S1 is elevated to a three-dimensional spatial location description with physical scale.

[0053] Step S2.2. Cross-temporal 3D geometric mapping and coordinate system alignment; Traffic accidents exhibit significant temporal characteristics, and the relative positions and changes in motion of the involved parties before and after the collision constitute crucial evidence for determining liability. However, due to the inherent motion of vehicle-mounted or surveillance cameras during data acquisition, the 3D point clouds in each frame are typically situated in different coordinate systems. Direct superposition of these frames will result in spatial drift and ghosting, thus affecting the accuracy of motion analysis.

[0054] To this end, this step achieves coordinate system one through cross-temporal rigid body mapping, aligning multiple frames of 3D observations within the candidate time slice to the same reference coordinate system. This allows for a stable characterization of the relative pose and motion changes of the object under a unified physical scale, and provides a consistent differential reference for dynamic quantities such as velocity and acceleration, avoiding pseudo-motions introduced by coordinate inconsistencies.

[0055] This embodiment assumes that three-dimensional points in the camera coordinate system have already been obtained. Based on this, firstly, based on the continuous risk score in step S1 Determine the peak risk moment .

[0056] Furthermore, to avoid selecting frames that are significantly affected by motion blur or optical flow disturbances, in Nearby local time window Inside( To preset the half-window width, the preferred value is [value]. (frame), introducing the frame-by-frame optical flow amplitude weighted centroid in step S1. Stability is used as an auxiliary constraint, and the moment with the smallest centroid displacement between adjacent frames is selected as the reference frame:

[0057] The resulting reference frame simultaneously satisfies the conditions of being close to the risk peak and having a stable spatial structure, thereby improving the robustness of 3D alignment.

[0058] Based on this, the results from any frame are obtained through camera extrinsic estimation. to reference frame Rigid body transformation relations This maps the 3D points of each frame to the reference frame coordinate system, thereby unifying the mapping of 3D points from each frame to the reference frame coordinate system. It is a rotation matrix; For the translation vector, the corresponding mapping relationship is:

[0059] Furthermore, the object-level three-dimensional position extracted from the above formula is... Align to reference frame Coordinate system:

[0060] in, The pixel and the i-th road are used in the reference frame respectively. Coordinates in a coordinate system.

[0061] Calculate 3D physical velocity and 3D physical acceleration based on the aligned object-level 3D position sequence:

[0062] in, The time interval between adjacent frames (unit: seconds). These are the 3D physical velocity and 3D physical acceleration (units: m / s, m / s) of the i-th road-using object at time t (frame t). ).

[0063] Finally, the cross-temporal 3D scene representation is defined as:

[0064] Through the above process, the two-dimensional detection box, center point and pixel-level motion observation in step S1 are upgraded to a three-dimensional spatial position description and motion state with physical scale, providing a unified measurement benchmark for subsequent blind spot completion.

[0065] Step S2.3. Blind spot semantic completion and global 3D accident snapshot generation; In real traffic accidents, many collisions occur in blind spots such as the sides or rear of vehicles or the edges of the field of view, such as "ghost pedestrians" or side-changing collisions. Because the objects involved rapidly enter or exit the field of view before and after the collision, or are obstructed, the observation information in the current frame is incomplete, thus interrupting the causal chain of the accident within the two-dimensional field of view. If inference is based solely on the pixels of the current frame, the multimodal large model will struggle to maintain causal consistency due to the lack of key targets. To recover crucial spatiotemporal information, this step performs blind spot semantic completion based on the cross-temporal 3D scene representation and generates a global 3D accident snapshot. Compared to 2D repair methods, 3D representation can maintain physical scale consistency and provide structured spatial information, thereby forming an interpretable chain of evidence.

[0066] Specifically, in this embodiment, a cross-temporal 3D scene representation is used. Prior to motion patterns Perform blind spot semantic completion, transferring the geometric structure and semantic state of the involved object that has been moved out of the current view in a historical frame to the 3D scene of the current frame:

[0067] in, This is a semantic completion network based on 3D sparse convolution and cross-attention mechanism. The motion pattern prior extracted from historical frames is based on the temporal aggregation of the object-level 3D position, velocity, and acceleration obtained in step 2.2.

[0068] in, As a temporal aggregation function, this embodiment preferably adopts the attention-weighted pooling method to jointly encode the object-level three-dimensional position, three-dimensional velocity and three-dimensional acceleration at each historical moment into a unified prior representation of motion mode.

[0069] Furthermore, in this embodiment, it is possible to first consider any given time... Constructing feature vectors from the motion state of objects :

[0070] in, This indicates a feature concatenation operation. This represents a feature encoding unit that maps the motion state of the concatenated object, used to map position, velocity, and acceleration information to a unified feature space. Subsequently, based on the features at each time step... Calculate the corresponding time series weights :

[0071] in, This represents a weighted scoring mapping unit, used to calculate its contribution to the overall motion pattern representation based on the motion state features at each time step. Finally, a weighted summation is performed on the motion state features at each time step to obtain the motion pattern prior:

[0072] Through the above aggregation method, the system can differentiate the motion state at different times before and after the accident, so that key moments with obvious acceleration, deceleration, turning or approach behavior can be given higher contributions, thereby preserving both the long-term evolution trend and instantaneous change characteristics of the object's motion.

[0073] Furthermore, such as Figure 2 As shown in this embodiment, the semantic completion network based on 3D sparse convolution and cross-attention mechanism... The structure operates according to the workflow of "voxelation encoding – blind zone candidate localization – cross-frame feature completion – motion prior fusion", specifically including a 3D sparse convolutional encoder, a cross-attention completion module, and a motion pattern prior fusion layer; wherein, the output of the 3D sparse convolutional encoder serves as the key of the cross-attention completion module. Sum The input and output of the cross-attention completion module are fed into the motion mode prior fusion layer, and finally output as a blind spot complete 3D scene representation. .

[0074] Specifically, the 3D sparse convolutional encoder represents the cross-temporal 3D scene in step S2.2. Using voxelized 3D point clouds as input, local 3D geometric features are extracted layer by layer using a 3-layer 3D sparse convolution (SparseConv3D). The kernel size of each 3D sparse convolution layer is [missing value]. The number of output channels is 64, 128, and 256, respectively, and the activation function is ReLU. Since the effective point cloud accounts for a small proportion of the overall 3D space in traffic accident scenes, sparse convolution is only performed at non-empty voxel locations, thereby avoiding redundant operations on blank areas and reducing overall computational overhead.

[0075] Furthermore, in this embodiment, the reference frame The portion of the current field of view 3D scene representation that can be directly obtained from the monocular depth estimation in step S2.1 and the coordinate alignment in step S2.2 is defined as... The spatial location of the current frame's blind zone is not manually specified, but rather determined by the aforementioned... The discrepancies between the observations and those observed across historical timeframes are determined. Specifically, those observed in historical frames but not in the reference frame are identified. Not appeared in The voxel positions within the blind zone are defined as the candidate voxel set. The cross-attention completion module generates a query based on the spatial location encoding of the candidate voxels in the blind zone. .

[0076] The cross-attention completion module uses the feature encoding of the blind zone spatial location in the current frame as the query, and the 3D geometric features of the historical frames as the key and value. Through a multi-head cross-attention mechanism (preferably with 4-8 heads), it transfers the 3D geometric features highly correlated with the blind zone location in the historical frames to the blind zone voxels in the current frame, achieving cross-spatial completion of the blind zone geometry and semantics, and obtaining preliminary 3D completion features. The formula for calculating attention is:

[0077] in, For candidate voxel sets based on blind zone Feature queries generated by spatial location encoding The keys and values ​​for the effective voxel features of historical frames. The attention head dimension (preferably 256).

[0078] The motion pattern prior fusion layer will integrate motion pattern priors. Encoding as preliminary 3D completion features Vectors of the same dimension are fused into the output of the cross-attention completion module through element-wise addition and layer normalization (LayerNorm), ultimately outputting a blind-spot-completed 3D scene representation. The completion result not only includes static geometric and semantic information, but also incorporates the prior knowledge of the dynamic motion trend of the object involved.

[0079] In particular, when the pre-accident history sequence is long (e.g., greater than 5 seconds), causing the weighted contribution of cross-attention to early information to be diluted, As a global memory item obtained through cross-temporal aggregation, it participates in the fusion process, compensating for early motion information and thus reducing the impact of attentional forgetting on the quality of remote cause completion in long-sequence scenes. The final result is a global 3D accident snapshot containing blind spot structures. :

[0080] in, This refers to the 3D scene representation portion corresponding to the visible area of ​​the current frame. This completes the 3D scene representation by filling in the blind spots calculated by the aforementioned network modules.

[0081] Furthermore, to ensure that the output of step S2 can be directly organized into a multimodal cue in step S4, the system further encodes the global 3D accident snapshot, the set of involved objects, and the object-level motion states into scene prior text. Specifically, a structured scene prior is constructed:

[0082] in, Indicates reference frame The set of object-level three-dimensional spatial positions of each involved object in the reference frame coordinate system at any given time. This represents the set of three-dimensional velocities and three-dimensional accelerations of each involved object at the reference frame time. Used to determine the road topology, relative object orientation, blind spot completion area location, key object interaction relationships and reference frames that exceed the proximity score. The corresponding scene state is encoded as natural language or structured text fragments, enabling the geometric space processing results of this step to be seamlessly invoked by the downstream step S4 in text form.

[0083] This step constructs a cross-temporal 3D geometric mapping and scene completion mechanism to restore the geometric structure and motion state of the objects involved in the blind spot, and improves the candidate time slice parameters obtained in step S1 into a 3D scene representation with a unified physical scale, thereby providing a consistent metric basis for subsequent completion and accident reasoning.

[0084] Furthermore, in this embodiment, step S3 includes the following steps: Step S3.1. Hierarchical uncertainty estimation: Traffic accidents often occur under adverse road conditions—mirror reflections from rain-soaked pavement, blurred target outlines due to low nighttime illumination, and glare at tunnel entrances and exits obscuring vehicle posture—these unique optical degradation factors in traffic scenarios severely interfere with the reliability of monocular depth perception. If depth estimation results are adopted indiscriminately, reflective areas from accumulated water will be misjudged as nearby obstacles, and vehicle outlines in dark areas will undergo geometric distortion. This leads to displacement of the collision contact point, distortion of the three-dimensional pose of the vehicle involved, and ultimately, a lack of physical basis for determining accident liability.

[0085] Therefore, this step starts from the perspective of traffic scene attribute perception, models the deep cognitive uncertainty at three traffic semantic levels: road surface texture unit, collision impact zone and traffic participants involved (the objects involved), and applies it to the three-dimensional mapping result of step S2 to achieve active reconstruction enhancement for the core area of ​​the accident.

[0086] Step S3.1.1. Uncertainty at the unit level of road surface texture (pixel-level depth uncertainty): For each pixel depth value in the depth estimation output Simultaneously extract its cognitive uncertainty This is used to quantify the reliability of depth sensing at that location:

[0087] in, For samples obtained via Monte Carlo Dropout Sub-depth prediction (preferred value is 8-16 times). The depth prediction obtained from the s-th Monte Carlo Dropout sampling. For variance calculation; The larger the value, the less reliable the 3D position estimate of that point. In scenarios such as rain, nighttime, or strong reflections, pixel-level uncertainty typically increases significantly, thus providing an explicit reliability indicator for the system and a basis for subsequent uncertainty-weighted reconstruction.

[0088] Step S3.1.2. Collision affects partition-level uncertainty: In traffic accident analysis, different areas of the scene possess drastically different traffic physical attributes and accident semantic roles: the collision stress zone (vehicle contact surface, pedestrian impact area) requires the highest precision 3D geometric reconstruction; the road surface friction zone (water accumulation reflective strip, asphalt-marking boundary) suffers from systematically low depth estimation due to optical interference; and the vulnerable participant activity zone (pedestrian crossing, non-motorized vehicle lane) has low confidence in depth estimation due to small target size and weak texture. If a fixed mesh is used, the same collision stress zone may be fragmented into different meshes, or water accumulation reflective strips may be improperly merged with normal road surfaces, resulting in partition-level uncertainty that fails to accurately reflect the true traffic physical attributes of the area. Therefore, this embodiment proposes an adaptive accident semantic partitioning method based on traffic scene attribute perception, ensuring that partition boundaries naturally conform to the physical structure of the traffic scene. Specifically, it includes the following steps: Step S3.1.2.1. Traffic scene attributes over-segmentation; For each frame of the candidate time slice In its visual appearance features and depth map Perform adaptive oversegmentation on the joint feature space to generate A compact initial scene fragment (Preferred value) ,in (For fragment indexing). Specifically, this oversegmentation process uses the joint distance, which includes pixel color, two-dimensional spatial coordinates, and three-dimensional depth values, as the clustering metric. When adjacent pixels cross regions of abrupt changes in depth or color, their joint distance increases significantly, thus preventing the same fragment from crossing physical edges. This oversegmentation iteratively converges in the joint feature space according to this metric, making the boundaries of the generated fragments naturally fit the physical structures in the traffic scene, such as vehicle outlines, road surface material boundaries, and pedestrian crossing edges. This ensures that pixels within the same fragment have similar traffic physical properties—pixel depths on the same vehicle surface are continuous and have a consistent appearance, while pixel depths in the same road surface material region are smooth and have similar reflective properties.

[0089] Step S3.1.2.2. Traffic scene attribute feature extraction; For each initial scene fragment Extracting 3D traffic attribute feature vectors The components are defined as follows:

[0090]

[0091]

[0092] in, The average depth distance within the segment reflects the distance relationship of the segment in the traffic scene (near vehicles vs. distant road surface). The uncertainty of average road surface optical degradation within a segment reflects the degree of traffic optical interference in that segment (reflective areas of water accumulation vs. normal dry road surface). The centroid coordinates of the segment reflect its spatial location within the scene (collision force zone vs. background area far from the accident). These three feature dimensions respectively characterize the segment's traffic distance attribute, traffic optical reliability attribute, and traffic spatial role attribute, providing a physically meaningful grouping basis for subsequent accident semantic partitioning.

[0093] Step S3.1.2.3. Merge the semantic partitions of the incident; Traffic attribute feature vectors for all initial scene fragments Perform hierarchical merging driven by traffic attribute consistency. Define the traffic attribute distance between any two spatially adjacent segments as the weighted feature difference:

[0094] in, This is a diagonal weight matrix, whose diagonal elements are respectively from step S3.1.2.2 , , Weighting coefficients for the three feature dimensions; This is used to determine whether two segments belong to the same traffic semantic level. When the two segments are located on the surface of the colliding vehicle and the distant background road surface, respectively, they are considered to belong to the same traffic semantic level. Completely different The two segments are too large, so the system determines they should not be merged; when the two segments belong to a waterlogged reflective area and a dry road surface respectively, they should be merged. Completely different The two values ​​are too large, so the system determines that they should not be merged.

[0095] In each merging iteration, the system selects the adjacent segments with the smallest traffic attribute distance to merge. The feature vector of the merged segment is then updated with a weighted average of the number of pixels, as shown in the following formula:

[0096] The above iterations are repeated until the number of remaining partitions reaches a learnable threshold. , The parameters, determined adaptively through training, are optimized by minimizing the deviation between the partition-level uncertainty estimate and the true depth error, resulting in the final accident semantic partition. (in (For partition indexing). Through traffic attribute consistency constraints, each accident semantic partition has internal consistency in depth distance, optical reliability, and spatial role, thereby ensuring that partition-level uncertainty can accurately reflect the real traffic physical state of the partition.

[0097] Step S3.1.2.4. Calculation of partition-level uncertainty; After obtaining the accident semantic partitions, the depth uncertainty of all road surface texture units within each partition is aggregated:

[0098] in, For the first The partition-level depth uncertainty of an accident semantic partition is an indicator that characterizes the overall depth perception reliability within a specific accident semantic partition. Since the partition boundary conforms to the irregular shape of the collision contact area and the outline of the vehicle involved, when a partition covers the collision force point, its uncertainty will directly affect the accuracy of collision angle calculation and vehicle pose recovery. When the partition corresponds to the road friction area, the increased uncertainty indicates that the depth observation in that area is affected by optical degradation interference such as water reflection, and needs to be corrected through cross-temporal optical compensation.

[0099] Step S3.1.3. Uncertainty at the level of traffic participants involved (object-level deep uncertainty); For each road use object detected in step S1 (Including vulnerable road users such as motor vehicles, non-motor vehicles, and pedestrians), based on their detection frame The covered pixel area calculates object-level depth uncertainty. :

[0100] This indicator has a direct impact on the determination of accident liability: the high uncertainty of the outline of the vehicle involved will cause the three-dimensional coordinates of the collision contact point to shift, and the high uncertainty of the surroundings of vulnerable participants (such as pedestrians and cyclists) may cause their spatial position in the 3D scene to drift, thereby affecting the determination of the collision sequence and liability.

[0101] Step S3.2. Uncertainty Weighting and Active Reconstruction; Based on the uncertainty estimation of the three traffic semantic levels mentioned above, the system performs differentiated quality control on the 3D scene reconstruction: the contribution of regions with high depth perception credibility in collision geometry reconstruction is enhanced, while the 3D observations of depth perception degradation regions are downweighted.

[0102] Specifically, the 3D points corresponding to each effective pixel It is assigned a collision geometry contribution weight that is inversely proportional to the pixel-level depth uncertainty and the uncertainty of the region of the object it belongs to:

[0103] in, To prevent stable terms with a denominator of zero (preferably within a certain range) When pixel (u,v) does not belong to the bounding box of the object in question, only the pixel-level uncertainty term can be retained for calculation. The traffic physics meaning of this weight is: when the pixel is located within the bounding box of the object in question, the object-level depth uncertainty... Further adjustments are made to the contribution weight of the corresponding 3D point for each pixel to suppress unreliable observations under conditions of blurred object contours, partial occlusion, or weak texture. Dry asphalt pavement texture units and areas with clear vehicle contours have high depth perception reliability due to good optical conditions, and their geometric contribution to the 3D coordinate localization of the collision contact point and the pose recovery of the involved vehicle should be strengthened. Conversely, low-confidence depth values ​​corresponding to areas of water accumulation and specular reflection or unlit dark areas are suppressed to prevent them from introducing erroneous depth offsets into the reconstruction of the core accident space, leading to distortion in collision angle calculations or spatial position drift of vulnerable participants.

[0104] At the same time, define a set of regions with high uncertainty. ,in A preset uncertainty threshold (preferably within a certain range) is set. ).

[0105] For any valid pixel If its location area The system identifies this area as a high-risk blind spot and proactively activates a cross-frame aggregation strategy: from candidate time slices... Reliable observations were searched from historical frames, and the aforementioned collision geometry contribution weights were used. As weights in a multi-frame weighted average, the output is a smoothed 3D point. :

[0106] in, For pixels In the The collision geometry contribution weight in the frame; the larger the weight, the more reliable the corresponding 3D observation of the frame, and the higher its contribution to the cross-frame aggregation result.

[0107] Let the 3D point after the above conditional judgment be... Its logic can be expressed as:

[0108] Finally, the effective 3D point cloud after uncertainty suppression and active reconstruction enhancement can be simply represented as:

[0109] Step S3.3. Enhance visual flow generation; Multimodal large models require intuitive visual evidence to support their analysis of traffic accidents: information such as the relative positions of the objects involved, the spatial distribution of the collision area, and the reliability distribution of depth estimates. If presented only in numerical form, large models struggle to understand this information effectively. The reason for using enhanced visual flow generation is to transform the abstract underlying 3D geometric mapping and depth reliability metrics into intuitive image features, enabling large models to "see" the confidence distribution of depth estimates, thereby assigning higher weights to high-confidence regions during inference.

[0110] Original visual fragments This is combined with the aforementioned computationally generated uncertainty features, enhanced point clouds, and global 3D accident snapshots for joint rendering. An enhanced fusion function is defined. Used to complete the blind spot results and the pixel-level uncertainty two-dimensional map (denoted as...) The image is then overlaid with 3D alignment information. Each frame of the image Performing this rendering operation yields the enhanced visual flow required for the multimodal large model in step S4. :

[0111] In this embodiment, to satisfy system implementability, the enhanced fusion function Specifically, a linear superposition mechanism based on Alpha channel blending and 3D reprojection is adopted to provide explicit and physically consistent visual evidence to multimodal large models.

[0112] Furthermore, in this embodiment, step S4 includes the following steps: This step follows the output of the global 3D accident snapshot from step S2. Object-level 3D motion state (velocity) With acceleration ) and scenario prior text (Including the spatial set of the objects involved) Set of velocities and accelerations of the objects involved (Including road topology, relative object orientation, and blind spot completion area location information), combined with the enhanced visual flow output from step S3. And the collision risk prior in step S1 (such as approximate collision time) Global motion intensity and the collection of parties involved To address the problem that pure visual large models have a blurred perception of pixel-level physical motion and struggle to capture underlying kinematic changes, this step proposes a multimodal input prompt construction mechanism driven by physical laws. This mechanism encodes quantitative dynamic data into the inference process, guiding the large model to output causal explanations of accidents that conform to real physical logic.

[0113] Step S4.1. Constructing a three-in-one multimodal input prompt: This embodiment designs a three-in-one input prompt structure integrating enhanced visual flow, scene text prior, and quantitative dynamic data. This is specifically designed for candidate time slices. Construct a multimodal input feature set :

[0114] in, This is an enhanced visual image sequence (enhanced visual flow) after being overlaid with uncertainty rendering. The context is prior text; This step involves constructing a "physical quantity encoding dynamic description".

[0115] Specifically, in this embodiment, the physical quantity encoding dynamic description The construction method is as follows: only for the set of involved objects output in step S1 The object in the middle, its 3D physical velocity 3D Physics Acceleration and approximate collision time Equal physical quantities are transformed into structured text according to the rules of physical kinematics; simultaneously, reference frames are... Corresponding global motion intensity Encode it as a scene-level motion disturbance index. For example, for a trajectory identified as... The dynamic description format of the vehicle involved in the accident was templated as: "trajectory" Current speed km / h, longitudinal acceleration m / s (Rapid deceleration), and trajectory "TTC = 1.2 s (extremely high collision risk)"; Scenario-level indicator format: "Global Motion Intensity" Pixels / frame (far exceeding normal levels) This indicates that a violent motion disturbance has occurred in the scene. This encoding method explicitly transforms abstract high-dimensional numerical arrays into natural language that large models can easily understand, and attaches physical semantic annotations (such as "sudden deceleration," "far exceeding normal levels") to force large models to follow these underlying quantitative facts in inference. Encoding only... The objects in the code can prevent irrelevant objects from interfering with reasoning, while The introduction of this feature enables large models to perceive the degree of anomalies in the overall motion of the scene, making up for the lack of perception of global disturbances by object-level physical quantities.

[0116] Step S4.2. Multimodal large model inference based on Transformer architecture; This embodiment uses a multimodal large model based on the Transformer architecture (preferably a model with a visual encoder and a language decoder aligned, such as InternVL2 or LLaVA-NeXT architecture) to perform structured inference.

[0117] Multimodal input prompts The data is fed into the inference engine to obtain a comprehensive structured inference output. :

[0118] in, Forward inference function of large model.

[0119] Overall output After parsing, it is decomposed into:

[0120] Each component is defined as follows: For the predicted accident category; The predicted moment of collision; The three-dimensional physical coordinates of the center of the predicted accident impact area; A collection of natural language interpretation texts, which contains scene context descriptions. Object behavior description And explanation of the cause of the accident .

[0121] Step S4.3. Structured reasoning prompt template driven by physical laws; To ensure that the multimodal large model can accurately fuse the above inputs and output strictly according to the format. For all dimensions in the dataset, this embodiment designs a corresponding prompt template to strongly bind cross-modal data using placeholders: Table 1 - Prompt Template for Multimodal Structured Inference Driven by Physical Laws

[0122] Furthermore, in this embodiment, step S5 includes the following steps: To quantitatively evaluate the reliability of multimodal large-scale model inference results and reduce the risk of misjudgment caused by visual model "illusions," this step introduces physically-anchored spatiotemporal-semantic cross-validation confidence (PA-SSC). This mechanism takes the comprehensive inference result output from step S4 as an example. The physical reference information extracted in steps S1 and S2 is used to perform consistency cross-validation of the reasoning results from three dimensions: time, space, and semantics.

[0123] Traffic accidents possess strict physical causal relationships and liability determination attributes. Therefore, this step selects "when it happened, where it happened, and what happened" as the core representation dimensions of the accident event. Temporal consistency measures the model's ability to locate moments of kinematic abrupt changes; spatial consistency assesses the model's accuracy in spatially locating the collision impact area; and semantic consistency constrains the correspondence between the model output and the legal accident classification. Furthermore, the system uses a harmonic averaging approach to fuse these scores, ensuring that a significant deviation in any dimension leads to a rapid decrease in overall confidence, thereby enhancing the system's reliability in traffic audit scenarios.

[0124] Step S5.1. Multi-dimensional cross-consistency scoring; In light of the unique characteristics of traffic accident scenarios, this step establishes the following three core consistency metrics: 1) Time-based consistency: Verify the model's accuracy in capturing the transient timing (when) of the extreme point of the accident. 2) Spatial positioning consistency: Verify the model's three-dimensional spatial positioning accuracy in the area affected by multi-agent collisions (where). 3) Semantic type consistency: Verify the qualitative accuracy of the model in classifying the cause and effect of accidents (what happened).

[0125] Furthermore, to achieve automated closed-loop verification, the system eliminates the need for additional manual annotation; instead, it directly reuses the parameters output from the normal calculations in the preceding steps to construct a physical reference benchmark. Specifically, the reference frame used as the spatial alignment benchmark in step S2 is... Directly defined as the physical reference time (i.e. Simultaneously, retrieve the proximity score at that moment. The two biggest key individuals involved And utilize its aligned 3D position and Calculate the location of the accident impact center ,in, and These represent the top-view two-dimensional coordinate components of the three-dimensional positions of the two key objects in the reference frame coordinate system.

[0126] For reference accident categories The system supports dual-mode operation: in offline audit mode, it accesses the historical database to obtain real tags; in real-time online mode, it automatically triggers classification based on whether the three-dimensional physical distance between objects exceeds a safety threshold. In real-time online mode, the three-dimensional physical distance between key involved objects is first calculated:

[0127] in, Indicates key parties involved and Three-dimensional physical distance in the reference frame coordinate system.

[0128] Furthermore, a reference accident category is automatically determined based on the relationship between the three-dimensional physical distance and the safety threshold. :

[0129] in, To determine the safety threshold for collisions, It is close to the collision detection threshold, and .

[0130] Based on this objective physical benchmark, the system calculates the cross-matching scores for the following three dimensions, with values ​​ranging from [value range missing]. : 1) Temporal Consistency Scoring: Traffic accidents are often accompanied by motion blur and occlusion. To avoid the vulnerability of hard time thresholds, the accident time is compared with the accident time predicted (inferred) by the model in step S4. With physical reference time Gaussian similarity scoring is used, defined as:

[0131] in, The time tolerance parameter (preferred value range is) The physical meaning of time in time in time in terms of seconds is: the smaller the deviation between the collision time predicted by the model and the underlying physical braking / collision time, the more reliable the time inference.

[0132] 2) Spatial Consistency Scoring: For spatial localization problems in multi-object interaction scenarios, the three-dimensional coordinates of the center of the accident impact area are compared with the model's predicted (inferred) coordinates. Compared with the physical reference location, i.e. the location of the accident's impact center Anisotropic Gaussian similarity is introduced, defined as:

[0133] in, This is a spatial tolerance parameter, whose preferred value is related to the physical width of the reference accident impact area in the top view. and physical length Correlation (unit: meters), in this embodiment, is taken as... , To adapt to the scale of different reference regions.

[0134] 3) Type consistency score: Compare the accident categories predicted by the model. With reference category Unlike traditional hard matching using univariate indicator functions, this invention innovatively introduces a flexible matching score based on an accident type semantic hierarchy tree:

[0135] in, This is a path similarity function based on a semantic hierarchy tree of accident types. Let the accident types form a semantic hierarchy tree, containing leaf nodes such as "rear-end collision" and "side collision," and abstract parent nodes such as "collision." The similarity is defined as:

[0136] in, for and The lowest common ancestor in the hierarchy tree, The node depth is defined. This mechanism allows the model to obtain a soft-match score when the prediction has a certain degree of reasonableness (such as belonging to the same collision class but with different subdivisions), preventing the confidence level from being over-penalized.

[0137] Step S5.2. Confidence fusion and risk warning; To comprehensively evaluate the model's consistency across the temporal, spatial, and semantic dimensions, the system employs a harmonic averaging approach to fuse the scores across all dimensions, yielding candidate time slices. Final PA-SSC confidence score :

[0138] Compared to the arithmetic mean, the harmonic mean is more sensitive to low scores, thus effectively suppressing high-confidence misjudgments caused by anomalies in a single dimension. When the score of any dimension is significantly low, the overall confidence will decrease accordingly, thereby ensuring that the model output is consistent with the underlying physical perception results at the three levels of temporal localization, spatial localization, and accident semantics.

[0139] When the final confidence score Below the preset confidence threshold (The preferred value range is) When a low-confidence event occurs, the system automatically triggers a risk review mechanism and marks the corresponding inference result as a low-confidence event to enter the manual review process, thereby reducing the impact of model illusions on practical applications.

[0140] Step S5.3. Final output packaging; The system ultimately encapsulates the candidate fragment boundaries from step S1, the structured reasoning from step S4, and the self-evaluation score from this step, and outputs a standard traffic accident analysis report for the incident. :

[0141] Through this structure, the generated report not only records in detail the causal logic deduced from the multimodal large model, but also... It also includes a quantitative confidence index for anti-hallucination. This greatly enhances the engineering practicality and auditability of the system of this invention in serious scenarios such as traffic management law enforcement and insurance liability determination.

[0142] The present invention also provides an electronic device, comprising: one or more processors and a memory; wherein the memory is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the above-described method for analyzing large-scale traffic accidents by fusing blind spot completion and physics-driven inference.

[0143] The present invention also provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described method for analyzing large-scale traffic accidents by fusing blind spot completion and physics-driven reasoning.

[0144] Those skilled in the art will understand that all or part of the functions of the various methods / modules in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the above functions can be implemented by executing the program with a computer. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be implemented.

[0145] In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the programs can also be stored in storage media such as servers, other computers, disks, optical discs, flash drives, or portable hard drives. They can be downloaded or copied to the memory of the local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be implemented.

[0146] The above-described specific examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention. Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A large model traffic accident analysis method fusing blind area completion and physical driving reasoning, characterized in that, The method includes the following steps: Step S1. Acquire a continuous video stream, extract global motion, object-level dynamics and multi-target interaction features in parallel based on the acquired data, filter candidate time slices based on risk scoring, and extract collision spatial localization information based on the global motion intensity within the candidate time slices. Step S2. performing monocular depth estimation on each frame image in the candidate time slice In each frame image in the candidate time slice, monocular depth estimation is performed to obtain a depth map; pixel points are back-projected to a 3D physical space in a camera coordinate system in combination with the depth map, and object-level three-dimensional positions are synchronously extracted; then the three-dimensional points of each frame are uniformly mapped to a reference frame coordinate system, and the extracted object-level three-dimensional positions are aligned to the reference frame coordinate system; finally, a semantic completion network based on 3D sparse convolution and cross-attention mechanism is used to perform blind area semantic completion, and a global 3D accident snapshot is generated. Step S3. Combine pixel-level depth uncertainty, collision-affected partition-level uncertainty, and object-level depth uncertainty to perform differentiated quality control on the 3D scene reconstruction. At the same time, combine global 3D accident snapshots and pixel-level depth uncertainty to generate the enhanced visual flow required in step S4. Step S4. Combining the outputs of steps S2 and S3, construct a dynamic description containing physical quantity encoding. Multimodal input feature set It utilizes a multimodal large model for reasoning, clearly defines reasoning constraint instructions and structured output instructions, and outputs the reasoning results; Step S5. Based on the output reasoning results and the physical reference information extracted in steps S1 and S2, perform consistency cross-validation on the reasoning results from three dimensions: time, space, and semantics.

2. The large-scale traffic accident analysis method integrating blind spot completion and physics-driven reasoning as described in claim 1, characterized in that, In step S1, when screening candidate time slices, the optical flow field between adjacent frames is calculated, and the average modulus of the optical flow field is used as the first... Global motion intensity of a frame ; The target detection and tracking algorithm is run to detect and track targets in each frame, and the velocity and acceleration of road users on the image plane are calculated based on the pixel coordinate sequence. For any pair of traffic participants, calculate their relative pixel distance and closing pixel velocity on the image plane to determine the collision time. And map it to proximity score ; Then, a time-axis risk score is generated using a linear weighting method based on global motion intensity, total average acceleration at the object level, and maximum proximity score. : ； in, For a moment The number of road users detected , and These are the weights for motion intensity, acceleration, and proximity, respectively. Finally, the continuous risk score sequence is compared with a preset threshold. Comparison, risk score at a certain moment When the threshold is exceeded, the moment is marked as a triggered state; subsequently, the consecutively triggered moments are merged into candidate time slices. For each candidate time slice Record its corresponding original visual fragment With the group of parties involved .

3. The large-scale traffic accident analysis method integrating blind spot completion and physics-driven reasoning as described in claim 2, characterized in that, In step S1, the amplitude-weighted centroid of the optical flow field is calculated based on the global motion intensity within the candidate time slice, and this centroid is used as the spatial location where the motion disturbance is most concentrated. ； in, and pixel coordinates Optical flow displacement components along the width and height directions of the image. and These are the image height and width, respectively. For the first Amplitude-weighted centroid of the frame light flow field.

4. The large-model traffic accident analysis method integrating blind spot completion and physics-driven reasoning as described in claim 3, is characterized in that... In step S2, the detected road usage objects are targeted. , detect the box The covered pixel space and the global pixel set that satisfies the effective depth constraint The intersection is obtained, and the average of the three-dimensional coordinates of all pixels in the intersection is calculated to extract the object-level three-dimensional position, that is, the three-dimensional spatial coordinates of the road-using object in the camera coordinate system.

5. The large-scale traffic accident analysis method integrating blind spot completion and physics-driven reasoning as described in claim 4, characterized in that, In step S2, the continuous risk scores from step S1 are used as a basis. Determine the peak risk moment In a local time window Within the frame, the moment with the smallest centroid displacement between adjacent frames is selected as the reference frame. : ； in, To preset half window width, For the first Amplitude-weighted centroid of the frame-optical flow field; Then, the 3D points of each frame are uniformly mapped to the coordinate system of the reference frame to obtain a cross-temporal 3D scene representation. The extracted object-level 3D positions are aligned to the reference frame coordinate system, and the 3D physical velocity is calculated based on the aligned object-level 3D position sequence. With 3D physics acceleration .

6. The large-scale traffic accident analysis method integrating blind spot completion and physics-driven reasoning as described in claim 5, characterized in that, The semantic completion network in step S2 includes a 3D sparse convolutional encoder, a cross-attention completion module, and a motion pattern prior fusion layer. Cross-temporal 3D scene representation The voxelized 3D point cloud is input to the 3D sparse convolutional encoder. The 3D sparse convolutional encoder uses 3 layers of 3D sparse convolution to extract local 3D geometric features layer by layer. Its output historical frame 3D geometric features are used as key and value inputs to the cross-attention completion module. The cross-attention completion module uses the feature encoding of the blind zone spatial location in the current frame as the query and the 3D geometric features of historical frames as the key and value. Through a multi-head cross-attention mechanism, it transfers the 3D geometric features related to the blind zone location in historical frames to the blind zone voxels in the current frame, achieving cross-temporal and spatiotemporal completion of blind zone geometry and semantics, and obtaining preliminary 3D completion features. The blind zone spatial location in the current frame is based on the current field of view 3D scene representation. The difference between the observations and those observed across time series is determined to be that which was observed in the historical frame but not in the reference frame. Not appeared in The voxel positions within the blind zone are defined as the candidate voxel set. ; The motion pattern prior fusion layer is used to integrate motion pattern priors. The encoded vector, with the same dimensions as the 3D completion features, is fused into the output of the cross-attention completion module through element-wise addition and layer normalization, ultimately outputting a blind-spot-completed 3D scene representation. Wherein, the motion pattern prior It is by using candidate time slices Object-level 3D position after inner alignment 3D physics speed With 3D physics acceleration Obtained by performing time-series aggregation; The global 3D accident snapshot Then, the global 3D accident snapshot, the set of involved objects, and the object-level motion states are encoded into scene prior text. This allows step S4 to be invoked seamlessly.

7. The large-scale traffic accident analysis method integrating blind spot completion and physics-driven reasoning as described in claim 6, characterized in that, The method for determining the pixel-level depth uncertainty in step S3 is as follows: for each pixel depth value output by the depth estimation, its cognitive uncertainty is extracted simultaneously. The method for determining the partition-level uncertainty of collision impact is as follows: for each frame image within a candidate time slice, its visual appearance features and depth map are compared. Adaptive oversegmentation is performed on the joint feature space to obtain initial scene fragments. For each initial scene fragment, a three-dimensional traffic attribute feature vector is extracted, and traffic attribute consistency-driven hierarchical merging is performed on the three-dimensional traffic attribute feature vectors of all initial scene fragments to obtain the final accident semantic partition. Finally, all pixel-level depth uncertainties in each accident semantic partition are aggregated to obtain the partition-level depth uncertainty of each accident semantic partition. The method for determining object-level depth uncertainty is as follows: for each detected road use object, based on its detection bounding box... The covered pixel area calculates object-level depth uncertainty. That's all; If the partition-level depth uncertainty of an accident semantic partition exceeds a set threshold, then the accident semantic partition is considered a high-uncertainty region. For any valid pixel, if its region belongs to the set of high-uncertainty regions, then the cross-frame aggregation strategy is actively activated: from candidate time slices... Reliable observations were searched from historical frames, and 3D points were output after smoothing correction through multi-frame weighted averaging. Conversely, the original mapping point is directly retained. Finally, an effective 3D point cloud was obtained after uncertainty suppression and active reconstruction enhancement. ; ； in, For pixels In the The collision geometry contribution weight in a frame is inversely proportional to the pixel-level depth uncertainty and the uncertainty of the region of the object involved. For the first Frame-time pixels Coordinates in the reference frame coordinate system.

8. The large-scale traffic accident analysis method integrating blind spot completion and physics-driven reasoning as described in claim 7, is characterized in that... In step S3, the enhanced fusion function is used. Original visual fragments With global 3D accident snapshot Pixel-level depth uncertainty two-dimensional graph and effective 3D point cloud Perform joint rendering to obtain the enhanced visual flow required for the multimodal large model in step S4. : ； in, The original visual fragment, To enhance the fusion function, a linear superposition mechanism based on Alpha channel blending and 3D reprojection is adopted; t is the image of the t-th frame in the original visual segment.

9. The large-scale traffic accident analysis method integrating blind spot completion and physics-driven reasoning as described in claim 8, characterized in that, In step S4, a multimodal large model based on the Transformer architecture is used, and the physical quantities are encoded as dynamic descriptions. The construction method is as follows: for the set of involved objects output in step S1 The object in the middle, its 3D physical velocity 3D Physics Acceleration as well as The text is transformed into structured text according to the rules of physical kinematics; at the same time, the reference frame is... Corresponding global motion intensity Encode as scene-level motion perturbation indicators; The multimodal input feature set .

10. The large-scale traffic accident analysis method integrating blind spot completion and physics-driven reasoning as described in claim 9, characterized in that, In step S5, the reference frame used as the spatial alignment reference in step S2 is... Defined as physical reference time Simultaneously, retrieve the proximity score at that moment. The two biggest key individuals involved And utilize its aligned three-dimensional position and Calculate the location of the accident impact center Simultaneously calculate the three-dimensional physical distance between key involved entities. The reference accident category is automatically determined based on the relationship between three-dimensional physical distance and safety threshold. Then compare the accident time in step S4 with the reasoning. With physical reference time A time consistency score is obtained; the three-dimensional coordinates of the center of the accident impact area are compared and inferred. Location of the accident's impact center A temporal and spatial consistency score is obtained; accident categories are compared and inferred. Compared with reference accident categories The type consistency score is obtained; finally, the harmonic average method is used to fuse the scores to obtain the final confidence score of the candidate time slice. When the final confidence score is lower than the preset confidence threshold, the risk review mechanism is automatically triggered and the corresponding inference result is marked as a low confidence event.

Images