Road accident stage judgment and video processing method based on multi-modal large model

Abstract

The present application relates to the technical field of intelligent transportation and vehicle-mounted data processing, and discloses a road accident stage judgment and video processing method based on a multi-modal large model, comprising: aligning sensor data and video streams to timestamps and preprocessing, and outputting standardized video frames; extracting multi-modal features, constructing joint representation data, and inputting the multi-modal large model, and outputting stage judgment results containing hidden danger period, collision period and post-incident period; performing stability verification to output high-confidence determination results; according to the high-confidence determination results, delimiting the time range boundary, intercepting the data stream to generate an accident record video file; and packaging the high-confidence determination results and the accident record video file into a structured evidence package for solidification and uploading. The present application fuses the vehicle operating state and the external visual environment, eliminates the misjudgment caused by the fluctuation of a single sensor, dynamically adjusts the video file cutting length to avoid picture omission, and ensures the evidence chain to be tamper-proof.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation and vehicle data processing technology, and in particular to a method for determining the stage of a road accident and processing video based on a multimodal large model. Background Technology

[0002] The vehicle is equipped with cameras, positioning modules, and onboard status sensors, continuously generating video streams and sensor data during operation. Currently, when processing data, the various sensors and video acquisition devices operate independently, resulting in a lack of a unified time reference for the video streams and sensor data. Changes in ambient light and vehicle vibrations cause video frames to become blurry and exhibit abnormal exposure. Using data without time synchronization and image preprocessing directly for subsequent analysis leads to biases in feature extraction dimensions.

[0003] In the road accident assessment process, existing mechanisms typically rely on a single sensor threshold triggering mechanism or a single visual image analysis mechanism. Data from a single sensor fluctuates under vehicle movement, leading to erroneous collision assessments. Pure visual analysis suffers from recognition bias in poor lighting conditions. Because a joint representation of visual information and vehicle operational status data is lacking, existing mechanisms struggle to provide continuous, phased assessments of the entire road accident process, including the hazard phase, collision phase, and post-accident recovery phase, and lack confidence screening and verification processes for the assessment results.

[0004] After a road accident, vehicle recording systems typically extract video files according to a pre-set fixed time length. In complex scenarios involving multiple vehicle collisions, this fixed-time extraction method means that footage of preceding lane changes or post-accident cleanup may be captured beyond the allotted time, leading to the omission of crucial footage. Furthermore, the extracted video files and trajectory data are stored locally in plaintext, lacking a mechanism to encapsulate the video files, trajectory data, and judgment results into structured data. The absence of timestamp-based encoding and encrypted transmission creates a closed-loop process, making the extracted data susceptible to tampering during transmission and hindering its use as a complete chain of evidence. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for determining the stage of a road accident and processing video based on a multimodal large model. This method solves the problems of inconsistent time references for multimodal data, inaccurate accident stage determination, lack of dynamic planning for video cropping time nodes, and missing closed-loop chain for evidence consolidation.

[0006] To achieve the above objectives, the first aspect of the present invention provides a method for determining the stage of a road accident and processing video based on a multimodal large model, comprising the following steps:

[0007] The system acquires various sensor data and video streams, extracts the Global Navigation Satellite System (GNSS) timestamp as a reference, aligns the various sensor data and video streams with the GNSS timestamp, performs preprocessing, and outputs standardized video frames.

[0008] Specifically, the time offset between the local received timestamp and the global navigation satellite system timestamp is calculated; based on the time offset, frame skipping or frame-level interpolation is performed on the real-time video stream and historical video stream of the vehicle to reconstruct the time axis, and the recording time series of various signals from external sensors is adjusted using the time offset to unify the time reference of multiple vehicles.

[0009] The original video frames are extracted from the real-time driving video stream and driving history video after timeline reconstruction. The original video frames are then subjected to resolution normalization, color space conversion, jitter reduction based on inter-frame displacement, dehazing based on dark channel prior algorithm, and backlight correction based on dynamically adjusting the pixel brightness gain value of overexposed and underexposed areas. The resulting standardized video frames are then compiled and output.

[0010] Multimodal features are extracted from standardized video frames and aligned sensor data, and joint representation data of a large multimodal model is constructed through a cross-modal attention network and a graph Transformer model.

[0011] Specifically, standardized video frames are input into the object detection network model and the semantic segmentation network model to generate spatial features containing entity object detection features and pixel-level lane line masks; continuous standardized video frames are input into the optical flow network model and the multi-object tracking model to generate temporal features containing optical flow motion vectors and trajectory data.

[0012] By combining pixel-level lane line masks, vehicle speed signals, vehicle steering signals, vehicle braking signals, vehicle attitude data, and vehicle impact data, collision time, lane departure angle, acceleration abrupt change value, and vehicle headway parameters are calculated and merged to generate semantic features.

[0013] Spatial graph representation data is generated by performing node feature aggregation operations on entity target detection features through a graph neural network model. Temporal evolution representation data is generated by calculating the evolution correlation parameters of trajectory data on the time axis through a self-attention network. The spatial graph representation data and the temporal evolution representation data are then combined to generate a video visual feature matrix.

[0014] The video visual feature matrix and semantic features are input into the cross-modal attention network to calculate the cross-attention weight distribution matrix, and a weighted mapping operation is performed to output cross-modal interaction feature data. The cross-modal interaction feature data is then input into the graph Transformer model to perform global correlation feature extraction operation to generate joint representation data.

[0015] The joint characterization data is input into a multimodal large model for hidden layer correlation inference, and the output includes stage judgment results for the hidden danger period, collision period, and recovery period. Specifically, when the acceleration mutation value extracted from the joint characterization data is greater than a preset acceleration mutation threshold, or the collision time is less than a preset collision time threshold, or the lane departure angle is greater than a preset lane departure angle threshold, the stage judgment result is output and assigned as the hidden danger period.

[0016] When the intersection-union ratio (CUI) of the bounding box coordinate sequences of different entities under the same global navigation satellite system timestamp is greater than the preset CUI threshold, and the optical flow motion vector is simultaneously determined to be greater than the preset optical flow mutation threshold and the acceleration mutation value is greater than the preset acceleration mutation threshold, or when the vehicle airbag signal is determined to be in the triggered activation state, or when the vehicle impact data is greater than the preset impact limit threshold, or when the detection features of an entity target belonging to the debris category are located, the stage judgment result is output and the stage judgment result is assigned as the collision period.

[0017] When the duration of the vehicle speed signal remaining at zero exceeds a preset static time threshold, or when the spatial coordinate transfer vector of an entity belonging to the pedestrian category is detected to extend from inside the vehicle to outside, thereby generating a person leaving the vehicle feature, the stage judgment result is output and the stage judgment result is assigned to the post-stage condition.

[0018] For consecutive stage judgment results, perform temporal stability verification and single-frame deterministic review, and output high-confidence judgment results. Specifically, establish a stage state cache queue, count the frequency values ​​of the hidden danger period, collision period and recovery period in the consecutive cached stage judgment results, and set the stage judgment result with the highest frequency value as the majority judgment label.

[0019] Extract the predicted probability data generated by the multimodal large model for different categories, calculate the information entropy value of the multimodal large model output based on the predicted probability data, compare the information entropy value with the preset information entropy benchmark threshold, and determine whether the judgment result has high deterministic attributes or low deterministic attributes.

[0020] When the judgment result in the judgment stage has a high degree of certainty and the category of the majority judgment label is consistent with the category with the lowest information entropy in the current frame, the majority judgment label is transformed into a high-confidence judgment result.

[0021] When the judgment result in the judgment stage has a low-determinism attribute, the comprehensive confidence level is calculated; when the comprehensive confidence level is greater than or equal to the preset comprehensive confidence level threshold, the majority judgment label is converted into a high-confidence judgment result; when the comprehensive confidence level is less than the preset comprehensive confidence level threshold, the automatic output task is suspended, a manual review request data packet containing standardized video frames and video visual feature matrices is generated and sent to the manual review terminal, the manual confirmation label returned by the manual review terminal is received and the manual confirmation label is converted into a high-confidence judgment result.

[0022] Based on the high confidence level determination results, the collision baseline time node is located, the scene complexity value is calculated to define the time axis boundary, the video interval revenue value is calculated based on the revenue function, and the target video data stream is extracted by combining the video interval revenue value to generate the accident record video file.

[0023] Specifically, the high-confidence determination result of the collision period is located in a continuous time series, and the global navigation satellite system timestamp corresponding to the high-confidence determination result is set as the collision reference time node.

[0024] The system counts the number of entities within standardized video frames adjacent to the collision baseline time point in the time dimension, the number of entities involved in the collision, and generates a trajectory interleaving index based on the overlap and temporal proximity of the trajectory data. The system then integrates the number of entities, the number of entities involved in the collision, and the trajectory interleaving index to calculate the scene complexity.

[0025] The scene complexity values ​​are compared with the first and second scene complexity benchmark values ​​to generate simple, medium, or complex scene labels. The corresponding pre-collision time offset and post-collision time offset are extracted based on the labels. The video cropping start and end time nodes are calculated by combining the collision benchmark time node and the offset. The video cropping start and end time nodes are used as the time axis boundaries for dynamic video cropping.

[0026] For standardized video frames located within the time axis boundary, calculate the statistical feature parameters of the natural scene to generate BRISQUE score values, calculate the variance of the high-frequency edge response matrix to generate ambiguity metric, calculate the overlapping pixel area of ​​the coordinate sequences of the bounding boxes of entity objects to generate occlusion ratio values, compare the proportion of contour pixels to generate target visibility evaluation parameters, and merge to generate a four-dimensional quality feature vector for the standardized video frame.

[0027] The sum of the pre-collision time offset and the post-collision time offset is set as the length of the dynamic video interval. Centered on the collision reference time node, multiple candidate video intervals with a duration equal to the length of the dynamic video interval are generated by a sliding window. The four-dimensional quality feature vectors in the candidate video intervals are weighted and summed to obtain the comprehensive quality evaluation value. The information implied metric value is calculated by combining the number of collision participants, the number of entity objects, and the trajectory intersection index.

[0028] Substituting the comprehensive quality evaluation value and the information implication metric value into the preset revenue function, multiple sets of video interval revenue values ​​are calculated. The candidate video interval bound to the video interval with the maximum revenue value is then identified as the target video interval.

[0029] The start and end points of the time corresponding to the target video interval are extracted and assigned as the editing start node and editing end node, respectively. The editing start node and editing end node are located in the original video stream sequence and the target video data stream is extracted and packaged to generate an accident record video file.

[0030] The high-confidence judgment results and accident recording video files are packaged into a structured evidence package, which is then encrypted and solidified before being uploaded in a closed loop.

[0031] Specifically, the retrieval and editing process covers trajectory data within the time range of the start and end nodes of the clipping, as well as the entity objects and unique identifiers (IDs) contained within the trajectory data. The trajectory data, unique identifiers (IDs), and global navigation satellite system timestamps are mapped to a unified time axis coordinate system and merged to generate spatiotemporal trajectory structured data.

[0032] The high-confidence judgment result, accident recording video file, spatiotemporal trajectory structured data, and comprehensive confidence value are sent to the packet storage unit for time axis correlation and alignment. The vehicle terminal identification code and system timing parameters are added to the front end of the data stream to perform data stream encapsulation and output a structured evidence package. This establishes a one-to-one mapping relationship between the high-confidence judgment result, accident recording video file, spatiotemporal trajectory structured data, and comprehensive confidence value within the structured evidence package.

[0033] The national cryptographic algorithm is invoked to perform data digest extraction and asymmetric encryption operations on the structured evidence package to generate an encrypted evidence stream. The global navigation satellite system timestamp at the time of generating the encrypted evidence stream is retrieved, and spatial bit alignment and merging are performed with the encrypted evidence stream to write it into an indelible storage sector, thus completing the timestamp solidification of the evidence data.

[0034] An encrypted communication tunnel is established using identity authentication credentials, and the solidified encrypted evidence stream is synchronously pushed to the external platform corresponding to the platform access parameters. The data reception confirmation receipt returned by the external platform is received, and the message integrity check value is compared with the original feature digest value of the encrypted evidence stream. If they are consistent, a reporting task completion instruction is generated and associated with the structured evidence package to complete the closed-loop verification of data reporting.

[0035] A second aspect of the present invention provides an in-vehicle data processing platform for implementing a road accident stage judgment and video processing method based on a multimodal large model. The in-vehicle data processing platform is equipped with a video access terminal and a multimodal sensor data access terminal.

[0036] The video access terminal is used to connect to the dashcam, enabling the dashcam to collect real-time video streams and historical video streams of the vehicle's surroundings, and transmit the real-time video streams and historical video streams to the vehicle data processing platform through the video access terminal.

[0037] The multimodal sensor data access terminal is used to connect external sensors, which include a GPS positioning module, a Beidou positioning module, an IMU measurement module, and a CAN bus interface. This enables the external sensors to continuously collect GPS location data, Beidou location data, vehicle attitude data, vehicle impact data, vehicle speed signals, vehicle steering signals, vehicle braking signals, and vehicle airbag signals, and transmit these data to the vehicle data processing platform.

[0038] The vehicle-mounted data processing platform is internally deployed with a multimodal large model inference environment and video encoding and decoding hardware. The vehicle-mounted data processing platform is used to perform multimodal fusion modeling operations and video cropping output based on the aggregated data of various types.

[0039] This invention provides a method for road accident stage determination and video processing based on a multimodal large model. It has the following beneficial effects:

[0040] 1. This invention reconstructs the timeline by calculating the time offset between the local received timestamp and the global navigation satellite system timestamp, performing frame skipping or frame-level interpolation on real-time and historical video streams of the vehicle. It adjusts the recording time series of various signals from external sensors using the time offset, and simultaneously performs jitter reduction, defogging, and backlight correction on the original video frames to output standardized video frames. This unifies the time reference of multimodal data, eliminates interference factors of ambient light differences on image feature extraction, and solves the problem of time asynchrony caused by independent acquisition by heterogeneous sensors.

[0041] 2. This invention generates joint representation data by inputting the video visual feature matrix and semantic features generated by combining various vehicle signals into a cross-modal attention network and a graph Transformer model. The joint representation data is used to perform threshold comparison and output stage judgment results including the potential danger period, collision period, and recovery period. The temporal stability verification and single-frame deterministic verification are performed on the continuous stage judgment results based on the information entropy value. This integrates the vehicle operating status and the external visual environment, eliminates misjudgments caused by fluctuations in data from a single sensor, and ensures the objectivity and certainty of the road accident stage judgment results through a confidence screening mechanism.

[0042] 3. This invention calculates the scene complexity value by integrating the number of entity objects, the number of entities involved in the collision, and the trajectory intersection index. Based on the scene complexity value, the offset is extracted to delineate the time axis boundary. Combined with the calculated video interval gain value, the target video data stream is extracted to generate an accident recording video file. The accident recording video file and the spatiotemporal trajectory structured data are encapsulated into a structured evidence package and encrypted and timestamped into the storage sector. The cropping length of the accident recording video file is dynamically adjusted to avoid the problem of missing key scenes caused by fixed-duration cropping. The immutable nature of the accident evidence chain is guaranteed by the non-erasable solidification mechanism. Attached Figure Description

[0043] Figure 1 This is a flowchart of the method of the present invention;

[0044] Figure 2 This is a hardware block diagram of the vehicle-mounted data processing platform of the present invention;

[0045] Figure 3 This is a schematic diagram illustrating the multimodal feature extraction and joint characterization construction of the present invention;

[0046] Figure 4 This is a schematic diagram of the accident stage reasoning and judgment based on a multimodal large model according to the present invention;

[0047] Figure 5 This is a schematic diagram illustrating the timing verification and confidence check of the stage judgment results of the present invention.

[0048] Figure 6 This is a schematic diagram of the dynamic video cropping and quality optimization based on scene complexity according to the present invention;

[0049] Figure 7 This is a comparison chart of the information entropy convergence and accuracy evolution at each time point in the accident stage judgment process of the present invention.

[0050] Figure 8 This is a comparison chart showing the invalid frame pruning and revenue function control during the evaluation process of the candidate video ranges in this invention. Detailed Implementation

[0051] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] See attached document Figure 1 and attached Figure 2 This invention provides a method for determining the stage of a road accident and processing video based on a multimodal large model, which is built on an on-board data processing platform; the on-board data processing platform is equipped with a video access terminal and a multimodal sensor data access terminal.

[0053] The video access terminal of the vehicle data processing platform is connected to the dashcam; the dashcam collects real-time driving video streams and historical driving videos of the vehicle's surrounding environment, and transmits the real-time driving video streams and historical driving videos to the vehicle data processing platform through the video access terminal.

[0054] The multimodal sensor data access terminal of the vehicle-mounted data processing platform connects to external sensors. These external sensors include a GPS positioning module, a BeiDou positioning module, an IMU measurement module, and a CAN bus interface. The GPS and BeiDou positioning modules connect to the multimodal sensor data access terminal, collect GPS and BeiDou location data, and continuously transmit these data to the vehicle-mounted data processing platform.

[0055] The IMU measurement module connects to the multimodal sensor data access terminal to measure the inertial state changes of the vehicle in real time, collect vehicle attitude data and vehicle impact data, and transmit the vehicle attitude data and vehicle impact data to the on-board data processing platform.

[0056] The CAN bus interface connects to the multimodal sensor data access terminal, extracts the vehicle's underlying controller status data, collects vehicle speed signals, vehicle steering signals, vehicle braking signals, and vehicle airbag signals, and transmits these signals to the onboard data processing platform.

[0057] The in-vehicle data processing platform aligns and aggregates real-time driving video streams, historical driving videos, GPS location data, BeiDou location data, vehicle attitude data, vehicle impact data, vehicle speed signals, vehicle steering signals, vehicle braking signals, and vehicle airbag signals. Internally, it deploys a multimodal large-model inference environment and video encoding / decoding hardware, performing multimodal fusion modeling operations and video cropping output based on the aggregated data.

[0058] See attached document Figure 1 -Appendix Figure 6The method for determining the stage of a road accident and processing video based on a multimodal large model provided by this invention specifically includes the following:

[0059] S1. Multimodal Data Spatiotemporal Synchronization and Video Preprocessing: After receiving various sensor data and video streams, the vehicle-mounted data processing platform extracts the Global Navigation Satellite System (GNSS) timestamps from the GPS and BeiDou location data, and sets them as a reference standard by combining network time protocols and precise time protocols. It acquires the local received timestamps corresponding to real-time driving video streams, historical driving videos, vehicle attitude data, vehicle impact data, vehicle speed signals, vehicle steering signals, vehicle braking signals, and vehicle airbag signals. It calculates the time offset between the local received timestamps and the GNSS timestamps.

[0060] The vehicle-mounted data processing platform reconstructs the timeline of the real-time driving video stream and historical driving video based on the time offset. When the time offset is positive and greater than the duration of a single video frame, frame skipping is performed on both the real-time and historical driving video streams, discarding redundant video frames. When the time offset is negative and its absolute value is greater than the duration of a single video frame, frame-level interpolation is performed on both the real-time and historical driving video streams to generate supplementary video frames.

[0061] The onboard data processing platform uses time offsets to adjust the time series of data recordings for vehicle attitude data, vehicle impact data, vehicle speed signals, vehicle steering signals, vehicle braking signals, and vehicle airbag signals. It aligns real-time driving video streams, historical driving videos, vehicle attitude data, vehicle impact data, vehicle speed signals, vehicle steering signals, vehicle braking signals, and vehicle airbag signals with the Global Navigation Satellite System timestamps, unifying the time reference for multiple vehicles.

[0062] The vehicle-mounted data processing platform receives real-time driving video streams and historical driving videos aligned with the timestamps of the Global Navigation Satellite System, triggers the internally deployed video encoding and decoding hardware to decode the real-time driving video streams and historical driving videos, and extracts continuous raw video frames from them.

[0063] The vehicle-mounted data processing platform reads the initial resolution parameters of the original video frames and compares them with the preset resolution standard. The preset resolution standard is limited to 1920×1080 and 1280×720. When the initial resolution parameters do not meet the preset resolution standard, resolution normalization is performed on the original video frames, and the pixel matrix of the original video frames is remapped and scaled to 1920×1080 or 1280×720.

[0064] The vehicle-mounted data processing platform extracts pixel-level color data from the original video frames and determines that the pixel-level color data belongs to the YUV color space. It then performs a color space conversion on the pixel-level color data, converting the YUV color space data to RGB color space data.

[0065] The onboard data processing platform synchronously reads vehicle attitude data aligned with the Global Navigation Satellite System timestamp. Based on this data, it calculates the inter-frame displacement offset of the original video frames and performs inverse compensation on the pixel coordinates to achieve jitter reduction. It then analyzes the global contrast distribution of the original video frames and filters out foggy pixels using a dark channel prior algorithm to achieve dehazing. Finally, it locates overexposed and underexposed areas within the original video frames and dynamically adjusts the pixel brightness gain values ​​in these areas to achieve backlight correction. The platform then assembles the original video frames after resolution normalization, color space conversion, jitter reduction, dehazing, and backlight correction, outputting standardized video frames.

[0066] S2. Multimodal feature extraction and joint representation construction: Refer to the appendix. Figure 3 The vehicle-mounted data processing platform inputs standardized video frames into an internally deployed object detection network model, which includes a YOLOv8 network model and a Faster R-CNN network model. The YOLOv8 and Faster R-CNN models are used to perform two-dimensional matrix convolution operations on the standardized video frames to locate entity objects within them. The platform outputs the bounding box coordinate sequence of the entity objects and the corresponding category label sequence, categorized into vehicle, pedestrian, traffic facility, and debris categories. The bounding box coordinate sequence and category label sequence are then combined to generate entity object detection features.

[0067] The vehicle-mounted data processing platform synchronously inputs standardized video frames into an internally deployed semantic segmentation network model, which includes a DeepLabV3+ network model and a SegFormer network model. The DeepLabV3+ and SegFormer network models are used to perform pixel-by-pixel classification on all discrete pixels of the standardized video frames, extracting the classification results to generate a pixel-level segmentation mask for the standardized video frames. The entity target detection features are then concatenated with the pixel-level segmentation mask to generate the spatial features of the standardized video frames. The pixel-level segmentation mask is limited to a pixel-level lane line mask.

[0068] The vehicle-mounted data processing platform acquires continuous standardized video frames in a time series and inputs these frames into an internally deployed optical flow network model, which includes a RAFT network model and a PWC-Net network model. The RAFT and PWC-Net network models are used to perform two-dimensional matching operations on the pixels within the continuous standardized video frames, extracting the two-dimensional displacement data of the pixels between the continuous standardized video frames, and generating optical flow motion vectors based on the two-dimensional displacement data.

[0069] The vehicle-mounted data processing platform synchronously extracts the bounding box coordinate sequence of entity objects, and inputs the bounding box coordinate sequence and optical flow motion vector into the internally deployed multi-target tracking model. The multi-target tracking model includes the ByteTrack network model and the DeepSORT network model. The ByteTrack network model and the DeepSORT network model are used to perform cross-frame association comparison of entity objects in consecutive standardized video frames, and assign a unique identity ID to the entity objects that have completed the cross-frame association comparison.

[0070] The vehicle-mounted data processing platform extracts the bounding box coordinate sequence of entity objects with unique identification IDs, connects the center points of the bounding box coordinate sequences within consecutive standardized video frames to generate trajectory data of the entity objects; combines optical flow motion vectors, trajectory data, and global navigation satellite system timestamps to calculate the real-time displacement change rate of the entity objects, and generates the velocity curve of the entity objects based on the real-time displacement change rate; and aggregates the optical flow motion vectors, unique identification IDs, trajectory data, and velocity curves to generate the temporal characteristics of standardized video frames.

[0071] The vehicle-mounted data processing platform extracts trajectory data and speed curves within temporal features, and extracts pixel-level lane line masks within spatial features; it also simultaneously extracts vehicle speed signals, vehicle steering signals, vehicle braking signals, vehicle attitude data, and vehicle impact data aligned with the global navigation satellite system timestamps.

[0072] The vehicle-mounted data processing platform combines GPS location data and trajectory data aligned with the Global Navigation Satellite System timestamps to extract the spatial relative distance between the vehicle and an entity with a unique identifier ID; it calculates the relative speed between the vehicle and the entity based on the speed curve and vehicle speed signal; and it calculates the collision time by dividing the spatial relative distance and the relative speed.

[0073] The vehicle data processing platform combines pixel-level lane line masks to extract lane line spatial extension direction parameters, and combines vehicle steering signals and vehicle attitude data to extract vehicle current driving orientation parameters; compares the lane line spatial extension direction parameters and the vehicle current driving orientation parameters, calculates the spatial angle difference between the lane line spatial extension direction parameters and the vehicle current driving orientation parameters, and generates the lane departure angle.

[0074] The vehicle-mounted data processing platform tracks the changes in vehicle impact data and vehicle speed signals over time; it performs differential calculations on the vehicle speed signal to extract the basic acceleration value, then performs further differentiation on the basic acceleration value, and calculates the instantaneous rate of change of the acceleration value in combination with the vehicle impact data to generate acceleration mutation values; it extracts the spatial relative distance between the vehicle and the entity directly in front in the same lane, divides the spatial relative distance by the current vehicle speed signal value, and calculates the vehicle headway parameter; it aggregates and merges the collision time, vehicle headway parameter, lane departure angle, acceleration mutation value, and vehicle braking signal to generate semantic features corresponding to standardized video frames.

[0075] The vehicle-mounted data processing platform extracts entity target detection features within spatial features, defines entity objects within the entity target detection features as graph nodes, and sets the spatial relative distance between entity objects as graph connection edges; it constructs a spatial relationship graph structure containing graph nodes and graph connection edges, and inputs the spatial relationship graph structure into an internally deployed graph neural network model; the graph neural network model performs node feature aggregation operations; the vehicle-mounted data processing platform extracts the output results of the graph neural network model to generate spatial graph representation data.

[0076] The vehicle-mounted data processing platform extracts unique identifiers (IDs) and trajectory data within the time-series features. It then arranges the trajectory data belonging to the same unique identifier (ID) into a time series according to the global navigation satellite system timestamp. The arranged trajectory data is then input into the internally deployed Transformer model. The self-attention network within the Transformer model calculates the evolution correlation parameters of the trajectory data on the time axis, generating time-series evolution representation data.

[0077] The vehicle-mounted data processing platform aggregates spatial graph representation data and temporal evolution representation data, and merges them to generate a video visual feature matrix. Semantic features are extracted simultaneously, and the video visual feature matrix and semantic features are input into an internally deployed cross-modal attention network. The cross-modal attention network calculates the cross-attention weight distribution matrix between the video visual feature matrix and the semantic features. The vehicle-mounted data processing platform performs a weighted mapping operation on the video visual feature matrix and semantic features based on the cross-attention weight distribution matrix, and outputs cross-modal interaction feature data.

[0078] The vehicle-mounted data processing platform inputs cross-modal interaction feature data into the internally deployed graph Transformer model; the graph Transformer model performs global correlation feature extraction operations on the cross-modal interaction feature data; the vehicle-mounted data processing platform obtains the computational output value of the graph Transformer model, generates joint representation data, and sets the joint representation data as the input data source of the multimodal large model.

[0079] S3. Accident stage reasoning and judgment based on multimodal large model: refer to the appendix. Figure 4 The vehicle-mounted data processing platform pre-sets various judgment thresholds in its internal storage unit: a boundary limit based on statistically determined extreme values ​​of acceleration and deceleration during normal driving is set as a preset acceleration mutation threshold; a time limit based on emergency braking safety distance conversion is set as a preset collision time threshold; an angle limit distinguishing between normal lane changes and loss of control deviation is set as a preset deviation angle threshold; a standard ratio representing the degree of overlap of the same entity in consecutive frames is set as a preset intersection-union ratio threshold; a motion vector jump limit distinguishing between normal motion and violent collision displacement is set as a preset optical flow mutation threshold; a lower limit of impact force representing a structural collision is set as a preset impact limit threshold; and the shortest time requirement representing the vehicle completely losing power and coming to a stop is set as a preset stationary time threshold. The vehicle-mounted data processing platform inputs the joint characterization data into a multimodal large model; the multimodal large model extracts the acceleration mutation value, collision time, and lane departure angle from the joint characterization data; compares the acceleration mutation value with the preset acceleration mutation threshold, and determines whether the acceleration mutation value conforms to the relational expression. , This represents the abrupt acceleration value; the collision time is compared with a preset collision time threshold to determine if the collision time conforms to the formula. , It represents the collision time; it compares the lane departure angle with the preset departure angle threshold and determines that the lane departure angle is greater than the preset departure angle threshold.

[0080] The multimodal large model maps joint representation data to a predefined semantic feature space, and performs hidden layer correlation reasoning on acceleration abrupt changes, collision time, and lane departure angle through an internal Transformer encoder. Based on the learned accident evolution pattern, it identifies semantic features of abnormal driving behavior or state instability presented in the joint representation. It establishes triggering conditions for potential hazard periods, which can be satisfied by any one of the following: the first condition is that the acceleration abrupt change conforms to the relational expression. The second term is characterized by a rapid decrease in collision time that conforms to the relation. The third condition is that a lane departure angle greater than a preset departure angle threshold represents abnormal driving behavior. When the joint characterization data meets any of the above-mentioned conditions for triggering a potential hazard period, the stage judgment result is output in conjunction with the semantic features, and the stage judgment result is assigned as a potential hazard period.

[0081] Multimodal large model continuously analyzes joint representation data, extracting bounding box coordinate sequences, optical flow motion vectors, and acceleration abrupt values ​​of entity objects within the joint representation data; it calculates the intersection-union ratio (CIU) of bounding box coordinate sequences of different entity objects under the same GNAS timestamp, and determines whether the CIU conforms to the relation. , The method involves comparing the optical flow motion vector with a preset optical flow mutation threshold and comparing the acceleration mutation value with a preset acceleration mutation threshold. It determines that the optical flow motion vector is greater than the preset optical flow mutation threshold and the acceleration mutation value is greater than the preset acceleration mutation threshold. It extracts vehicle airbag signals and vehicle impact data aligned with the global navigation satellite system timestamp, determines that the vehicle airbag signal is in the triggered activation state, and determines that the vehicle impact data is greater than a preset impact limit threshold. It performs fine-grained visual analysis on the joint characterization data to locate entity target detection features belonging to the fragment category within the joint characterization data.

[0082] The multimodal large model establishes collision period triggering conditions, which employ multi-dimensional judgment logic. Triggering occurs when any one of the following dimensional scenarios is met: Scenario 1: Triggering based on visual feature dimension, determining the intersection-union ratio (IU) according to the following relation. In scenario one, the optical flow motion vector of the entity object is determined to be greater than the preset optical flow mutation threshold and the acceleration mutation value is greater than the preset acceleration mutation threshold. In scenario two, based on the sensor dimension trigger, the vehicle airbag signal is determined to be in the triggered opening state, or the vehicle impact data is greater than the preset impact limit threshold. In scenario three, based on the semantic detection dimension trigger, the entity target detection features belonging to the fragment category are located. When the joint representation data meets the collision period triggering conditions set by any of the above scenario dimensions, a stage judgment result is generated, and the stage judgment result is assigned as the collision period.

[0083] The multimodal large model extracts vehicle speed signals from the fused joint representation data and continuously monitors changes in vehicle speed signal values. It determines when the vehicle speed signal value drops to zero and calculates the duration of this zero-speed state using a Global Navigation Satellite System (GNSS) timestamp. The duration is then compared to a preset static time threshold to determine if the duration conforms to the relevant formula. , This involves analyzing the entity detection features contained in the joint representation data to locate entities belonging to the pedestrian category; extracting the unique identifier ID and trajectory data bound to the entity belonging to the pedestrian category; calculating the spatial coordinate transfer vector of the entity based on the trajectory data and extracting the spatial boundary coordinates of the vehicle; comparing the spatial coordinate transfer vector with the spatial boundary coordinates to determine that the spatial coordinate transfer vector of the entity shows an extension trend from inside the vehicle to outside the vehicle, generating a person leaving the vehicle feature; integrating the duration determination result and the person leaving the vehicle feature to establish a post-traumatic stress relief trigger condition, which is that the duration of the vehicle speed signal conforms to the following relationship. Alternatively, it can simultaneously detect the characteristics of people leaving the vehicle; when the input data meets any of the conditions for triggering the post-completion phase, it outputs the phase judgment result and assigns the phase judgment result as the post-completion phase.

[0084] S4. Time-series verification and confidence check of stage judgment results: Refer to Appendix Figure 5 The vehicle-mounted data processing platform receives continuous stage judgment results from the multimodal large model; establishes a stage state cache queue in the internal storage unit; and sequentially stores the stage judgment results associated with N consecutive standardized video frames into the stage state cache queue, setting the frame number parameter. .

[0085] The vehicle data processing platform extracts the five stage judgment results from the stage status cache queue; calculates the frequency values ​​of the stage judgment results assigned the values ​​of the hidden danger period, collision period, and recovery period respectively; performs a descending sort comparison on the frequency values ​​of the hidden danger period, collision period, and recovery period; locates the stage judgment result with the largest frequency value, and sets the stage judgment result with the largest frequency value as the majority judgment label.

[0086] When extracting the stage judgment results from the multimodal large model output, the vehicle-mounted data processing platform generates predicted probability data for all candidate categories (including the potential hazard period, collision period, and recovery period); based on the predicted probability data, it calculates the information entropy value of the multimodal large model output results, using the following formula:

[0087] ;

[0088] in: Represents the value of information entropy; This represents performing a cumulative summation operation on all candidate categories; Represents the predicted probability data for the first The probability values ​​of each candidate category.

[0089] The vehicle-mounted data processing platform extracts a preset information entropy benchmark threshold (setting the information entropy critical limit used to distinguish the high and low certainty of model prediction results as the preset information entropy benchmark threshold), and compares the information entropy value with the information entropy benchmark threshold. The information entropy value and the classification certainty of the stage judgment result have an inverse proportional mapping relationship. When the information entropy value is less than the information entropy benchmark threshold, the stage judgment result output by the multimodal large model is determined to have a high certainty attribute. When the information entropy value is greater than or equal to the information entropy benchmark threshold, the stage judgment result output by the multimodal large model is determined to have a low certainty attribute.

[0090] When the judgment result in the judgment stage has a high degree of certainty and the category of the majority judgment label is consistent with the category with the lowest information entropy of the current frame, the vehicle data processing platform simultaneously extracts the majority judgment label and converts it into a high-confidence judgment result, and performs time-series stability verification and single-frame deterministic review; when the judgment result in the judgment stage has a low degree of certainty, a review judgment instruction for the judgment result in the stage is triggered.

[0091] When a review and judgment instruction is received, the vehicle data processing platform extracts the frequency of occurrence of the majority judgment tag in the stage status cache queue, divides the frequency of occurrence by the frame number parameter to generate the majority percentage value, calculates the comprehensive confidence level based on the majority percentage value and the information entropy value, compares the comprehensive confidence level with the preset comprehensive confidence level threshold, and sets the preset comprehensive confidence level to 0.75.

[0092] When the overall confidence level is less than 0.75, the vehicle data processing platform automatically outputs the judgment result during the suspension phase; extracts standardized video frames and video visual feature matrices for the corresponding time period, merges the standardized video frames and video visual feature matrices to generate a manual review request data packet; sends the manual review request data packet to the externally accessed manual review terminal; continuously monitors the data receiving port, receives the manual confirmation tag returned by the manual review terminal based on the manual review request data packet; and converts the manual confirmation tag into a high-confidence judgment result.

[0093] When the overall confidence level is greater than or equal to 0.75, the vehicle data processing platform directly extracts the majority decision label and transforms it into a high-confidence decision result. After generating the high-confidence decision result, the automatic output task of the recovery stage decision result generates a downstream execution instruction containing the high-confidence decision result and transmits the downstream execution instruction to the downstream data receiving module.

[0094] S5. Dynamic video cropping and quality optimization based on scene complexity: Refer to the appendix. Figure 6 The vehicle-mounted data processing platform extracts the high-confidence judgment results and the Global Navigation Satellite System (GNSS) timestamps aligned with the high-confidence judgment results; locates the high-confidence judgment results assigned to the collision period in a continuous time series; extracts the GNSS timestamps corresponding to the high-confidence judgment results assigned to the collision period, and sets the located GNSS timestamps as the collision reference time nodes.

[0095] The vehicle-mounted data processing platform extracts the pre-collision time offset and post-collision time offset from the pre-installed internal storage module; it then combines the collision reference time node, the pre-collision time offset, and the post-collision time offset to calculate the video cropping start time node and the video cropping end time node. The calculation formula is as follows:

[0096] ;

[0097] ;

[0098] in: This represents the start time point of video cropping; Represents the baseline time point for the collision; This represents the time offset prior to the collision; This represents the end time of video cropping; This represents the time offset after the collision.

[0099] The vehicle-mounted data processing platform sets the start and end times of video cropping as the timeline boundaries for dynamic video cropping; it then encapsulates and merges the data from the start and end times of video cropping to generate parameters for establishing the cropping baseline nodes.

[0100] The vehicle-mounted data processing platform extracts standardized video frames that are adjacent to the collision reference time node in the time dimension; analyzes the entity target detection features within the standardized video frames and counts the number of entity objects contained in the entity target detection features; locates the entity objects that collide within the standardized video frames and counts the number of collision-participating entities; extracts trajectory data belonging to different unique identifiers, and performs joint quantitative analysis based on the degree of intersection and overlap of different trajectory data in the two-dimensional spatial plane and the degree of proximity on the time axis to generate a trajectory intersection index.

[0101] The vehicle-mounted data processing platform integrates the number of entities, the number of entities involved in the collision, and the trajectory intersection index to calculate the scene complexity of the current standardized video frame sequence. The calculation formula is as follows:

[0102] ;

[0103] in: This represents the numerical value of scene complexity; This represents the first preset weighting coefficient; Represents the number of entity objects; This represents the second preset weighting coefficient; This represents the number of entities involved in the collision; This represents the third preset weighting coefficient; The index represents the trajectory intersection.

[0104] The vehicle-mounted data processing platform establishes a mapping table between scene complexity levels and time offsets in its internal storage unit. It sets benchmark parameters for classifying scene levels: the lower complexity assessment threshold derived from historical routine traffic accident data is set as the first scene complexity benchmark value, and the higher complexity assessment threshold derived from historical multi-vehicle collision data is set as the second scene complexity benchmark value. When the scene complexity value is less than the first scene complexity benchmark value, a simple scene label is generated based on the mapping table. Under the simple scene label, parameter assignment operations are performed, setting the pre-collision time offset to 5 seconds and simultaneously setting the post-collision time offset to 10 seconds.

[0105] When the scene complexity value is greater than or equal to the first scene complexity benchmark value and less than the second scene complexity benchmark value, the vehicle data processing platform generates a medium scene label according to the mapping relationship table; under the medium scene label, the parameter assignment operation is performed, the pre-collision time offset is set to 10s, and the post-collision time offset is set to 20s simultaneously.

[0106] When the scene complexity value is greater than or equal to the second scene complexity benchmark value, the vehicle data processing platform generates a complex scene label according to the mapping relationship table; under the complex scene label, the parameter assignment operation is performed, setting the pre-collision time offset to 15s and simultaneously setting the post-collision time offset to 30s; the assigned pre-collision time offset and post-collision time offset are input to the time node calculation unit to complete the task of dynamically defining the time axis boundary of the standardized video frame.

[0107] The vehicle-mounted data processing platform extracts standardized video frames whose Global Navigation Satellite System timestamps fall within the time axis boundary (i.e., the interval formed by the start and end times of video cropping). For each standardized video frame, a no-reference quality analysis operation is performed based on spatial features to calculate natural scene statistical feature parameters. A BRISQUE score corresponding to the standardized video frame is generated based on these parameters. A second-order spatial partial derivative filtering operation is performed on the standardized video frame using the Laplacian operator to obtain the high-frequency edge response matrix of the image pixels. The variance of all pixels within the high-frequency edge response matrix is ​​calculated, and the calculated variance value is directly set as a blur metric representing the image's sharpness.

[0108] The vehicle-mounted data processing platform analyzes the entity target detection features contained in the standardized video frame; extracts the bounding box coordinate sequences of multiple entity objects in the same time dimension, calculates the overlapping pixel area between the bounding box coordinate sequences of different entity objects; divides the overlapping pixel area by the total pixel area of ​​the entity objects participating in the overlap, and outputs the occlusion ratio value.

[0109] The vehicle-mounted data processing platform extracts the pixel distribution array of the entity object within the standardized video frame and compares the pixel distribution array with the preset complete entity object outline benchmark template; it calculates the proportion of outline pixels of the entity object that are not covered by the environmental background pixels and establishes the proportion of uncovered outline pixels as the target visibility evaluation parameter; it simultaneously summarizes the BRISQUE score value, blur metric, occlusion ratio value and target visibility evaluation parameter, and merges them to generate a four-dimensional quality feature vector for the standardized video frame.

[0110] The vehicle data processing platform extracts the pre-collision time offset and post-collision time offset after the scene complexity is assigned, and sets the sum of the two as the length of the dynamic video interval. Centered on the collision reference time node, multiple sets of candidate video intervals with a duration equal to the length of the dynamic video interval are generated by sliding windows according to the preset time step.

[0111] The vehicle-mounted data processing platform extracts the four-dimensional quality feature vectors corresponding to all standardized video frames within the candidate video interval; performs normalization and weighted summation operations on the data items contained within the four-dimensional quality feature vectors to calculate the comprehensive quality evaluation value corresponding to the candidate video interval; extracts the number of entity objects, the number of collision-involved entities, and the trajectory intersection index contained within the candidate video interval, and generates an information implication metric value through the following formula:

[0112] ;

[0113] in: The information represents a metric value; This represents the first adjustment coefficient; This represents the density of entities involved in the collision when the collision event occurs. This represents the second adjustment coefficient; The index represents the trajectory intersection.

[0114] The vehicle-mounted data processing platform retrieves a preset weight allocation table to establish a revenue function. The comprehensive quality evaluation value and the information implication metric value are then substituted into the revenue function to calculate the revenue value for each video interval. The calculation formula is as follows:

[0115] ;

[0116] in: This represents the revenue value for a given video segment. This represents the preset information weighting coefficient; The information represents a metric value; This represents the preset quality weighting coefficient; This represents the overall quality evaluation value.

[0117] The vehicle-mounted data processing platform repeatedly executes calculation instructions for all candidate video intervals within the global time boundary interval to obtain a set of values ​​consisting of multiple video interval revenue values; it then compares the revenue values ​​of the video intervals within the set to locate the video interval revenue value that is at its maximum value.

[0118] The vehicle-mounted data processing platform identifies the candidate video interval bound to the video interval with the maximum value as the target video interval, and extracts the start and end points of time corresponding to the target video interval; the start point of time is assigned as the editing start node, and the end point of time is assigned as the editing end node.

[0119] The vehicle-mounted data processing platform locates the start and end points of the editing process in the original video stream sequence, extracts the target video data stream based on the start and end points, encapsulates the target video data stream into an accident recording video file, and outputs it to the internal long-term storage unit.

[0120] S6. Encryption, solidification, and closed-loop uploading of structured evidence packages: The vehicle-mounted data processing platform assembles structured evidence packages; retrieves high-confidence judgment results determined through confidence assessment; synchronously extracts accident recording video files aligned with the high-confidence judgment results in the time dimension; retrieves trajectory data within the time range covered by the editing start node and editing end node; and extracts entity objects and unique identification IDs contained within the trajectory data.

[0121] The vehicle-mounted data processing platform maps trajectory data, unique identifier ID, and GNSS timestamp to a unified time axis coordinate system; merges the mapped trajectory data, unique identifier ID, and GNSS timestamp to generate spatiotemporal trajectory structured data; and extracts the comprehensive confidence score calculated based on the stage judgment results.

[0122] The vehicle-mounted data processing platform executes preset data encapsulation rules; sends the high-confidence judgment result, accident recording video file, spatiotemporal trajectory structured data, and comprehensive confidence value to the packet storage unit; performs time axis association and alignment of data items within the packet storage unit; adds the vehicle terminal identification code and system timing parameters to the front end of the aligned data stream; performs data stream encapsulation on the high-confidence judgment result, accident recording video file, spatiotemporal trajectory structured data, comprehensive confidence value, vehicle terminal identification code, and system timing parameters; outputs a structured evidence package, establishing a one-to-one mapping association between the high-confidence judgment result, accident recording video file, spatiotemporal trajectory structured data, and comprehensive confidence value within the structured evidence package.

[0123] The vehicle-mounted data processing platform extracts the structured evidence package; calls the national cryptographic encryption algorithm integrated in the internal hardware security module; and uses the national cryptographic encryption algorithm to perform data digest extraction and asymmetric encryption operations on the structured evidence package to generate an encrypted evidence stream.

[0124] The vehicle-mounted data processing platform retrieves the Global Navigation Satellite System (GNSS) timestamp at the moment the encrypted evidence stream was generated; it then aligns and merges the GNSS timestamp with the encrypted evidence stream's execution space bits; finally, it writes the encrypted evidence stream containing the GNSS timestamp into a non-erasable storage sector within the storage unit, thus completing the timestamp solidification of the evidence data.

[0125] The vehicle-mounted data processing platform retrieves pre-stored platform access parameters, which include network access addresses and identity authentication credentials for the police platform, emergency platform, and insurance platform. It then uses the identity authentication credentials to establish encrypted communication tunnels with the police platform, emergency platform, and insurance platform respectively. Finally, it pushes the solidified encrypted evidence stream to the police platform, emergency platform, and insurance platform simultaneously through the encrypted communication tunnels.

[0126] The vehicle-mounted data processing platform continuously monitors the backhaul port of the encrypted communication tunnel; receives data reception confirmation receipts from the police platform, emergency platform, and insurance platform; parses the message integrity check value contained in the data reception confirmation receipt; compares the message integrity check value with the original feature digest value of the encrypted evidence stream; if the message integrity check value and the feature digest value are consistent, it generates a reporting task completion instruction; and associates the reporting task completion instruction with the structured evidence package to complete the closed-loop verification of data reporting.

[0127] Specific application examples:

[0128] Based on the on-site stage assessment and video cropping optimization of specific road rear-end collision accidents, the constants and initial parameters set for the vehicle-mounted data processing platform are as follows:

[0129] When the multimodal large model outputs the stage judgment results, the information entropy baseline threshold preset in the instruction storage unit is set to 0.80; the frame number parameter of the stage state cache queue is set to a constant of 5; the first preset weight coefficient in the time node calculation unit is set to 0.1; the second preset weight coefficient is set to 1.0; the third preset weight coefficient is set to 0.5; the first adjustment coefficient of the global dynamic delineation evaluation stage is set to 10.0; and the second adjustment coefficient is set to 2.0.

[0130] During the result verification process at the execution stage of the vehicle-mounted data processing platform, the multimodal large model infers and decouples the multimodal correlation factors in the video stream. At a certain prediction execution moment, the vehicle-mounted data processing platform extracts the prediction probability data output by the multimodal large model for the current standardized video frame, where the prediction probability values ​​for the potential hazard period, collision period, and recovery period are 0.85, 0.10, and 0.05, respectively. Combining the current data stream status, the vehicle-mounted data processing platform extracts the probability values ​​of all candidate categories and substitutes them into the logarithmic summation formula of information entropy to calculate the current information entropy value of this frame:

[0131] ;

[0132] The vehicle-mounted data processing platform uses the calculated value of 0.518 as the pure classification deterministic bias feature for that frame, and simultaneously extracts the information entropy baseline threshold of 0.80 set in the internal database. Since the calculated value of 0.518 is less than the preset threshold of 0.80, the vehicle-mounted data processing platform marks the judgment result of this stage as having high deterministic attributes, thereby eliminating false biases caused by transient sensor occlusion or drastic changes in illumination, and preventing excessive fluctuations in subsequent confidence levels.

[0133] When the vehicle-mounted data processing platform actively defines the time range boundary conditions to detect the optimal capture state, the internal evaluation engine dynamically quantifies the visual image characteristics. The platform filters out feature entities that meet the overlap attenuation criteria based on the cross-frame matching operator, extracting 15 entity objects at that timestamp. Simultaneously, it captures 2 collision-related entities at that timestamp, and extracts the corresponding trajectory intersection index of 2.50 through two-dimensional spatial joint quantization analysis. The platform then calculates the current scene complexity using the weight allocation formula.

[0134] ;

[0135] The calculated scene complexity value of 4.75 is called by the time node calculation unit. Combined with the judgment requirement of the second scene complexity benchmark value of 4.0, a complex scene label is automatically generated to constrain the time range search space of the dynamic delineation stage. The time offsets before and after the collision are strictly expanded to ensure that the key features of the accident recording cycle are not truncated in a destructive way.

[0136] As global time boundary intervals are continuously issued and executed, the vehicle-mounted data processing platform completes the evaluation and division of multiple sets of consecutive candidate video intervals. After the latest round of video interval evaluation, the vehicle-mounted data processing platform extracts the four-dimensional quality features of this consecutive candidate video interval and calculates a comprehensive quality evaluation value of 0.85. Simultaneously, based on the previously extracted number of entity objects, number of collision-involved entities, and trajectory intersection index, the information implication metric formula is substituted:

[0137] ;

[0138] The platform extracts a preset information weight coefficient of 0.6 and a quality weight coefficient of 0.4, and substitutes them into the revenue evaluation formula to calculate the revenue value for this continuous candidate video interval:

[0139] ;

[0140] Since the calculated latest video interval gain value of 2.240 is the maximum value among all interval values, the judgment condition is triggered. The vehicle data processing platform immediately generates a target video interval fixation instruction internally, ends the optimization loop, and locks the editing start node and editing end node of the current interval.

[0141] Actual operation tests were conducted and data were compared. Specific verification results are detailed in the appendix. Figure 7 and attached Figure 8 The experiment compared the video and data processing results of the traditional fixed-duration clipping method and the method of this invention under the same real road collision test set and the same initial fluctuation distribution conditions of multimodal sensors. The experimental data is recorded in Table 1.

[0142] Table 1. Comparison of operational data between traditional fixed-duration cropping methods and the method of this invention.

[0143] Method type Accuracy rate of stage judgment (%) Key information retention rate (%) Invalid frame redundancy ratio (%) Average volume of evidence package (MB) Traditional fixed-duration cutting method 78.5 81.6 42.5 125.4 Method of the present invention 96.2 98.4 11.5 48.6

[0144] See attached document Figure 7 And Table 1, Appendix Figure 7 This diagram compares the convergence of information entropy and the evolution of accuracy at each time point in the accident stage judgment process. The horizontal axis represents the time point, in seconds (s), ranging from 40 to 60. The left vertical axis represents the information entropy value, dimensionless, ranging from 0 to 1.5. The right vertical axis represents the stage judgment accuracy, in percentages (%), ranging from 60 to 100. The diagram includes the accuracy curves of the traditional single-modal method, the method of this invention, and the information entropy curve of this invention.

[0145] In the initial stage of collision feature bursts from 45s to 50s, traditional single-modal methods fail to decouple multimodal interaction parameters, leading to a baseline shift in target judgment. This causes a rapid decline in accuracy, which then stagnates, ultimately settling at 78.5% and failing to improve further. The method of this invention introduces a cross-attention distribution matrix to eliminate single-modal abrupt changes and performs temporal verification optimization based on comprehensive confidence. The accuracy curve of this method exhibits a stable and highly resistant-to-interference trend. During the strong distortion phase at 50s, the information entropy value rapidly penetrates the 0.80 baseline threshold, triggering suspension protection. This ensures that the final judgment accuracy is strictly controlled within 96.2%, completely eliminating the accumulation of logical misjudgments caused by a single visual blind spot.

[0146] See attached document Figure 8 And Table 1, Appendix Figure 8This diagram illustrates the invalid frame pruning and revenue function control during the evaluation process for candidate video intervals. The horizontal axis represents the candidate video interval number, dimensionless, ranging from 1 to 20. The left vertical axis represents the invalid frame redundancy ratio, in percentages, ranging from 0 to 50; the right vertical axis represents the video interval revenue value, dimensionless, ranging from 0 to 3.0. The diagram includes a redundancy ratio line from the traditional method, a redundancy ratio line from this invention, and a video interval revenue status curve from this invention.

[0147] When the candidate video interval number reaches around 10, the target occlusion ratio increases sharply due to severe field of view shaking and a large amount of debris flying. Traditional methods, lacking multi-dimensional scene complexity constraints, directly include low-value peripheral images in the time-capture operation, leading to a surge in useless features. The highest peak of the redundancy ratio shown by the traditional method in the polygonal line reaches 42.5%.

[0148] This invention dynamically updates the true high-frequency edge response matrix by extracting the attenuation features of the first derivative of optical flow motion, and constructs an optimization equation matrix combining information implication metrics and comprehensive quality evaluation values ​​in the revenue function calculation unit. As can be seen from the revenue state curve of the video interval in this invention, when the video revenue approaches its extreme value, the platform automatically reduces and locks the tiny increments of redundant data stream packets, limiting their revenue comparison optimization to around the peak of 2.240. Under this globally maximum value state locking, the redundancy ratio piecewise linear curve of this invention rapidly reduces dimensionality throughout the entire pruning cycle and remains below 11.5%, ensuring a very high level of lightweightness and information density in the final output structured evidence package.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for road accident stage judgment and video processing based on a multi-modal large model, characterized in that, Includes the following steps: Acquire various sensor data and video streams, extract the global navigation satellite system timestamp as a reference, align the various sensor data and video streams with the global navigation satellite system timestamp, perform preprocessing, and output standardized video frames; Multimodal features are extracted from the standardized video frames and aligned sensor data, and joint representation data of a large multimodal model is constructed through a cross-modal attention network and a graph Transformer model. The joint representation data is input into the multimodal large model for hidden layer correlation reasoning, and the output includes the stage judgment results of the hidden danger period, the collision period and the recovery period. Perform temporal stability verification and single-frame deterministic review on the continuous stage judgment results, and output high-confidence judgment results; Based on the high confidence determination result, the collision reference time node is located, the scene complexity value is calculated to delineate the time axis boundary, the video interval benefit value is calculated according to the benefit function, and the target video data stream is extracted and an accident record video file is generated by combining the video interval benefit value. The high-confidence determination result and the accident recording video file are packaged into a structured evidence package, which is then encrypted and solidified before being uploaded in a closed loop.

2. The method of claim 1, wherein the method further comprises: The process of acquiring various sensor data and video streams and outputting standardized video frames includes: Calculate the time offset between the local received timestamp and the global navigation satellite system timestamp, perform frame skipping or frame-level interpolation on the real-time driving video stream and the driving history video based on the time offset to reconstruct the time axis, and use the time offset to adjust the recording time series of various signals from external sensors to unify the time reference of multiple vehicles. The real-time driving video stream and the driving history video after timeline reconstruction are video decoded to extract the original video frames. The original video frames are then subjected to resolution normalization, color space conversion, jitter reduction processing based on inter-frame displacement, dehazing processing based on dark channel prior algorithm, and backlight correction processing based on dynamically adjusting the pixel brightness gain value of overexposed and underexposed areas. The standardized video frames are then compiled and output. 3.The method of claim 1, wherein, The process of extracting multimodal features and constructing joint representation data includes: The standardized video frames are input into the target detection network model and the semantic segmentation network model to generate spatial features containing entity target detection features and pixel-level lane line masks. The continuous standardized video frames are input into the optical flow network model and the multi-target tracking model to generate temporal features containing optical flow motion vectors and trajectory data. By combining the pixel-level lane line mask, vehicle speed signal, vehicle steering signal, vehicle braking signal, vehicle attitude data, and vehicle impact data, the collision time, lane departure angle, acceleration abrupt value, and vehicle headway parameter are calculated and merged to generate semantic features. Spatial graph representation data is generated by performing node feature aggregation operations on the entity target detection features through a graph neural network model, and temporal evolution representation data is generated by calculating the evolution correlation parameters of the trajectory data on the time axis through a self-attention network. The spatial graph representation data and the temporal evolution representation data are then combined to generate a video visual feature matrix. The video visual feature matrix and the semantic features are input into the cross-modal attention network to calculate the cross-attention weight distribution matrix, and a weighted mapping operation is performed to output cross-modal interaction feature data. The cross-modal interaction feature data is then input into the graph Transformer model to perform a global association feature extraction operation to generate the joint representation data.

4. The method for road accident stage judgment and video processing based on a multimodal large model according to claim 1, characterized in that, The process of outputting the stage judgment results, including the hidden danger period, the collision period, and the recovery period, includes: When the acceleration mutation value extracted from the joint characterization data is greater than the preset acceleration mutation threshold, or the collision time is less than the preset collision time threshold, or the lane departure angle is greater than the preset lane departure angle threshold, the stage judgment result is output and the stage judgment result is assigned the hidden danger period. When the intersection-union ratio of the bounding box coordinate sequences of different entity objects under the same global navigation satellite system timestamp is determined to be greater than the preset intersection-union ratio threshold, and the optical flow motion vector is determined to be greater than the preset optical flow mutation threshold and the acceleration mutation value is determined to be greater than the preset acceleration mutation threshold, or when the vehicle airbag signal is determined to be in the triggered opening state, or the vehicle impact data is greater than the preset impact limit threshold, or when the entity target detection feature belonging to the debris category is located, the stage judgment result is output and the stage judgment result is assigned as the collision period; When the duration of the vehicle speed signal remaining at zero exceeds a preset static time threshold, and the spatial coordinate transfer vector of the entity object belonging to the pedestrian category is detected to show an extension trend from inside the vehicle to outside the vehicle, thereby generating a person leaving the vehicle feature, the stage judgment result is output and the stage judgment result is assigned to the post-stage.

5. The method for road accident stage judgment and video processing based on a multimodal large model according to claim 3, characterized in that, The process of performing temporal stability verification and single-frame deterministic review on the continuous stage judgment results, and outputting high-confidence judgment results includes: Establish a stage state cache queue, count the frequency values ​​of the hidden danger period, collision period and recovery period in the continuously cached stage judgment results, and set the stage judgment result with the largest frequency value as the majority judgment label; Extract the predicted probability data generated by the multimodal large model for different categories, calculate the information entropy value of the output result of the multimodal large model based on the predicted probability data, compare the information entropy value with the preset information entropy benchmark threshold, and determine whether the stage judgment result has a high deterministic attribute or a low deterministic attribute. When the stage judgment result is determined to have a high degree of certainty and the category of the majority judgment label is consistent with the category with the lowest information entropy in the current frame, the majority judgment label is transformed into the high confidence judgment result. When the stage judgment result is determined to have a low-determinism attribute, a comprehensive confidence level is calculated. When the comprehensive confidence level is greater than or equal to a preset comprehensive confidence level threshold, the majority judgment label is converted into the high-confidence judgment result. When the comprehensive confidence level is less than the preset comprehensive confidence level threshold, the automatic output task is suspended, a manual review request data packet containing the standardized video frame and the video visual feature matrix is ​​generated and sent to the manual review terminal, the manual confirmation label returned by the manual review terminal is received, and the manual confirmation label is converted into the high-confidence judgment result.

6. The method for determining the stage of a road accident and processing video based on a multimodal large model according to claim 1, characterized in that, The process of locating the collision reference time point and defining the time axis boundaries includes: Locate the high-confidence determination result assigned the value of the collision period in a continuous time series, and set the global navigation satellite system timestamp corresponding to the high-confidence determination result as the collision reference time node; The number of entity objects in the standardized video frames that are adjacent to the collision reference time node in the time dimension and the number of collision-participating entities that have collided are counted, and a trajectory intersection index is generated based on the cross-over degree and time proximity of the trajectory data. The scene complexity is calculated by integrating the number of entity objects, the number of entities involved in the collision, and the trajectory intersection index. The scene complexity value is compared with the first scene complexity benchmark value and the second scene complexity benchmark value to generate simple scene labels, medium scene labels or complex scene labels, and the corresponding pre-collision time offset and post-collision time offset are extracted according to the label. The video cropping start time node and video cropping end time node are calculated by combining the collision reference time node and the offset, and the video cropping start time node and video cropping end time node are used as the time axis boundaries of dynamic video cropping.

7. The method for determining the stage of a road accident and processing video based on a multimodal large model according to claim 6, characterized in that, The process of extracting target video data streams by combining video interval gain values ​​to generate accident record video files includes: For the standardized video frame located within the time axis boundary, calculate the statistical feature parameters of the natural scene to generate a BRISQUE score value, calculate the variance of the high-frequency edge response matrix to generate a blur metric, calculate the overlapping pixel area of ​​the coordinate sequences of the bounding boxes of the entity objects to generate an occlusion ratio value, compare the proportion of contour pixels to generate target visibility evaluation parameters, and merge to generate a four-dimensional quality feature vector for the standardized video frame. Within the global time boundary interval formed by the video cropping start time node and the video cropping end time node, multiple consecutive candidate video intervals are divided. The four-dimensional quality feature vectors in the candidate video intervals are weighted and summed to obtain a comprehensive quality evaluation value. The information implied metric value is calculated by combining the number of collision participating entities, the number of entity objects, and the trajectory intersection index. Substitute the comprehensive quality evaluation value and the information implication metric value into a preset revenue function to calculate multiple sets of video interval revenue values, and locate the candidate video interval bound to the video interval revenue value that is in the maximum value state as the target video interval. The start and end points of the time corresponding to the target video interval are extracted and assigned as the editing start node and editing end node, respectively. The editing start node and the editing end node are located in the original video stream sequence and the target video data stream is extracted and encapsulated to generate the accident record video file.

8. The method for determining the stage of a road accident and processing video based on a multimodal large model according to claim 1, characterized in that, The process of packaging evidence into a structured evidence package includes: Retrieve trajectory data within the time range covered by the start and end nodes of the clip, as well as the entity objects and unique identifiers (IDs) contained within the trajectory data; The trajectory data, the unique identifier ID, and the global navigation satellite system timestamp are mapped to a unified time axis coordinate system and merged to generate spatiotemporal trajectory structured data. The high confidence level determination result, the accident recording video file, the spatiotemporal trajectory structured data, and the comprehensive confidence level value are sent to the packet storage unit for time axis association and alignment. The vehicle terminal identification code and system timing parameters are added to the front end of the data stream to perform data stream encapsulation and output the structured evidence package, so that the high confidence level determination result, the accident recording video file, the spatiotemporal trajectory structured data, and the comprehensive confidence level value in the structured evidence package establish a one-to-one mapping association.

9. The method for determining the stage of a road accident and processing video based on a multimodal large model according to claim 8, characterized in that, The process of encrypting and solidifying the structured evidence package and then uploading it in a closed loop includes: The national cryptographic algorithm is invoked to perform data digest extraction and asymmetric encryption operations on the structured evidence package, generating an encrypted evidence stream; Retrieve the Global Navigation Satellite System timestamp at the moment the encrypted evidence stream was generated, perform spatial bit alignment with the encrypted evidence stream, merge and write it into a non-erasable storage sector to complete the timestamp solidification of the evidence data; An encrypted communication tunnel is established using identity authentication credentials, and the solidified encrypted evidence stream is synchronously pushed to the external platform corresponding to the platform access parameters. The system receives a data reception confirmation receipt from the external platform, compares the message integrity check value with the original feature digest value of the encrypted evidence stream, and generates a reporting task completion instruction if they match. The system then associates the instruction with the structured evidence package to complete the closed-loop verification of the data reporting.

10. An in-vehicle data processing platform, characterized in that, For implementing the road accident stage judgment and video processing method based on a multimodal large model as described in any one of claims 1-9, the vehicle-mounted data processing platform is equipped with a video access terminal and a multimodal sensor data access terminal; The video access terminal is used to connect to the dashcam, enabling the dashcam to collect real-time driving video streams and historical driving videos of the vehicle's surrounding environment, and to transmit the real-time driving video streams and historical driving videos to the vehicle data processing platform through the video access terminal; The multimodal sensor data access terminal is used to connect to external sensors, which include a GPS positioning module, a Beidou positioning module, an IMU measurement module, and a CAN bus interface. This enables the external sensors to continuously collect GPS location data, Beidou location data, vehicle attitude data, vehicle impact data, vehicle speed signals, vehicle steering signals, vehicle braking signals, and vehicle airbag signals, and transmit these data to the vehicle data processing platform. The vehicle-mounted data processing platform is internally deployed with a multimodal large model inference environment and video encoding and decoding hardware. The vehicle-mounted data processing platform is used to perform multimodal fusion modeling operations and video cropping output based on the aggregated data using the multimodal large model inference environment and the video encoding and decoding hardware.

Images