New energy vehicle accident time accurate identification method based on multi-modal data fusion

Abstract

This invention discloses a method for accurate identification of the moment of an accident in a new energy vehicle based on multimodal data fusion. The method acquires vehicle motion sensor data, dynamic parameters, and visual image data, and then performs the following parallel processing: The physical layer processes the data to obtain motion features and compares them with preset trigger conditions; when a suspected collision is detected, it outputs the physical layer trigger time and acceleration amplitude; the dynamic layer processes the data based on the deviation between the vehicle's dynamic model and its actual motion state, outputting a vehicle trajectory status label; the visual layer processes the data to detect targets and analyze their motion state; when visual evidence of a collision is detected, it outputs the visual layer trigger time and the type of collision object; the fusion layer performs spatiotemporal alignment of the three layers' outputs and makes a comprehensive judgment, outputting the fused accurate collision trigger time and accident scene classification information. This invention effectively filters false alarms and improves the detection rate of low-speed accidents through multimodal fusion, achieving accurate identification of the moment of an accident and scene reconstruction.

Description

Technical Field

[0001] This invention relates to the fields of active safety technology for automobiles, autonomous driving perception technology, and traffic accident identification, and more specifically, to a method for accurate identification of the moment of an accident in a new energy vehicle based on multimodal data fusion. Background Technology

[0002] As the level of intelligence in new energy vehicles improves, the boundaries between autonomous driving perception systems and passive safety systems are gradually merging. Traditional collision detection technology mainly relies on MEMS acceleration sensors deployed in the airbag control unit (ACU) or T-BOX. Its core logic is to monitor the vehicle's acceleration value (G-value) on the X-axis (longitudinal) or Y-axis (lateral). Once this value exceeds a preset threshold (usually 1.5G-2.0G), a collision is determined and EDR data locking is triggered.

[0003] However, this single-dimensional physical triggering mechanism faces serious challenges in practical engineering applications: 1. False Positives Caused by Road Interference: When a vehicle travels at a high speed (e.g., 60 km / h) over a rigid speed bump, pothole, or protrusion on an unpaved road, the instantaneous impact acceleration transmitted from the suspension system to the vehicle body can often exceed 2.0G, causing the system to incorrectly lock onto accident data. This not only occupies valuable storage space but may also lead to a backlog of invalid alarms in the back-end monitoring center.

[0004] 2. False Negative for low-speed minor collisions: In congested traffic, low-speed crawling (<10km / h) or parking lot scraping scenarios, the peak acceleration generated by the vehicle collision may only be 0.6G-0.8G, which is lower than the traditional high trigger threshold. As a result, such accidents with frequent disputes over liability cannot be captured by the system.

[0005] 3. Lack of semantic information: Purely physical sensors can only sense "force" but not "receiver". That is, the system cannot distinguish whether the vehicle crashed into the guardrail due to loss of control, was rear-ended by another vehicle, or had its chassis impacted by a foreign object on the road. Summary of the Invention

[0006] The purpose of this invention is to propose a method for accurate identification of new energy vehicle accidents based on multimodal data fusion, which solves the problem of high false alarm rate in complex road conditions and missed detection of low-speed minor collisions in existing single acceleration threshold triggering mechanisms based on EDR and T-BOX, thereby improving the accuracy of accident identification.

[0007] To achieve the above objectives, this invention proposes a method for accurate accident identification of new energy vehicles based on multimodal data fusion, comprising: Real-time acquisition of vehicle motion sensor data, dynamic parameters, and visual image data; Perform the following processing respectively: Physical layer processing: The motion sensor data is processed to obtain motion features. Based on the comparison between the motion features and preset trigger conditions, it is determined whether a suspected collision has occurred. When a suspected collision is determined, the physical layer collision trigger time and the corresponding acceleration amplitude are output. Dynamics layer processing: Based on the vehicle dynamics model, the theoretical motion state of the vehicle is calculated according to the dynamic parameters, the actual motion state is obtained, and the vehicle trajectory state label is output based on the deviation between the theoretical motion state and the actual motion state. Visual layer processing: The visual image data is processed to detect targets in the image and analyze the motion state of the targets. Based on the comparison between the motion state of the targets and preset visual conditions, it is determined whether there is visual evidence of collision. When it is determined that there is visual evidence of collision, the visual layer collision trigger time and collision object type are output. The outputs obtained from the physical layer processing, dynamics layer processing, and visual layer processing are subjected to fusion layer processing, including: spatiotemporal alignment of the heterogeneous data output from each layer, and synthesis of the outputs from each layer according to preset rules, outputting the fused precise collision trigger time and accident scene classification information.

[0008] Optionally, the motion sensor data includes inertial measurement unit data and wheel speed sensor data; The dynamic parameters include steering wheel angle and actual yaw rate; The visual image data refers to image data collected in real time by the vehicle-mounted camera.

[0009] Optionally, in the physical layer processing, processing the motion sensor data to obtain motion features includes: The motion sensor data is filtered to obtain an acceleration estimate. Calculate the jerk based on the acceleration estimate; The motion characteristics include the acceleration estimate and the jerk; The filtering process for the motion sensor data includes: A linear Kalman filter is constructed, with the vehicle's current position, speed, and acceleration as the state vector. State prediction is performed based on a uniform acceleration model, with the speed observed by the wheel speed sensor and the acceleration observed by the inertial measurement unit as the observation values, and a smooth acceleration estimate is output.

[0010] Optionally, in the physical layer processing, determining whether a suspected collision has occurred based on a comparison between the motion characteristics and preset triggering conditions includes: The dynamic threshold of acceleration is determined based on the current vehicle speed, and the dynamic threshold of acceleration changes continuously with the vehicle speed. When the magnitude of the acceleration estimate exceeds the acceleration dynamic threshold and the magnitude of the jerk exceeds the preset impact threshold, a suspected collision is determined to have occurred.

[0011] Optionally, the acceleration dynamic threshold is calculated based on the current vehicle speed using the Sigmoid function, and the calculation formula is as follows:

[0012] in, For acceleration threshold, This is the lower limit of low-speed sensitivity. The upper limit of high-speed robustness, Current vehicle speed For the center conversion speed, This is the curve steepness coefficient; The conditions for determining a suspected collision are expressed as follows:

[0013] in, This indicates the judgment result of the physical layer processing output. This is the smoothed acceleration estimate. To accelerate, This is the preset jerk threshold.

[0014] Optionally, in the dynamics layer processing, the vehicle dynamics model is a two-degree-of-freedom vehicle model; the theoretical motion state includes the theoretical yaw rate, and the actual motion state includes the actual yaw rate; Based on the deviation between the theoretical motion state and the actual motion state, a vehicle trajectory status label is output, including: Calculate the magnitude of the deviation between the theoretical yaw rate and the actual yaw rate; The cumulative error energy is obtained by performing a sliding window integration on the magnitude of the deviation. Based on a finite state machine, the trajectory state of the vehicle is determined according to whether the accumulated error energy exceeds a preset energy threshold and the duration, wherein the trajectory state includes at least one of normal driving, understeering, oversteering and spin.

[0015] Optionally, in the visual layer processing, detecting targets in the image and analyzing the motion state of the targets includes: Using a lightweight YOLOv11 neural network that has been pruned and quantized, target detection and tracking are performed on visual image data acquired in real time by vehicle cameras to obtain the bounding box and identification of the target. The target includes at least one of pedestrians, vehicles, cyclists, and obstacles.

[0016] Optionally, in the visual layer processing, determining whether there is visual evidence of a collision based on a comparison between the target's motion state and preset visual conditions includes: The width of the target bounding box and its rate of change are smoothed using Kalman filtering; The visual collision time is calculated based on the bounding box width change rate, and the formula is as follows:

[0017] in, For visual collision time, The width of the target bounding box. The rate of change of the target bounding box width over time; When the visual collision time is less than a preset time threshold and the ratio of the target bounding box width to the image width exceeds a preset ratio, visual evidence of a collision is determined to exist.

[0018] Optionally, the visual layer processing further includes: Optical flow analysis is performed on the image to detect at least one anomalous feature among full-image instantaneous shift, sudden increase in optical flow direction entropy, or continuous image signal loss; When the abnormal feature is detected, a visual impact indicator is output as supplementary visual evidence of the collision.

[0019] Optionally, the fusion layer processing includes: The output data obtained from physical layer processing, dynamics layer processing, and vision layer processing are stored in their respective circular buffers. Based on the timestamp of the data output by the visual layer, linear interpolation is used to align the data output by the physical layer and the dynamics layer to obtain the aligned multimodal determination result. The weights of the physical layer processing, dynamics layer processing, and visual layer processing are dynamically adjusted based on the current vehicle speed. A weighted confidence score is then calculated for the aligned multimodal decision result, using the following formula:

[0020] in, , , These are the normalized outputs from the physical layer processing, dynamics layer processing, and visual layer processing, respectively. , , The dynamic weighting coefficient varies with vehicle speed; The accident level is determined based on the threshold range in which the confidence score falls; Based on the aligned multimodal determination results and the accident level, a fused accurate collision trigger time and accident scene classification information are generated; the accident scene classification information includes at least one of vehicle trajectory status label, collision object type and accident level. The incident levels include serious incidents, suspected incidents, and interference events; serious incidents trigger automatic alarms, data locking, and cloud upload operations; suspected incidents trigger local data locking and user interaction inquiry operations; and interference events trigger cached data discarding operations.

[0021] The beneficial effects of this invention are as follows: This invention achieves high-confidence and accurate identification of accident moments by constructing a four-in-one parallel processing architecture of "physical layer, dynamics layer, vision layer, and fusion layer." Compared with existing technologies, this invention significantly improves the accuracy of collision detection in the physical layer by extracting motion features and comparing preset triggering conditions. In the dynamics layer, it achieves accurate identification of vehicle out-of-control states based on the deviation analysis between the vehicle dynamics model and the actual motion state. In the vision layer, it endows the system with semantic understanding capabilities through target detection and motion state analysis. In the fusion layer, it outputs the precise collision trigger moment and accident scene classification information after fusion by aligning heterogeneous data in time and space and making comprehensive judgments. Finally, through deep fusion of multimodal data, it achieves accurate identification of accident moments and scene reconstruction, significantly improving the accuracy and robustness of accident identification.

[0022] Furthermore, this invention effectively distinguishes between rigid collisions and disturbances such as road impacts and heavy braking by analyzing dynamic thresholds related to vehicle speed at the physical layer through acceleration analysis; at the dynamic layer, it accurately identifies pre-control precursors such as understeer, oversteer, and spin based on a two-degree-of-freedom model and finite state machine; and at the visual layer, it provides collision object types and visual evidence through lightweight YOLOv11 and optical flow methods. These three parallel processing layers mutually corroborate each other, overcoming the shortcomings of traditional single acceleration threshold methods, which suffer from high false alarm rates in complex road conditions and high false negative rates in low-speed scenarios.

[0023] Furthermore, this invention achieves millisecond-level heterogeneous data synchronization through a circular buffer and linear interpolation. Scene-adaptive weights dynamically adjust the contribution of each layer based on vehicle speed. At low speeds, visual evidence is emphasized to capture minor scrapes, while at high speeds, physical and dynamic evidence is prioritized to ensure robustness, thus guaranteeing high confidence in the fusion results. The final output, including the precise collision trigger time T0 and accident scene classification information (vehicle trajectory status, collision object type, and accident level), provides comprehensive and accurate data support for traffic accident identification.

[0024] Furthermore, this invention classifies incidents into serious incidents, suspected incidents, and interference events based on confidence scores, and triggers differentiated response actions (automatic alarm, local query, cache discard), which ensures that real incidents are handled in a timely manner, while avoiding invalid alarms from occupying storage and communication resources, thus balancing real-time response and resource optimization.

[0025] The present invention has other features and advantages, which will be apparent from or will be set forth in detail in the accompanying drawings and the following detailed description, which together serve to explain the particular principles of the invention. Attached Figure Description

[0026] The above and other objects, features and advantages of the present invention will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0027] Figure 1 The flowchart of a method for accurate identification of incidents in new energy vehicles based on multimodal data fusion according to Embodiment 1 of the present invention is shown.

[0028] Figure 2 The diagram shows a schematic of a new energy vehicle accident-time accurate identification system based on multimodal data fusion according to Embodiment 2 of the present invention. Detailed Implementation

[0029] The invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0030] Example 1

[0031] This embodiment provides a method for accurate accident identification of new energy vehicles based on multimodal data fusion. The overall process is as follows: Figure 1 As shown, the specific steps include the following.

[0032] S1: Data Acquisition: Real-time acquisition of vehicle motion sensor data, dynamic parameters, and visual image data; Specifically, this step involves data acquisition, obtaining multi-source heterogeneous data about the vehicle in real time, including motion sensor data, dynamic parameters, and visual image data. The motion sensor data includes inertial measurement unit (IMU) data and wheel speed sensor data. The dynamic parameters include steering wheel angle and actual yaw rate. The visual image data is image data acquired in real time by the onboard camera. In this embodiment, the IMU outputs triaxial acceleration signals at a sampling rate of 50Hz, the wheel speed sensors provide vehicle speed signals, the steering wheel angle is converted to the front wheel angle through the steering ratio, and the camera acquires images of the vehicle's surrounding environment at a sampling rate of 30Hz.

[0033] Then, the following processing steps S2-S4 are executed in parallel.

[0034] S2: Physical layer processing: Process the motion sensor data to obtain motion features, and determine whether a suspected collision has occurred based on the comparison between the motion features and preset trigger conditions. When a suspected collision is determined, output the physical layer collision trigger time and the corresponding acceleration amplitude. Physical layer processing is the cornerstone of the entire system, responsible for extracting clean motion signals from high-frequency noise and providing a primary trigger flag.

[0035] In one embodiment, processing the motion sensor data to obtain motion features includes: filtering the motion sensor data to obtain an acceleration estimate; calculating jerk based on the acceleration estimate; and the motion features include the acceleration estimate and the jerk.

[0036] The filtering process for the motion sensor data includes: constructing a linear Kalman filter, using the vehicle's current position, speed, and acceleration as state vectors, performing state prediction based on a uniform acceleration model, using the speed observed by the wheel speed sensor and the acceleration observed by the inertial measurement unit as observed values, and outputting a smooth acceleration estimate.

[0037] Specifically, to smooth the signal and obtain the optimal estimate of acceleration, a linear Kalman filter is constructed to filter the motion sensor data. The state vector is defined as follows: , including location ,speed acceleration .

[0038] State transition equation:

[0039] in, Here is the state transition matrix. The noise is a process noise that follows a Gaussian distribution. For discrete-time indexing; Where the transition matrix Based on the uniform acceleration model:

[0040] Observation matrix Only velocity (from wheel speed sensor) and acceleration (from IMU) were observed:

[0041] By adjusting the process noise covariance matrix and observation noise covariance matrix The Kalman filter can correct for instantaneous spikes in sensor readings (such as electromagnetic interference) using the state prediction value from the previous moment, thereby outputting a smooth acceleration estimate with minimal hysteresis. .

[0042] In one implementation, determining whether a suspected collision has occurred based on a comparison between the motion characteristics and preset triggering conditions includes: An acceleration dynamic threshold is determined based on the current vehicle speed, and this threshold changes continuously with the vehicle speed. When the amplitude of the estimated acceleration exceeds the acceleration dynamic threshold and the amplitude of the jerk exceeds a preset impact threshold, a suspected collision is identified, and the physical layer trigger time is output. And the corresponding acceleration amplitude.

[0043] Traditional techniques typically employ discrete threshold segmentation (step function), for example: , This method exhibits a significant physical discontinuity. A 0.7G collision will be detected when the vehicle is traveling at 19.9 km / h, but the same collision will be ignored when accelerating to 20.1 km / h. This "jump" can lead to logical flaws in accident investigation.

[0044] This invention proposes a continuous adaptive acceleration dynamic threshold model, which utilizes the S-shaped curve characteristics of the Sigmoid function to construct the acceleration threshold. With vehicle speed Nonlinear mapping relationship:

[0045] in, This is the lower limit of low-speed sensitivity. The upper limit of high-speed robustness, Current vehicle speed For the center conversion speed, This is the curve steepness coefficient. It is a natural constant.

[0046] In a preferred embodiment, the above parameters are set as follows: Low-speed sensitivity lower limit set to The setting is based on the fact that minor scrapes in a parking lot (such as bumper collisions) typically produce lateral acceleration of 0.5G-0.8G. At this time, the vehicle speed is low, the road surface is flat, the ambient noise is low, and the high sensitivity will not lead to false alarms.

[0047] High-speed robustness upper limit set to The threshold is set based on the fact that when a vehicle travels at speeds greater than 80 km / h on a highway, it experiences an instantaneous impact of approximately 1.0G when passing over bridge joints or expansion joints. To prevent false triggering, the threshold needs to be increased to 1.5G or higher.

[0048] Center conversion speed set to This is a typical speed limit on urban roads, and also the critical point for the transition from low-speed congestion to medium-to-high-speed cruising conditions.

[0049] The curve steepness coefficient is set to This parameter controls the rate at which the threshold transitions from sensitive to robust. The higher the value, the more drastic the transition. Through extensive real-vehicle calibration, 0.15 ensures that the threshold increases linearly with speed within the commonly used speed range of 30-70 km / h, which meets the driver's psychological expectations for vehicle stability.

[0050] This acceleration threshold continuity function ensures that, at any velocity point, the acceleration threshold remains constant. All are differentiable and continuous: when hour, Since the denominator is huge, the second term approaches 0.

[0051] when hour, Since the exponent approaches 0, the denominator approaches 1.

[0052] when At that time, the function automatically calculates an intermediate value (e.g., 0.9G).

[0053] This design not only eliminates logical jumps, but also naturally conforms to the laws of physics: the faster the speed, the higher the energy threshold required for "confirming the collision".

[0054] Furthermore, this step also introduces "jerk" feature analysis. Since acceleration amplitude alone is sometimes insufficient to distinguish between a "collision" and "forced braking," and when AEB (Automatic Emergency Braking) is triggered, the vehicle deceleration can reach 1.0G or even 1.2G, which may falsely trigger the low-speed threshold.

[0055] Therefore, we introduce the physical quantity Jerk (jerk / impact), which is the derivative of acceleration with respect to time:

[0056] in, To increase jerk (impact). Let acceleration be a function of time. The acceleration sample value at the current moment. The acceleration sample value from the previous moment. The sampling time interval, This is a discrete sampling index.

[0057] Rigid collision (Crash) is an extremely brief physical process (contact time < 50 ms). Metal structures undergo plastic deformation, and acceleration spikes from 0 to its peak value within milliseconds. Therefore, the Jerk value of a rigid collision is extremely high, typically... During hard braking, the hydraulic pressure build-up in the braking system and the exertion of tire grip are relatively slow processes (build-up time > 200ms). The acceleration curve shows a smooth upward trend, therefore the Jerk value during hard braking is relatively small, typically... Therefore, the preset impact threshold is set. The speed is set to 30G / s to differentiate between rigid collisions and AEB braking scenarios.

[0058] The final trigger condition of the physical layer (determining a suspected collision) is designed as an AND logic:

[0059] in, This indicates the decision result of the physical layer processing output, and is a boolean value (True / False). This is the filtered and smoothed acceleration estimate. To accelerate, The preset jerk (impact) threshold is set to... , This is the logical AND operator.

[0060] The system only identifies a suspected collision when it detects a huge acceleration amplitude that occurs instantaneously. This logic completely eliminates false alarms caused by AEB braking and long-wave road undulations (such as wavy roads).

[0061] S3: Dynamics layer processing: Based on the vehicle dynamics model, calculate the theoretical motion state of the vehicle according to the dynamic parameters, obtain the actual motion state, and output the vehicle trajectory state label based on the deviation between the theoretical motion state and the actual motion state. The physical layer confirms that "a force was applied," while the dynamics layer is responsible for answering whether "the vehicle lost control." This is the key to distinguishing between ordinary road impacts (where the vehicle trajectory remains unchanged) and real accidents (where the vehicle trajectory changes). The dynamics layer reconstructs the vehicle's trajectory state by establishing a vehicle dynamics model to answer whether "the vehicle lost control."

[0062] In one implementation, the vehicle dynamics model in the dynamics layer processing is a two-degree-of-freedom vehicle model; the theoretical motion state includes the theoretical yaw rate, and the actual motion state includes the actual yaw rate. Based on the deviation between the theoretical motion state and the actual motion state, a vehicle trajectory status label is output, including: Calculate the magnitude of the deviation between the theoretical yaw rate and the actual yaw rate; The cumulative error energy is obtained by performing a sliding window integration on the magnitude of the deviation. Based on a finite state machine, the trajectory state of the vehicle is determined according to whether the accumulated error energy exceeds a preset energy threshold and the duration, wherein the trajectory state includes at least one of normal driving, understeering, oversteering and spin.

[0063] Specifically, to determine whether a vehicle is in an abnormal state, it is necessary to first establish a theoretical behavioral benchmark for the vehicle under normal conditions. This method employs the classic linear two-degree-of-freedom "bicycle model".

[0064] Assume the vehicle is undergoing steady-state steering motion on a level road surface, and the tires are in a linear region with a small slip angle. According to the Newton-Euler equations, the theoretical yaw rate is... With vehicle speed and front wheel steering angle The relationship is:

[0065] in: The wheelbase (distance between the centers of the front and rear axles) is a geometrical parameter of the vehicle. The actual steering angle of the front wheels, the steering wheel angle read by the steering wheel angle sensor. It needs to be divided by the steering gear ratio. Therefore ; For stability factor, unit is a core dynamic parameter characterizing the steering characteristics of a vehicle, and it is defined as:

[0066] in, For the overall vehicle quality, These are the distances from the center of mass to the front and rear axles, respectively. These are the lateral stiffness of the front and rear tires, respectively.

[0067] The calibration methods for the above parameters are as follows: The value is usually measured through a real-vehicle "steady-state yaw rate test". In the constant-circle test, the steering wheel angle and yaw rate are recorded at different vehicle speeds, and the slope of the fitted curve contains the values. Value. For most passenger vehicles, understeer characteristics are required by design, therefore .

[0068] This model is based on the assumption of small-angle linearity. This means that when the tire's lateral acceleration exceeds 0.4G, the tire enters the nonlinear region, and the model's calculations will no longer be accurate. It is precisely by utilizing this that the model can be used to calculate the actual yaw rate (actual yaw rate). ) and model calculated values ​​(theoretical yaw rate) When a huge deviation occurs, it means that the vehicle has broken out of the linear stability region and entered a "out-of-control" state.

[0069] Furthermore, the motion state of the vehicle is abstracted into a finite state machine (FSM), and the vehicle trajectory state label is output by the finite state machine as the determination result of the dynamics layer.

[0070] The state transitions of a finite state machine are based on the deviation rate. and lateral acceleration The state decision matrix is ​​defined as shown in the table below, where, For yaw rate deviation, Actual yaw rate (sensor measurement). The theoretical yaw rate, The threshold used to determine whether the deviation is within the normal range (this threshold is usually set based on sensor noise and system accuracy). This is the actual steering angle of the front wheels. This is the saturation threshold for lateral acceleration (typically corresponding to the limit of the tire's linear region).

[0071]

[0072] Due to the presence of sensor noise, the deviation rate Momentary glitches may occur. To prevent frequent transitions between state 0 and S2, a time window is introduced. The points system.

[0073] Define cumulative error energy :

[0074] The state transition trigger condition has been upgraded to:

[0075] That is, only when the accumulated error energy exceeds the preset energy threshold. Furthermore, the state machine only transitions when the duration exceeds 100ms. This logic ensures that only continuous and genuine dynamic deviations will trigger the loss-of-control state determination, thereby distinguishing between "instantaneous deviation caused by driving over one side of water accumulation" and "vehicle fishtailing caused by a collision".

[0076] S4: Visual layer processing: Process the visual image data, detect targets in the image and analyze the motion state of the targets, and determine whether there is visual evidence of collision based on the comparison between the motion state of the targets and preset visual conditions. When it is determined that there is visual evidence of collision, output the visual layer collision trigger time and collision object type. The visual layer endows the system with the ability of "semantic understanding," which can not only verify the occurrence of a collision but also identify the collision object. It is a key basis for determining responsibility for accidents in multimodal fusion.

[0077] In one implementation, detecting a target in an image and analyzing the target's motion state includes: Using a lightweight YOLOv11 neural network that has undergone pruning and quantization, target detection and tracking are performed on visual image data acquired in real time by an onboard camera to obtain the bounding box and identification of the target; the target includes at least one of pedestrians, vehicles, cyclists, and obstacles.

[0078] Specifically, considering the computing power limitations of the vehicle-mounted system, a lightweight YOLOv11 architecture with pruning and quantization is adopted to significantly reduce the amount of computation while ensuring detection accuracy.

[0079] In a preferred embodiment, the lightweight YOLOv11 neural network uses a modified CSPDarknet as its backbone. The CSP (Cross Stage Partial) structure splits the feature map channels in two: one part is computed through convolutional layers, and the other part is directly connected to the end via a Skip Connection. This design reduces computational cost (FLOPs) and GPU memory usage by approximately 50% while maintaining feature extraction capabilities.

[0080] In the input layer, the Focus module is used instead of traditional convolutional downsampling. The Focus module transfers spatial information to the channel dimension through slicing operations, effectively reducing information loss during the downsampling process.

[0081] To enhance the model's detection capabilities in complex backgrounds, a CBAM (Convolutional Block Attention Module) attention mechanism is embedded in the Neck layer. CBAM comprises two independent dimensions: Channel Attention: Learns the weights of different channels through Global Average Pooling and MLP, emphasizing "what" is an important feature (such as headlights and license plate texture).

[0082] Spatial Attention: By learning spatial weights through max pooling and convolution, it emphasizes "where" is the target region and suppresses interfering information such as trees and buildings in the background.

[0083] This allows the model to accurately lock onto the outline of vehicles ahead even in low signal-to-noise ratio environments such as rainy days and nighttime.

[0084] To achieve accurate detection of multi-scale targets, a structure combining FPN (Feature Pyramid Network) and PANet (Path Aggregation Network) is adopted. Specifically: Deep features (P5): After multiple downsampling, they have strong semantic information and are used to detect large targets (such as nearby vehicles).

[0085] Shallow features (P3): Preserve rich geometric details for detecting small targets (such as pedestrians in the distance).

[0086] FPN transmits deep semantic information to shallow layers, while PANet further enhances bottom-up path aggregation, shortening information paths and improving the fusion effect of multi-scale features. This multi-scale fusion ensures that the system can detect distant risks as early as possible and maintain the stability of the detection box just before a collision (when the target fills the screen).

[0087] By using target detection and tracking (such as the ByteTrack algorithm), a unique ID is assigned to each detected target to achieve stable tracking across frames and obtain the target's bounding box and identity.

[0088] Furthermore, in this step, determining whether there is visual evidence of a collision based on a comparison between the target's motion state and preset visual conditions includes: The width of the target bounding box and its rate of change are smoothed using Kalman filtering; The visual collision time is calculated based on the bounding box width change rate, and the formula is as follows:

[0089] in, For the collision time, The width of the target bounding box. The rate of change of the target bounding box width over time; When the visual collision time is less than a preset time threshold and the ratio of the target bounding box width to the image width exceeds a preset ratio, visual evidence of a collision is determined to exist.

[0090] Specifically, TTC (Time To Collision) is the core indicator for assessing the degree of danger. Traditional TTC relies on millimeter-wave radar ranging, while this method uses only a monocular camera to achieve visual TTC estimation.

[0091] The TTC formula derivation is based on the pinhole camera model, as follows: Based on the pinhole camera imaging model, the width of the object in the image With its actual width and its distance from the camera The relationship is:

[0092] in For camera focal length. For time. Differentiate:

[0093] in This refers to relative velocity.

[0094] According to the TTC definition (Assuming the relative velocity is constant), substituting into the above equation and simplifying, we get:

[0095] Kalman filtering is used to smooth the width and its rate of change, thus avoiding jitter in a single frame.

[0096] The robust implementation logic for determining the existence of visual evidence of collision in this step is as follows: Target tracking: The ByteTrack algorithm is used to assign an ID to the detection box in each frame to ensure that the size change of the same target is being calculated.

[0097] Smoothing: Jitter in the width of the original detection box will cause... Drastic fluctuations. A Kalman filter is used to... and Perform a smoothing estimate.

[0098] Triggering condition: When the condition is met and When the target percentage exceeds 30%, visual evidence of a collision is determined, the visual layer outputs a "Pre-Crash" warning signal, and outputs the trigger time of the visual layer. and the type of collision object.

[0099] In another example, the visual layer processing further includes: performing optical flow analysis on the image to detect at least one anomalous feature among full-image instantaneous shift, sudden increase in optical flow direction entropy, or continuous image signal loss; and when the anomalous feature is detected, outputting a visual impact indicator as supplementary visual evidence of a collision.

[0100] Specifically, at the instant the collision occurs ( At certain times, due to excessive distance or camera impact, the YOLO model often fails (target loss or image blurring). In this case, optical flow becomes the final "black box" determination method.

[0101] Shi-Tomasi corner points were selected in the image, and the sparse optical flow vector was calculated using the Lucas-Kanade algorithm. During normal driving, the optical flow vector radiates outwards from the image center (FOE), and its length changes continuously. During an accident impact, the following characteristics may appear: (1) Full-image instantaneous shift: The optical flow vectors of all feature points undergo a displacement with the same direction and huge amplitude within 1-2 frames. This indicates that the camera bracket has undergone physical displacement or deformation due to a severe impact.

[0102] (2) Sudden increase in entropy: The directional entropy of the optical flow vector field spikes instantly, indicating that the image is caught in a chaotic vibration blur.

[0103] (3) Signal loss: If the histogram of N consecutive frames is completely black (pixel mean <10) or remains unchanged, it indicates that the camera module is damaged or the wiring harness is broken.

[0104] The system will mark the timestamp of detecting the above features as... It has a very high confidence weight.

[0105] S5: Fusion layer processing: The outputs obtained from the physical layer processing, dynamics layer processing and visual layer processing are fused, including: spatiotemporally aligning the heterogeneous data output from each layer, and integrating the outputs from each layer according to preset rules to output the fused precise collision trigger time and accident scene classification information.

[0106] The fusion layer is the "brain" of the system, which resolves the inconsistencies in time, space and logic of heterogeneous sensor data and outputs the final decision results.

[0107] In one embodiment, the fusion layer processing includes: storing the output data obtained from the physical layer processing, the dynamics layer processing, and the visual layer processing into their respective circular buffers; Based on the timestamp of the data output by the visual layer, linear interpolation is used to align the data output by the physical layer and the dynamics layer to obtain the aligned multimodal determination result. The weights of physical layer processing, dynamics layer processing, and visual layer processing are dynamically adjusted according to the current vehicle speed, and a weighted confidence score is calculated for the aligned multimodal determination result. The accident level is determined based on the threshold range in which the confidence score falls; Based on the aligned multimodal determination results and the accident level, a fused accurate collision trigger time and accident scene classification information are generated; the accident scene classification information includes at least one of vehicle trajectory status label, collision object type and accident level. The incident levels include serious incidents, suspected incidents, and interference events; serious incidents trigger automatic alarms, data locking, and cloud upload operations; suspected incidents trigger local data locking and user interaction inquiry operations; and interference events trigger cached data discarding operations.

[0108] Specifically, since the IMU sampling rate is 50Hz (20ms / frame) and the camera sampling rate is 30Hz (33ms / frame), there is an inherent time difference between the two. Direct comparison can lead to logical errors such as "the visual system saw the impact, but the physical signal had not yet arrived" or vice versa.

[0109] Therefore, two shared memory regions are allocated as FIFO queues: Buffer A (IMU): Length 250, stores the past 5 seconds. data.

[0110] Buffer B (Vision): Length 150, stores the past 5 seconds. data.

[0111] timestamps of visual frames This serves as the baseline (because visual data has a low frequency and is sparse). Whenever the visual layer outputs a frame, the fusion engine searches for the timestamp in the IMU buffer. and ,satisfy:

[0112] Calculate the precise physical state at that moment using linear interpolation:

[0113] in, The interpolated physical state value (such as acceleration) is represented by a unit that depends on the variable being interpolated. This represents the physical state value at the previous moment. This represents the physical state value at the next moment. Indicates the timestamp of the previous moment. A timestamp indicating the next moment. This indicates the trigger time of the visual layer (interpolation target time).

[0114] This mechanism ensures that each frame corresponds to a precisely calculated physical state value, achieving millisecond-level spatiotemporal alignment.

[0115] The processing results from the physical layer, dynamics layer, and vision layer are spatiotemporally aligned to generate aligned multimodal feature vectors. The physical layer output includes the collision trigger time. The dynamics layer outputs the vehicle trajectory status label, including normal driving, understeer, oversteer, or spin; the visual layer outputs the visual trigger moment. The system also identifies the type of collision object, such as pedestrians, vehicles, or obstacles. These aligned feature vectors serve as input to the fusion layer for subsequent confidence scoring and accident level determination.

[0116] To adapt to different operating conditions, the fusion logic processed by the fusion layer is not a simple "majority rules," but rather adopts a weighted scoring model based on Bayesian probability. The scoring formula is as follows:

[0117] in, , , These are the normalized outputs (0 or 1) for physical layer processing, dynamics layer processing, and visual layer processing, respectively. , , The dynamic weighting coefficient varies with vehicle speed; In a preferred embodiment, the scene adaptive weight allocation strategy is as follows: (1) Low-speed scenarios ): The physical signal is weak (small impact force), while the visual signal is strong (clear target, no motion blur).

[0118] Strategy: Significantly increase visual weight.

[0119] .

[0120] explain:As long as the visual system confirms that the TTC is zero and there is a target, even if the acceleration is only 0.6G, it is considered a scrape.

[0121] (2) High-speed scenarios ): The physical signal is strong (highly destructive), the dynamic characteristics are obvious (out of control), and the vision may fail due to motion blur.

[0122] Strategy: Increase the weights of the dynamics and physics layers.

[0123] .

[0124] explain: If spin and high G values ​​are detected, an alarm must be triggered immediately, even if it is not clearly seen visually (e.g., a side impact).

[0125] Decision-making hierarchy: Score > 0.8 (Severe Incident): Immediately trigger eCall, lock the highest level EDR data, and upload it to the cloud.

[0126] 0.5 < Score < 0.8 (Suspected / Minor Accident): Data is locked locally, a pop-up window appears on the vehicle's infotainment system asking the user "Has an accident occurred?", and no automatic alarm is triggered.

[0127] Score < 0.5 (Interference event): If identified as a speed bump or pothole, cached data will be automatically discarded and memory will be released.

[0128] Finally, the precise collision trigger time after fusion is generated. (by physical layer) and visual layer The accident assessment report (determined through spatiotemporal alignment and confidence-weighted calculation) includes accident scenario classification information. This classification information includes at least one of the following: vehicle trajectory status label, collision object type, and accident level. This final accident assessment report can be used for subsequent accident investigation, insurance claims, and other applications.

[0129] Example 2

[0130] This embodiment provides a system for accurate identification of accident moments in new energy vehicles based on multimodal data fusion. This system is used to implement the method described in Embodiment 1. Figure 2 As shown, the system specifically includes: Data acquisition module 1 is used to acquire multi-source heterogeneous data of the vehicle, including: Motion sensor data (triaxial acceleration, angular velocity) from the inertial measurement unit; Vehicle speed signal from wheel speed sensors; Steering wheel angle signal from the steering wheel angle sensor; Visual image data from the vehicle's onboard camera.

[0131] Physical layer processing module 2, connected to the data acquisition module 1, is used to perform physical layer processing on the motion sensor data. This module specifically includes: The Kalman filter unit filters the acceleration signal and wheel speed signal, and outputs a smooth acceleration estimate. The jerk calculation unit calculates the jerk based on the acceleration estimate. The dynamic threshold generation unit calculates a dynamic threshold related to the vehicle speed based on the current vehicle speed using the Sigmoid function; The physical layer determination unit determines a suspected collision when the magnitude of the acceleration estimate exceeds the dynamic threshold and the magnitude of the jerk exceeds the preset impact threshold, and outputs the physical layer trigger time and the corresponding acceleration magnitude.

[0132] The dynamics layer processing module 3, connected to the data acquisition module 1, is used to perform dynamics layer processing on the dynamic parameters. This module specifically includes: The two-degree-of-freedom model calculation unit calculates the theoretical yaw rate based on vehicle speed and steering wheel angle. The deviation calculation unit calculates the magnitude of the deviation between the theoretical yaw rate and the actual yaw rate. The sliding window integration unit performs sliding window integration on the deviation amplitude to obtain the cumulative error energy; The finite state machine unit determines the trajectory state of the vehicle based on the accumulated error energy and duration, and outputs the vehicle trajectory state label.

[0133] The visual layer processing module 4, connected to the data acquisition module 1, is used to perform visual layer processing on the visual image data. This module specifically includes: The target detection and tracking unit uses a lightweight YOLOv11 neural network that has been pruned and quantized to perform target detection and tracking on the image, and obtains the bounding box and identification of the target. The TTC calculation unit calculates the visual collision time TTC based on the bounding box width change rate and smooths the width and its change rate using Kalman filtering. The visual determination unit determines the existence of visual evidence of collision when the TTC is less than the preset threshold and the target occupies more than the preset proportion of the image, and outputs the visual layer trigger time and the type of collision object. The optical flow analysis unit performs optical flow analysis on the image, detects abnormal features such as instantaneous shifts in the entire image, sudden increases in optical flow direction entropy, or signal loss, and outputs visual impact indicators as supplementary evidence.

[0134] The spatiotemporal alignment module 5 is connected to the physical layer processing module 2, the dynamics layer processing module 3, and the vision layer processing module 4, respectively, and is used to perform temporal alignment on the data output from the three layers. This module specifically includes: The circular buffer unit stores the output data of each layer into its respective circular buffer. The linear interpolation unit performs linear interpolation on the physical layer and dynamics layer data based on the visual layer trigger time to obtain the aligned multimodal determination results, including the physical layer trigger time and acceleration amplitude, the dynamics layer trajectory status label, and the visual layer trigger time and collision object type.

[0135] The fusion decision module 6, connected to the spatiotemporal alignment module 5, is used to synthesize the outputs of each layer according to preset rules. This module specifically includes: The weight generation unit dynamically adjusts the weights of the physical layer, dynamics layer, and visual layer based on the current vehicle speed. The confidence scoring unit calculates a weighted confidence score based on the aligned multimodal judgment results and the weights. The accident level determination unit determines the accident level (serious accident, suspected accident, interference event) based on the confidence score. The output unit generates the fused precise collision trigger time and accident scene classification information, and triggers corresponding response actions according to the accident level.

[0136] The above modules are implemented in software, run on the vehicle computing platform, acquire sensor data through the vehicle bus, and output accident judgment results to the vehicle display terminal, remote information processing platform or data storage unit.

[0137] This invention constructs a four-in-one parallel processing architecture consisting of a physical layer, a dynamics layer, a visual layer, and a fusion layer, achieving high-confidence and accurate identification of accident moments and comprehensive reconstruction of accident scenes. Compared with existing technologies, it has the following advantages: (1) Strong anti-interference capability, significantly reducing false alarm rate. The physical layer smooths the signal through Kalman filtering, introduces acceleration analysis to distinguish between rigid collisions and flexible impacts, and combines a Sigmoid dynamic threshold model to eliminate logical jumps at the velocity critical point; the visual layer verifies the existence of the collision object through YOLOv11 target detection and optical flow analysis. The multi-level verification mechanism effectively filters out false alarms caused by pure physical vibrations such as speed bumps and road potholes, solves the problem of high false alarm rate of traditional single acceleration threshold method under complex road conditions, and avoids invalid alarms occupying storage space and background resources.

[0138] (2) High scene restoration degree and clear accident liability determination. Based on the two-degree-of-freedom model and the finite state machine, the dynamics layer can accurately identify precursors of out-of-control situations such as understeer, oversteer, and spin, and can distinguish whether it is the vehicle's own out-of-control (abnormal trajectory first) or passive impact (sudden G-value change first and then abnormal trajectory); the vision layer identifies the type of collision object (person / vehicle / object) through object detection, providing key semantic information for accident liability determination. The fusion layer synthesizes information from each layer to output the vehicle trajectory state, the type of collision object, and the accident level, achieving a comprehensive restoration of the accident scene.

[0139] (3) Covers low-speed scenarios and eliminates blind spots of missed reports. Aiming at the problem that traditional methods cannot capture low-speed scratches (G-value is only 0.6G - 0.8G), the physical layer sets a lower limit for low-speed sensitivity , and the vision layer assigns a higher visual weight in low-speed scenarios through TTC estimation based on the bounding box expansion rate ( ). Even if the acceleration amplitude does not reach the traditional threshold, as long as the vision confirms the collision risk, the system can determine an accident, significantly improving the detection rate of minor accidents in parking lots and congested road conditions.

[0140] (4) High recognition accuracy and output of accident determination reports with high confidence. The fusion layer achieves millisecond-level spatio-temporal alignment of the IMU (50Hz) and the camera (30Hz) through a circular buffer and linear interpolation, ensuring the synchronization of heterogeneous data; the scene adaptive weighted confidence scoring model dynamically adjusts the weights of each layer according to the vehicle speed, emphasizing visual evidence at low speeds and physical and dynamics evidence at high speeds. The finally output accurate collision trigger moment T0 and accident scene classification information have the highest confidence, providing reliable data support for traffic accident identification, insurance claims, and emergency rescue.

[0141] (5) The hierarchical decision-making mechanism balances real-time response and resource optimization. Based on the confidence score, accidents are classified into serious accidents (Score > 0.8), suspected accidents (0.5 < Score ≤ 0.8), and interference events (Score ≤ 0.5), and different response actions are triggered: serious accidents immediately trigger eCall, lock EDR data, and upload it to the cloud; suspected accidents locally lock data and pop up a window on the vehicle console to ask the user; interference events automatically discard cached data. This mechanism not only ensures that real accidents are promptly responded to but also avoids the backlog of invalid alarms, optimizing the utilization of system resources.

[0142] In summary, through the deep fusion of multi-modal data, the present invention has achieved a technological leap from "single-threshold triggering" to "intelligent perception and verification", and has made significant progress in aspects such as anti-interference ability, scene restoration degree, low-speed detection rate, recognition accuracy, and system resource optimization, having important engineering application value and broad market prospects.

[0143] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A method for accurate identification of real-time accidents in new energy vehicles based on multimodal data fusion, characterized in that, include: Real-time acquisition of vehicle motion sensor data, dynamic parameters, and visual image data; Perform the following processing respectively: Physical layer processing: The motion sensor data is processed to obtain motion features. Based on the comparison between the motion features and preset trigger conditions, it is determined whether a suspected collision has occurred. When a suspected collision is determined, the physical layer collision trigger time and the corresponding acceleration amplitude are output. Dynamics layer processing: Based on the vehicle dynamics model, the theoretical motion state of the vehicle is calculated according to the dynamic parameters, the actual motion state is obtained, and the vehicle trajectory state label is output based on the deviation between the theoretical motion state and the actual motion state. Visual layer processing: The visual image data is processed to detect targets in the image and analyze the motion state of the targets. Based on the comparison between the motion state of the targets and preset visual conditions, it is determined whether there is visual evidence of collision. When it is determined that there is visual evidence of collision, the visual layer collision trigger time and collision object type are output. The outputs obtained from the physical layer processing, dynamics layer processing, and visual layer processing are subjected to fusion layer processing, including: spatiotemporal alignment of the heterogeneous data output from each layer, and synthesis of the outputs from each layer according to preset rules, outputting the fused precise collision trigger time and accident scene classification information.

2. The method according to claim 1, characterized in that, The motion sensor data includes inertial measurement unit data and wheel speed sensor data; The dynamic parameters include steering wheel angle and actual yaw rate; The visual image data refers to image data collected in real time by the vehicle-mounted camera.

3. The method according to claim 1, characterized in that, In the physical layer processing, the motion sensor data is processed to obtain motion features, including: The motion sensor data is filtered to obtain an acceleration estimate. Calculate the jerk based on the acceleration estimate; The motion characteristics include the acceleration estimate and the jerk; The filtering process for the motion sensor data includes: A linear Kalman filter is constructed, with the vehicle's current position, speed, and acceleration as the state vector. State prediction is performed based on a uniform acceleration model, with the speed observed by the wheel speed sensor and the acceleration observed by the inertial measurement unit as the observation values, and a smooth acceleration estimate is output.

4. The method according to claim 3, characterized in that, In the physical layer processing, determining whether a suspected collision has occurred based on a comparison of the motion characteristics with preset triggering conditions includes: The dynamic threshold of acceleration is determined based on the current vehicle speed, and the dynamic threshold of acceleration changes continuously with the vehicle speed. When the magnitude of the acceleration estimate exceeds the acceleration dynamic threshold and the magnitude of the jerk exceeds the preset impact threshold, a suspected collision is determined to have occurred.

5. The method according to claim 4, characterized in that, The acceleration dynamic threshold is calculated based on the current vehicle speed using the Sigmoid function. The calculation formula is as follows: in, For acceleration threshold, This is the lower limit of low-speed sensitivity. The upper limit of high-speed robustness, Current vehicle speed For the center conversion speed, This is the curve steepness coefficient; The conditions for determining a suspected collision are expressed as follows: in, This indicates the judgment result of the physical layer processing output. This is the smoothed acceleration estimate. To accelerate, This is the preset jerk threshold.

6. The method according to claim 1, characterized in that, In the dynamics layer processing, the vehicle dynamics model is a two-degree-of-freedom vehicle model; the theoretical motion state includes the theoretical yaw rate, and the actual motion state includes the actual yaw rate. Based on the deviation between the theoretical motion state and the actual motion state, a vehicle trajectory status label is output, including: Calculate the magnitude of the deviation between the theoretical yaw rate and the actual yaw rate; The cumulative error energy is obtained by performing a sliding window integration on the magnitude of the deviation. Based on a finite state machine, the trajectory state of the vehicle is determined according to whether the accumulated error energy exceeds a preset energy threshold and the duration, wherein the trajectory state includes at least one of normal driving, understeering, oversteering and spin.

7. The method according to claim 1, characterized in that, In the visual layer processing, detecting targets in the image and analyzing the motion state of the targets includes: Using a lightweight YOLOv11 neural network that has been pruned and quantized, target detection and tracking are performed on visual image data acquired in real time by vehicle cameras to obtain the bounding box and identification of the target. The target includes at least one of pedestrians, vehicles, cyclists, and obstacles.

8. The method according to claim 7, characterized in that, In the visual layer processing, determining whether there is visual evidence of a collision based on a comparison between the target's motion state and preset visual conditions includes: The width of the target bounding box and its rate of change are smoothed using Kalman filtering; The visual collision time is calculated based on the bounding box width change rate, and the formula is as follows: in, For visual collision time, The width of the target bounding box. The rate of change of the target bounding box width over time; When the visual collision time is less than a preset time threshold and the ratio of the target bounding box width to the image width exceeds a preset ratio, visual evidence of a collision is determined to exist.

9. The method according to claim 7, characterized in that, The visual layer processing also includes: Optical flow analysis is performed on the image to detect at least one anomalous feature among full-image instantaneous shift, sudden increase in optical flow direction entropy, or continuous image signal loss; When the abnormal feature is detected, a visual impact indicator is output as supplementary visual evidence of the collision.

10. The method according to claim 1, characterized in that, The fusion layer processing includes: The output data obtained from physical layer processing, dynamics layer processing, and vision layer processing are stored in their respective circular buffers. Based on the timestamp of the data output by the visual layer, linear interpolation is used to align the data output by the physical layer and the dynamics layer to obtain the aligned multimodal determination result. The weights of the physical layer processing, dynamics layer processing, and visual layer processing are dynamically adjusted based on the current vehicle speed. A weighted confidence score is then calculated for the aligned multimodal decision result, using the following formula: in, , , These are the normalized outputs from the physical layer processing, dynamics layer processing, and visual layer processing, respectively. , , The dynamic weighting coefficient varies with vehicle speed; The accident level is determined based on the threshold range in which the confidence score falls; Based on the aligned multimodal determination results and the accident level, a fused accurate collision trigger time and accident scene classification information are generated; the accident scene classification information includes at least one of vehicle trajectory status label, collision object type and accident level. The incident levels include serious incidents, suspected incidents, and interference events; serious incidents trigger automatic alarms, data locking, and cloud upload operations; suspected incidents trigger local data locking and user interaction inquiry operations; and interference events trigger cached data discarding operations.

Images