Vehicle collision intelligent detection system and method based on edge-cloud cooperation and multi-agent evolution

Abstract

The application relates to a vehicle collision intelligent detection system and method based on edge-cloud cooperation and multi-agent evolution. A light-weight AI model is used in the vehicle-mounted edge layer to perform two-stage feature extraction and suspicious degree scoring on six-axis IMU data, a dynamic threshold decision maker based on historical false alarm rate, scene similarity and cloud feedback weight is combined to realize hierarchical triggering, and only medium and high risk events are collected and uploaded; a multi-agent cooperative architecture is used in the cloud analysis layer, a task scheduling agent distributes resources, a multi-modal analysis agent calls a pre-trained large model to perform cross-attention multi-modal fusion, and event classification, severity, collision type and structured scene description are output, a decision agent generates a disposal suggestion; an optimization agent collects user feedback to form real labels, and through error analysis, incremental training and knowledge distillation, the cloud and the vehicle-mounted model realize closed-loop evolution. The application greatly reduces the false alarm rate and data transmission cost, has the scene understanding and self-optimization capability, and can be widely deployed.

Description

Technical Field

[0001] This invention relates to the field of vehicle intelligent safety technology, specifically to a vehicle collision intelligent detection system and method based on edge-cloud collaboration and multi-agent evolution. Background Technology

[0002] Currently, vehicle collision detection technology primarily serves scenarios such as airbag deployment, accident recording, emergency rescue, and insurance claims. Existing solutions typically employ the following two technical approaches:

[0003] 1. Collision detection method based on a single physical sensor threshold

[0004] This method is widely deployed in dashcams, in-vehicle T-Boxes, and airbag controllers. The system monitors vehicle impact acceleration in real time using an accelerometer (Gsensor) or inertial measurement unit (IMU). When the detected instantaneous acceleration value exceeds a preset fixed threshold (such as 2.5g, 5g, etc.), a collision is determined to have occurred, triggering actions such as data saving, alarm, or airbag ignition.

[0005] 2. Collision warning / detection methods based on vision or radar

[0006] Some high-end vehicles and pre-installed ADAS systems use cameras, millimeter-wave radar, or lidar to perceive the surrounding environment and achieve forward collision warning (FCW) or automatic emergency braking (AEB) through target tracking and trajectory prediction. After an accident, some systems can also identify the collision by observing changes in the image.

[0007] While the aforementioned technologies have proven effective in their respective application scenarios, they still suffer from the following four core shortcomings in large-scale deployments:

[0008] (1) High false alarm rate and lack of scene differentiation ability;

[0009] Solutions based on single sensor thresholds cannot effectively distinguish between real collisions and normal driving disturbances. Sudden braking, speed bumps, uneven roads, and vehicles driving over potholes all generate instantaneous impact acceleration, easily triggering threshold alarms. Numerous false alarms not only disrupt the user experience but also result in a significant waste of resources for cloud storage, network transmission, and manual verification. According to actual measurements, the false alarm rate of such systems often exceeds 60%, leading users to gradually disable the alarm function, rendering the system virtually useless.

[0010] (2) The judgment dimension is too narrow and lacks contextual understanding;

[0011] Current technologies can only output a binary "yes / no" collision conclusion, failing to answer questions such as "what type of collision occurred," "how severe was it," and "were other objects involved?" The lack of multimodal information, including visual context and vehicle status, severely limits the application value of collision data in areas such as insurance damage assessment, accident liability determination, and emergency rescue prioritization. Even in solutions that incorporate cameras, only simple video recording is performed without deep fusion analysis with sensor data.

[0012] (3) The system structure is closed and cannot be continuously optimized;

[0013] Current vehicle collision detection systems mostly employ an "open-loop" architecture: sensor triggering → data saving → process completion. The system lacks the ability to verify the judgment results and cannot use actual feedback to correct the detection logic. Detection thresholds and algorithm parameters are fixed at the factory, failing to adapt to changes in different vehicle models, driving habits, and regional road conditions. As usage time increases, system performance declines rather than improves, and the level of intelligence stagnates.

[0014] (4) The data transmission strategy is crude and resource utilization is low;

[0015] Many existing solutions indiscriminately upload sensor data and video clips to the cloud, with over 90% of this data consisting of normal driving data without collisions. This not only consumes valuable 4G / 5G network bandwidth and increases user data costs, but also leads to the waste of cloud storage and computing resources on invalid data. Emergency events are delayed due to the excessive data volume, severely impacting real-time requirements in rescue and claims scenarios.

[0016] In summary, existing vehicle collision detection technologies have significant shortcomings in terms of high precision, scene understanding, self-evolution, and resource efficiency. There is an urgent need for a new collision detection solution that can integrate multimodal information, has collaborative decision-making capabilities, and can continuously self-optimize. Summary of the Invention

[0017] This invention designs a vehicle collision intelligent detection system and method based on edge-cloud collaboration and multi-agent evolution. The technical problems it solves are the high false alarm rate, lack of context understanding, inability of the system to evolve itself, and serious waste of data transmission resources in existing vehicle collision detection technologies.

[0018] To solve the aforementioned technical problems, the present invention adopts the following solution:

[0019] A vehicle collision intelligent detection system based on edge-cloud collaboration and multi-agent evolution includes: an in-vehicle edge layer deployed on the vehicle terminal for real-time acquisition of vehicle motion data, real-time screening and suspicion assessment using a lightweight AI model, generation of graded trigger signals based on dynamic thresholds, and targeted acquisition of multimodal data; and a cloud analysis layer communicating with the in-vehicle edge layer via a wireless network, comprising multiple collaborative intelligent agents, including at least a task scheduling agent, a multimodal analysis agent, a decision agent, and an optimization agent; wherein the task scheduling agent receives multimodal data uploaded by the in-vehicle edge layer and allocates computing resources; the multimodal analysis agent calls a pre-trained multimodal large model to perform deep fusion analysis on the multimodal data, outputting event classification, severity, collision type, and scene description; the decision agent combines a knowledge base to generate final decisions and handling suggestions; and the optimization agent collects feedback to generate real labels, compares model prediction results to identify system weaknesses, formulates incremental training strategies, and updates the lightweight AI model of the in-vehicle edge layer and the large model in the cloud.

[0020] Preferably, the vehicle edge layer includes: a sensor data acquisition module for real-time acquisition of six-axis IMU data and vehicle CAN bus status information; a lightweight AI screening model employing a two-stage inference architecture, wherein the first stage extracts multi-dimensional features and outputs a preliminary suspicion score through a traditional machine learning model, and the second stage uses a CNN-GRU hybrid network to perform secondary inference on samples within a preset ambiguity range and outputs a corrected score; a dynamic threshold decision-maker that calculates the trigger threshold in real-time based on historical false alarm rates, the similarity between waveforms and typical scene template libraries, and cloud feedback weights; a hierarchical triggering and multimodal acquisition unit that classifies events into low, medium, and high levels based on the comparison results of the suspicion score and the dynamic threshold, and records video and packages multimodal data only at medium and high risk levels; and a hierarchical transmission and bandwidth adaptive module that prioritizes data according to event levels and dynamically adjusts the upload strategy based on network status.

[0021] Preferably, the threshold calculation formula of the dynamic threshold decision-maker is:

[0022] T=T base +α⋅ΔR false +β⋅ΔS sim +γ⋅W cloud , among which, T base As the baseline threshold, ΔR false ΔS represents the change in historical false alarm rate. sim W represents the cosine similarity offset between the current waveform and the typical scene template library. cloud α, β, and γ are the cloud-based feedback weighting factors, and are configurable coefficients.

[0023] Preferably, the input of the multimodal large model includes: video modality tensor, sensor temporal tensor, and metadata tensor; the output includes: event classification probability distribution, severity continuous score, collision type one-hot encoding, structured text summary, and confidence score; the multimodal large model adopts a bidirectional cross-attention mechanism for multimodal fusion: sensor feature sequences and visual feature sequences are used as queries and keys / values ​​for cross-attention calculation, and the fused features are embedded and concatenated with metadata before being input into the Transformer decoder.

[0024] Preferably, the multiple agents coordinate with each other through an asynchronous communication mechanism based on message queues. The message format includes an event identifier, a source / target agent identifier, an action type, and payload data. When the multimodal analysis agent detects that the input data quality is insufficient, it sends a data retransmission request to the task scheduling agent. After verification, the task scheduling agent issues a retransmission instruction to the vehicle edge layer and reschedules the analysis task.

[0025] Preferably, the optimization agent performs closed-loop optimization steps including:

[0026] Collect user confirmation, correction, or rejection actions for cloud-pushed results to generate real labels; compare model predictions with real labels to generate a confusion matrix and identify high-frequency error scenarios; extract error samples, high-confidence correct samples, and historical representative samples from the feedback database to construct an incremental training dataset; update the cloud-based large model using either full fine-tuning or online learning strategies based on the incremental data scale; verify model performance on a fixed test set, and automatically roll back if core metrics decline; use the cloud-based large model as the teacher model and the in-vehicle lightweight AI model as the student model to transfer knowledge through soft labeling, feature map, or attention map distillation; and push the updated in-vehicle model to the in-vehicle terminal via OTA.

[0027] A vehicle collision intelligent detection method based on edge-cloud collaboration and multi-agent evolution is characterized by the following steps:

[0028] S1: The vehicle edge layer collects vehicle motion data in real time, extracts features and scores suspicion through a lightweight AI model, and determines the event level by combining dynamic thresholds. Only medium and high risk events are activated for camera recording and multimodal data packets are generated and uploaded to the cloud in a hierarchical manner.

[0029] S2: The cloud task scheduling agent receives multimodal data packets and assigns them to the multimodal analysis agent. The multimodal analysis agent calls a pre-trained multimodal large model to perform cross-attention fusion on video, sensor and metadata, and outputs event classification, severity, collision type and structured scene description.

[0030] S3: The decision agent combines the knowledge base to generate the final decision and disposal suggestions and pushes them to the user terminal, and receives user feedback to generate real tags;

[0031] S4: Optimize the agent to perform error analysis based on feedback data, build an incremental training dataset, update the cloud-based large model by full fine-tuning or online learning, and transfer the cloud model capabilities to the vehicle-mounted lightweight AI model through knowledge distillation, and then issue updates via OTA.

[0032] Preferably, the two-stage inference of the lightweight AI model in step S1 specifically involves:

[0033] Temporal, frequency, and statistical features are extracted from six-axis IMU data within a 10-second window and input into an XGBoost classifier to obtain a preliminary score. If the preliminary score falls within the ambiguous range of 0.3 to 0.7, the original time-series data is input into a CNN-GRU hybrid network to obtain a corrected score. The dynamic threshold is calculated using the formula T = T base +α⋅ΔR false +β⋅ΔS sim +γ⋅W cloud Adjustments can be made online; among them, T base As the baseline threshold, ΔR false ΔS represents the change in historical false alarm rate. sim W represents the cosine similarity offset between the current waveform and the typical scene template library. cloud The cloud-based feedback weighting factors are α, β, and γ, which are configurable coefficients. The event level classification rules are as follows: if the score is <0.6T, it is low risk and only local caching is used; if the score is 0.6T ≤ 1.2T, it is medium risk and 10 seconds of compressed video and sensor data before and after the collision are uploaded; if the score is ≥1.2T, it is high risk and the original video and full data for 30 seconds before and 60 seconds after the collision are uploaded.

[0034] Preferably, the input to the multimodal large model in step S2 is specifically:

[0035] The video modal tensor dimensions are [450, 3, 240, 320], corresponding to 90 seconds, 5fps, RGB, and a resolution of 240×320; the sensor modal tensor dimensions are [1000, 6], corresponding to 10 seconds and 100Hz six-axis data; the metadata includes vehicle state time series [90, 20][90, 20] and vehicle inherent features

[70] ; the cross-attention fusion specifically involves: the sensor feature sequence S and the visual feature sequence V are mutually used as Query and Key / Value for bidirectional attention calculation, and after fusion, they are spliced ​​together with the metadata embedded in M ​​and input into the decoder.

[0036] Preferably, in step S4, when constructing the incremental training dataset, erroneous samples are assigned a 3x sampling weight, and high-confidence correct samples are assigned a 1x sampling weight. A core set selection algorithm is used to extract a 20% representative subset from the historical data. The knowledge distillation specifically involves using the soft labels output by the large cloud model as teacher signals to train the vehicle-mounted lightweight AI model to minimize cross-entropy loss. For high-computing vehicle platforms with NPUs, feature map distillation or attention map distillation is further employed. The OTA delivery uses a hot-loading method during idle periods to update the vehicle-mounted model.

[0037] The intelligent vehicle collision detection system and method based on edge-cloud collaboration and multi-agent evolution has the following beneficial effects:

[0038] (1) This invention breaks through the limitations of traditional single-sensor threshold detection. Through a two-stage inference architecture of a lightweight AI model at the vehicle edge layer, it integrates time domain, frequency domain, and statistical features for multi-dimensional analysis. Combined with a dynamic threshold decision-maker, it adjusts the trigger threshold in real time based on historical false alarm rate, scene similarity, and cloud feedback weights. Compared with existing fixed threshold schemes, this invention can effectively distinguish between real collisions and normal driving interference such as sudden braking, bumpy roads, and speed bumps. The measured false alarm rate is reduced from over 60% to below 5%, significantly improving the user experience, avoiding users from turning off the alarm function due to frequent false alarms, and ensuring the long-term effective operation of the system.

[0039] (2) This invention, through a cloud-based multimodal large model and agent collaborative architecture, achieves, for the first time, deep fusion analysis of multimodal data in the field of vehicle collision detection. The system not only outputs a binary "yes / no" collision conclusion, but also accurately identifies collision types (rear-end collision, side collision, frontal collision, rollover), assesses severity (1-10 point continuous scoring), and generates structured scene descriptions (involving objects, relative positions, and relative speeds). This breakthrough enables collision data to have substantial application value in scenarios such as insurance damage assessment, accident liability determination, and emergency rescue priority ranking, laying the foundation for subsequent business process automation.

[0040] (3) This invention overturns the traditional "open-loop" system architecture and constructs a complete "detection-verification-learning-evolution" closed-loop mechanism by optimizing the Agent and user feedback loop. The system can automatically collect user confirmation, correction or rejection operations on cloud push results, generate real labels and continuously accumulate high-quality training data; identify high-frequency misjudgment scenarios through error analysis and construct incremental training datasets accordingly; update the cloud-based large model by combining full fine-tuning and online learning, and transfer cloud capabilities to the vehicle-mounted lightweight model through knowledge distillation. Actual tests show that the overall accuracy of the system increases by more than 30% after 6 months of use, and the performance continues to improve with the increase of usage time, completely solving the pain point of traditional solutions being "fixed at the factory and becoming more and more outdated with use".

[0041] (4) This invention adopts an edge-cloud two-level collaborative architecture. On the vehicle side, a lightweight AI model performs real-time intelligent filtering of Gsensor data, triggering video recording and multimodal data upload only for medium- and high-risk events, and implementing a graded transmission strategy based on the event level. Compared with the traditional solution that uploads all data indiscriminately, this invention reduces the amount of invalid data uploaded by more than 85%, significantly reducing 4G / 5G network bandwidth usage and user traffic costs, while also reducing the ineffective consumption of cloud storage and computing resources. The graded transmission and bandwidth adaptive mechanism ensure that high-risk events are prioritized, significantly improving the response efficiency of scenarios with high real-time requirements such as emergency rescue and insurance claims.

[0042] (5) This invention is based on a multi-Agent modular architecture design. Each agent achieves loosely coupled collaboration through a standardized message interface, possessing strong adaptability and scalability. The dynamic threshold decision-maker can adaptively adjust the trigger threshold according to different vehicle models, driving habits, and regional road conditions, without requiring factory-fixed parameters. The system architecture can quickly adapt to other driving event detection tasks such as fatigue driving recognition, dangerous lane change warning, and road anomaly monitoring, requiring only adjustments to the task definition of the multimodal analysis agent and corresponding fine-tuning of the dataset. This design significantly reduces system maintenance and upgrade costs and significantly extends the commercial lifecycle of the technical solution.

[0043] (6) The structured collision data output by this invention can directly generate standard reports that meet the requirements of insurance claims, legal evidence and maintenance, simplifying the accident handling process by more than 70%; in emergency rescue scenarios, the system automatically dispatches rescue resources according to the severity score, improving the response speed of serious injury accidents; the aggregated anonymized collision data can provide high-quality data support for urban road safety planning, high-risk road section identification and differentiated pricing by insurance companies, generating significant economic benefits and social value. Attached Figure Description

[0044] Figure 1This invention presents an overall architecture diagram of a vehicle collision intelligent detection system based on edge-cloud collaboration and multi-agent evolution.

[0045] Figure 2 Flowchart of the intelligent filtering and hierarchical triggering mechanism for vehicle-side G-sensor data in this invention.

[0046] Figure 3 This invention presents a schematic diagram of the cloud-based multi-agent collaborative large-scale model analysis architecture and its interaction logic.

[0047] Figure 4 Flowchart of the closed-loop optimization and model self-evolution mechanism of this invention. Detailed Implementation

[0048] The following is combined Figures 1 to 4 The present invention will be further described as follows:

[0049] Example 1: System Overall Architecture

[0050] like Figure 1 As shown, the intelligent vehicle collision detection system based on edge-cloud collaboration and multi-agent evolution of the present invention consists of two main parts: the vehicle edge layer and the cloud analysis layer, and realizes bidirectional data interaction through wireless network.

[0051] The in-vehicle edge layer is deployed in vehicle intelligent terminals (such as T-Box, dashcam, ADAS domain controller), including:

[0052] Sensor data acquisition module: Real-time acquisition of six-axis IMU data (accelerometer three-axis x, y, z + gyroscope three-axis x, y, z), sampling rate 100Hz, and simultaneous reading of vehicle CAN bus information (vehicle speed, GPS location, timestamp, steering angle, braking status, etc.).

[0053] Lightweight AI screening model: An embedded AI inference engine (such as TensorFlow Lite or ONNX Runtime) deployed at the edge performs real-time analysis of the Gsensor time series, outputs an event suspicion score, and dynamically adjusts the trigger threshold.

[0054] Multimodal data acquisition unit: Activated only when the suspicion of an event exceeds a preset threshold, it calls the vehicle's front / rear view camera to record video data (320×240 resolution, RGB format, 5fps) for 30 seconds before and 60 seconds after the trigger point, and packages it with sensor data and vehicle status metadata for the corresponding time period.

[0055] Tiered transmission and bandwidth adaptive module: Based on the event level, the transmission priority is marked. When the network is congested, the complete data of high-risk events is uploaded first, while key frames or compressed streams are transmitted for medium and low-risk events.

[0056] The cloud analytics layer is deployed on cloud servers or edge computing nodes and contains five core agents. Each agent communicates and collaborates asynchronously through message queues (such as RabbitMQ and Kafka).

[0057] Task scheduling agent: Receives data packets uploaded from the vehicle, parses information such as event ID, priority, and data size, and dynamically allocates computing resources to the corresponding analysis queue.

[0058] Multimodal Analysis Agent: Calls a fine-tuned multimodal large model to deeply fuse and understand sensor time series, video frame sequences and metadata, and outputs event classification, severity score, collision type and structured scene description.

[0059] Decision Agent: Combines the knowledge base (traffic regulations, historical cases, insurance claims rules) with the output of the analysis agent to generate the final decision and handling suggestions.

[0060] Optimize the Agent: Regularly compare the model's predictions with the real labels from user feedback to identify system weaknesses, develop incremental training strategies, and perform knowledge distillation to update the lightweight in-vehicle model.

[0061] Resource Management Agent: Monitors cloud computing power, storage and bandwidth usage, dynamically adjusts resource quotas, and ensures timely response to high-priority tasks.

[0062] Figure 1 The complete data flow of the present invention is also shown: continuous monitoring on the vehicle side → edge intelligent filtering → targeted data collection → hierarchical uploading → cloud agent collaborative analysis → result push and feedback collection → model evolution and parameter distribution, forming a closed-loop intelligent system.

[0063] Example 2: Intelligent Filtering and Hierarchical Triggering Mechanism for Vehicle-Side G-Sensor Data

[0064] like Figure 2 As shown, the core innovation of this invention on the vehicle side lies in "lightweight AI model-driven adaptive screening and hierarchical triggering".

[0065] 2.1 Feature Extraction and Suspicion Scoring In a specific embodiment of the present invention, the lightweight AI model built into the vehicle terminal adopts a two-stage inference architecture:

[0066] Phase 1: Extracting multi-dimensional statistical features from the raw six-axis sensor data (10-second window, 1000 sampling points), including:

[0067] Temporal characteristics: peak acceleration, impact duration, rise slope, pulse area;

[0068] Frequency domain characteristics: energy distribution and peak frequency in the main frequency band (0-20Hz) after FFT;

[0069] Statistical features: variance, skewness, kurtosis, and zero-crossing rate. These features are input into a pre-trained XGBoost classifier, which outputs a suspicion score (a continuous value of 0-1). This score indicates the probability that the current event is a real collision.

[0070] The second stage: For "fuzzy samples" with XGBoost output scores between 0.3 and 0.7 (especially in complex road conditions such as gravel roads and continuous bumps), the system automatically triggers a small deep learning model for secondary judgment. This model adopts a CNN+GRU hybrid architecture: the CNN layer extracts local temporal features, the GRU layer captures long-range dependencies, and finally, a fully connected layer outputs a corrected suspicion score. This small model has fewer than 1MB of parameters and can complete inference in milliseconds.

[0071] 2.2 Dynamic Threshold Adjustment

[0072] This invention abandons the fixed threshold and introduces a dynamic threshold decision-maker. The decision-maker calculates the current trigger threshold T in real time based on the following three aspects:

[0073] Historical false alarm rate: The percentage of false alarm events verified by the cloud within the past 7 days for this vehicle is statistically analyzed. The threshold is automatically raised when the false alarm rate is high.

[0074] Typical scenario database: The vehicle terminal pre-stores typical collision waveform templates and common interference waveform features. By calculating the cosine similarity between the current waveform and the template library, the threshold offset is dynamically adjusted.

[0075] Cloud-based feedback weights: The judgment results of past uploaded events by the cloud-based large model (such as "suspected false alarm" and "confirmed collision") are transmitted as weight factors, which affect the subsequent threshold settings.

[0076] The final trigger threshold T=T base +α⋅ΔR false +β⋅ΔS sim +γ⋅W cloud , among which, T base As the baseline threshold, ΔR false ΔS represents the change in historical false alarm rate. sim W represents the cosine similarity offset between the current waveform and the typical scene template library. cloudThe cloud feedback weighting factors are α, β, and γ, which are configurable coefficients with values ​​ranging from [0, 1]. Typical values ​​can be set to α=0.4, β=0.3, and γ=0.3, corresponding to the balanced weights of historical false alarm rate, scene similarity, and cloud feedback, respectively. In actual deployment, α, β, and γ can be automatically tuned based on the validation set through grid search or Bayesian optimization, or dynamically distributed by the cloud according to the overall false alarm statistics of the fleet to achieve adaptive threshold calibration.

[0077] 2.3 Tiered Triggering Strategy

[0078] Based on the suspicion level score and dynamic threshold, the system classifies events into three levels:

[0079] Low risk (score < 0.6×T): Only record sensor data to a local circular buffer, do not upload;

[0080] Medium risk (0.6×T ≤ ​​score < 1.2×T): Activate camera recording, upload key frames (10 seconds before and after the collision, a total of 20 seconds of compressed video) and complete sensor data;

[0081] High risk (score ≥ 1.2×T): Record a complete 90-second video, upload the original high-definition video (H.265 compression optional) and the full amount of multimodal data.

[0082] This tiered strategy reduces the amount of data that needs to be uploaded by more than 85%, significantly reducing bandwidth and storage costs.

[0083] Example 3: Cloud-based Multi-Agent Collaborative Large-Scale Model Analysis Architecture

[0084] like Figure 3 As shown, the cloud-based multi-agent collaborative large model analysis architecture is another core invention of this invention, which solves the problem that traditional solutions lack contextual understanding and collaborative decision-making capabilities.

[0085] 3.1 Input and output formats of multimodal large models

[0086] In a preferred embodiment of the invention, the multimodal large model uses QwenOmni as its base and performs efficient parameter fine-tuning (PEFT) specifically for vehicle collision detection scenarios. The model input consists of three parts:

[0087] Video modality: Keyframe sequences from front and rear cameras. Each event segment includes 30 seconds before and 60 seconds after the collision, for a total of 90 seconds. The sampling rate is 5fps, with a total of 450 frames. Each frame is 240×320 pixels and has three RGB channels. The tensor dimensions are [450, 3, 240, 320].

[0088] Sensor modalities: six-axis IMU data (accelerometer x, y, z + gyroscope x, y, z), sampling rate 100Hz, window length 10 seconds (covering key time periods before and after the collision), forming a temporal tensor of [1000, 6].

[0089] Metadata: Vehicle status information (speed, GPS location, timestamp, road type code, etc.), a total of 90 time points, each time point has 20 features, dimensions [90, 20]; Vehicle inherent features (wheelbase, curb weight, sensor installation location, etc.), dimensions

[70] .

[0090] The model output is a joint output of multiple tasks:

[0091] Event classification probability distribution: 4-dimensional vector [normal driving, minor interference, minor collision, severe collision];

[0092] Severity score: a continuous value from 1 to 10, generated by the regression head;

[0093] Collision type recognition: one-hot encoding [rear-end collision, side collision, frontal collision, rollover];

[0094] Key scenario description: Structured text summary, including the objects involved (cars, pedestrians, two-wheeled vehicles, etc.), relative positions (left front, front, etc.), and relative speeds (low speed, medium speed, high speed);

[0095] Confidence score: a scalar of 0-1, representing the overall reliability of the judgment.

[0096] 3.2 Multimodal fusion mechanism

[0097] This invention employs a cross-attention mechanism to achieve deep semantic alignment of multimodal features. The specific implementation is as follows:

[0098] Sensor data is processed through a 1D convolutional layer and a Transformer encoder to extract temporal features, resulting in a feature sequence S∈R. 100×256 ;

[0099] Video frames are processed through 2D convolution (such as EfficientNet) and temporal positional encoding to obtain a visual feature sequence V∈R. 45 ×512 ;

[0100] Metadata is embedded into a feature vector M∈R using MLP. 256 .

[0101] A bidirectional cross-attention approach is employed: using S as the query and V as the key / value, sensor-enhanced visual features are obtained; simultaneously, using V as the query and S as the key / value, visually enhanced sensor features are obtained. The fused features are concatenated with M and fed into the Transformer decoder to generate the final output.

[0102] 3.3 Inter-Agent Collaboration Mechanism

[0103] This invention implements multi-agent collaboration based on the MCP interface design and Langgraph computation graph. Each agent communicates through a standardized message structure. The message format is JSON, and the required fields are: event_id, agent_source, agent_target, action_type, payload, and timestamp.

[0104] Typical collaborative scenario example:

[0105] When the multimodal analysis agent processes video data, if it detects that keyframes are blurred due to overexposure or occlusion, making it impossible to accurately determine the collision object, it sends a `request_extend_data` request to the task scheduling agent. The parameters include `event_id`, `required_duration` (e.g., requesting an extension to 120 seconds after the collision), and `reason` ("insufficient visual evidence"). After verifying resource availability, the task scheduling agent sends a command to the vehicle terminal to request the transmission of additional video keyframes for the extended period. This process demonstrates the system's intelligence in proactively perceiving missing information and collaboratively completing it.

[0106] Example 4: Closed-loop optimization and model self-evolution mechanism

[0107] like Figure 4 As shown, this invention constructs a continuously self-evolving intelligent system by introducing an optimized Agent and a user feedback loop. Figure 4 The nine core steps of closed-loop optimization and model self-evolution are illustrated in detail below, and are explained one by one with reference to the attached figures:

[0108] Step 401: Push the collision analysis results to the cloud;

[0109] After the decision agent generates the final judgment result (event classification, severity, collision type, and handling recommendations), it pushes it to users (vehicle owners, fleet managers) and relevant organizations (insurance companies, rescue centers) via mobile APP, SMS, WeChat mini program or third-party platform API.

[0110] Step 402: On-site confirmation and feedback from the user;

[0111] Users can view the analysis results and on-site video thumbnails on the terminal and choose "Confirm," "Correct," or "Reject":

[0112] Confirmation: Indicates agreement with the cloud-based judgment result;

[0113] Correction: Users can manually correct the event type (e.g., change "minor collision" to "speed bump impact") or adjust the severity rating;

[0114] Rejection: The system is deemed to have given a false alarm, and the result is rejected.

[0115] Feedback actions are accompanied by timestamps and user identification, forming the original feedback record.

[0116] Step 403: Feedback data aggregation and real label generation;

[0117] The cloud-based feedback collection module cleans, deduplicates, and anonymizes the original records. It then associates the event IDs with historical data to generate high-quality samples with ground truth labels, which are stored in the feedback database. Each sample contains complete input multimodal data and its corresponding ground truth vector.

[0118] Step 404: Optimize the Agent to periodically trigger error analysis;

[0119] Optimize the agent to scan for new feedback samples at fixed intervals (e.g., daily / weekly) and perform the following analysis:

[0120] Calculate the confusion matrix between the model's predicted labels and the actual labels;

[0121] Identify high-frequency error scenarios (such as specific vehicle models, specific road surfaces, specific lighting conditions, and specific speed ranges);

[0122] Statistically analyze the percentage and confidence level distribution of each error type to generate a system vulnerability report.

[0123] Step 405: Construct the incremental training dataset;

[0124] Based on the error analysis results, the optimized agent extracts data from the feedback database.

[0125] Error sample set: All samples that were corrected or rejected by the user (weighted 3 times);

[0126] High-confidence correct sample set: Samples confirmed by the user and with a model output confidence score > 0.9 (weighted by 1).

[0127] Historical representative sample set: A core set selection algorithm is used to extract a subset (accounting for 20% of the total training set) covering different scenarios from the full historical data to prevent catastrophic forgetting.

[0128] The three types of samples are mixed proportionally to form an incremental training dataset.

[0129] Step 406: Select and execute the training strategy;

[0130] Optimize the agent's assessment of incremental dataset size and current system load:

[0131] If the number of new erroneous samples exceeds 1000, initiate full fine-tuning and retrain the large cloud model using all historical data and incremental data;

[0132] If the number of erroneous samples is less than 1000, online learning is used to update some parameters of the model in a mini-batch manner (only the LoRA adaptation layer or the last few Transformer layers are updated).

[0133] If sufficient samples of new scenarios / new vehicle models are detected, domain-adaptive fine-tuning will be triggered.

[0134] Step 407: Model performance verification;

[0135] The updated model is comprehensively evaluated on a fixed test set (a diverse set of scenarios updated quarterly, including real accidents, synthetic accidents, and normal driving interference), comparing accuracy, recall, and F1 score before and after the update. If any core metric drops by more than 1 percentage point, the model is automatically rolled back to the previous version and an alert is triggered.

[0136] Step 408: Knowledge distillation and edge model update;

[0137] To migrate the capabilities of large cloud-based models to lightweight in-vehicle models, optimize agent-based knowledge distillation:

[0138] Teacher Model: A finely tuned cloud-based large model;

[0139] Student model: In-vehicle lightweight AI model (XGBoost+CNN-GRU);

[0140] Distillation Method: The teacher model generates soft labels (probability distributions for each category) on the distillation dataset. The student model uses these soft labels as supervision and is trained using the cross-entropy loss function. During distillation, all parameters of the teacher model are frozen, and only the weights of the student model are updated to ensure that the generalization ability of the large cloud model is not forgotten, while avoiding the influence of noisy feedback on the teacher model. For high-computing-power in-vehicle platforms with NPUs, further feature map distillation (to make the student's intermediate layer features approximate those of the teacher) and attention map distillation (to align the student's attention distribution with that of the teacher) are employed. After distillation, the accuracy of the student model is improved, while the number of parameters remains unchanged.

[0141] Step 409: OTA Push and Vehicle Evolution;

[0142] The optimized agent packages the updated lightweight vehicle model and pushes it to the vehicle terminal via OTA (Over-The-Air) download technology. The terminal downloads the new model during idle periods, verifies its integrity, and then hot-loads it, achieving synchronous evolution of models across the entire fleet. To ensure upgrade security, the OTA package must be digitally signed. The terminal rigorously verifies the signature validity and model hash value before loading to prevent firmware tampering or unauthorized injection. Simultaneously, the system retains the model files from the previous two versions. If the new model triggers three consecutive false alarms or the user feedback rejection rate exceeds a threshold (e.g., >10%), a version rollback is automatically triggered, restoring to the previous stable version. Evolution logs are transmitted back to the cloud, completing the closed loop.

[0143] Through the above nine-step closed-loop mechanism, after six months of use, the overall accuracy of the system improved by more than 30%, the false alarm rate dropped to below 5%, and the ability to continuously optimize was significantly enhanced.

[0144] Example 5: Alternatives and Extended Applications

[0145] The architecture and method described in this invention can also be implemented in other alternative ways, while still achieving the objectives of this invention:

[0146] 5.1 Pure edge computing scheme;

[0147] The enhanced AI model (such as MobileNetV4+Transformer) is fully deployed on a high-performance in-vehicle computing platform (such as NVIDIA Orin or Horizon Robotics Journey 5), with all analysis completed locally, uploading only key metadata and compressed videos of extremely high-risk events. This solution reduces cloud dependency and communication costs, but is limited by in-vehicle computing power and makes it difficult to achieve continuous global model learning (requiring aggregation and updates from roadside or central cloud).

[0148] 5.2 Multi-level physical threshold scheme;

[0149] Without introducing an AI model, a cascaded decision logic based on multi-dimensional physical quantities such as G-sensor peak value, pulse width, pulse area, and frequency domain energy is designed to achieve low-complexity collision detection. This solution is suitable for low-cost devices with severely limited computing power (such as aftermarket simple dashcams), but it lacks adaptive capabilities and scene understanding depth, resulting in a higher false alarm rate than the solution in this patent.

[0150] 5.3 Expanding application scenarios;

[0151] The system architecture of this invention has strong modularity and scalability, and can be quickly adapted to other driving event detection tasks:

[0152] Fatigue driving recognition: Add an in-vehicle driver camera, and use a multimodal analysis agent to fuse facial features, steering wheel operation, and lane departure signals to output the fatigue level;

[0153] Dangerous lane change warning: Integrating lateral millimeter-wave radar and surround-view cameras, the decision agent generates a lane change risk score;

[0154] Road surface anomaly monitoring: Using suspension acceleration sensors and GPS, road surface smoothness is analyzed to generate a road surface defect map.

[0155] Functionality can be expanded simply by adjusting the task definition of the multimodal analysis agent and fine-tuning the dataset and knowledge base rules, without having to refactor the underlying architecture.

[0156] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the implementation of the present invention is not limited to the above-described manner. Any improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without modification, are all within the protection scope of the present invention.

Claims

1. A vehicle collision intelligent detection system based on edge-cloud collaboration and multi-agent evolution, characterized in that, include: The vehicle edge layer, deployed on the vehicle terminal, is used to collect vehicle motion data in real time, perform real-time screening and suspicion assessment through a lightweight AI model, generate graded trigger signals based on dynamic thresholds, and collect multimodal data in a targeted manner. The cloud-based analytics layer communicates with the vehicle edge layer via a wireless network and includes multiple collaborative intelligent agents, which at least include a task scheduling agent, a multimodal analysis agent, a decision-making agent, and an optimization agent. The task scheduling agent receives multimodal data uploaded from the vehicle edge layer and allocates computing resources; the multimodal analysis agent calls a pre-trained multimodal large model to perform deep fusion analysis on the multimodal data, outputting event classification, severity, collision type, and scene description; the decision agent combines a knowledge base to generate final decisions and handling suggestions; and the optimization agent collects feedback to generate real labels, compares model prediction results to identify system weaknesses, formulates incremental training strategies, and updates the lightweight AI model of the vehicle edge layer and the large model in the cloud.

2. The vehicle collision intelligent detection system based on edge-cloud collaboration and multi-agent evolution as described in claim 1, characterized in that: The vehicle-mounted edge layer includes: The sensor data acquisition module is used to acquire six-axis IMU data and vehicle CAN bus status information in real time; The lightweight AI screening model adopts a two-stage inference architecture. In the first stage, a traditional machine learning model is used to extract multi-dimensional features and output an initial suspicion score. In the second stage, a CNN-GRU hybrid network is used to perform secondary inference on samples in the preset ambiguity range and output a corrected score. The dynamic threshold decision-maker calculates the trigger threshold in real time based on the historical false alarm rate, the similarity between the waveform and the typical scenario template library, and the cloud feedback weight. The graded triggering and multimodal acquisition unit divides events into three levels—low, medium, and high—based on the comparison results of suspicion scores and dynamic thresholds, and records videos and packages multimodal data only at medium and high risk levels. The hierarchical transmission and bandwidth adaptive module prioritizes data based on event levels and dynamically adjusts the upload strategy according to network conditions.

3. The vehicle collision intelligent detection system based on edge-cloud collaboration and multi-agent evolution as described in claim 2, characterized in that: The threshold calculation formula for the dynamic threshold decision-maker is as follows: T=T base +α⋅ΔR false +β⋅ΔS sim +γ⋅W cloud , Among them, T base As the baseline threshold, ΔR false ΔS represents the change in historical false alarm rate. sim W represents the cosine similarity offset between the current waveform and the typical scene template library. cloud α, β, and γ are the cloud-based feedback weighting factors, and are configurable coefficients.

4. The vehicle collision intelligent detection system based on edge-cloud collaboration and multi-agent evolution as described in claim 1, characterized in that: The inputs of the multimodal large model include: video modality tensor, sensor temporal tensor, and metadata tensor; the outputs include: event classification probability distribution, severity continuous score, collision type one-hot encoding, structured text summary, and confidence score. The multimodal large model employs a bidirectional cross-attention mechanism for multimodal fusion: the sensor feature sequence and the visual feature sequence are used as queries and key / value pairs for cross-attention calculation, and the fused features are embedded and concatenated with metadata before being input into the Transformer decoder.

5. The vehicle collision intelligent detection system based on edge-cloud collaboration and multi-agent evolution as described in claim 1, characterized in that: The multiple agents coordinate with each other through an asynchronous communication mechanism based on message queues. The message format includes event identifier, source / target agent identifier, action type, and payload data. When the multimodal analysis agent detects that the input data quality is insufficient, it sends a data retransmission request to the task scheduling agent. After verification, the task scheduling agent sends a retransmission instruction to the vehicle edge layer and reschedules the analysis task.

6. The intelligent vehicle collision detection system based on edge-cloud collaboration and multi-agent evolution as described in claim 1, characterized in that: The optimized Agent performs closed-loop optimization steps, including: Collect user confirmation, correction, or rejection actions regarding cloud-pushed results to generate real-world tags; By comparing model predictions with real labels, a confusion matrix is ​​generated and high-frequency error scenarios are identified. An incremental training dataset is constructed by extracting error samples, high-confidence correct samples, and historical representative samples from the feedback database. Based on the scale of incremental data, choose either full fine-tuning or online learning strategies to update the large cloud model; Validate model performance on a fixed test set, and automatically roll back if core metrics decline. Using a large cloud-based model as the teacher model and a lightweight in-vehicle AI model as the student model, knowledge is transferred through soft labeling, feature map, or attention map distillation. The updated vehicle model will be pushed to the vehicle terminal via OTA.

7. A vehicle collision intelligent detection method based on edge-cloud collaboration and multi-agent evolution, characterized in that, Includes the following steps: S1: The vehicle edge layer collects vehicle motion data in real time, extracts features and scores suspicion through a lightweight AI model, and determines the event level by combining dynamic thresholds. Only medium and high risk events are activated for camera recording and multimodal data packets are generated and uploaded to the cloud in a hierarchical manner. S2: The cloud task scheduling agent receives multimodal data packets and assigns them to the multimodal analysis agent. The multimodal analysis agent calls a pre-trained multimodal large model to perform cross-attention fusion on video, sensor and metadata, and outputs event classification, severity, collision type and structured scene description. S3: The decision agent combines the knowledge base to generate the final decision and disposal suggestions and pushes them to the user terminal, and receives user feedback to generate real tags; S4: Optimize the agent to perform error analysis based on feedback data, build an incremental training dataset, update the cloud-based large model by full fine-tuning or online learning, and transfer the cloud model capabilities to the vehicle-mounted lightweight AI model through knowledge distillation, and then issue updates via OTA.

8. The intelligent vehicle collision detection method based on edge-cloud collaboration and multi-agent evolution according to claim 7, characterized in that: The two-stage inference of the lightweight AI model in step S1 is specifically as follows: Time-domain, frequency-domain, and statistical features were extracted from the six-axis IMU data in a 10-second window and input into the XGBoost classifier to obtain a preliminary score. If the initial score is in the ambiguous range of 0.3 to 0.7, the original time series data is input into the CNN-GRU hybrid network to obtain the corrected score; The dynamic threshold is based on the formula T=T base +α⋅ΔR false +β⋅ΔS sim +γ⋅W cloud Adjustments can be made online; among them, T base As the baseline threshold, ΔR false ΔS represents the change in historical false alarm rate. sim W represents the cosine similarity offset between the current waveform and the typical scene template library. cloud For cloud-based feedback weighting factors, α, β, and γ are configurable coefficients; The event severity classification rules are as follows: if the score is <0.6T, it is considered low risk and only local caching is required; if the score is 0.6T ≤ 1.2T, it is considered medium risk and 10 seconds of compressed video and sensor data before and after the collision are uploaded; if the score is ≥1.2T, it is considered high risk and the original video and full data for 30 seconds before and 60 seconds after the collision are uploaded.

9. The intelligent vehicle collision detection method based on edge-cloud collaboration and multi-agent evolution according to claim 7, characterized in that: The input to the multimodal large model mentioned in step S2 is specifically as follows: The video modal tensor dimensions are [450, 3, 240, 320], corresponding to 90 seconds, 5fps, RGB, and a resolution of 240×320; The sensor modal tensor has dimensions [1000, 6], corresponding to 10 seconds, 100Hz six-axis data; Metadata includes vehicle status temporal characteristics [90, 20] and vehicle inherent characteristics [70]; The cross-attention fusion specifically involves: performing bidirectional attention calculations on the sensor feature sequence S and the visual feature sequence V, which serve as the query and key / value pairs respectively, and then concatenating the fused sequence with the metadata embedding M before inputting it into the decoder.

10. The intelligent vehicle collision detection method based on edge-cloud collaboration and multi-agent evolution according to claim 7, characterized in that: In step S4, when constructing the incremental training dataset, erroneous samples are given a 3x sampling weight, high-confidence correct samples are given a 1x sampling weight, and a core set selection algorithm is used to extract a 20% representative subset from the historical data. The knowledge distillation specifically involves using the soft labels output by the large cloud model as teacher signals to train a lightweight in-vehicle AI model and minimize cross-entropy loss. For high-computing in-vehicle platforms with NPUs, feature map distillation or attention map distillation is further employed. The OTA update uses a hot-loading method during idle periods to update the vehicle model.

Images