Vehicle-mounted intelligent equipment collision detection method and system based on multi-modal large model

Abstract

The invention relates to a vehicle-mounted collision detection method and system based on a multi-modal large model, and relates to the technical field of automotive electronics. The method comprises the following steps: monitoring sensor data through vehicle-mounted equipment, and when the data exceeds a preset threshold value, triggering a primary collision event and uploading a multi-modal data packet; the cloud server receives the data packet, performs feature extraction and fusion analysis by using a multi-modal large model, and outputs an event classification result and confidence; and pushing a result to a user terminal for field confirmation, and performing incremental training optimization on the model based on feedback information. The system correspondingly comprises a vehicle-mounted intelligent equipment end, a cloud server and a closed-loop optimization system. Through multi-modal fusion analysis and a closed-loop self-evolution mechanism, the technical problems of high false alarm rate, lack of scene understanding and incapability of self-optimization of a traditional scheme are fundamentally solved, and high-precision and intelligent vehicle-mounted collision detection is realized.

Description

Technical Field

[0001] This invention relates to the field of automotive electronics and artificial intelligence, specifically to a collision detection method and system for in-vehicle intelligent devices based on a multimodal large model. Background Technology

[0002] Currently, the mainstream solution in the field of vehicle collision detection is a threshold judgment mechanism based on a single sensor. This solution typically relies on an embedded sensor, such as an accelerometer, to monitor the vehicle's dynamic information. Its working principle is as follows: the system presets a fixed physical threshold; when the data collected by the sensor exceeds this preset threshold, the system determines that a collision has occurred and triggers subsequent response procedures, such as recording event data, issuing alarm signals, or connecting to a rescue center.

[0003] Although the above technical solutions are simple to implement and low in cost, their inherent limitations have led to a series of technical problems that urgently need to be solved in practical applications. The specific shortcomings are as follows: Because it relies solely on a single, instantaneous impact signal, the system cannot effectively distinguish between real collisions and normal driving disturbances. Common driving scenarios, such as emergency braking, driving over bumpy roads, speed bumps, or high-speed potholes, all generate brief impact signals, easily triggering false collision warnings. This high frequency of false alarms not only severely disrupts drivers, creating a "boy who cried wolf" effect and making them insensitive to genuine warnings, but also results in a significant waste of data storage space, wireless communication resources, and the cost of manual verification in the background.

[0004] The existing technology is essentially a binary "yes / no" judgment system, and its output does not contain any semantic information. It cannot answer "what happened," that is, it cannot identify the type of collision (e.g., rear-end collision, side impact, or frontal collision), cannot assess the severity of the collision, and cannot obtain crucial visual contextual information (e.g., whether there was contact with other vehicles or objects). Therefore, the system has very limited value in scenarios requiring in-depth analysis and cannot provide effective and automated decision-making basis for insurance companies' rapid claims processing, accurate determination of liability in traffic accidents, and priority dispatching of emergency rescue services.

[0005] Existing collision detection solutions are typical "open-loop" systems. Once a preset action is triggered and executed, the entire process ends. The system lacks a mechanism to verify its own judgments, and it has no ability to learn from errors and optimize itself. Its detection algorithms and judgment thresholds are usually fixed at the factory and cannot be dynamically adjusted as vehicle operating data accumulates, different driving styles are adopted, or complex and changing road conditions are encountered. This causes the system's accuracy and intelligence level to stagnate after deployment, making it difficult to iterate and improve, and thus lacking long-term competitiveness. Summary of the Invention

[0006] The present invention designs a collision detection method and system for vehicle-mounted intelligent devices based on multimodal large model. The technical problems it solves are: (1) the false alarm rate in the existing technology is extremely high, the interference is serious and the resources are wasted; (2) the existing technology lacks the ability of context perception and deep analysis, and the application value is limited; (3) the existing technology system is isolated and rigid, and cannot optimize and evolve itself.

[0007] To solve the aforementioned technical problems, the present invention adopts the following solution: A vehicle collision detection method based on a multimodal large model includes the following steps: S1, event triggering and data acquisition step: continuously monitoring sensor data through an in-vehicle intelligent device; when the sensor data continuously exceeds a preset physical threshold, triggering a primary collision event and collecting and uploading multimodal data packets; S2, cloud-based multimodal large model analysis step: receiving the multimodal data packets through a cloud server and inputting them into a pre-trained multimodal large model for analysis, outputting the classification results and confidence levels of the collision event; S3, on-site verification and model optimization step: pushing the classification results to a user terminal for on-site confirmation, collecting on-site feedback information as real labels, and incrementally training and optimizing the multimodal large model based on the multimodal data packets, classification results, and real labels.

[0008] Preferably, step S1, the event triggering and data acquisition step, includes: setting a physical threshold for collision detection, the physical threshold including at least a gravitational acceleration threshold and a velocity threshold; triggering a primary collision event when sensor data continuously exceeds the physical threshold; acquiring and uploading a multimodal data packet, the multimodal data packet including: raw data from a six-axis IMU sensor for a preset period before and after the triggering time; video data from an onboard two-way camera for a preset period before and after the triggering time; and vehicle status data, including GPS location and speed information.

[0009] Preferably, in step S2, the cloud-based multimodal large model analysis, the multimodal large model performs the following operations: feature extraction: extracting time-domain and frequency-domain features from the six-axis IMU sensor data, and extracting spatiotemporal visual features from the video data; multimodal fusion: fusing sensor features, visual features, and vehicle state features to generate a comprehensive event feature representation; event classification: classifying collision events based on the comprehensive event feature representation, and outputting the classification results and confidence levels.

[0010] Preferably, the multimodal fusion operation employs a multimodal feature fusion mechanism based on cross-attention to achieve deep semantic alignment between different modal features; the classification result includes at least one of "normal event", "minor collision", "serious collision" and "false alarm".

[0011] Preferably, in step S3, on-site verification and model optimization, the incremental training optimization includes: storing the multimodal data packet, model classification results and real labels of this event into the training database; periodically or based on the accumulated data volume to trigger model retraining; and updating the parameters of the multimodal large model through an incremental learning algorithm to improve model accuracy.

[0012] A vehicle-mounted collision detection system based on a multimodal large model includes: an in-vehicle intelligent device for continuously monitoring sensor data and triggering a primary collision event when the sensor data continuously exceeds a preset physical threshold; the in-vehicle intelligent device also collects multimodal data packets and uploads them via a wireless network; a cloud server for receiving multimodal data packets from the in-vehicle intelligent device; analyzing the data packets using the multimodal large model to output the classification result and confidence level of the collision event; a closed-loop optimization system for pushing the classification result to a user terminal for on-site confirmation; collecting on-site feedback information as real labels; and incrementally training and optimizing the multimodal large model based on the multimodal data packets, classification results, and real labels; and a user terminal configured to receive the classification result pushed from the cloud and provide a user confirmation interface for users to provide feedback on the actual situation of the event.

[0013] Preferably, the in-vehicle intelligent device includes: a sensor module, including a six-axis IMU sensor and a GPS module, for collecting vehicle motion data and location information; The video acquisition module includes a front-facing camera and an in-vehicle camera, used to collect video data inside and outside the vehicle; the data processing and communication module is equipped with a circular data buffer, used to cache video data for a preset duration, and freezes the buffer when a primary collision event is triggered, packaging and uploading data for a specific time period before and after the trigger point.

[0014] Preferably, the multimodal large model in the cloud server includes: a feature extraction layer with a multi-branch architecture: a sensor branch, including 1D-CNN and LSTM / Transformer networks, for extracting temporal and frequency domain features of sensor data; a video branch, including 3D-CNN / Video Transformer networks, for extracting spatiotemporal visual features of video data; a metadata branch, including a fully connected network, for processing structured data of vehicle status; a multimodal fusion layer, configured to deeply fuse the features extracted from each branch; and a decision output layer, configured to output event classification results and confidence levels based on the fused features.

[0015] Preferably, the multimodal fusion layer adopts a feature fusion mechanism based on cross-attention, which enables features from different modalities to query and weight each other, thereby achieving deep semantic alignment.

[0016] Preferably, the closed-loop optimization system includes: a feedback collection module configured to collect on-site confirmation information through a user terminal APP or a third-party platform interface; a data management module configured to store multimodal data packets, model prediction results, and real labels to build a training database; and a model optimization module configured to perform incremental training on a large multimodal model based on the training database to achieve continuous self-evolution of the model.

[0017] The collision detection method and system for in-vehicle intelligent devices based on a multimodal large model have the following beneficial effects: (1) This invention introduces a multimodal large model to deeply fuse and analyze sensor data and video visual information. This solution enables the system to deeply understand the physical dynamics and visual context of events, thereby intelligently and accurately distinguishing between real collisions and normal driving interference. It fundamentally solves the industry pain point of extremely high false alarm rate of traditional single threshold detection, upgrading the detection result from "coarse alarm" to "precise diagnosis", greatly improving the reliability of the system and user experience, while avoiding a large waste of data storage, communication resources and manual verification costs caused by false alarms.

[0018] (2) This invention goes beyond simple "yes / no" binary judgments and can accurately output the nature of the collision, the severity level, and related visual scene information. This provides an unprecedented, automated, and intelligent decision-making basis for insurance companies' rapid automatic claims settlement, accurate determination of liability in traffic accidents, and priority scheduling of emergency rescue resources. It greatly improves the efficiency and fairness of post-event processing, realizes the leap from "detection" to "understanding," and significantly enhances the application value of the system output.

[0019] (3) By introducing an on-site verification feedback mechanism, the invention constructs a continuous optimization closed loop of detection-verification-learning. This means that the system can continuously learn incrementally from misjudgment cases and new scenarios in actual use, adapting to different vehicle models, driving styles and complex road conditions, and achieving self-evolution that becomes more accurate with use. This breaks the limitations of the rigid and unupdable algorithms of traditional systems, giving the system continuous vitality and building long-term technological barriers and market competitive advantages for enterprises.

[0020] (4) From the automatic triggering of events and the intelligent encapsulation and uploading of data to the in-depth analysis in the cloud and the automatic push of results, this invention greatly reduces the necessity of manual verification and intervention. For commercial users such as fleet management and insurance companies, this can significantly reduce their manpower and time costs in event screening, verification and processing, achieving significant cost reduction and efficiency improvement. At the same time, the system's high-precision judgment can also prevent social public rescue resources from being ineffectively occupied due to false reports, generating positive social benefits. Attached Figure Description

[0021] Figure 1This is an overall flowchart of the vehicle collision detection method based on a multimodal large model provided in the embodiments of the present invention; Figure 2 This is a schematic diagram of the architecture of a cloud-based multimodal large model provided in an embodiment of the present invention; Figure 3 This is a detailed flowchart of the system closed-loop optimization process provided in the embodiments of the present invention. Detailed Implementation

[0022] The following is combined Figures 1 to 3 The present invention will be further described as follows: Example 1: Vehicle Collision Detection Method Based on Multimodal Large Model like Figure 1 As shown in the figure, this embodiment provides a vehicle collision detection method based on a multimodal large model, which specifically includes the following steps: S1: Event Triggering and Data Acquisition; Physical thresholds for collision detection are preset in in-vehicle intelligent devices, including gravitational acceleration thresholds (e.g., 3G) and rate of change of velocity thresholds. During vehicle operation, a six-axis IMU sensor (including a three-axis accelerometer and a three-axis gyroscope) and a GPS module continuously operate, collecting real-time data on the vehicle's acceleration, angular velocity, position, and speed. Simultaneously, the front-facing camera and the in-vehicle camera continuously record video, and the video data is written to a circular data buffer of fixed duration (e.g., 10 minutes) for cyclical overwriting.

[0023] When the composite acceleration calculated in real time by the algorithm (derived from triaxial acceleration data) continuously exceeds a preset threshold (e.g., 3G) for more than 100 milliseconds, the system immediately triggers a "primary collision event." Upon triggering, the system performs the following operations: 1. Freeze the current video circular data buffer and save video data for specific time periods before and after the trigger point (e.g., 5 minutes before and 5 minutes after); 2. Extract the raw high-frequency sampling data from the six-axis IMU sensor for 15 seconds before and after the trigger point; 3. Obtain vehicle status data at the trigger time, including GPS location, speed, and vehicle model information; The aforementioned sensor data, video clips, and vehicle status information are packaged into a multimodal data packet; the data packet is then uploaded to the cloud server via a 4G / 5G wireless network.

[0024] S2: Cloud-based multimodal large model analysis; like Figure 2 As shown, after receiving the multimodal data packet, the cloud server calls a pre-trained multimodal large model for analysis and processing, specifically including: S21: Multimodal feature extraction; The model employs a multi-branch architecture to process data from different modalities in parallel, including sensor branches, video branches, and metadata branches.

[0025] The sensor branch involves inputting the time-series data from a six-axis IMU into a 1D-CNN network to extract temporal features such as local impact waveforms and vibration patterns. Subsequently, long-term temporal dependencies are captured through LSTM or Transformer networks to understand the dynamic evolution of events, and finally a 256-dimensional feature vector is output, which encodes the physical dynamic characteristics of the event.

[0026] The video branch involves inputting video frame sequences from the front and rear cameras into a 3D-CNN or Video Transformer network, extracting spatial features (such as object shape and scene structure) and temporal features (such as object trajectory and scene changes), identifying the interaction patterns between the vehicle and its surrounding environment, and finally outputting a 1024-dimensional feature vector that encodes the visual context information of the event.

[0027] The metadata branch inputs structured vehicle status data (such as GPS speed and vehicle model) into a fully connected network, learns the impact of different vehicle types and speed ranges on collision judgment, and outputs a 64-dimensional feature vector to provide auxiliary decision-making information.

[0028] S22: Multimodal feature fusion; The feature vectors (256-dimensional, 1024-dimensional, and 64-dimensional) output from the three branches are input into the multimodal fusion layer. The fusion layer employs an advanced fusion mechanism based on cross-attention, allowing sensor modal features and visual modal features to mutually query and weight each other, achieving deep semantic alignment rather than simple feature concatenation. For example, the system can learn the strong correlation between a "specific impact waveform" and "the scene of two cars colliding in a video." The fused features undergo nonlinear transformation and dimensionality reduction through a fully connected layer to generate a comprehensive event feature representation.

[0029] S23: Event Classification and Output; Based on the fused comprehensive features, a fully connected layer and a Softmax classifier output probability distributions for four categories: normal events, minor collisions, severe collisions, and false alarms. Simultaneously, the model outputs a confidence score, representing the degree of certainty regarding the judgment result. The final classification result and confidence score are then sent to subsequent processing modules.

[0030] S3: Field validation and model optimization, such as Figure 3 As shown, this stage achieves closed-loop optimization of the system: S31: Results push and on-site verification; The cloud server pushes the classification results and key evidence (such as screenshots of the moment of collision) to the user's mobile app. The user can view the results through the app interface and confirm whether the actual situation matches the system's judgment. For example, the user can provide feedback such as "It was indeed a serious rear-end collision," "It was just a sudden braking incident, the system gave a false alarm," or "It was a minor scratch, the system's judgment was correct."

[0031] S32: Feedback Collection and Realistic Tag Generation; The system collects user confirmation information and uses it as a "verification tag." Simultaneously, the system also supports receiving third-party verification information from insurance company damage assessment platforms or repair shops via API interfaces, further enriching the feedback channels.

[0032] S33: Incremental training and optimization of the model; The multimodal data package, model prediction results, and ground truth labels for this event are stored together in the training database. When a sufficient number of new samples (e.g., 1000) are accumulated or a preset period (e.g., weekly) is reached, the system automatically triggers the incremental training process. Using an incremental learning algorithm, the multimodal large model is fine-tuned with new training data based on the original model parameters, focusing on optimizing previously misjudged cases, thereby improving the model's accuracy in similar future scenarios and achieving the system's self-evolution.

[0033] Example 2: Vehicle collision detection system based on multimodal large model, see [link / reference] Figure 3 .

[0034] This embodiment provides a vehicle collision detection system for implementing the above method, including: The in-vehicle intelligent device is integrated inside the vehicle and includes: a sensor module: using an MPU-6050 six-axis IMU sensor to collect motion data and a U-blox GPS module to collect location information; a video acquisition module: using a wide-angle high-definition front camera and an infrared in-vehicle camera to record road conditions outside the vehicle and the driver's status inside the vehicle; and a data processing and communication module: using a Huawei Kirin 990 chip, configured with 8GB of memory, running a Linux system, and responsible for data caching, threshold judgment, and 4G / 5G communication transmission.

[0035] The cloud server is deployed on the Alibaba Cloud platform, including: a data receiving service: using a Kafka message queue to receive uploaded data from a large number of vehicle terminals at high concurrency; and a multimodal large model analysis service: deployed using the PyTorch framework, such as... Figure 2 The neural network model shown uses an NVIDIA A100 GPU for accelerated inference; the results push and service module pushes analysis results to the user's APP via Huawei Push service.

[0036] The closed-loop optimization system includes: a feedback collection module, which receives user confirmation information through the APP backend service and receives feedback from third-party platforms through a RESTful API; a data management module, which uses a MySQL database to store structured data and MinIO object storage to store video and unstructured data; and a model optimization module, which uses PyTorch's incremental learning toolkit to periodically fine-tune and update the model in the production environment.

[0037] The user terminal is an app installed on the user's mobile phone, which provides a visual interface to display the collision analysis results and has a one-click confirmation button for users to provide feedback on the actual situation.

[0038] Working principle: After the system is started, the vehicle-mounted terminal continuously monitors the system. When the triggering conditions are met, the data is automatically uploaded to the cloud. After in-depth analysis by the large model in the cloud, the results are pushed to the user. User feedback forms a closed loop, driving the system to continuously optimize and form an intelligent detection system that becomes more and more accurate with use.

[0039] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the implementation of the present invention is not limited to the above-described manner. Any improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without modification, are all within the protection scope of the present invention.

Claims

1. A method for vehicle collision detection based on a multi-modal large model, characterized in that, The method comprises the following steps: S1, event triggering and data collection step: continuously monitoring sensor data through the vehicle-mounted intelligent device, triggering a primary collision event when detecting that the sensor data continuously exceeds a preset physical threshold, and collecting and uploading a multi-modal data package; S2, cloud multi-modal large model analysis step: receiving the multi-modal data package through the cloud server and inputting it into a pre-trained multi-modal large model for analysis, outputting the classification result and confidence of the collision event; S3, on-site verification and model optimization step: pushing the classification result to the user terminal for on-site confirmation, collecting on-site feedback information as real labels, and incrementally training and optimizing the multi-modal large model based on the multi-modal data package, classification result and real label. 2.The vehicle-mounted collision detection method based on a multi-modal large model according to claim 1, wherein: S1, the event triggering and data collection step comprises: a preset collision detection physical threshold, the physical threshold comprising at least a gravitational acceleration threshold and a speed threshold; triggering a primary collision event when the sensor data continuously exceeds the physical threshold; collecting and uploading a multi-modal data package, the multi-modal data package comprising: six-axis IMU sensor raw data in a preset time period before and after the triggering time; vehicle-mounted bidirectional camera video data in a preset time period before and after the triggering time; and state data of the vehicle, including GPS position and speed information. 3.The vehicle-mounted collision detection method based on a multi-modal large model according to claim 1, characterized in that: In the S2, cloud multi-modal large model analysis step, the multi-modal large model performs the following operations: feature extraction operation: extracting time domain and frequency domain features from the six-axis IMU sensor data and extracting spatio-temporal visual features from the video data; multi-modal fusion operation: fusing sensor features, visual features and vehicle state features to generate comprehensive event feature representation; event classification operation: classifying the collision event based on the comprehensive event feature representation, and outputting the classification result and confidence. 4.The vehicle-mounted collision detection method based on a multi-modal large model according to claim 3, characterized in that: The multi-modal fusion operation adopts a multi-modal feature fusion mechanism based on cross-attention to realize deep semantic alignment between different modal features; the classification result comprises at least one of "normal event", "minor collision", "serious collision" and "false alarm". 5.The multi-modal large model-based vehicle collision detection method according to claim 1, wherein: In the S3, on-site verification and model optimization step, the incremental training and optimization comprises: storing the multi-modal data package, model classification result and real label of the event in the training database; triggering model retraining regularly or based on accumulated data volume; updating the parameters of the multi-modal large model through incremental learning algorithm to improve the accuracy of the model. 6.A vehicle-mounted collision detection system based on a multi-modal large model, characterized by: It comprises: a vehicle-mounted intelligent device end for continuously monitoring sensor data and triggering a primary collision event when detecting that the sensor data continuously exceeds a preset physical threshold, the vehicle-mounted intelligent device end also collecting a multi-modal data package and uploading it through a wireless network; a cloud server for receiving the multi-modal data package from the vehicle-mounted intelligent device end; analyzing the data package through a multi-modal large model, and outputting the classification result and confidence of the collision event; a closed-loop optimization system for pushing the classification result to the user terminal for on-site confirmation; collecting on-site feedback information as real labels; incrementally training and optimizing the multi-modal large model based on the multi-modal data package, classification result and real label. A user terminal configured to receive the cloud-pushed classification result and provide a user confirmation interface for the user to feed back the real situation of the event.

7. The multi-modal large model-based vehicle collision detection system of claim 6, wherein: The vehicle-mounted intelligent device end comprises: A sensor module comprising a six-axis IMU sensor and a GPS module for collecting vehicle motion data and position information; A video acquisition module comprising a front camera and an in-vehicle camera for collecting vehicle internal and external video data; A data processing and communication module configured with a ring-shaped data buffer for buffering video data for a preset time length, and freezing the buffer when a primary collision event is triggered, and packaging and uploading data of a specific time period before and after the trigger point. 8.The multi-modal large model-based vehicle collision detection system of claim 7, wherein: The multi-modal large model in the cloud server comprises: A feature extraction layer with a multi-branch architecture: A sensor branch comprising a 1D-CNN and an LSTM / Transformer network for extracting time domain and frequency domain features of sensor data; A video branch comprising a 3D-CNN / Video Transformer network for extracting spatio-temporal visual features of video data; A metadata branch comprising a fully connected network for processing vehicle state structured data; A multi-modal fusion layer configured to deeply fuse the features extracted by each branch; A decision output layer configured to output event classification results and confidence based on the fused features. 9.The multi-modal large model-based vehicle collision detection system of claim 8, wherein: The multi-modal fusion layer adopts a cross-attention-based feature fusion mechanism, which enables different modalities of features to query and weight each other, achieving deep semantic alignment.

10. The multi-modal large model based vehicle collision detection system of claim 7, wherein: The closed-loop optimization system comprises: A feedback collection module configured to collect on-site confirmation information through a user terminal APP or a third-party platform interface; A data management module configured to store multi-modal data packets, model prediction results and real labels, and build a training database; A model optimization module configured to perform incremental training of the multi-modal large model based on the training database, realizing continuous self-evolution of the model.

Images