A vehicle-mounted multi-source heterogeneous data fusion method, device and equipment

Abstract

The application discloses a kind of fusion method, device and equipment of vehicle-mounted multi-source heterogeneous data, comprising: real-time acquisition of the original data stream of multiple heterogeneous vehicle-mounted data sources on vehicle, the original data stream includes EDR data stream and non-EDR data stream;The local preprocessing including data cleaning, feature extraction, lightweight compression is carried out to the non-EDR data stream, the feature data after preprocessing is obtained, and the timestamp mark generated by the edge computing gateway local clock is attached to the feature data;Reference event is selected from the EDR data stream, and the EDR data stream is time-aligned with the feature data based on the reference event, to obtain the multi-source data set after time alignment;The multi-source data set is carried out data fusion processing, generates fusion data segment with time label, and carries out criticality evaluation to the fusion data segment, and the fusion data segment evaluated as high priority is transmitted to cloud data center as priority data segment.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus and equipment for fusing multi-source heterogeneous data in vehicles. Background Technology

[0002] As intelligent connected vehicles evolve towards high automation and deep interconnection, vehicles have transformed from simple transportation tools into complex data terminals integrating perception, decision-making, and control. In this process, the key vehicle state parameters recorded by the onboard event data recording system (EDR) have become the core legal basis for accident retrospective analysis, liability determination, and safety regulation compliance verification. However, faced with increasingly complex traffic scenarios and accident patterns, single-dimensional EDR data falls short in reconstructing the full picture of an event and understanding its causes. To construct an accurate and complete event map, the industry generally recognizes the need to integrate heterogeneous data from multiple sources, such as visual sensors, radar, and vehicle bus systems, to achieve collaborative perception of the accident environment and vehicle state.

[0003] Currently, achieving effective fusion of multi-source vehicle data faces significant technical challenges. First, different sensors and data sources have varying sampling frequencies, data formats, and clock references. In particular, EDR data itself may not contain high-precision absolute timestamp information, causing traditional alignment methods based on absolute timestamps to fail or produce significant errors in actual vehicle systems. This makes it difficult to achieve accurate time series alignment without strict clock synchronization, posing a fundamental challenge to subsequent correlation analysis. Second, in traditional cloud-centric centralized processing architectures, the full upload of massive amounts of raw data not only puts enormous pressure on the bandwidth of the vehicle-cloud communication link but also makes it difficult to meet the stringent requirements of high-real-time applications such as accident warning and real-time analysis due to unavoidable network transmission and queuing delays. Furthermore, EDR data in existing technologies typically exists as isolated snapshots, lacking effective correlation and synchronization with data from other vehicle systems such as advanced driver assistance systems and pedestrian protection systems, further restricting the depth and reliability of multi-source data fusion analysis. Summary of the Invention

[0004] In view of this, the purpose of this invention is to propose a method, apparatus and device for fusing multi-source heterogeneous vehicle data, which aims to solve the technical problems of insufficient time alignment accuracy and large data transmission and processing delay in the fusion of multi-source heterogeneous vehicle data in the prior art.

[0005] To achieve the above objectives, the present invention provides a method for fusing multi-source heterogeneous data in vehicles. The method is implemented based on an edge computing gateway deployed on the vehicle. The method includes: Real-time acquisition of raw data streams from multiple heterogeneous vehicle-mounted data sources, including EDR data streams and non-EDR data streams; The non-EDR data stream is subjected to local preprocessing including data cleaning, feature extraction, and lightweight compression to obtain preprocessed feature data, and a timestamp generated by the local clock of the edge computing gateway is added to the feature data. A reference event is selected from the EDR data stream, and the EDR data stream and the feature data are time-aligned based on the reference event to obtain a time-aligned multi-source dataset. The multi-source datasets are fused to generate fused data segments with time labels. The fused data segments are then evaluated for criticality, and the fused data segments evaluated as high priority are transmitted to the cloud data center as priority data segments.

[0006] To achieve the above objectives, the present invention also provides a device for fusing multi-source heterogeneous data in vehicles. The device is implemented based on an edge computing gateway deployed on the vehicle. The device includes: The acquisition unit is used to acquire raw data streams from multiple heterogeneous vehicle-mounted data sources in real time, including EDR data streams and non-EDR data streams. The preprocessing unit is used to perform local preprocessing on the non-EDR data stream, including data cleaning, feature extraction, and lightweight compression, to obtain preprocessed feature data, and to attach a timestamp generated by the local clock of the edge computing gateway to the feature data. A time alignment unit is used to select a reference event from the EDR data stream, and perform time alignment between the EDR data stream and the feature data based on the reference event to obtain a time-aligned multi-source dataset. The fusion processing unit is used to perform data fusion processing on the multi-source dataset, generate fused data segments with time labels, and perform criticality evaluation on the fused data segments, and transmit the fused data segments evaluated as high priority to the cloud data center as priority data segments.

[0007] To achieve the above objectives, the present invention also proposes an in-vehicle multi-source heterogeneous data fusion device, including a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of the in-vehicle multi-source heterogeneous data fusion method as described in the above embodiments.

[0008] To achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a computer program that is executed by a processor to implement the steps of a method for fusing multi-source heterogeneous data in a vehicle as described in the above embodiments.

[0009] Beneficial effects: The above solution, by introducing an edge computing gateway at the vehicle end, performs local preprocessing on non-EDR data streams and adds local clock timestamps, significantly reducing the amount of data that needs to be uploaded and fundamentally reducing network bandwidth consumption and transmission latency. Simultaneously, time alignment is performed based on reference events selected from the EDR data streams, achieving time synchronization with other heterogeneous data streams without relying on absolute timestamp synchronization of each data source. This effectively overcomes the dependence on strict absolute clock synchronization of each data source, significantly improving the robustness and accuracy of time alignment. By conducting critical evaluation of fused data segments, only high-priority segments are transmitted to the cloud first, ensuring real-time response to critical events and further optimizing edge-cloud communication efficiency, achieving intelligent scheduling of computing and communication resources. Overall, this improves the real-time performance, accuracy, and overall system efficiency of data fusion processing.

[0010] By identifying physical features corresponding to reference events in multi-source data and establishing a dynamic time window using the occurrence time of the reference event in the EDR data stream and feature data as a common reference point, each data stream is mapped to a unified time scale anchored by the reference event. This effectively overcomes the alignment difficulties caused by different sensor sampling frequencies, inconsistent clock references, and the lack of high-precision timestamps in the EDR data. In particular, the use of timestamp interpolation and nearest neighbor matching algorithms, along with the setting of tolerance ranges to handle time jitter, enables time alignment operations to achieve both high precision and high fault tolerance in complex real-world vehicle environments. This improves the robustness and accuracy of time alignment, ensuring stable and accurate correlation of multi-source data on the time axis, and providing a reliable time synchronization foundation for subsequent fusion analysis.

[0011] Building upon time alignment, this approach employs strategies such as weighted averaging, Kalman filtering, or sensor confidence-based priority selection to fuse multi-source data describing the same physical quantity. This yields more accurate and stable estimates of the physical quantity than from a single data source, improving the reliability of these estimates. Simultaneously, it retains unique parameters from different data sources, enriching the feature dimensions of the fused dataset. Furthermore, by extracting vehicle state parameters such as longitudinal deceleration, yaw rate, and seatbelt locking signals from the fused data segments and automatically identifying high-risk periods using threshold conditions, it achieves intelligent judgment and priority marking of critical periods such as collision events or emergency braking. This ensures that critical accident information is reliably captured and prioritized, allowing subsequent transmission and cloud analysis to focus on the most valuable data segments.

[0012] In the local preprocessing stage, a dynamic adaptive mechanism based on driving status is further introduced. This significantly reduces the generation and processing load of invalid data under normal conditions while ensuring that critical events are not lost. It achieves intelligent scheduling of edge node computing resources and network bandwidth, and realizes a dynamic balance between data value density and system processing load. In other words, by reducing resource consumption under normal conditions and immediately entering a "panoramic high-frequency recording" state and marking priorities in advance when abnormal signs appear, the system maximizes the utilization of edge node computing, storage, and communication resources while ensuring that no critical event details are lost, demonstrating the intelligence and economy of the system.

[0013] By conducting in-depth analysis and 3D reconstruction of the accident process on high-priority fused data segments received through cloud data centers, and continuously optimizing the weight parameters and risk trigger thresholds in the fusion strategy based on statistical analysis of a large amount of fused data, the updated model parameters are then distributed to the edge computing gateway, forming a closed-loop iterative mechanism of "end-edge-cloud" collaboration. This enables the edge processing capabilities deployed on massive vehicle terminals to continuously iterate, giving the entire system the ability to learn online and evolve on its own, improving the system's long-term performance and adaptability, thereby enhancing the accuracy of accident identification and the adaptability of fused processing. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating a method for fusing multi-source heterogeneous data in a vehicle, as provided in an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of the structure of a vehicle-mounted multi-source heterogeneous data fusion device provided in an embodiment of the present invention.

[0017] The realization of the invention's objective, its functional characteristics, and advantages will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The present invention will be described in detail below with reference to the embodiments.

[0020] Reference Figure 1 The diagram illustrates a flowchart of a method for fusing multi-source heterogeneous data in a vehicle, according to an embodiment of the present invention. In this embodiment, the method is implemented based on an edge computing gateway deployed on the vehicle. The edge computing gateway is also configured with a local cache for caching preprocessed feature data and fused data fragments. The local cache automatically deletes the oldest or lowest-value cached data according to a set storage capacity limit and time eviction policy, reserving space to store newly arrived high-priority data.

[0021] Specifically, the method includes: S11, Real-time acquisition of raw data streams from multiple heterogeneous vehicle-mounted data sources on the vehicle, including EDR data streams and non-EDR data streams.

[0022] The heterogeneous vehicle data sources include EDR systems, vehicle cameras, millimeter-wave radar, and vehicle sensors.

[0023] In this embodiment, the method is implemented based on an edge computing gateway deployed on the vehicle (i.e., an in-vehicle edge computing gateway). The edge computing gateway is configured with a local cache for caching preprocessed feature data and fused data fragments. The local cache automatically deletes the oldest or lowest value cached data according to a set storage capacity limit (e.g., 1GB) and a time eviction policy (e.g., retaining data from the most recent 24 hours, or prioritizing the deletion of smooth driving data marked as "low value"), in order to reserve space for storing newly arrived high-priority data.

[0024] Edge computing gateways collect raw data streams from multiple heterogeneous vehicle data sources in real time via vehicle network buses (such as CAN bus, in-vehicle Ethernet, etc.). These heterogeneous vehicle data sources include, but are not limited to: EDR systems (usually integrated into the airbag control unit ACU), vehicle cameras (such as front-view cameras, surround-view cameras), millimeter-wave radars (such as forward radar, side radar), and various vehicle sensors (such as wheel speed sensors, accelerometers, gyroscopes, brake pedal position sensors, seat belt buckle sensors, etc.). The raw data streams collected include two categories: EDR data streams and non-EDR data streams. The EDR data streams record snapshots of key vehicle status parameters before and after the collision event, including but not limited to vehicle longitudinal speed, longitudinal and lateral acceleration, braking status (brake pedal opening, ABS activation indicator), seat belt usage status (pretensioner trigger indicator, buckle status), and airbag deployment indicator. The non-EDR data streams cover video streams (or image sequences) collected by cameras, point cloud data collected by millimeter-wave radar (including target distance, relative speed, angle, etc.), and periodic vehicle status signals on the CAN bus (such as wheel speed, steering wheel angle, yaw rate, etc.).

[0025] S12, perform local preprocessing on the non-EDR data stream, including data cleaning, feature extraction, and lightweight compression, to obtain preprocessed feature data, and attach a timestamp generated by the local clock of the edge computing gateway to the feature data.

[0026] Furthermore, the local preprocessing also includes: For non-EDR data streams during smooth driving, the acquisition frequency or reporting frequency is reduced. When the deceleration exceeds the collision warning threshold, high-frequency recording of all data streams is triggered and the corresponding data segments are automatically marked as high priority.

[0027] Feature extraction of the non-EDR data stream includes: Extract keyframes and target detection boxes from camera video streams, and / or perform clustering and target tracking on millimeter-wave radar point cloud data.

[0028] In this embodiment, the edge computing gateway performs local preprocessing on the collected non-EDR data streams. The preprocessing first includes data cleaning: the edge computing gateway sets a dynamic data anomaly judgment threshold based on the physical characteristics of each sensor and the distribution of historical data, and removes anomalies that are obviously beyond the reasonable range (such as noise points with acceleration exceeding the vehicle's physical limits, isolated noise points in radar point clouds, etc.). For detected abnormal data, the edge computing gateway can automatically increase the collection frequency of the data source for intensive monitoring and send alarm information to the local or remote server; when the data returns to normal, the processing frequency is adjusted back to the normal frequency.

[0029] After data cleaning, the edge computing gateway performs feature extraction on the cleaned non-EDR data stream. For video streams captured by vehicle cameras, the edge computing gateway uses lightweight target detection algorithms (such as the YOLO series lightweight algorithms) to extract keyframes from the video stream (e.g., extracting 1-2 keyframes per second, or automatically triggering keyframe extraction when scene changes are detected). It then identifies and locates targets such as vehicles, pedestrians, and obstacles within the keyframes, outputting the bounding box coordinates, target category, and confidence score. For point cloud data acquired by millimeter-wave radar, the edge computing gateway uses clustering algorithms to segment the point cloud, grouping points belonging to the same target into one category. It then tracks the clustered targets, using Kalman filtering or extended Kalman filtering algorithms to estimate the target's motion state (including relative position, relative speed, and direction of motion), and maintaining a unique ID for each target to achieve target association between consecutive frames. For periodic vehicle status signals on the CAN bus (such as wheel speed and steering wheel angle), the edge computing gateway can directly retain them as time-series numerical features, or smooth them using moving average filtering or other methods to obtain more stable features.

[0030] After feature extraction, the edge computing gateway performs lightweight compression on the extracted feature data: for example, video features (keyframe images, bounding box coordinate lists) are encoded and compressed (e.g., JPEG compression), while discarding redundant information (e.g., blank frames where no target was detected); radar tracking results are stored in structured JSON or Protobuf format, retaining only key fields such as target ID, location, and velocity, while discarding noise points in the original point cloud. Through this series of preprocessing steps, the original non-EDR data stream (e.g., video frames at the MB level) is transformed into lightweight structured data (typically only at the KB level) containing only key feature information.

[0031] After feature extraction and compression, the edge computing gateway appends a high-precision timestamp generated by its local clock to each frame of preprocessed feature data. This local timestamp is independent of the individual sensor's clock and is based on the edge computing gateway's system time, thus avoiding the problem of clock asynchrony between different sensors. If this timestamp is denoted as t_local, its accuracy can reach the millisecond level. The timestamp-added feature data is then temporarily stored in a local buffer, awaiting subsequent time alignment processing.

[0032] Furthermore, the local preprocessing also includes a dynamic processing strategy based on the vehicle's driving state. The edge computing gateway continuously monitors the vehicle's state parameters (especially longitudinal deceleration). When the vehicle is in a stable driving phase (i.e., the absolute value of longitudinal deceleration is consistently below a first preset threshold), the edge computing gateway automatically reduces the feature extraction frequency and reporting frequency of non-EDR data streams. For example, it reduces the keyframe extraction frequency of the camera from 10 frames per second to 1 frame per second, and changes the clustering and tracking cycle of the radar point cloud from once per frame to once every 5 frames, in order to reduce the computational load of the edge computing nodes and the storage occupation of the local cache. When the edge computing gateway detects through the CAN bus that the absolute value of the vehicle's longitudinal deceleration suddenly exceeds the preset collision warning threshold, the system determines that an emergency braking or collision event may occur, and immediately triggers all data streams to enter a high-frequency processing mode: the keyframe extraction frequency of the camera is restored to the highest level, clustering and tracking of the radar point cloud is performed every frame, and all data segments within the corresponding time window are automatically marked as high priority, ensuring that these data segments are preferentially retained in the local cache and are not overwritten by subsequent normal data. This dynamic adaptation strategy based on "value density" achieves an intelligent balance between data preprocessing load and storage resources, significantly reducing resource consumption during normal operation while ensuring the integrity of critical event data.

[0033] Through the above, the edge computing gateway transforms the original non-EDR data stream into lightweight structured feature data carrying key feature information and bearing a unified local timestamp, providing a high-quality, low-data-volume input foundation for subsequent time alignment and multi-source data fusion.

[0034] S13, Select a reference event from the EDR data stream, and perform time alignment between the EDR data stream and the feature data based on the reference event to obtain a time-aligned multi-source dataset.

[0035] Furthermore, in step S13, the step of aligning the EDR data stream and the feature data in time based on the reference event to obtain a time-aligned multi-source dataset includes: S13-1, Identify the physical features corresponding to the reference event in the feature data to obtain the matching time of the reference event in the feature data; S13-2, A dynamic time window is established using the occurrence time of the reference event in the EDR data stream and the matching time in the feature data as a common reference point; S13-3, Based on the dynamic time window, the data points in the EDR data stream and the feature data are mapped onto a unified time scale with the reference event as the anchor point to obtain the time-aligned multi-source dataset.

[0036] Furthermore, in step S13-3, mapping the EDR data stream and data points in the feature data onto a unified time scale anchored by the reference event, based on the dynamic time window, includes: The data points in the EDR data stream and the feature data are aligned within the dynamic time window using timestamp interpolation and nearest neighbor matching algorithms, and time jitter is handled during alignment based on a preset tolerance range.

[0037] Furthermore, the reference event is an event recorded in the EDR data stream and having a corresponding physical feature in the feature data. The reference event includes an airbag deployment event, a seatbelt pretensioner triggering event, or an anti-lock braking system activation event.

[0038] The length of the dynamic time window is dynamically adjusted according to the type of the reference event or the current driving status of the vehicle; the tolerance range is ±50 milliseconds.

[0039] In this embodiment, the edge computing gateway first selects one or more reference events with significant characteristics (such as high contrast, high uniqueness, and high response value) from the acquired EDR data stream. These reference events are events explicitly recorded in the EDR data stream and have corresponding physical characteristics in the preprocessed feature data, such as airbag deployment events, seatbelt pretensioner triggering events, or anti-lock braking system (ABS) activation events. Taking a side-impact collision as an example, the EDR data stream records the airbag deployment event and its occurrence time T_edr (represented by an internal counter or relative timestamp of the EDR system, which may not include absolute clock information). The edge computing gateway then uses an image recognition algorithm (such as a deep learning-based object detection network) to search frame-by-frame for visual / signal features corresponding to the airbag deployment in the preprocessed camera feature data. When a critical moment from airbag deployment to full inflation is detected in a video frame, the local timestamp corresponding to that frame is recorded as the matching time T_cam. This is the matching time of the reference event in the feature data.

[0040] After obtaining the occurrence time T_edr of the reference event in the EDR data stream and the matching time T_cam in the feature data, the edge computing gateway uses these two times as a common reference point to establish a dynamic time window (updating the prior information used for estimation in real time through a moving average). The length of this window can be dynamically adjusted according to actual alignment needs (e.g., appropriately widened or narrowed based on the confidence level of the reference event or the degree of data jitter), thereby ensuring sufficient data points participate in the matching while avoiding the introduction of too much irrelevant noise. For example, the window can be dynamically adjusted based on the type of reference event and the current driving state of the vehicle: for instantaneous and feature-significant events such as airbag deployment, the window can be set to a narrower range to improve alignment accuracy; for events with longer durations and relatively gradual feature changes, such as ABS activation, the window can be appropriately widened to ensure sufficient data points participate in the matching. At the same time, if the vehicle is traveling at high speed or under severe braking, the system will also shrink the window accordingly to capture more refined dynamic changes.

[0041] After establishing a dynamic time window, the edge computing gateway maps the EDR data stream and all data points in the feature data within the window onto a unified time scale anchored by a reference event, using a combination of timestamp interpolation (such as linear interpolation) and nearest neighbor matching. Specifically, using the timestamp of each data point in the EDR data stream as the target (or conversely, selecting the data stream with the highest sampling rate as the benchmark), the nearest neighbor data point with the smallest time difference is found in the cache of each feature data stream as the matching point. Due to differences in sampling periods and transmission delays among different sensors, this matching process allows a preset tolerance range, such as ±50 milliseconds: as long as the absolute value of the time difference between two data points does not exceed 50 milliseconds, it is considered a valid alignment at the "same time point," which effectively tolerates clock drift and random jitter in the sensor network. For isolated points whose time differences exceed the tolerance range but are still within the window, linear interpolation is used to estimate the approximate feature value at the target time using the time-value relationship between adjacent matching points, thereby filling the sampling gaps and obtaining a continuous and equally spaced alignment sequence. An equivalent implementation could be to use the Pandas library's merge_asof function, set the direction to "nearest," and specify the tolerance parameter, while deploying a similar lightweight matching algorithm on the edge computing nodes.

[0042] After the alignment process described above, each key event in the EDR data stream (such as airbag deployment and ABS activation) becomes a fixed anchor point. All feature data from cameras, radar, and the CAN bus are mapped onto a relative time axis with this anchor point as the origin, forming a time-synchronized, unified-scale multi-source dataset. This dataset retains the original sampling accuracy of each data source while eliminating phase deviations caused by clock asynchrony or inconsistent trigger times, laying a reliable foundation for subsequent accurate data fusion.

[0043] S14, perform data fusion processing on the multi-source dataset to generate fused data segments with time labels, and perform criticality evaluation on the fused data segments. The fused data segments evaluated as high priority are transmitted to the cloud data center as priority data segments.

[0044] Furthermore, in step S14, the process of fusing the multi-source dataset to generate fused data segments with time labels includes: S14-1, A preset fusion strategy is used to fuse data from different data sources that describe the same physical quantity to obtain a preliminary fused dataset. The preset fusion strategy includes weighted average, Kalman filtering, or a priority selection strategy based on sensor confidence. S14-2, merge data from different data sources that describe different physical quantities into the preliminary fused dataset, and add time tags to the merged fused dataset according to the time information of the multi-source datasets on a unified time scale to generate the fused data segment.

[0045] Furthermore, in step S14, the critical evaluation of the fused data segment includes: S14-3, extract vehicle status parameters from the fused data segment. When the vehicle status parameters meet the preset risk triggering conditions, the corresponding fused data segment is identified as a high-risk period and assigned a high-priority mark. The vehicle status parameters include longitudinal deceleration, yaw rate, or seat belt locking signal. The risk triggering conditions include the longitudinal deceleration exceeding a first threshold, the yaw rate exceeding a second threshold, or the seat belt locking signal being triggered.

[0046] When transmitting priority data segments to the cloud data center, the MQTT-SN lightweight communication protocol is used for priority transmission, and local caching is enabled to wait for retransmission when the network is interrupted.

[0047] In this embodiment, the edge computing gateway performs data fusion processing on the multi-source dataset obtained after time alignment, including fusing data from different data sources that describe the same physical quantity and merging data from different data sources that describe different physical quantities.

[0048] For data describing the same physical quantity from different data sources, such as vehicle longitudinal acceleration, which may originate simultaneously from EDR accelerometers, millimeter-wave radar target velocity differentials, and camera motion estimates, the edge computing gateway employs a pre-defined fusion strategy to obtain a preliminary fused dataset. This pre-defined fusion strategy includes weighted averaging, Kalman filtering, or a sensor confidence-based priority selection strategy. Taking the weighted averaging strategy as an example, the edge computing gateway assigns different weights to each sensor based on their confidence level (or historical accuracy statistics) under specific operating conditions, and then sums the weighted measurements from each sensor at the same time to obtain the fused physical quantity estimate. If a Kalman filtering strategy is used, a state vector containing the vehicle's motion state (e.g., position, velocity, acceleration) is constructed, using the sensor measurements as input, and the optimal state estimate is obtained recursively through prediction and update steps. If a sensor confidence-based priority selection strategy is used, the edge computing gateway dynamically evaluates the quality of each data source (e.g., signal strength, noise level, historical consistency) at each time step, selecting the data source with the highest current confidence score as the fused output.

[0049] For data from different data sources describing different physical quantities, such as target types detected by cameras (e.g., pedestrians, vehicles), target distance and relative speed output by millimeter-wave radar, steering wheel angle and wheel speed on the CAN bus, the edge computing gateway directly merges these independent parameters into the initial fusion dataset, retaining their respective information dimensions without performing cross-physical quantity fusion. After the above merging is completed, the edge computing gateway adds a unified time label to the fusion result at each time point based on the time information of the multi-source datasets on a unified time scale (i.e., the aligned relative time anchored to the reference event), generating a fusion data segment with a time label. This segment covers multi-dimensional features such as vehicle motion state, environmental perception information, and driver operation behavior within the complete time window from before to after the collision.

[0050] Next, the edge computing gateway performs a critical evaluation on the generated fused data segments. The edge computing gateway extracts vehicle state parameters from each fused data segment, including longitudinal deceleration, yaw rate, and seatbelt locking signal. The edge computing gateway internally presets risk trigger conditions: when the longitudinal deceleration exceeds a first preset threshold, or the yaw rate exceeds a second preset threshold, or the seatbelt locking signal is triggered, the current fused data segment is determined to be a high-risk period. For example, in a side-impact collision scenario, sudden changes in longitudinal deceleration recorded by the EDR and the seatbelt pretensioner trigger signal can both trigger this condition. Once any of the above risk trigger conditions is met, the edge computing gateway classifies the corresponding fused data segment as a high-risk period and assigns it a high-priority label. In practical applications, the entire time window from 2 seconds before the collision to 1 second after the collision can be marked as a "critical time period." The edge computing gateway can then trim fused data segments based on this time range and assign a uniform priority label to the data within it.

[0051] After completing criticality assessment and labeling, the edge computing gateway transmits high-priority fused data segments as priority data segments to the cloud data center. To ensure that critical data arrives at the cloud in a timely and reliable manner, the edge computing gateway uses the MQTT-SN lightweight communication protocol for priority transmission. This protocol is designed for low-bandwidth, unreliable network environments, supports publish / subscribe modes, and has low power consumption. During transmission, the edge computing gateway places high-priority segments at the head of the transmission queue and sets a high quality of service level (e.g., at least once). If a network interruption is detected (e.g., a vehicle entering a tunnel or a signal blind spot), the edge computing gateway automatically activates a local buffer to temporarily store priority data segments that have not yet been successfully transmitted and continuously monitors the network connection status. Once the network is restored, these segments are immediately retrieved from the buffer and retransmitted in priority order to ensure that critical data is not lost and arrives at the cloud in an orderly manner. For non-high-priority, normal data segments (e.g., low-rate-of-change data during smooth driving), the edge computing gateway can choose to delay transmission (e.g., upload in batches after accumulating a certain amount of data) or only store summary information locally, thereby effectively saving vehicle-to-cloud communication bandwidth and cloud storage resources.

[0052] Through the above process, the edge computing gateway completes the generation of fused data segments from multi-source datasets, intelligent determination of high-risk periods, and efficient and reliable uploading of priority data segments, providing a timely, accurate, and critical data foundation for subsequent cloud-based in-depth analysis and incident reconstruction.

[0053] In another embodiment, the method further includes: S15, the cloud data center is used to perform in-depth analysis and three-dimensional reconstruction of the received priority data fragments of the accident process, and the model parameters are optimized based on the analysis results. The optimized model parameters are then sent to the edge computing gateway. The model parameters include weight parameters or thresholds of risk triggering conditions in the preset fusion strategy.

[0054] In this embodiment, the cloud data center, acting as the cloud brain of the entire converged architecture, receives priority data segments (i.e., high-priority fused data segments marked as high-risk periods) uploaded from the edge computing gateway and performs deep analysis and model optimization. Specifically, the cloud data center first performs deep reconstruction of the accident process on the received priority data segments. Since the data uploaded by the edge computing gateway has already undergone time alignment and preliminary fusion, it possesses a unified time scale anchored by a reference event and precise correlation of multi-source heterogeneous data. The cloud data center can directly utilize this high-precision synchronized data, combined with physical models and machine learning algorithms, to perform millisecond-level and millimeter-level precise 3D reconstruction. Taking a side-impact collision accident as an example, the priority data segments received by the cloud data center include vehicle speed, longitudinal and lateral acceleration, braking status, seat belt locking signal recorded by EDR, keyframes and target bounding boxes extracted from the camera video stream (recording the position, size, and direction of movement of the other vehicle at the moment of collision), and target distance, relative speed, and angle information output by millimeter-wave radar. Based on the aforementioned multi-source data, the cloud-based data center uses kinematic models (such as uniform acceleration models or more advanced vehicle dynamics models) to infer the instantaneous velocity and trajectory before the collision. Simultaneously, combining target detection and tracking results, it reconstructs the relative pose, collision angle, and overlapping area of ​​the two vehicles at the moment of collision. Through frame-by-frame matching on the timeline, the system can generate a complete 3D animation sequence of vehicle state and environmental changes from several seconds before to several seconds after the collision. Finally, it outputs an accident analysis report containing details such as collision speed, collision energy, and energy absorption distribution. This report can meet the needs of regulatory compliance assessments and forensic identification.

[0055] Based on this, the cloud data center continuously optimizes various model parameters through statistical analysis of a large number of fused data fragments, including but not limited to: weight parameters in the preset fusion strategy (for example, in the weighted average strategy, the confidence weight of each data source is dynamically adjusted according to the statistical deviation between the data of each sensor in a large number of real scenarios and the actual true value, so that the fusion result is closer to the real physical state), thresholds for risk triggering conditions (such as longitudinal deceleration threshold, yaw rate threshold, etc., which can be adaptively calibrated according to historical accident data under different vehicle models, different road conditions or different driving scenarios to reduce false alarm rate or false negative rate), and internal parameters of accident recognition algorithm and feature extraction model (such as the weights of the target detection neural network, the process noise and measurement noise covariance matrix of the Kalman filter, etc.).

[0056] After optimization, the cloud data center encapsulates the updated model parameters into lightweight configuration files or model parameter packages, and periodically distributes them to the edge computing gateway in the vehicle via vehicle-to-everything (V2X) communication (such as 4G / 5G, satellite communication, etc.). Upon receiving the model parameters from the cloud, the edge computing gateway performs a local hot update (i.e., loading the new parameters into the running fusion algorithm module without restarting the system or interrupting normal data processing), thereby immediately or as needed activating the optimized fusion strategy and risk assessment rules. Through this closed-loop iterative mechanism, the entire system can continuously evolve using massive amounts of real-world operational data, improving the accuracy of data fusion, the sensitivity and specificity of accident identification, and the comprehensiveness and reliability of the final accident analysis from the "end-edge-cloud" collaborative level.

[0057] Reference Figure 2 The diagram shown is a structural schematic of a vehicle-mounted multi-source heterogeneous data fusion device provided in an embodiment of the present invention.

[0058] In this embodiment, the device is implemented based on an edge computing gateway deployed on the vehicle. The device 20 includes: The acquisition unit 21 is used to acquire raw data streams from multiple heterogeneous vehicle-mounted data sources in real time, including EDR data streams and non-EDR data streams. Preprocessing unit 22 is used to perform local preprocessing on the non-EDR data stream, including data cleaning, feature extraction, and lightweight compression, to obtain preprocessed feature data, and to attach a timestamp generated by the local clock of the edge computing gateway to the feature data. The time alignment unit 23 is used to select a reference event from the EDR data stream, and perform time alignment between the EDR data stream and the feature data based on the reference event to obtain a time-aligned multi-source dataset. The fusion processing unit 24 is used to perform data fusion processing on the multi-source dataset, generate fused data segments with time labels, and perform criticality evaluation on the fused data segments, and transmit the fused data segments evaluated as high priority to the cloud data center as priority data segments.

[0059] In another embodiment, the device 20 further includes: The optimization unit is used to perform in-depth analysis and three-dimensional reconstruction of the received priority data fragments of the accident process using the cloud data center, and optimize the model parameters based on the analysis results, and send the optimized model parameters to the edge computing gateway. The model parameters include weight parameters in the preset fusion strategy or thresholds of risk triggering conditions.

[0060] Each unit module of the device 20 can execute the corresponding steps in the above method embodiment, so the details of each unit module will not be elaborated here. Please refer to the description of the corresponding steps above for details.

[0061] This invention also provides a device for fusing multi-source heterogeneous data in vehicles. This device includes the multi-source heterogeneous data fusion apparatus described above, wherein the multi-source heterogeneous data fusion apparatus can employ… Figure 2 The structure of the embodiment, correspondingly, can be executed Figure 1 The technical solutions of the method embodiments shown are similar in implementation principle and technical effect. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0062] The device includes: a server, personal computer, smartphone or tablet computer, or other device with image / text processing and computing functions. The device may include components such as memory, processor, input unit, display unit, and power supply.

[0063] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory can also include a memory controller to provide access to the memory for the processor and input units.

[0064] The input unit can be used to receive input numerical, character, or image information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, in addition to a camera, the input unit of this embodiment may also include a touch-sensitive surface (e.g., a touch screen) and other input devices.

[0065] The display unit can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The display unit may include a display panel, optionally configured as an LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar display panel. Furthermore, a touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to the processor to determine the type of touch event. Subsequently, the processor provides corresponding visual output on the display panel based on the type of touch event.

[0066] This invention also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement... Figure 1 The method for fusing multi-source heterogeneous data in a vehicle is shown. The computer-readable storage medium may be a read-only memory, a disk, or an optical disk, etc.

[0067] This invention also provides a computer program product, including a computer program / instructions, which are loaded and executed by a processor to implement... Figure 1 This paper presents a method for fusing multi-source heterogeneous data in a vehicle.

[0068] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the device embodiments, equipment embodiments, and storage medium embodiments, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions in the method embodiments.

[0069] Furthermore, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0070] The foregoing description illustrates and describes preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept by means of the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for fusing multi-source heterogeneous data in a vehicle, characterized in that, The method is implemented based on an edge computing gateway deployed on the vehicle, and the method includes: Real-time acquisition of raw data streams from multiple heterogeneous vehicle-mounted data sources, including EDR data streams and non-EDR data streams; The non-EDR data stream is subjected to local preprocessing including data cleaning, feature extraction, and lightweight compression to obtain preprocessed feature data, and a timestamp generated by the local clock of the edge computing gateway is added to the feature data. A reference event is selected from the EDR data stream, and the EDR data stream and the feature data are time-aligned based on the reference event to obtain a time-aligned multi-source dataset. The multi-source datasets are fused to generate fused data segments with time labels. The fused data segments are then evaluated for criticality, and the fused data segments evaluated as high priority are transmitted to the cloud data center as priority data segments.

2. The method for fusing multi-source heterogeneous data in a vehicle according to claim 1, characterized in that, The step of aligning the EDR data stream and the feature data in time based on the reference event to obtain a time-aligned multi-source dataset includes: Identify the physical features corresponding to the reference event in the feature data to obtain the matching time of the reference event in the feature data; A dynamic time window is established using the occurrence time of the reference event in the EDR data stream and the matching time in the feature data as a common reference point; Based on the dynamic time window, the data points in the EDR data stream and the feature data are mapped onto a unified time scale anchored by the reference event to obtain the time-aligned multi-source dataset.

3. The method for fusing multi-source heterogeneous data in a vehicle according to claim 2, characterized in that, The step of mapping the EDR data stream and data points in the feature data to a unified time scale anchored by the reference event, based on the dynamic time window, includes: The data points in the EDR data stream and the feature data are aligned within the dynamic time window using timestamp interpolation and nearest neighbor matching algorithms, and time jitter is handled during alignment based on a preset tolerance range.

4. A method for fusing multi-source heterogeneous data in a vehicle according to any one of claims 1-3, characterized in that, The reference event is an event recorded in the EDR data stream and having a corresponding physical feature in the feature data. The reference event includes an airbag deployment event, a seat belt pretensioner trigger event, or an anti-lock braking system activation event.

5. The method for fusing multi-source heterogeneous data in a vehicle according to claim 1, characterized in that, The step of performing data fusion processing on the multi-source dataset to generate fused data segments with time labels includes: A preset fusion strategy is used to fuse data from different data sources that describe the same physical quantity to obtain a preliminary fused dataset. The preset fusion strategy includes weighted average, Kalman filtering, or a priority selection strategy based on sensor confidence. Data from different data sources and describing different physical quantities are merged into the preliminary fused dataset, and time labels are added to the merged fused dataset according to the time information of the multi-source datasets on a unified time scale to generate the fused data fragment.

6. The method for fusing multi-source heterogeneous data in a vehicle according to claim 1, characterized in that, The critical evaluation of the fused data fragment includes: Vehicle status parameters are extracted from the fused data segments. When the vehicle status parameters meet preset risk triggering conditions, the corresponding fused data segments are identified as high-risk periods and assigned a high-priority label. The vehicle status parameters include longitudinal deceleration, yaw rate, or seat belt locking signal. The risk triggering conditions include the longitudinal deceleration exceeding a first threshold, the yaw rate exceeding a second threshold, or the seat belt locking signal being triggered.

7. The method for fusing multi-source heterogeneous data in a vehicle according to claim 1, characterized in that, The local preprocessing also includes: For non-EDR data streams during smooth driving, the acquisition frequency or reporting frequency is reduced. When the deceleration exceeds the collision warning threshold, high-frequency recording of all data streams is triggered and the corresponding data segments are automatically marked as high priority.

8. The method for fusing multi-source heterogeneous data in a vehicle according to claim 1, characterized in that, The method further includes: The cloud data center is used to perform in-depth analysis and three-dimensional reconstruction of the received priority data fragments of the accident process, and the model parameters are optimized based on the analysis results. The optimized model parameters are then sent to the edge computing gateway. The model parameters include weight parameters or thresholds of risk triggering conditions in the preset fusion strategy.

9. A vehicle-mounted multi-source heterogeneous data fusion device, characterized in that, The device is implemented based on an edge computing gateway deployed on the vehicle. The device includes: The acquisition unit is used to acquire raw data streams from multiple heterogeneous vehicle-mounted data sources in real time, including EDR data streams and non-EDR data streams. The preprocessing unit is used to perform local preprocessing on the non-EDR data stream, including data cleaning, feature extraction, and lightweight compression, to obtain preprocessed feature data, and to attach a timestamp generated by the local clock of the edge computing gateway to the feature data. A time alignment unit is used to select a reference event from the EDR data stream, and perform time alignment between the EDR data stream and the feature data based on the reference event to obtain a time-aligned multi-source dataset. The fusion processing unit is used to perform data fusion processing on the multi-source dataset, generate fused data segments with time labels, and perform criticality evaluation on the fused data segments, and transmit the fused data segments evaluated as high priority to the cloud data center as priority data segments.

10. A vehicle-mounted multi-source heterogeneous data fusion device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory, wherein the computer program, when executed by the processor, implements the steps of the method for fusing multi-source heterogeneous data in a vehicle as described in any one of claims 1 to 8.

Images