Vehicle safety hazard warning method, device and equipment based on multi-sensor fusion

Abstract

The present application relates to the field of data processing technology, and discloses a vehicle safety hazard warning method, device and equipment based on multi-sensor fusion. The method includes: performing outlier detection and feature extraction processing on multi-modal vehicle safety situation data to obtain vehicle-environment interaction features; performing multi-modal deep feature learning processing on vehicle-environment interaction features to obtain high-level vehicle safety semantic features; performing knowledge mapping and context association processing on high-level vehicle safety semantic features to obtain semantically enhanced vehicle safety status representation; performing dynamic probabilistic reasoning processing on the semantically enhanced vehicle safety status representation to obtain multi-dimensional vehicle safety risk indicators; performing threshold judgment and multi-level warning generation processing on multi-dimensional vehicle safety risk indicators to obtain real-time vehicle safety hazard warning information. The present application improves the efficiency and accuracy of vehicle safety hazard warning based on multi-sensor fusion.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a vehicle safety hazard warning method, device and equipment based on multi-sensor fusion. Background Art

[0002] Existing vehicle safety warning systems mainly rely on single or a few sensor data for analysis and judgment. For example, some systems only use radar ranging to achieve collision avoidance warnings, or only rely on camera image recognition for lane departure warnings. These systems usually use simple threshold judgments or rule-based methods to trigger warnings, such as issuing a warning when the distance to the vehicle in front is less than a fixed value. At the same time, some more advanced systems have begun to introduce machine learning algorithms, such as support vector machines (SVM) or decision trees, to handle more complex scenarios and improve the accuracy of warnings. However, these methods often treat different types of safety risks separately and lack a comprehensive assessment of the overall safety status of the vehicle.

[0003] However, these existing technologies have several obvious shortcomings. First, a single sensor or simple fusion method cannot fully capture the various risk factors in a complex traffic environment, and is prone to misjudgment or missed judgment. Second, methods with fixed thresholds or simple rules are difficult to adapt to different driving environments and the personalized needs of drivers, and may lead to too many false alarms or missed alarms. Furthermore, the separate treatment of different types of safety risks ignores the mutual influence and cumulative effects between various risk factors, and cannot accurately assess the overall safety situation. Finally, most existing systems lack the ability to analyze the dynamic changes and development trends of safety risks, making it difficult to provide forward-looking warning information. Summary of the invention

[0004] The present application provides a vehicle safety hazard warning method, device and equipment based on multi-sensor fusion, which are used to improve the efficiency and accuracy of vehicle safety hazard warning based on multi-sensor fusion.

[0005] In a first aspect, the present application provides a vehicle safety hazard warning method based on multi-sensor fusion, and the vehicle safety hazard warning method based on multi-sensor fusion includes: performing time alignment and spatial registration processing on the original data collected by the vehicle-mounted multi-source sensors to obtain multimodal vehicle safety situation data; performing outlier detection and feature extraction processing on the multimodal vehicle safety situation data to obtain vehicle-environment interaction features; performing multimodal deep feature learning processing on the vehicle-environment interaction features to obtain high-level vehicle safety semantic features; performing knowledge mapping and context association processing on the high-level vehicle safety semantic features to obtain a semantically enhanced vehicle safety status representation; performing dynamic probabilistic reasoning processing on the semantically enhanced vehicle safety status representation to obtain a multidimensional vehicle safety risk index; performing threshold judgment and multi-level warning generation processing on the multidimensional vehicle safety risk index to obtain real-time vehicle safety hazard warning information.

[0006] In a second aspect, the present application provides a vehicle safety hazard warning device based on multi-sensor fusion, and the vehicle safety hazard warning device based on multi-sensor fusion includes:

[0007] The registration module is used to perform time alignment and spatial registration processing on the raw data collected by the multi-source sensors on the vehicle to obtain multi-modal vehicle safety situation data;

[0008] An extraction module, used to perform outlier detection and feature extraction processing on the multimodal vehicle safety situation data to obtain vehicle-environment interaction features;

[0009] A processing module, used for performing multimodal deep feature learning processing on the vehicle-environment interaction features to obtain high-level vehicle safety semantic features;

[0010] An association module, used for performing knowledge mapping and context association processing on the advanced vehicle safety semantic features to obtain a semantically enhanced vehicle safety status representation;

[0011] A reasoning module, used for performing dynamic probabilistic reasoning processing on the semantically enhanced vehicle safety status representation to obtain a multi-dimensional vehicle safety risk index;

[0012] The generation module is used to perform threshold judgment and multi-level warning generation processing on the multi-dimensional vehicle safety risk indicators to obtain real-time vehicle safety hazard warning information.

[0013] The third aspect of the present application provides a computer device, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate via a bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned vehicle safety hazard warning method based on multi-sensor fusion are performed.

[0014] In the technical solution provided by this application, the raw data collected by the multi-source sensors on the vehicle are time-aligned and spatially registered to ensure the consistency of data from different sensors in the time and space dimensions, provide a reliable data basis for subsequent analysis, and effectively reduce the risk of misjudgment caused by data asynchrony or inconsistent spatial references. The outlier detection and feature extraction processing of multimodal vehicle safety situation data not only improves the data quality, eliminates noise and outliers that may affect the judgment, but also extracts the most representative and discriminative features, greatly improving the efficiency and accuracy of subsequent analysis. Then, through multimodal deep feature learning processing, the system can automatically learn and capture complex nonlinear feature patterns, greatly enhancing the ability to understand complex traffic scenes, making the assessment of safety status more comprehensive and in-depth. The knowledge mapping and context association processing of advanced vehicle safety semantic features organically combines data-driven feature learning with domain expert knowledge, improves the interpretability and generalization ability of the system, and makes the safety assessment results more reliable and practical. Subsequently, the semantically enhanced vehicle safety status representation is processed by dynamic probabilistic reasoning, so that the system can make reasonable risk assessments in uncertain environments and dynamically update risk estimates as new information is input, improving the system's ability to adapt to complex and changing environments. Finally, by performing threshold judgment and multi-level warning generation processing on multi-dimensional vehicle safety risk indicators, the system can generate targeted warning information according to different risk levels and types, avoiding the troubles that may be caused by excessive warnings and ensuring timely reminders in real dangerous situations. This multi-level and multi-angle safety hazard warning method not only comprehensively improves the accuracy, timeliness and reliability of warnings, but also makes warning information more intelligent and humanized through semantic understanding and knowledge fusion, greatly enhancing driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0016] Figure 1 A schematic diagram of an embodiment of a vehicle safety hazard early warning method based on multi-sensor fusion in an embodiment of the present application;

[0017] Figure 2 This is a schematic diagram of an embodiment of a vehicle safety hazard warning device based on multi-sensor fusion in an embodiment of the present application;

[0018] Figure 3It is a schematic diagram of the structure of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The embodiments of the present application provide a method, device and equipment for warning of vehicle safety hazards based on multi-sensor fusion. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 In the embodiment of the present application, an embodiment of the vehicle safety hazard warning method based on multi-sensor fusion includes:

[0021] Step S101, performing time alignment and spatial registration processing on the raw data collected by the vehicle-mounted multi-source sensor to obtain multi-modal vehicle safety situation data;

[0022] Step S102: performing outlier detection and feature extraction processing on the multimodal vehicle safety situation data to obtain vehicle-environment interaction features;

[0023] Step S103, performing multimodal deep feature learning processing on the vehicle-environment interaction features to obtain high-level vehicle safety semantic features;

[0024] Step S104, performing knowledge mapping and context association processing on the advanced vehicle safety semantic features to obtain a semantically enhanced vehicle safety status representation;

[0025] Step S105, performing dynamic probability reasoning processing on the semantically enhanced vehicle safety status representation to obtain a multi-dimensional vehicle safety risk index;

[0026] Step S106: Perform threshold judgment and multi-level warning generation processing on the multi-dimensional vehicle safety risk indicators to obtain real-time vehicle safety hazard warning information.

[0027] It is understandable that the execution subject of the present application may be a vehicle safety hazard warning device based on multi-sensor fusion, or a terminal or a server, which is not limited here. The present application embodiment is described by taking the server as the execution subject as an example.

[0028] Specifically, the raw data collected by multi-source sensors on the vehicle are processed. These sensors include cameras, radars, GPS, inertial measurement units, etc., which collect data at different frequencies. The time alignment process synchronizes the data of different frequencies to a unified time axis through an interpolation algorithm, while the spatial registration process converts the coordinate systems of different sensors into the unified reference coordinate system of the vehicle. The result of this step is multimodal vehicle safety situation data, which contains comprehensive information about the vehicle status, surrounding environment, and potential risk factors. Outlier detection and feature extraction are performed on the multimodal vehicle safety situation data. Outlier detection uses statistical methods such as Z-score or interquartile range method to identify data points that deviate from the normal range. Feature extraction extracts meaningful features from the raw data, such as vehicle speed, acceleration, relative distance to surrounding vehicles, etc. The output of this step is the vehicle-environment interaction feature, which describes the interaction between the vehicle and the surrounding environment.

[0029] Subsequently, the vehicle-environment interaction features are processed through multimodal deep feature learning. This step uses deep neural networks, such as convolutional neural networks to process image data, recurrent neural networks to process time series data, and attention mechanisms to fuse information from different modalities. Through multiple layers of nonlinear transformations, the network learns high-level vehicle safety semantic features that can better express complex safety-related patterns. Next, the high-level vehicle safety semantic features are processed through knowledge mapping and context association. This step uses a pre-built traffic safety knowledge graph to associate the learned features with known safety concepts and rules. Through technologies such as graph neural networks, the system can infer a wider range of contextual information and obtain a semantically enhanced representation of the vehicle's safety status.

[0030] Then, the semantically enhanced vehicle safety status representation is processed by dynamic probabilistic reasoning. This step uses probabilistic models such as dynamic Bayesian networks to consider dependencies in time series and infer multidimensional vehicle safety risk indicators. These indicators may include multiple dimensions such as collision risk, lane departure risk, and speeding risk. Finally, threshold judgment and multi-level warning generation processing are performed on the multidimensional vehicle safety risk indicators. The system judges the severity of each risk indicator based on the preset risk threshold and generates warning information of the corresponding level. Warning information may include sound and light warnings, driver prompts, or automatic intervention control.

[0031] For example, suppose a car is driving on a highway. The camera captures 30 frames per second, the millimeter-wave radar scans the surrounding environment every 0.1 seconds, and the GPS updates the position information once a second. Time alignment processing unifies this data to a frequency of 10Hz, and the spatial matching rule converts the position of the object detected by the radar to the same coordinate system as the on-board camera. Outlier detection finds a sudden jump in the relative distance data of the vehicle in front, which the system marks as an anomaly and smoothes. Feature extraction calculates the speed of the vehicle to be 120km / h, the relative speed to the vehicle in front to be -5km / h, and the relative distance to the vehicle in front to be 50m. The deep learning network identifies a potential pattern of following too closely from these features. Knowledge mapping associates this pattern with the safe following distance rule, and considering the current weather condition of light rain, the system infers a high risk of following. Dynamic probability reasoning calculates the collision risk index to be 0.7, which exceeds the high risk threshold of 0.6. Finally, the system generates a first-level warning message, displays a warning "Please pay attention to the safe distance" on the dashboard, and slightly activates the braking system to create more sufficient reaction time for the driver. The entire process from data collection to warning generation takes only 0.5 seconds, providing drivers with timely and effective safety assistance.

[0032] In the embodiment of the present application, the raw data collected by the multi-source sensors on the vehicle are subjected to time alignment and spatial registration processing, which ensures the consistency of data from different sensors in the time and space dimensions, provides a reliable data basis for subsequent analysis, and effectively reduces the risk of misjudgment caused by data asynchrony or inconsistent spatial references. The outlier detection and feature extraction processing of multimodal vehicle safety situation data not only improves the data quality, eliminates noise and outliers that may affect the judgment, but also extracts the most representative and discriminative features, greatly improving the efficiency and accuracy of subsequent analysis. Then, through multimodal deep feature learning processing, the system can automatically learn and capture complex nonlinear feature patterns, greatly enhancing the ability to understand complex traffic scenes, making the assessment of safety status more comprehensive and in-depth. The knowledge mapping and context association processing of advanced vehicle safety semantic features organically combines data-driven feature learning with domain expert knowledge, improves the interpretability and generalization ability of the system, and makes the safety assessment results more reliable and practical. Subsequently, the semantically enhanced vehicle safety status representation is processed by dynamic probabilistic reasoning, so that the system can make reasonable risk assessments in uncertain environments and dynamically update risk estimates as new information is input, improving the system's ability to adapt to complex and changing environments. Finally, by performing threshold judgment and multi-level warning generation processing on multi-dimensional vehicle safety risk indicators, the system can generate targeted warning information according to different risk levels and types, avoiding the troubles that may be caused by excessive warnings and ensuring timely reminders in real dangerous situations. This multi-level and multi-angle safety hazard warning method not only comprehensively improves the accuracy, timeliness and reliability of warnings, but also makes warning information more intelligent and humanized through semantic understanding and knowledge fusion, greatly enhancing driving safety.

[0033] In a specific embodiment, the process of executing step S101 may specifically include the following steps:

[0034] (1) The raw data collected by the vehicle-mounted multi-source sensors are processed into a unified data format to obtain multi-source sensor data in a standard format, and the multi-source sensor data in the standard format are processed for timestamp extraction to obtain a multi-source sensor data sequence with a timestamp;

[0035] (2) Performing time window division processing on the multi-source sensor data series with time stamps to obtain data subsets of multiple time windows, and performing interpolation processing on the data subsets of multiple time windows to obtain data samples with equal time intervals;

[0036] (3) performing coordinate system conversion processing on data samples with equal time intervals to obtain multi-source sensor data in a unified coordinate system, and performing spatial registration processing on the multi-source sensor data in the unified coordinate system to obtain spatially aligned multi-source sensor data;

[0037] (4) performing noise suppression processing on the spatially aligned multi-source sensor data through a Kalman filter algorithm to obtain the denoised multi-source sensor data, and performing data fusion processing on the denoised multi-source sensor data to obtain preliminary fused multi-modal data;

[0038] (5) Performing feature extraction processing on the initially fused multimodal data to obtain a multidimensional feature vector, and performing feature selection processing on the multidimensional feature vector to obtain multimodal vehicle safety situation data.

[0039] Specifically, the raw data collected by the multi-source sensors on the vehicle are processed in a unified data format. Data from different sensors, such as image data from cameras, distance data from radars, and location data from GPS, are converted into a standard data format. The standardized data structure contains fields such as sensor type, data value, and time information. Subsequently, the multi-source sensor data in the standard format is subjected to timestamp extraction processing, and each piece of data is associated with its acquisition time to form a multi-source sensor data sequence with a timestamp. The multi-source sensor data sequence with a timestamp is divided into time windows. The size of the time window is determined according to the data acquisition frequency and application requirements, and is usually between several hundred milliseconds and several seconds. After division, multiple data subsets of time windows are obtained, each of which contains multi-source sensor data within a period of time. Then, these data subsets are interpolated, and the data of different sensors are aligned to the same time point using methods such as linear interpolation or spline interpolation to obtain data samples with equal time intervals.

[0040] Subsequently, the data samples with equal time intervals are processed by coordinate system conversion. Different sensors may use different coordinate systems, such as the image coordinate system for cameras, the geographic coordinate system for GPS, and the polar coordinate system for radars. Through the coordinate transformation matrix, all data are converted to the unified coordinate system of the vehicle, usually a Cartesian coordinate system with the center of the vehicle as the origin. After the conversion, the multi-source sensor data in the unified coordinate system are spatially registered to ensure that the same object observed by different sensors corresponds to the same spatial position, and the spatially aligned multi-source sensor data is obtained. The spatially aligned multi-source sensor data is subjected to noise suppression by the Kalman filter algorithm. Kalman filtering is a recursive filtering algorithm that estimates the optimal state value based on the dynamic model and observation model of the system and combines multiple measurement results. This step effectively reduces the random noise and measurement error in the sensor data and improves the reliability of the data. The multi-source sensor data after noise reduction is subjected to data fusion processing, and the characteristics and reliability of each sensor are comprehensively considered to obtain the preliminary fused multi-modal data.

[0041] Finally, the preliminarily fused multimodal data is subjected to feature extraction. Feature extraction includes the calculation of time domain features (such as mean, variance, peak, etc.) and frequency domain features (such as spectrum energy distribution, main frequency, etc.), as well as the extraction of more complex statistical features and morphological features. This step obtains a multidimensional feature vector, which contains rich vehicle status and environmental information. Subsequently, feature selection is performed on the multidimensional feature vector, using methods such as principal component analysis (PCA) or recursive feature elimination (RFE) to select the most representative and discriminative feature subsets, and finally obtain multimodal vehicle safety situation data.

[0042] For example, suppose a vehicle equipped with a forward-facing camera, millimeter-wave radar, and GPS is traveling on a highway. The camera captures an image frame every 0.1 seconds with a resolution of 1920x1080 pixels in RGB format; the millimeter-wave radar scans every 0.05 seconds with a measurement range of 0-200 meters and an angular resolution of 1 degree; the GPS updates the location information every 1 second with an accuracy of 5 meters. Data format unification converts these heterogeneous data into a unified JSON format, which contains sensor type, timestamp, and data value fields. Timestamp extraction associates each piece of data with the exact moment of collection with millisecond accuracy.

[0043] The time window is divided into 0.5 second units, dividing the data into continuous time segments. In each time window, the radar and GPS data are interpolated to the same 10Hz sampling rate as the camera through linear interpolation method to obtain data samples with equal time intervals. The coordinate system transformation converts the image coordinates of the camera, the polar coordinates of the radar, and the geographic coordinates of the GPS into a vehicle coordinate system with the center of the vehicle as the origin, the front as the positive direction of the x-axis, and the left as the positive direction of the y-axis. Spatial registration ensures the consistency of the position of the same object in different sensor data, for example, the position of the front vehicle in the camera image and the radar scan corresponds.

[0044] Kalman filtering is used to estimate the position and speed of the vehicle. Assume that the vehicle's motion model is uniform linear motion, and the observation model includes GPS position and radar ranging. Through the two steps of prediction and update, the Kalman filter continuously optimizes the state estimation and effectively suppresses the noise in GPS and radar measurements. Data fusion combines the filtered position and speed information with the lane line information recognized by the camera to form a more complete environmental perception result. Feature extraction calculates a series of features from the fused data, such as vehicle speed, acceleration, distance from the vehicle in front, relative speed, lane deviation degree, etc. In the feature selection process, the first few principal components that explain 95% of the variance are retained through principal component analysis, and the feature vectors that may have dozens or even hundreds of dimensions are reduced to about 10-20 dimensions, obtaining the final multimodal vehicle safety situation data.

[0045] In a specific embodiment, the process of executing step S102 may specifically include the following steps:

[0046] (1) Performing statistical analysis on the multimodal vehicle safety situation data to obtain data distribution characteristics, and performing outlier identification processing on the data distribution characteristics to obtain an outlier candidate set;

[0047] (2) Performing context correlation analysis on the outlier candidate set to obtain an outlier verification result, and performing outlier data correction on the outlier verification result to obtain cleaned security situation data;

[0048] (3) performing time domain feature extraction processing on the cleaned security situation data to obtain a time domain feature set, and performing frequency domain transformation processing on the time domain feature set to obtain a frequency domain feature set;

[0049] (4) Perform feature fusion processing on the time domain feature set and the frequency domain feature set to obtain a multi-dimensional feature vector, and perform dimensionality reduction processing on the multi-dimensional feature vector to obtain the vehicle-environment interaction feature.

[0050] Specifically, the multimodal vehicle safety situation data is statistically analyzed and processed. The statistics of each feature, including mean, variance, skewness and kurtosis, are calculated to obtain the data distribution characteristics. For continuous data, such as vehicle speed and acceleration, the mean and standard deviation are calculated; for discrete data, such as lane number, the frequency distribution is calculated. Then, the data distribution characteristics are processed for outlier identification, and the box plot method or Z-score method is used to identify potential outliers. The box plot method marks data points that exceed 1.5 times the interquartile range as abnormal, while the Z-score method regards data points that deviate from the mean by more than 3 standard deviations as abnormal. These methods obtain an outlier candidate set. The outlier candidate set is processed for contextual relevance analysis. This step considers the temporal continuity and spatial relevance of the data to determine whether the outlier is a real anomaly. For example, for vehicle speed data, an abnormally high speed at a single time point may be noise, but if high speed is shown at multiple consecutive time points, it may be a real speeding behavior. Context analysis also considers the consistency between different sensor data, such as the comparison of GPS speed and wheel speed sensor data. Through this analysis, the outlier verification results are obtained. The outlier verification results are processed for abnormal data correction, and the outliers confirmed as noise are interpolated or smoothed, while the data confirmed as real anomalies are retained for subsequent analysis. In this way, the cleaned security situation data is obtained.

[0051] Subsequently, the cleaned security situation data is processed for time domain feature extraction. Time domain features include statistical features (such as mean, variance, skewness, kurtosis), morphological features (such as number of peaks, zero crossing rate) and correlation features (such as autocorrelation coefficient, mutual correlation coefficient), etc. These features reflect the changing characteristics of the data in the time dimension, and a time domain feature set is obtained. At the same time, the time domain feature set is processed by frequency domain transformation, and the time domain signal is converted to the frequency domain using fast Fourier transform (FFT). Features such as power spectral density, main frequency, band energy ratio, etc. are extracted in the frequency domain to obtain a frequency domain feature set. Frequency domain features can capture the periodic changes and frequency composition of data. Finally, feature fusion processing is performed on the time domain feature set and the frequency domain feature set. Feature fusion adopts an early fusion strategy to directly splice the time domain and frequency domain features into a large feature vector. This method retains all the information of the original features, but also increases the feature dimension. Therefore, the multi-dimensional feature vector is subjected to dimensionality reduction processing, and methods such as principal component analysis (PCA) or linear discriminant analysis (LDA) are used to reduce the feature dimension while retaining the main information. PCA converts variables that may be correlated into linearly independent variables through orthogonal transformation, and selects several principal components with the largest variance contribution rate. In this way, the vehicle-environment interaction characteristics after dimensionality reduction are obtained, which not only retains the main information of the original data, but also reduces the complexity of subsequent processing.

[0052] For example, suppose a vehicle equipped with multiple sensors is traveling on a highway. The onboard system collects data every 0.1 seconds, including GPS speed, accelerometer readings, steering wheel angle, brake pedal position, etc. For the GPS speed data, first calculate its mean and standard deviation. Assume that the mean is 100km / h and the standard deviation is 10km / h. Use the Z-score method to detect outliers, and mark data points that deviate from the mean by more than 3 standard deviations as outliers. For example, if the speed at a certain moment is 135km / h, its Z-score is (135-100) / 10=3.5, which exceeds the threshold of 3, so it is marked as an outlier candidate. Perform context correlation analysis on this outlier and check the speed data within 1 second before and after. If it is found that the speed gradually increases from 100km / h to 135km / h and remains at a high speed in the following seconds, then this "outlier" is likely to be a real speeding behavior rather than a measurement error. Therefore, this data point is retained during the abnormal data correction stage.

[0053] In the time domain feature extraction stage, the statistical features of the speed data, such as the mean, variance, skewness, and kurtosis, are calculated. At the same time, the mean and maximum values ​​of the speed change rate (acceleration) are calculated. These features reflect the dynamic behavior of the vehicle. For the steering wheel angle data, the mean and variance of its change rate are calculated to reflect the driver's operating characteristics. In the frequency domain transformation processing, the speed data is subjected to a fast Fourier transform to obtain its spectrum. The main frequency components and the energy distribution of each frequency band are extracted from the spectrum. For example, a significant peak may be found near 0.1 Hz, which may correspond to the acceleration-deceleration cycle of the vehicle every 10 seconds or so.

[0054] Feature fusion combines time domain and frequency domain features to form a high-dimensional feature vector. Assuming that the original feature vector has 100 dimensions, PCA dimensionality reduction can reduce the feature dimension to about 20 by retaining the principal components required to explain 95% of the variance. This reduced feature vector is the final vehicle-environment interaction feature, which integrates the vehicle's motion state, the driver's operating characteristics, and the time variation mode of these characteristics.

[0055] In a specific embodiment, the process of executing step S103 may specifically include the following steps:

[0056] (1) Perform multimodal data alignment on vehicle-environment interaction features to obtain aligned multimodal features, and perform feature mapping on the aligned multimodal features to obtain a unified feature space representation;

[0057] (2) Perform nonlinear transformation on the representation of the unified feature space to obtain a deep feature representation, and perform attention mechanism processing on the deep feature representation to obtain a weighted feature representation;

[0058] (3) performing multi-scale feature extraction processing on the weighted feature representation to obtain a multi-scale feature set, and performing feature aggregation processing on the multi-scale feature set to obtain a fused feature representation;

[0059] (4) The fused feature representation is reconstructed through a self-encoding algorithm to obtain reconstructed features, and the reconstructed features are semantically annotated to obtain high-level vehicle safety semantic features.

[0060] Specifically, the vehicle-environment interaction features are subjected to multimodal data alignment. This solves the problem of inconsistency between different sensor data in time and space. Temporal alignment uses interpolation technology to adjust all sensor data to a unified timestamp sequence. Spatial alignment uses coordinate transformation to convert the spatial reference system of different sensors into the vehicle coordinate system. The aligned multimodal features ensure the consistency and comparability of data from different sources. Subsequently, feature mapping is performed on the aligned multimodal features to project the features of different modalities into a unified feature space. This step usually uses neural network structures such as multilayer perceptrons or autoencoders to learn the nonlinear mapping relationship between different modal features and obtain a representation of a unified feature space. The representation of the unified feature space is subjected to nonlinear transformation. This step uses the powerful nonlinear fitting ability of deep neural networks to capture the complex interactions between features. The specific implementation can use a multilayer feedforward neural network, each layer of which contains a nonlinear activation function such as ReLU (Rectified Linear Unit). Through multilayer nonlinear transformations, the network can learn more abstract and advanced feature representations and obtain deep feature representations. Then, the deep feature representation is processed by the attention mechanism. The attention mechanism allows the model to dynamically focus on the importance of different features, which is particularly effective when processing sequence data or multimodal data. Here, the self-attention mechanism can be used to calculate the correlation between features and assign attention weights accordingly. By weighting the attention, a weighted feature representation is obtained, which highlights the features that are more important for judging the current security status.

[0061] Subsequently, the weighted feature representation is subjected to multi-scale feature extraction. Multi-scale feature extraction aims to capture patterns at different temporal and spatial scales. This can be achieved by using convolution kernels of different sizes or time windows of different lengths. For example, for time series data, one-dimensional convolutions of different lengths can be used to extract short-term, medium-term, and long-term feature patterns. For spatial data, such as image data, two-dimensional convolution kernels of different sizes can be used. The resulting multi-scale feature set contains various features from fine-grained to coarse-grained. Next, the multi-scale feature set is subjected to feature aggregation. The purpose of feature aggregation is to effectively combine features of different scales to form a comprehensive feature representation. This can be achieved by feature concatenation or weighted summation. The result of feature aggregation is a fused feature representation that integrates information from multiple scales. Finally, the fused feature representation is reconstructed by an autoencoder algorithm. An autoencoder is an unsupervised learning algorithm that attempts to learn an identity function so that the output is as similar to the input as possible. An autoencoder consists of two parts: an encoder and a decoder. The encoder compresses the input to a low-dimensional representation (called the encoding), and the decoder attempts to reconstruct the original input from the encoding. By minimizing the reconstruction error, the autoencoder is able to learn the intrinsic structure and most important features of the data. This step results in the reconstructed features, which are a more compact and information-rich representation of the original features. Finally, the reconstructed features are semantically annotated. The purpose of semantic annotation is to associate low-level features with high-level semantic concepts. This can be achieved through pre-trained classification models or clustering algorithms. For example, a pre-trained deep learning model can be used to map features to predefined semantic categories, or an unsupervised clustering algorithm can be used to discover natural groupings of features and assign semantic meaning to each grouping. Through semantic annotation, advanced vehicle safety semantic features are obtained, which not only contain the information of the original data, but also carry a higher level of semantic understanding.

[0062] For example, suppose an intelligent vehicle equipped with a forward-facing camera, millimeter-wave radar, GPS, and inertial measurement unit (IMU) is driving on a highway. The camera captures a 1920x1080 RGB image frame every 0.1 seconds, the radar scans the surrounding environment every 0.05 seconds, the GPS updates the position information every second, and the IMU provides acceleration and angular velocity data at a frequency of 100Hz. First, these heterogeneous data are time-aligned to synchronize all data to a unified time series of 10Hz. Spatial alignment transforms the object position detected by the radar and the GPS position information into a coordinate system with the center of the vehicle as the origin. The feature mapping step uses a multi-layer perceptron to map image features (extracted by a pre-trained CNN), radar point cloud features, GPS position, and IMU motion features into a 256-dimensional unified feature space.

[0063] The nonlinear transformation uses a 4-layer feedforward neural network with ReLU activation function in each layer to gradually transform the 256-dimensional features into a 512-dimensional deep feature representation. The attention mechanism calculates the correlation matrix between these 512 features and assigns weights to each feature based on it. For example, when a vehicle changes lanes, features related to lateral distance and lane lines may receive higher attention weights. Multi-scale feature extraction uses one-dimensional convolution kernels of different sizes (such as 3, 5, 7) on weighted features to extract temporal patterns, and two-dimensional convolution kernels of different sizes (such as 3x3, 5x5, 7x7) on image data to extract spatial patterns. The resulting multi-scale features contain a variety of information from instantaneous states to long-term trends. Feature aggregation concatenates these multi-scale features to form a longer feature vector, assuming a length of 1024.

[0064] The encoder part of the autoencoder compresses the 1024-dimensional fused features to 128 dimensions, and the decoder attempts to reconstruct the original 1024-dimensional features from these 128 dimensions. Through repeated training, the autoencoder learns the 128 key features that best represent the original data. Finally, semantic annotation uses a pre-trained multi-label classifier to map these 128-dimensional features to predefined safety-related semantic categories, such as "lane keeping status", "safety of distance from the vehicle in front", "vehicle speed adaptability", etc. The resulting high-level vehicle safety semantic features not only encode the information of the original sensor data, but also contain the semantic meaning of this data in safety assessment.

[0065] In a specific embodiment, the process of executing step S104 may specifically include the following steps:

[0066] (1) Performing ontology matching on the advanced vehicle safety semantic features to obtain semantic concept mapping results, and performing knowledge graph entity linking on the semantic concept mapping results to obtain an entity association network;

[0067] (2) Performing relational reasoning on the entity association network to obtain a semantic relationship graph, and performing context expansion on the semantic relationship graph to obtain an extended semantic network;

[0068] (3) Perform subgraph extraction processing on the extended semantic network to obtain a set of related subgraphs, and perform semantic aggregation processing on the set of related subgraphs to obtain a comprehensive semantic representation;

[0069] (4) Performing temporal association processing on the comprehensive semantic representation to obtain a temporal semantic sequence, and performing state encoding processing on the temporal semantic sequence to obtain a semantically enhanced vehicle safety status representation.

[0070] Specifically, the advanced vehicle safety semantic features are processed by ontology matching. Ontology matching is the process of matching data features with a predefined concept system. In the field of vehicle safety, the ontology includes concept categories such as vehicle status, road environment, and driving behavior. The matching process uses semantic similarity algorithms, such as cosine similarity or word embedding models, to compare feature vectors with ontology concepts to obtain semantic concept mapping results. For example, speed features may be mapped to the concept of "vehicle dynamics", while steering wheel angle features may be mapped to the concept of "driving operation". Subsequently, the semantic concept mapping results are processed by knowledge graph entity linking. A knowledge graph is a structured knowledge base containing entities, attributes, and relationships. Entity linking associates the mapped concepts with specific entity nodes in the knowledge graph. This step uses an entity disambiguation algorithm to consider contextual information and determine the most appropriate entity link. Through entity linking, an entity association network is obtained, which is a graph structure with entities as nodes and relationships as edges. Relational reasoning is performed on the entity association network. Relational reasoning aims to discover implicit and indirect relationships between entities. This step uses graph traversal algorithms, such as depth-first search or breadth-first search, to explore paths between entities. At the same time, inference rules, such as transitivity rules (if A is related to B and B is related to C, then A may be related to C), are applied to discover potential relationships. The result of inference forms a semantic relationship graph, which not only contains directly observed relationships, but also includes implicit relationships derived from reasoning. The semantic relationship graph is then processed for context extension. Context extension takes into account a wider range of situational information, such as time, location, weather, etc. This step uses graph extension algorithms, such as random walks or PageRank variants, to add relevant contextual nodes and edges to the original graph structure. The extension process takes into account the importance and relevance of the nodes to obtain an extended semantic network, which provides a more comprehensive semantic context.

[0071] Subsequently, the extended semantic network is processed for subgraph extraction. The purpose of subgraph extraction is to obtain the most relevant parts of the current situation from a complex network. This step uses subgraph isomorphism algorithms or graph pattern matching techniques to extract relevant subgraphs from the extended semantic network based on predefined important patterns or core entities of current concern. The extraction process considers factors such as the importance of nodes, the weight of edges, and the length of paths to obtain a set of relevant subgraphs. This set contains multiple small-scale semantic networks that are highly relevant to the current security situation. Next, the set of relevant subgraphs is processed for semantic aggregation. Semantic aggregation combines the information of multiple subgraphs to form a unified semantic representation. This step uses graph embedding techniques such as DeepWalk or Node2Vec to convert each subgraph into a vector representation of fixed dimensions. Then, these vectors are aggregated into a comprehensive semantic representation through weighted averaging or attention mechanisms. This representation not only retains the key information of each subgraph, but also provides a global semantic perspective. Finally, the comprehensive semantic representation is processed for temporal association. Temporal association takes into account the changes of semantic information over time. This step uses a recurrent neural network (RNN) or a long short-term memory network (LSTM) to take the comprehensive semantic representation at continuous time points as the input sequence. The network learns to capture the temporal dependency of semantic information and obtains a temporal semantic sequence. This sequence not only contains the semantic information of the current moment, but also encodes the impact of historical information on the current state. Finally, the temporal semantic sequence is state-encoded. State encoding converts the temporal information into a compact state representation. This step can use an attention mechanism or a gating mechanism to extract the most critical information from the temporal sequence. Finally, a semantically enhanced vehicle safety state representation is obtained, which integrates multi-source sensor data, knowledge graph information, and temporal dependencies, providing a rich semantic basis for safety hazard warnings.

[0072] For example, suppose an intelligent vehicle is driving on a highway. The vehicle system generates high-level vehicle safety semantic features once a second, including vehicle speed, acceleration, steering wheel angle, lane deviation degree, etc. Ontology matching maps these features to predefined concepts, such as mapping a vehicle speed of 80km / h to the concept of "medium speed driving". Knowledge graph entity linking associates "medium speed driving" with specific entity nodes in the knowledge base, such as "Highway_Speed_State_1". Relational reasoning may find that "medium speed driving" is associated with "safe following distance" because speed affects the judgment of safe following distance. Context extension further considers that it is currently raining (from the weather sensor) and adds the "driving in rainy weather" node to the network. Subgraph extraction extracts the most relevant parts of the extended network that are most relevant to the current driving state, including nodes such as "medium speed driving", "safe following distance", "driving in rainy weather" and their relationships. Assume that 5 relevant subgraphs are extracted, each containing 10-15 nodes. Semantic aggregation uses graph embedding technology to convert each subgraph into a 128-dimensional vector, and then aggregates these 5 vectors into a comprehensive semantic representation through the attention mechanism. Temporal association processing considers the semantic representation sequence of the past 10 seconds. The LSTM network receives the 128-dimensional vectors of these 10 time steps as input and outputs a 256-dimensional vector that encodes the temporal information. Finally, state encoding uses a multi-head attention mechanism to extract the most critical information from the 256-dimensional vector and generate a 64-dimensional semantically enhanced vehicle safety state representation. This final representation not only includes the safety meaning of the current vehicle speed on rainy days, but also takes into account the temporal patterns such as speed changes and following distance adjustments in the past 10 seconds, providing a rich semantic and temporal information basis for subsequent safety hazard warnings.

[0073] In a specific embodiment, the process of executing step S105 may specifically include the following steps:

[0074] (1) Performing probability distribution estimation processing on the semantically enhanced vehicle safety state representation to obtain the state probability distribution, and performing conditional probability calculation processing on the state probability distribution to obtain a conditional probability table;

[0075] (2) performing Bayesian network construction processing on the conditional probability table to obtain an initial Bayesian network structure, and performing parameter learning processing on the initial Bayesian network structure to obtain a parameterized Bayesian network;

[0076] (3) Perform evidence propagation processing on the parameterized Bayesian network to obtain the posterior probability distribution, and perform risk factor identification processing on the posterior probability distribution to obtain a set of key risk factors;

[0077] (4) Perform risk quantification processing on the key risk factor set to obtain a risk score vector, and perform multi-dimensional aggregation processing on the risk score vector to obtain a multi-dimensional vehicle safety risk index.

[0078] Specifically, the semantically enhanced vehicle safety state representation is processed for probability distribution estimation. The kernel density estimation (KDE) method is used to convert the discrete state representation into a continuous probability distribution. KDE obtains a smooth probability density function by placing a kernel function (usually a Gaussian kernel) around each data point and then adding these kernel functions. The state probability distribution obtained in this way reflects the possibility of different safety states. Subsequently, the state probability distribution is processed for conditional probability calculation to calculate the probability of a specific state occurring under certain conditions. This step uses Bayes' theorem to consider the mutual influence between various factors and obtain a conditional probability table. The conditional probability table describes the dependency between different safety factors and provides a basis for subsequent risk assessment. The conditional probability table is processed for Bayesian network construction. Bayesian network is a probabilistic graph model that uses a directed acyclic graph to represent the conditional dependency between variables. The construction process first determines the nodes of the network, each node represents a safety-related variable, such as vehicle speed, road conditions, driving behavior, etc. Then, directed edges are added between nodes based on the dependency in the conditional probability table. This step uses a structural learning algorithm, such as the K2 algorithm or greedy search, to obtain the initial Bayesian network structure. Subsequently, the initial Bayesian network structure is processed by parameter learning. Parameter learning is to determine the conditional probability distribution of each node in the network. This step uses the maximum likelihood estimation or expectation maximization (EM) algorithm to estimate the parameters of each node based on historical data. Through parameter learning, a parameterized Bayesian network is obtained, which contains not only the network structure, but also the specific probability parameters of each node.

[0079] Then, the parameterized Bayesian network is processed for evidence propagation. Evidence propagation is the process of updating the probability distribution of other nodes in the network given some observation evidence. This step uses message passing algorithms, such as variable elimination or belief propagation algorithms. When new sensor data is input, these data are input into the network as evidence, triggering the propagation of probability updates. Through evidence propagation, the posterior probability distribution is obtained, which reflects the latest probability estimates of each safety factor under the current observation conditions. Subsequently, the posterior probability distribution is processed for risk factor identification. This step uses sensitivity analysis methods, such as variance contribution analysis or influence analysis, to identify the factors that have the greatest impact on safety risks. By comparing the impact of probability changes of different factors on the overall risk, a set of key risk factors is obtained. This set contains several factors that have the most significant impact on the current safety situation. Finally, the set of key risk factors is processed for risk quantification. Risk quantification converts qualitative risk factors into quantitative risk scores. This step uses risk scoring models, such as linear weighted models or fuzzy comprehensive evaluation methods, to assign a numerical score to each risk factor. The calculation of the score takes into account the importance of the factor, the degree of deviation from the current state, and the potential scope of impact. This results in a risk score vector, where each component corresponds to a quantitative score of a key risk factor. Finally, the risk score vector is subjected to multi-dimensional aggregation. Multi-dimensional aggregation aims to integrate multiple risk dimensions into an overall risk assessment. This step can use weighted summation, geometric mean, or more complex nonlinear aggregation methods. The aggregation process takes into account the relative importance and potential interactions of different risk dimensions, and finally obtains a multi-dimensional vehicle safety risk index. This index is a comprehensive risk score that reflects the current overall safety status of the vehicle.

[0080] For example, suppose an intelligent vehicle is driving on a highway. The semantically enhanced representation of the vehicle's safety state contains information in multiple dimensions, such as vehicle speed, distance from the vehicle in front, lane deviation, driver fatigue, etc. The KDE method is used to estimate the probability distribution of these states and obtain the probability density function of each dimension. For example, the probability distribution of vehicle speed may be normally distributed with a mean of 100km / h and a standard deviation of 10km / h. Conditional probability calculation takes into account the mutual influence of these factors, such as calculating the probability that the distance from the vehicle in front is less than the safe distance under the condition of a speed of 120km / h. Bayesian network construction takes these factors as nodes and adds edges based on the dependencies between them. For example, the vehicle speed node is connected to the distance from the vehicle in front node, indicating that the vehicle speed affects the judgment of the safe following distance. Parameter learning estimates the conditional probability table of each node based on historical data. For example, it may be learned that in high-speed (>100km / h) situations, a safe distance greater than 50m needs to be maintained in 70% of cases.

[0081] During the process of evidence propagation, when new sensor data shows that the current vehicle speed is 130km / h and the distance to the vehicle in front is 40m, this information is input into the network as evidence to update the probabilities of other nodes. Through sensitivity analysis, it is identified that excessive speed and close following distance are the two most critical risk factors at present. Risk quantification converts these two factors into specific risk scores. Assuming a scoring scale of 0-10, the risk score for excessive speed is 8 (very high) and the risk score for close following distance is 7 (high). Multi-dimensional aggregation considers the combined impact of these two factors and may use a weighted average method, assuming that the speed weight is 0.6 and the distance weight is 0.4, resulting in a final risk index of 7.6.

[0082] In a specific embodiment, the process of executing step S106 may specifically include the following steps:

[0083] (1) Normalizing the multi-dimensional vehicle safety risk indicators to obtain standardized risk indicators, and performing multi-dimensional threshold setting processing on the standardized risk indicators to obtain risk level classification standards;

[0084] (2) Perform risk level determination processing on the standardized risk indicators to obtain preliminary risk level results, and perform time series smoothing processing on the preliminary risk level results to obtain a smoothed risk level sequence;

[0085] (3) Perform trend analysis on the smoothed risk level sequence to obtain the risk change trend, and determine the warning level of the risk change trend to obtain a multi-level warning result;

[0086] (4) Performing warning information generation processing on the multi-level warning results to obtain a warning description text, and performing user interface adaptation processing on the warning description text to obtain the real-time vehicle safety hazard warning information.

[0087] Specifically, the multi-dimensional vehicle safety risk indicators are normalized. Normalization is the process of converting indicators of different scales to the same scale range, usually using the minimum-maximum normalization or Z-score normalization method. This step ensures that risk indicators of different dimensions can be compared fairly. After normalization, a standardized risk indicator is obtained, and its value is usually limited to between 0 and 1. Subsequently, the standardized risk indicator is subjected to multi-dimensional threshold setting. Threshold setting is based on historical data analysis, expert experience and safety standards to determine the boundaries of multiple risk levels for each risk dimension. Generally, risk levels can be divided into three levels of low, medium and high or more detailed multi-level divisions. The output of this step is the risk level division standard, which defines the risk level corresponding to each risk dimension within different numerical ranges. The standardized risk indicator is subjected to risk level determination. This step compares the standardized value of each risk dimension with the previously set threshold to determine the risk level to which it belongs. The determination process takes into account multiple risk dimensions, and may use weighted voting or fuzzy logic to synthesize the risk levels of each dimension. In this way, a preliminary risk level result is obtained, which reflects the risk status at the current moment. Then, the preliminary risk level results are smoothed in time series. Time series smoothing aims to reduce short-term fluctuations in risk level determination and improve the stability of early warning. This step can use algorithms such as moving average, exponential smoothing or Kalman filtering to consider the changes in risk levels over a period of time. Through time series smoothing, a smoothed risk level sequence is obtained, which reflects both the current risk status and the recent risk change trend.

[0088] Subsequently, the smoothed risk level sequence is processed for trend analysis. Trend analysis aims to identify the change pattern of risk level over time, such as an upward trend, a downward trend or a fluctuation pattern. This step can use methods such as linear regression, ARIMA model or time series decomposition to analyze the long-term trend and cyclical changes of risk level. Through trend analysis, the risk change trend is obtained, which describes the dynamic change characteristics of the risk level. Next, the risk change trend is processed for early warning level determination. The determination of the early warning level not only considers the current risk level, but also the speed of risk change and possible future development. For example, a medium-risk but rapidly rising situation may require a higher level of early warning. This step may use methods such as decision trees or fuzzy rules to map risk levels and change trends to predefined early warning levels. Finally, a multi-level early warning result is obtained, which includes the urgency and specific type of the early warning.

[0089] Finally, the multi-level warning results are processed for warning information generation. The numerical warning results are converted into text descriptions that can be understood by humans. Warning information generation can use template filling or natural language generation technology to generate corresponding warning texts, recommended measures, etc. according to different warning levels and types. In this way, the warning description text is obtained, which details the current risk situation and the recommended actions. Finally, the warning description text is adapted to the user interface. Interface adaptation takes into account the characteristics of different display devices (such as vehicle-mounted display screens, mobile phone apps), as well as the needs of different usage scenarios (such as driving and parking). This step may include text formatting, color coding, icon selection, etc., to ensure that the warning information is presented to the user in the most appropriate way. Through interface adaptation, real-time vehicle safety hazard warning information is finally obtained, which is an intuitive, easy-to-understand safety prompt suitable for the current usage environment.

[0090] For example, suppose an intelligent vehicle is driving on a highway, and its multidimensional safety risk indicators include speeding risk, following distance risk, and lane departure risk. The original risk indicators are 75, 60, and 40 points (the full score is 100, and the higher the score, the greater the risk). First, the minimum-maximum normalization process is performed. Assuming that the minimum value of the historical data is 0 and the maximum value is 100, the standardized risk indicators are 0.75, 0.60, and 0.40, respectively. The multidimensional threshold setting divides the risk level into three levels: low (0-0.3), medium (0.3-0.7), and high (0.7-1.0). The risk level determination is based on the standardized indicators and thresholds, and the preliminary risk level results are obtained: speeding risk is high, following distance risk is medium, and lane departure risk is medium. Time series smoothing considers the risk level of the past 10 seconds and uses the simple moving average method. Assuming that the average risk level of the first 9 seconds is medium, the current risk level after smoothing is still medium.

[0091] The trend analysis process uses linear regression to analyze the risk level sequence of the last 60 seconds. Assume that the analysis results show that the speeding risk and the following distance risk are on an upward trend, with slopes of 0.01 and 0.008 per second, respectively, while the lane departure risk remains stable. The warning level determination comprehensively considers the current risk level (medium) and the upward trend, and raises the warning level to "high-level warning". The warning information generation process converts this data into specific text: "High-level warning: The vehicle speed is too fast and there is a trend of acceleration. At the same time, the distance to the vehicle in front is reduced, and there is a potential risk of collision. It is recommended to slow down immediately and increase the following distance." Finally, the user interface adaptation displays this warning in red font in the center of the vehicle screen, and triggers a sound prompt and a slight vibration of the steering wheel to ensure that the driver can pay attention to this important safety warning information in time.

[0092] The above describes the vehicle safety hazard warning method based on multi-sensor fusion in the embodiment of the present application. The following describes the vehicle safety hazard warning device based on multi-sensor fusion in the embodiment of the present application. Figure 2 In the embodiment of the present application, an embodiment of a vehicle safety hazard warning device based on multi-sensor fusion includes:

[0093] The registration module 201 is used to perform time alignment and spatial registration processing on the raw data collected by the vehicle-mounted multi-source sensors to obtain multi-modal vehicle safety situation data;

[0094] An extraction module 202 is used to perform outlier detection and feature extraction processing on the multimodal vehicle safety situation data to obtain vehicle-environment interaction features;

[0095] A processing module 203 is used to perform multimodal deep feature learning processing on the vehicle-environment interaction features to obtain high-level vehicle safety semantic features;

[0096] An association module 204 is used to perform knowledge mapping and context association processing on the advanced vehicle safety semantic features to obtain a semantically enhanced vehicle safety status representation;

[0097] A reasoning module 205, configured to perform dynamic probability reasoning processing on the semantically enhanced vehicle safety status representation to obtain a multi-dimensional vehicle safety risk index;

[0098] The generation module 206 is used to perform threshold judgment and multi-level warning generation processing on the multi-dimensional vehicle safety risk index to obtain real-time vehicle safety hazard warning information.

[0099] Through the collaboration of the above components, the raw data collected by the multi-source sensors on the vehicle are time-aligned and spatially registered, ensuring the consistency of data from different sensors in the temporal and spatial dimensions, providing a reliable data basis for subsequent analysis, and effectively reducing the risk of misjudgment caused by data asynchrony or inconsistent spatial references. Outlier detection and feature extraction processing of multimodal vehicle safety situation data not only improves data quality, eliminates noise and outliers that may affect judgment, but also extracts the most representative and discriminative features, greatly improving the efficiency and accuracy of subsequent analysis. Then, through multimodal deep feature learning processing, the system can automatically learn and capture complex nonlinear feature patterns, greatly enhancing the ability to understand complex traffic scenes, making the assessment of safety status more comprehensive and in-depth. The knowledge mapping and context association processing of advanced vehicle safety semantic features organically combines data-driven feature learning with domain expert knowledge, improves the interpretability and generalization ability of the system, and makes the safety assessment results more reliable and practical. Subsequently, the semantically enhanced vehicle safety status representation is processed by dynamic probabilistic reasoning, so that the system can make reasonable risk assessments in uncertain environments and dynamically update risk estimates as new information is input, improving the system's ability to adapt to complex and changing environments. Finally, by performing threshold judgment and multi-level warning generation processing on multi-dimensional vehicle safety risk indicators, the system can generate targeted warning information according to different risk levels and types, avoiding the troubles that may be caused by excessive warnings and ensuring timely reminders in real dangerous situations. This multi-level and multi-angle safety hazard warning method not only comprehensively improves the accuracy, timeliness and reliability of warnings, but also makes warning information more intelligent and humanized through semantic understanding and knowledge fusion, greatly enhancing driving safety.

[0100] Based on the same technical concept, the embodiment of the present application also provides an electronic device. Figure 3 As shown, it is a schematic diagram of the structure of the electronic device 300 provided in the embodiment of the present application, including a processor 301, a memory 302, and a bus 303. Among them, the memory 302 is used to store execution instructions, including a memory 3021 and an external memory 3022; the memory 3021 here is also called an internal memory, which is used to temporarily store the operation data in the processor 301, and the data exchanged with the external memory 3022 such as a hard disk. The processor 301 exchanges data with the external memory 3022 through the memory 3021. When the electronic device 300 is running, the processor 301 communicates with the memory 302 through the bus 303.

[0101] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the vehicle safety hazard warning method based on multi-sensor fusion.

[0102] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0103] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0104] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A vehicle safety hazard early warning method based on multi-sensor fusion, characterized in that: The vehicle safety hazard early warning method based on multi-sensor fusion includes: Perform time alignment and spatial registration processing on the raw data collected by multi-source sensors on the vehicle to obtain multi-modal vehicle safety situation data; Performing outlier detection and feature extraction processing on the multimodal vehicle safety situation data to obtain vehicle-environment interaction features; Performing multimodal deep feature learning processing on the vehicle-environment interaction features to obtain high-level vehicle safety semantic features; Performing knowledge mapping and context association processing on the advanced vehicle safety semantic features to obtain a semantically enhanced vehicle safety status representation; Performing dynamic probability reasoning processing on the semantically enhanced vehicle safety status representation to obtain a multi-dimensional vehicle safety risk index; Threshold judgment and multi-level warning generation processing are performed on the multi-dimensional vehicle safety risk indicators to obtain real-time vehicle safety hazard warning information.

2. The vehicle safety hazard early warning method based on multi-sensor fusion according to claim 1 is characterized in that: The raw data collected by the vehicle-mounted multi-source sensor is subjected to time alignment and spatial registration processing to obtain multi-modal vehicle safety situation data, including: Performing data format unification processing on the raw data collected by the vehicle-mounted multi-source sensor to obtain multi-source sensor data in a standard format, and performing timestamp extraction processing on the multi-source sensor data in the standard format to obtain a multi-source sensor data sequence with a timestamp; Performing time window division processing on the multi-source sensor data sequence with time stamps to obtain data subsets of multiple time windows, and performing interpolation processing on the data subsets of the multiple time windows to obtain data samples with equal time intervals; Performing coordinate system conversion processing on the data samples with equal time intervals to obtain multi-source sensor data in a unified coordinate system, and performing spatial registration processing on the multi-source sensor data in the unified coordinate system to obtain spatially aligned multi-source sensor data; Performing noise suppression processing on the spatially aligned multi-source sensor data by using a Kalman filter algorithm to obtain denoised multi-source sensor data, and performing data fusion processing on the denoised multi-source sensor data to obtain preliminarily fused multi-modal data; The preliminarily fused multimodal data is subjected to feature extraction processing to obtain a multidimensional feature vector, and the multidimensional feature vector is subjected to feature selection processing to obtain the multimodal vehicle safety situation data.

3. The vehicle safety hazard early warning method based on multi-sensor fusion according to claim 1 is characterized in that: The performing of outlier detection and feature extraction processing on the multimodal vehicle safety situation data to obtain vehicle-environment interaction features includes: Performing statistical analysis on the multimodal vehicle safety situation data to obtain data distribution characteristics, and performing outlier identification on the data distribution characteristics to obtain an outlier candidate set; Performing context correlation analysis on the outlier candidate set to obtain an outlier verification result, and performing outlier data correction on the outlier verification result to obtain cleansed security situation data; Performing time domain feature extraction processing on the cleaned security situation data to obtain a time domain feature set, and performing frequency domain transformation processing on the time domain feature set to obtain a frequency domain feature set; The time domain feature set and the frequency domain feature set are subjected to feature fusion processing to obtain a multi-dimensional feature vector, and the multi-dimensional feature vector is subjected to dimensionality reduction processing to obtain the vehicle-environment interaction feature.

4. The vehicle safety hazard early warning method based on multi-sensor fusion according to claim 1 is characterized in that: The multimodal deep feature learning process is performed on the vehicle-environment interaction features to obtain high-level vehicle safety semantic features, including: Performing multimodal data alignment processing on the vehicle-environment interaction features to obtain aligned multimodal features, and performing feature mapping processing on the aligned multimodal features to obtain a representation of a unified feature space; Performing nonlinear transformation processing on the representation of the unified feature space to obtain a deep feature representation, and performing attention mechanism processing on the deep feature representation to obtain a weighted feature representation; Performing multi-scale feature extraction processing on the weighted feature representation to obtain a multi-scale feature set, and performing feature aggregation processing on the multi-scale feature set to obtain a fused feature representation; The fused feature representation is subjected to feature reconstruction processing by a self-encoding algorithm to obtain a reconstructed feature, and the reconstructed feature is subjected to semantic annotation processing to obtain the advanced vehicle safety semantic feature.

5. The vehicle safety hazard early warning method based on multi-sensor fusion according to claim 1 is characterized in that: The performing knowledge mapping and context association processing on the advanced vehicle safety semantic features to obtain a semantically enhanced vehicle safety status representation includes: Performing ontology matching processing on the advanced vehicle safety semantic features to obtain a semantic concept mapping result, and performing knowledge graph entity linking processing on the semantic concept mapping result to obtain an entity association network; Performing relational reasoning processing on the entity association network to obtain a semantic relationship graph, and performing context expansion processing on the semantic relationship graph to obtain an extended semantic network; Performing subgraph extraction processing on the extended semantic network to obtain a related subgraph set, and performing semantic aggregation processing on the related subgraph set to obtain a comprehensive semantic representation; The comprehensive semantic representation is subjected to temporal association processing to obtain a temporal semantic sequence, and the temporal semantic sequence is subjected to state encoding processing to obtain the semantically enhanced vehicle safety state representation.

6. The vehicle safety hazard early warning method based on multi-sensor fusion according to claim 1 is characterized in that: The dynamic probability reasoning process is performed on the semantically enhanced vehicle safety status representation to obtain a multi-dimensional vehicle safety risk index, including: Performing probability distribution estimation processing on the semantically enhanced vehicle safety state representation to obtain a state probability distribution, and performing conditional probability calculation processing on the state probability distribution to obtain a conditional probability table; Performing a Bayesian network construction process on the conditional probability table to obtain an initial Bayesian network structure, and performing a parameter learning process on the initial Bayesian network structure to obtain a parameterized Bayesian network; Performing evidence propagation processing on the parameterized Bayesian network to obtain a posterior probability distribution, and performing risk factor identification processing on the posterior probability distribution to obtain a set of key risk factors; The key risk factor set is subjected to risk quantification processing to obtain a risk score vector, and the risk score vector is subjected to multi-dimensional aggregation processing to obtain the multi-dimensional vehicle safety risk index.

7. The vehicle safety hazard early warning method based on multi-sensor fusion according to claim 1 is characterized in that: The threshold judgment and multi-level warning generation processing are performed on the multi-dimensional vehicle safety risk indicators to obtain real-time vehicle safety hazard warning information, including: Normalizing the multi-dimensional vehicle safety risk index to obtain a standardized risk index, and performing multi-dimensional threshold setting processing on the standardized risk index to obtain a risk level classification standard; Performing risk level determination processing on the standardized risk indicator to obtain a preliminary risk level result, and performing time series smoothing processing on the preliminary risk level result to obtain a smoothed risk level sequence; Performing trend analysis on the smoothed risk level sequence to obtain a risk change trend, and performing warning level determination on the risk change trend to obtain a multi-level warning result; The multi-level warning results are processed for generating warning information to obtain a warning description text, and the warning description text is processed for user interface adaptation to obtain the real-time vehicle safety hazard warning information.

8. A vehicle safety hazard warning device based on multi-sensor fusion, used to implement the vehicle safety hazard warning method based on multi-sensor fusion as described in any one of claims 1 to 7, characterized in that: The vehicle safety hazard warning device based on multi-sensor fusion includes: The registration module is used to perform time alignment and spatial registration processing on the raw data collected by the multi-source sensors on the vehicle to obtain multi-modal vehicle safety situation data; An extraction module, used to perform outlier detection and feature extraction processing on the multimodal vehicle safety situation data to obtain vehicle-environment interaction features; A processing module, used for performing multimodal deep feature learning processing on the vehicle-environment interaction features to obtain high-level vehicle safety semantic features; An association module, used for performing knowledge mapping and context association processing on the advanced vehicle safety semantic features to obtain a semantically enhanced vehicle safety status representation; A reasoning module, used for performing dynamic probabilistic reasoning processing on the semantically enhanced vehicle safety status representation to obtain a multi-dimensional vehicle safety risk index; The generation module is used to perform threshold judgment and multi-level warning generation processing on the multi-dimensional vehicle safety risk indicators to obtain real-time vehicle safety hazard warning information.

9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the vehicle safety hazard warning method based on multi-sensor fusion are performed as described in any one of claims 1 to 7.

Images