Safety detection method and system for intelligent networked automobile based on multi-modal sensing fusion

Abstract

The invention discloses a multi-modal sensing fusion-based safety detection method and system for an intelligent networked automobile, and relates to the technical field of safety detection of intelligent networked automobiles. The multi-modal sensing fusion-based safety detection method for the intelligent networked automobile comprises the following steps: acquiring multi-source sensing data of the intelligent networked automobile in a driving process based on a set sampling period; a multi-mode driving feature set of the intelligent network connection automobile is constructed; based on the multi-modal driving feature set, analyzing a driving state index set of the intelligent networked vehicle; analyzing a driving state fusion index of the intelligent networked automobile; according to the method, the image connectivity feature, the approach evolution feature and the attitude fluctuation feature are constructed and are subjected to joint analysis to form judgment indexes of the regional connectivity index, the relative displacement index and the attitude response index, so that judgment deviation caused by distortion or missing of single sensing data is effectively avoided.

Description

Technical Field

[0001] This invention relates to the field of intelligent connected vehicle safety detection technology, specifically to a method and system for safety detection of intelligent connected vehicles based on multimodal sensor fusion. Background Technology

[0002] With the continuous development of intelligent connected vehicle technology, vehicles are increasingly reliant on environmental information for perception, decision-making, and control. Achieving comprehensive perception and accurate assessment of driving conditions has become a key challenge in improving the safety of autonomous driving. Traditional single-sensor perception methods, limited by their field of view, anti-interference capabilities, and information coverage, struggle to provide stable and reliable detection results in complex traffic environments.

[0003] The limitations of existing technologies include at least the following problems: Current common methods for detecting the driving safety of intelligent connected vehicles mainly rely on visual image recognition or single vehicle attitude analysis. In real-world complex road environments, these methods often face uncertainties such as highly dynamic obstacles, numerous road obstructions and interferences, and frequent fluctuations in the vehicle's own attitude. Relying solely on a single modality makes it difficult to accurately reconstruct the vehicle's true driving state at any given moment. For example, when there are large areas of obstruction in the visual image or the road edges are blurred, it is difficult to reliably determine traffic safety based solely on the image. Similarly, during frequent approach of obstacles or acceleration and change of direction, it is difficult to determine whether the vehicle is stable and controllable based solely on obstacle detection data. Likewise, when the vehicle's attitude is disturbed, such as a sudden change in pitch angle or yaw acceleration, it is difficult to determine whether a real risk exists without combining road and surrounding environmental data. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a safety detection method and system for intelligent connected vehicles based on multimodal sensor fusion. This solves the problem that the lack of multi-source perception fusion in existing technologies leads to one-sided and inaccurate judgments of vehicle driving status, making it difficult to comprehensively assess driving safety levels.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a safety detection method for intelligent connected vehicles based on multimodal sensor fusion, comprising the following steps: acquiring multi-source perception data of the intelligent connected vehicle during its driving process based on a set sampling period, the multi-source perception data including road image sequences, obstacle perception time-series data, and vehicle attitude time-series data; constructing a multimodal driving feature set of the intelligent connected vehicle based on the multi-source perception data, the multimodal driving feature set including image connectivity features, proximity evolution features, and attitude fluctuation features; analyzing the driving state index set of the intelligent connected vehicle based on the multimodal driving feature set, the driving state index set including regional connectivity index, relative displacement index, and attitude response index; analyzing the driving state fusion index of the intelligent connected vehicle based on the driving state index set; and determining the driving safety level of the intelligent connected vehicle based on the driving state fusion index.

[0006] Furthermore, the road image sequence includes road images at multiple time points, each road image consisting of several pixels, with each pixel corresponding to a pixel brightness value. The obstacle perception time-series data includes obstacle recording data at multiple time points, including the three-dimensional spatial coordinates, relative velocity estimate, acceleration estimate, and obstacle detection confidence score for each obstacle. The vehicle attitude time-series data includes yaw angle, pitch angle, roll angle, lateral acceleration, longitudinal acceleration, and yaw angular velocity values ​​at multiple time points.

[0007] Furthermore, the specific steps for constructing a multimodal driving feature set for intelligent connected vehicles based on multi-source perception data are as follows: read the multi-source perception data of the intelligent connected vehicle during driving; input the multi-source perception data into a pre-trained multimodal driving feature analysis model, extract the image connectivity features, proximity evolution features, and attitude fluctuation features of the intelligent connected vehicle, and construct a multimodal driving feature set. The multimodal driving feature analysis model includes a road feature extraction sub-network, an obstacle perception feature extraction sub-network, and an attitude feature extraction sub-network.

[0008] Furthermore, the specific steps for extracting image connectivity features, proximity evolution features, and attitude fluctuation features of intelligent connected vehicles are as follows: In the road feature extraction subnetwork of the multimodal driving feature analysis model, connectivity and occlusion analysis is performed on the road image sequence to obtain the image connectivity features of intelligent connected vehicles; in the obstacle perception feature extraction subnetwork of the multimodal driving feature analysis model, distance and speed evolution analysis is performed on the obstacle perception time series data to obtain the proximity evolution features of intelligent connected vehicles; in the attitude feature extraction subnetwork of the multimodal driving feature analysis model, attitude change analysis is performed on the vehicle attitude time series data to obtain the attitude fluctuation features of intelligent connected vehicles.

[0009] Furthermore, image connectivity features include the proportion of passable areas in the image, edge pixel perturbation density, central region occlusion rate, and image grayscale complexity; proximity trend features include minimum obstacle distance value, confidence proximity clustering intensity, average relative velocity change rate, acceleration direction variation rate, and obstacle orientation concentration; attitude fluctuation features include yaw angle standard deviation value, pitch angle change rate, acceleration direction switching frequency, and heading angle abrupt change value.

[0010] Further, the specific steps for analyzing the regional connectivity index of intelligent connected vehicles are as follows: read the edge pixel perturbation density and image grayscale complexity of the intelligent connected vehicle, and perform normalization processing; combine the normalized edge pixel perturbation density and image grayscale complexity with the corresponding passable area ratio and central area occlusion rate for comprehensive analysis to obtain the regional connectivity index of the intelligent connected vehicle.

[0011] Further, the specific steps for analyzing the relative displacement index of intelligent connected vehicles are as follows: read the approach trend characteristics of intelligent connected vehicles and perform normalization processing; conduct a comprehensive analysis of the normalized approach trend characteristics to obtain the relative displacement index of intelligent connected vehicles.

[0012] Furthermore, the specific steps for analyzing the attitude response index of intelligent connected vehicles are as follows: read the attitude fluctuation characteristics of intelligent connected vehicles and perform normalization processing; conduct a comprehensive analysis of the normalized attitude fluctuation characteristics to obtain the attitude response index of intelligent connected vehicles.

[0013] Furthermore, based on the driving state fusion index, the specific steps for determining the driving safety level of intelligent connected vehicles are as follows: read the driving state fusion index of the intelligent connected vehicle and compare it with multiple preset driving state intervals, with each driving state interval corresponding to a driving safety level; when the driving state fusion index falls into a preset driving state interval, the driving safety level corresponding to that interval is taken as the driving safety level of the intelligent connected vehicle.

[0014] The safety detection system for intelligent connected vehicles based on multimodal sensor fusion includes: a multi-source data acquisition unit, used to acquire multi-source perception data of the intelligent connected vehicle during driving based on a set sampling period, including road image sequences, obstacle perception time-series data, and vehicle attitude time-series data; a multimodal feature construction unit, used to construct a multimodal driving feature set of the intelligent connected vehicle based on the multi-source perception data, including image connectivity features, proximity evolution features, and attitude fluctuation features; a state analysis unit, used to analyze the driving state index set of the intelligent connected vehicle based on the multimodal driving feature set, including regional connectivity index, relative displacement index, and attitude response index; a fusion analysis unit, used to analyze the driving state fusion index of the intelligent connected vehicle based on the driving state index set; and a safety level determination unit, used to determine the driving safety level of the intelligent connected vehicle based on the driving state fusion index.

[0015] The present invention has the following beneficial effects:

[0016] (1) The safety detection method for intelligent connected vehicles based on multimodal sensor fusion constructs image connectivity features, proximity evolution features, and attitude fluctuation features, and analyzes them together to form judgment indicators such as regional connectivity index, relative displacement index, and attitude response index. This effectively avoids judgment bias caused by distortion or lack of single sensor data. For example, when the camera image is affected by strong light interference or occlusion and the image connectivity decreases, the relative speed and acceleration trend in the obstacle perception data can still be used to make auxiliary judgments based on the vehicle's own attitude stability. A collaborative compensation relationship is constructed between different data. Even if there is data fluctuation in a certain direction, the recognition basis can still be supplemented by information from other dimensions. Through this mechanism, a stable judgment basis can be guaranteed under complex road conditions. It does not rely on the judgment result of a certain modality and enhances the mutual verification effect between data as a whole, reducing the probability of false alarms and false alarms.

[0017] (2) The safety detection method for intelligent connected vehicles based on multimodal sensor fusion clearly analyzes the changing trends of time-series features. For example, the relative speed change rate and acceleration direction variation rate are introduced into the approach evolution features, which can reflect the motion trend between the obstacle and the vehicle, rather than making a judgment based on just a frame of image or a single speed value. Similarly, the attitude fluctuation features also characterize the degree of motion instability of the vehicle by analyzing the angle change rate and acceleration switching frequency. By establishing a time-series index system, it is possible to make a prediction when fluctuation signals appear in the early stage of risk, without having to wait for the collision probability to rise sharply before intervening in control, leaving sufficient time for subsequent response. This method is closer to the evolution process of risk from weak to strong in actual driving, and can support earlier and more stable driving state assessment.

[0018] (3) The safety detection method for intelligent connected vehicles based on multimodal sensor fusion no longer outputs the judgment result in the form of raw data points or fuzzy scores based on the driving safety level judged by the driving state fusion index. Instead, it gives a clear level distinction by dividing the interval. Each level corresponds to a judgment interval. This design facilitates direct connection with vehicle control strategy and reduces uncertainty in the intermediate translation and interpretation process. For example, when the fusion index falls into a higher interval, the control system can be linked to execute a deceleration command. When it falls into a middle interval, only prompts or observations can be given. If it is in a low interval, the current state remains unchanged. This segmented processing method does not rely on the confidence output of complex models, which is convenient for adjustment and parameter tuning. It is also convenient for quickly deploying a unified judgment standard in different scenarios, so that the result can be directly called and executed by other systems.

[0019] (4) The safety detection system for intelligent connected vehicles based on multimodal sensor fusion sets up five consecutive units: multi-source data acquisition, multimodal feature construction, state analysis, fusion analysis and safety level determination. Each unit has a clear function and can be deployed or integrated and debugged independently according to needs, which significantly improves the engineering feasibility of the system. Taking the multimodal feature construction unit as an example, after receiving image, obstacle and attitude time series data, it calls different sub-networks to complete feature extraction and outputs a driving feature set in a unified format. The state analysis unit can directly read the set and independently complete index construction without additional intervention, forming a stable data path. For actual deployment scenarios, such as edge computing nodes, vehicle-side AI chips or cloud centralized processing platforms, the fusion analysis and level determination modules can be flexibly selected to achieve on-demand calling and distributed deployment, avoiding system bloat or excessive data coupling. At the same time, the structure of each unit is clear, which facilitates later model replacement, algorithm update or function expansion, and supports continuous product iteration and scenario adaptation.

[0020] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0021] Figure 1 This is a flowchart of the safety detection method for intelligent connected vehicles based on multimodal sensor fusion according to the present invention.

[0022] Figure 2 This is a flowchart illustrating the specific steps involved in analyzing the regional connectivity index of an intelligent connected vehicle in the safety detection method for intelligent connected vehicles based on multimodal sensor fusion, as described in this invention.

[0023] Figure 3 This is a block diagram of the safety detection system for intelligent connected vehicles based on multimodal sensor fusion according to the present invention. Detailed Implementation

[0024] Please see Figure 1 This invention provides a technical solution: a safety detection method for intelligent connected vehicles based on multimodal sensor fusion, comprising the following steps: acquiring multi-source perception data of the intelligent connected vehicle during its driving process based on a set sampling period, the multi-source perception data including road image sequences, obstacle perception time-series data, and vehicle attitude time-series data; constructing a multimodal driving feature set of the intelligent connected vehicle based on the multi-source perception data, the multimodal driving feature set including image connectivity features, proximity evolution features, and attitude fluctuation features; analyzing a driving state index set of the intelligent connected vehicle based on the multimodal driving feature set, the driving state index set including regional connectivity index, relative displacement index, and attitude response index; analyzing a driving state fusion index of the intelligent connected vehicle based on the driving state index set; and determining the driving safety level of the intelligent connected vehicle based on the driving state fusion index.

[0025] The specific formula for calculating the driving state fusion index of intelligent connected vehicles is as follows: ;in, , , , The four indices, in order, are the driving state fusion index, regional connectivity index, relative displacement index, and attitude response index for intelligent connected vehicles. , The values ​​are the relative displacement adjustment coefficient and the attitude response adjustment coefficient stored in the database, respectively, and in this embodiment, they are 1.2 and 1.1, respectively.

[0026] The relative displacement adjustment coefficient ranges from [0.5, 2.5].

[0027] The attitude response adjustment coefficient ranges from [0.5, 2.5].

[0028] The road image sequence includes road images at multiple time points. Each road image consists of several pixels, and each pixel corresponds to a pixel brightness value. The obstacle perception time series data includes obstacle recording data at multiple time points. The obstacle recording data includes the three-dimensional spatial coordinates, relative velocity estimate, acceleration estimate, and obstacle detection confidence score for each obstacle. The vehicle attitude time series data includes yaw angle, pitch angle, roll angle, lateral acceleration, longitudinal acceleration, and yaw angular velocity values ​​at multiple time points.

[0029] The pixel brightness value represents the brightness information of each pixel in the road image in the grayscale or RGB channels. It is obtained by capturing road images using a vehicle-mounted camera (including but not limited to a monocular or binocular stereo camera) and acquiring the pixel matrix data for each frame. For grayscale images, the grayscale value of each pixel can be directly read as the brightness value; for RGB color images, the brightness value of each pixel can be obtained by weighted averaging of the three channel values ​​(red, green, and blue) or by extracting the maximum component.

[0030] Three-dimensional spatial coordinates represent the spatial distribution of obstacles within the vehicle's coordinate system, typically represented by X (front / back), Y (left / right), and Z (up / down) axes. They are obtained as follows: Point cloud data of the external environment is collected using an onboard LiDAR system; a multi-frame point cloud registration algorithm is used to extract the obstacle target point set; and then, the point cloud coordinates are transformed using vehicle self-localization information to obtain the obstacle's three-dimensional spatial position within the vehicle's coordinate system. Alternatively, millimeter-wave radar combined with visual camera data can be used to construct a scene map and locate the target using a visual SLAM algorithm, thereby extracting the target's three-dimensional position coordinates.

[0031] The relative velocity estimate represents the real-time speed and direction of an obstacle relative to a vehicle. It is obtained by: performing differential calculations on the spatial position of the obstacle at different time points based on radar point cloud data from multiple consecutive sampling periods; combining this with the vehicle's current speed and direction data to calculate the target's velocity vector relative to the vehicle. During this process, a Kalman filter can be introduced to smooth and predict the velocity values ​​over time, eliminating noise and improving estimation accuracy.

[0032] Acceleration estimates represent the change in velocity of an obstacle per unit time, reflecting its dynamic motion trend. They are obtained by calculating the velocity difference between two adjacent time points based on their relative velocity estimates, and then dividing this difference by the sampling time interval between the two time points to obtain the instantaneous acceleration value of the obstacle. This process does not rely on direct sensor measurements but is derived through velocity sequence differencing combined with the sampling period interval.

[0033] The obstacle detection confidence score represents the degree of confidence in identifying each detected obstacle. It is obtained by inputting the image data of the current frame into a pre-trained deep object detection model (such as YOLOv5, CenterPoint, etc.). During the model's inference output phase, the confidence score generated for each obstacle is read. The value typically ranges from 0 to 1, reflecting the probability that the model classifies the region as an obstacle.

[0034] The yaw angle represents the angle of rotation of a vehicle around its vertical axis (Z-axis), i.e., the change in the angle of the vehicle's facing. It is obtained by: measuring the angular velocity around the Z-axis using the gyroscope component in the onboard inertial measurement unit (IMU) and integrating it over time to obtain the angle change; or by using a combined navigation system (such as RTK+IMU) formed by the IMU and a Global Navigation Satellite System (GNSS) to perform attitude calculations and obtain the vehicle's yaw angle value in real time.

[0035] The pitch angle represents the angle by which a vehicle pitches forward or backward around its lateral axis (Y-axis), and is used to characterize the vehicle's forward or backward tilt. It is obtained by measuring the angular velocity around the Y-axis in real time using a three-axis gyroscope in the IMU, and then calculating the pitch angle by integrating the angular velocity. If an attitude fusion algorithm is used, the angle can also be corrected by incorporating the static component of the accelerometer in the vertical direction.

[0036] The roll angle value represents the angle of rotation of the vehicle around its longitudinal axis (X-axis), i.e., the degree of left and right tilt of the vehicle body. It is obtained by continuously collecting the angular velocity around the X-axis using the three-axis gyroscope in the IMU, performing angular velocity integration in the attitude calculation module, and combining the static perception of the ground direction by the accelerometer to perform roll angle correction, thus obtaining the current roll state of the vehicle body.

[0037] Lateral acceleration values ​​represent the acceleration components of a vehicle in the lateral direction (i.e., left-right direction). They are obtained by reading the real-time acceleration values ​​of the vehicle along the lateral axis (usually the Y-axis) from the triaxial accelerometer in the IMU, measured in meters per second squared (m / s²). This reflects the intensity of lateral acceleration during lane changes, turns, and other maneuvers.

[0038] The longitudinal acceleration value represents the acceleration component of the vehicle along the longitudinal direction (usually the X-axis). It is obtained by the IMU's accelerometer component measuring the degree of acceleration or deceleration of the vehicle in real time during forward movement or braking. The positive or negative value represents the acceleration or deceleration state and is used to determine driving stability and power control strategy.

[0039] The yaw angular velocity value represents the instantaneous rotational speed of a vehicle around its vertical axis, reflecting whether the vehicle is currently in a change of direction. It is obtained by directly measuring the angular velocity component around the Z-axis using a three-axis gyroscope in the IMU, typically in degrees per second (° / s) or radians per second (rad / s). It can be used to identify dynamic behaviors such as sharp turns and drifts under high-frequency sampling.

[0040] Specifically, the steps for constructing a multimodal driving feature set for intelligent connected vehicles based on multi-source perception data are as follows: read multi-source perception data of intelligent connected vehicles during driving; input the multi-source perception data into a pre-trained multimodal driving feature analysis model, extract the image connectivity features, proximity evolution features, and attitude fluctuation features of intelligent connected vehicles, and construct a multimodal driving feature set. The multimodal driving feature analysis model includes a road feature extraction sub-network, an obstacle perception feature extraction sub-network, and an attitude feature extraction sub-network.

[0041] Image connectivity features include the proportion of passable areas in the image, edge pixel perturbation density, central region occlusion rate, and image grayscale complexity. Proximity trend features include minimum obstacle distance value, confidence proximity clustering intensity, average relative velocity change rate, acceleration direction variation rate, and obstacle orientation concentration. Attitude fluctuation features include yaw angle standard deviation value, pitch angle change rate, acceleration direction switching frequency, and heading angle abrupt change value.

[0042] The pre-training steps for the multimodal driving feature analysis model are as follows:

[0043] The road feature extraction subnetwork, obstacle sensing feature extraction subnetwork, and pose feature extraction subnetwork are trained separately:

[0044] The specific steps for pre-training the road feature extraction subnetwork are as follows:

[0045] First, a video dataset containing different road scenarios was constructed. This dataset includes road image sequences under complex traffic environments such as daytime, nighttime, rain, snow, and backlight. The passing area, occlusion area, and grayscale changes of each frame of the image were manually annotated by annotators.

[0046] Next, an initial model is constructed based on an image semantic segmentation network (such as DeepLabv3+ or U-Net). The input is a sequence of road images at continuous time points, and the output is a semantic probability map of each region in the image, including accessibility, edge perturbation intensity, and center occlusion degree.

[0047] Then, the network is trained through supervised learning, using manually labeled region masks as labels. The cross-entropy loss function and Dice coefficient loss function are jointly optimized, and data augmentation strategies (such as brightness perturbation, image rotation, mirror flipping, etc.) are introduced during training to improve the robustness of the model.

[0048] Finally, the model parameters that perform best on the validation set are selected as the pre-trained model weights to complete the training process of the road feature extraction sub-network.

[0049] The specific steps for pre-training the obstacle sensing feature extraction subnetwork are as follows:

[0050] First, an obstacle perception dataset containing dynamic changes of obstacles around the vehicle is constructed. The data sources include LiDAR perception records, millimeter-wave radar detection results, and synchronous image target detection data. This dataset labels the three-dimensional coordinate position, relative velocity, acceleration, orientation angle, and detection confidence score of each obstacle at each time point.

[0051] Next, an obstacle temporal modeling network is constructed based on a temporal graph neural network (such as ST-GCN) or LSTM structure. The position, velocity, and acceleration features of each obstacle at multiple time points are encoded into temporal feature vectors. The training objective is to predict its approach trend, variation rate, and aggregation direction with respect to vehicles.

[0052] Then, the future state of the obstacle is predicted by regression using the MSE loss function, and classification loss is introduced to assist in judging its risk level. The model is optimized by using a joint loss function, and a sliding time window strategy is adopted to improve the model's sensitivity to short-term evolution.

[0053] Finally, the network parameters after training are frozen and used as the initial weights for the obstacle sensing feature extraction subnetwork.

[0054] The specific steps for pre-training the pose feature extraction subnetwork are as follows:

[0055] First, the vehicle's attitude data is collected under different turning, lane changing, braking, and evasive maneuvers. The data includes yaw angle, pitch angle, roll angle, lateral and longitudinal acceleration, and yaw angular velocity. Key attitude events (such as sudden attitude changes, acceleration direction switching, abnormal yaw, etc.) are then labeled by professional testers.

[0056] Next, a temporal feature modeling network based on Transformer or LSTM structure is constructed. The input is the sequence of vehicle attitude parameters at multiple consecutive time points, and the output is the numerical estimation of various attitude fluctuation features or the event discrimination results.

[0057] Then, a weighted regression loss function is used to learn continuous variables, while an event classification task (such as whether there is a sudden change or whether there is a drastic fluctuation) is introduced as an auxiliary task. The feature representation ability of the model is improved through multi-task joint training.

[0058] Finally, the network parameters that perform best are retained as the pre-trained model for the pose feature extraction sub-network.

[0059] The specific steps for extracting image connectivity features, proximity evolution features, and attitude fluctuation features of intelligent connected vehicles are as follows:

[0060] In the road feature extraction subnetwork of the multimodal driving feature analysis model, connectivity and occlusion analysis are performed on the road image sequence to obtain the image connectivity features of intelligent connected vehicles, specifically:

[0061] Regarding the percentage of passable area in the image:

[0062] The pixel brightness value of each pixel in each frame of the road image is compared with the set traffic brightness threshold one by one. The number of pixels with brightness values ​​higher than the traffic threshold is counted, and the ratio of this number to the total number of pixels in the frame is processed to obtain the proportion of the trafficable area of ​​the frame. Then, the proportion of the trafficable area of ​​the image at all time points is averaged to obtain the proportion of the trafficable area of ​​the image.

[0063] For edge pixel perturbation density:

[0064] For each frame of road image at consecutive time points, the Canny edge detection algorithm is first used to extract the edges of the image to obtain the set of edge pixels in each frame. Then, edge pixels with similar positions in adjacent frames are compared one by one. If the difference in their gray values ​​exceeds the set perturbation threshold, they are judged as perturbed edge points. In each frame, the ratio of the number of perturbed edge points to the image area is calculated, and the ratio is averaged over all time points to obtain the edge pixel perturbation density.

[0065] For the occlusion rate in the central area:

[0066] Each frame of the image is divided into a nine-grid area, and the pixel set of the central region of the image is extracted. For the central region of each frame, the number of pixels whose pixel brightness value is lower than the occlusion judgment threshold is counted, and the ratio of this number to the total number of pixels in the central region is processed to obtain the central region occlusion rate of each frame. Then, the occlusion rates at all time points are averaged to obtain the central region occlusion rate.

[0067] Regarding image grayscale complexity:

[0068] For each frame of the image, the pixel brightness values ​​of all pixels are expanded into a brightness sequence in row and column order. The absolute value of the difference between the pixel brightness values ​​of adjacent pixels is calculated, and all absolute values ​​of the difference are summed to obtain the total brightness jump of the frame. Then, the total brightness jump at all time points is averaged to obtain the image grayscale complexity.

[0069] In the obstacle perception feature extraction subnetwork of the multimodal driving feature analysis model, distance and speed evolution analysis is performed on the obstacle perception time series data to obtain the approach evolution characteristics of intelligent connected vehicles, which are as follows:

[0070] For the minimum obstacle distance value:

[0071] For each time point, based on the three-dimensional position coordinates of the obstacles and the current position coordinates of the vehicle, calculate the Euclidean space distance values ​​from all obstacles to the vehicle, and take the minimum value as the nearest distance at that time point; then take the overall minimum value from the minimum distance values ​​at all time points as the minimum obstacle distance value.

[0072] For confidence levels close to cluster strength:

[0073] The obstacle record data at all time points is traversed to extract the spatial location and confidence score of each obstacle; the confidence scores of obstacles that are close to the vehicle (below the set distance threshold) are accumulated and the accumulated values ​​are normalized to serve as the proximity confidence strength value at each time point; then the proximity confidence strength values ​​at all time points are averaged to obtain the confidence proximity cluster strength.

[0074] For the average rate of change of relative velocity:

[0075] Extract the relative velocity estimate of each obstacle at all time points, and calculate the velocity difference between two adjacent time points for each obstacle in chronological order; then process the absolute value of the velocity difference of all obstacles at all time points and average it to obtain the average relative velocity change rate.

[0076] For the rate of change of acceleration direction:

[0077] For each obstacle, the acceleration estimate at consecutive time points is used to calculate the change of its direction vector in three-dimensional space, i.e., the angle between the acceleration vectors of two adjacent time points, and the number of changes with the angle greater than a set threshold is counted. Then, the ratio of this number of changes to the total number of time frames is processed to obtain the acceleration direction variation rate.

[0078] Regarding the concentration of obstacle orientation:

[0079] Based on the spatial coordinates of each obstacle and the vehicle's position coordinates, calculate the azimuth angle value of the obstacle relative to the vehicle; plot all azimuth angle values ​​into an angle distribution histogram, and calculate the ratio of the number of obstacles in the angle interval of the main direction angle to the total number of obstacles, as the obstacle azimuth concentration.

[0080] In the attitude feature extraction subnetwork of the multimodal driving feature analysis model, attitude change analysis is performed on the vehicle attitude time series data to obtain the attitude fluctuation characteristics of intelligent connected vehicles, specifically:

[0081] For the standard deviation of the yaw angle:

[0082] The vehicle yaw angle values ​​at consecutive time points are combined into an angle sequence, and the standard deviation of this angle sequence is calculated as the standard deviation of the vehicle's yaw angle within a set time window.

[0083] For the rate of change of pitch angle:

[0084] The pitch angle values ​​at consecutive time points are calculated by taking the difference between adjacent values, and the pitch angle difference between every two time points is compared with the time interval to obtain the pitch angle change rate sequence per unit time. Then, the average of this sequence is calculated to obtain the pitch angle change rate.

[0085] Regarding the frequency of acceleration direction switching:

[0086] The lateral acceleration value and the longitudinal acceleration value at each time point are used to construct a planar acceleration vector. The angle of change of acceleration direction between two adjacent time points is counted. If the change of direction angle exceeds a set threshold, it is judged as a switching event. The total number of switching events in the entire time series is counted as the acceleration direction switching frequency.

[0087] For abrupt changes in heading angle:

[0088] The heading angular velocity values ​​at consecutive time points are compared one by one, the difference between two adjacent time points is calculated, and the change with the largest absolute value is extracted as the heading angular velocity mutation value.

[0089] This implementation scheme can comprehensively reflect the current road conditions and driving dynamic risks of intelligent connected vehicles from the perspectives of image, obstacle perception, and vehicle attitude. Image connectivity features can effectively identify whether there are obstructions or obstacles on the road ahead, providing basic support for path judgment. Proximity evolution features quantify the potential collision trend between obstacles and vehicles through factors such as distance, speed, and direction, making it easier to perceive dangerous areas in advance. Attitude fluctuation features intuitively reflect whether the vehicle has unstable behaviors such as deflection, violent acceleration and deceleration, and frequent direction changes in the current period, which helps to identify whether the driving state is abnormal. Compared with the traditional method that relies on a single sensor or simple threshold judgment, this method can extract fine-grained features with evolutionary trends and spatial distribution characteristics from multiple dimensions, which helps to improve the vehicle's perception ability in complex road environments.

[0090] Specifically, such as Figure 2 As shown, the specific steps for analyzing the regional connectivity index of intelligent connected vehicles are as follows: read the edge pixel perturbation density and image grayscale complexity of the intelligent connected vehicle, and perform normalization processing; combine the normalized edge pixel perturbation density and image grayscale complexity with the corresponding passable area ratio and central area occlusion rate for comprehensive analysis to obtain the regional connectivity index of the intelligent connected vehicle.

[0091] The specific formula for calculating the regional connectivity index of intelligent connected vehicles is as follows: ;in, , , , , The following are, in order: regional connectivity index of intelligent connected vehicles, percentage of passable areas in the image, (normalized) image grayscale complexity, (normalized) edge pixel perturbation density, and central region occlusion rate. , , , The values ​​are, in order, the access area adjustment coefficient, the grayscale complexity adjustment coefficient, the edge disturbance adjustment coefficient, and the area occlusion adjustment coefficient stored in the database, and in this embodiment, they are 0.68, 0.42, 1.73, and 2.25, respectively.

[0092] The range of the passage area adjustment coefficient is [0.0, 1.0].

[0093] The range of the grayscale complexity adjustment coefficient is [0.0, 1.0].

[0094] The range of the edge disturbance adjustment coefficient is [0.5, 3.0].

[0095] The range of the regional occlusion adjustment coefficient is [0.5, 5.0].

[0096] In this implementation scheme, the construction of the regional connectivity index can comprehensively reflect the passage space status of intelligent connected vehicles within the current field of vision. Compared with the traditional method that relies solely on the presence of obstacles, this method not only considers the proportion of passable areas in the image but also introduces dynamic factors such as edge perturbation density and grayscale complexity. This allows it to identify visual interference caused by factors such as water accumulation, strong light, shadows, and occlusion. Furthermore, by setting reasonable adjustment coefficients, the index can be flexibly adjusted according to different road scenarios or driving strategies. For example, when large areas of dynamic shadows or complex texture interference appear in the image, the weights of edge perturbation and grayscale complexity can be increased accordingly to enhance the index's ability to perceive complex visual environments. The regional connectivity index ultimately output by this method can serve as an important basis for determining whether a vehicle possesses stable passage conditions.

[0097] Specifically, the steps for analyzing the relative displacement index of intelligent connected vehicles are as follows: read the approach trend characteristics of intelligent connected vehicles and perform normalization processing; conduct a comprehensive analysis of the normalized approach trend characteristics to obtain the relative displacement index of intelligent connected vehicles.

[0098] The specific formula for calculating the relative displacement index of intelligent connected vehicles is as follows: ;in, , , , , , The following are, in order: relative displacement index of intelligent connected vehicles, (normalized) average relative velocity change rate, (normalized) minimum obstacle distance value, (normalized) confidence level close clustering intensity, (normalized) acceleration direction variation rate, and (normalized) obstacle orientation concentration. , , The values ​​are, in order, the speed change adjustment coefficient, the distance confidence adjustment coefficient, and the joint disturbance adjustment coefficient stored in the database, and in this embodiment, they are 1.04, 0.91, and 1.13, respectively.

[0099] The range of the speed variation adjustment coefficient is [0.95, 1.25].

[0100] The range of the distance confidence adjustment coefficient is [0.85, 1.15].

[0101] The range of the joint disturbance adjustment coefficient is [1.05, 1.30].

[0102] In this implementation scheme, the construction of the relative displacement index can express the dynamic approach trend of obstacles around the vehicle in a more three-dimensional way. It no longer relies on a single distance or speed judgment, but integrates multiple dimensions of evolutionary characteristics for comprehensive analysis. For example, the speed trend of obstacle approach can be perceived through the average relative velocity change rate, the minimum obstacle distance value provides a distance reference for the most dangerous target at present, the confidence proximity clustering intensity enhances the ability to identify dense approaching situations of high-confidence obstacles, and the acceleration direction variation rate and obstacle orientation concentration further characterize the stability and spatial clustering of obstacle movement, which helps to identify complex risk situations such as side-by-side crossing and clustered traversing. The final output relative displacement index can quantify the dynamic approach risk between obstacles and vehicles in the current scenario.

[0103] Specifically, the steps for analyzing the attitude response index of intelligent connected vehicles are as follows: read the attitude fluctuation characteristics of intelligent connected vehicles and perform normalization processing; conduct a comprehensive analysis of the normalized attitude fluctuation characteristics to obtain the attitude response index of intelligent connected vehicles.

[0104] The specific formula for calculating the attitude response index of intelligent connected vehicles is as follows: ;in, , , , , The following are, in order: attitude response index of intelligent connected vehicles, standard deviation of yaw angle (normalized), rate of change of pitch angle (normalized), frequency of acceleration direction switching (normalized), and abrupt change of yaw angle (normalized). This is a natural constant, and in this embodiment, it is taken as 2.71. , The values ​​are, in order, the yaw angle adjustment coefficient and the pitch angle change adjustment coefficient stored in the database, and in this embodiment, they are respectively taken as 4.73 and 1.27.

[0105] The range of the yaw angle adjustment coefficient is [1.0, 10.0].

[0106] The range of the pitch angle adjustment coefficient is [0.0, 2.0].

[0107] In this implementation scheme, the construction of the attitude response index effectively quantifies the dynamic fluctuations of the vehicle's attitude during continuous driving, enabling more sensitive capture of unstable changes in the vehicle's state. By introducing the standard deviation of the yaw angle, it reflects the lateral stability of the vehicle during steering; the pitch angle change rate reveals the longitudinal attitude adjustment of the vehicle under acceleration, deceleration, or sloping road conditions; the acceleration direction switching frequency can identify whether the vehicle frequently performs emergency lane changes, obstacle avoidance, or offset adjustments; and the yaw angle abrupt change value further reflects the sudden intensity of directional changes under extreme conditions. After these features are normalized, a natural constant attenuation mechanism is introduced to prevent the impact of attitude anomalies from being linearly amplified, thereby enhancing the tolerance to slight fluctuations. At the same time, the setting of the adjustment coefficient allows the system to flexibly adapt to different vehicle models or driving styles. The resulting attitude response index can accurately describe the dynamic stability of the vehicle.

[0108] Specifically, the steps for determining the driving safety level of an intelligent connected vehicle based on the driving state fusion index are as follows: read the driving state fusion index of the intelligent connected vehicle and compare it with multiple preset driving state intervals, with each driving state interval corresponding to a driving safety level; when the driving state fusion index falls into a preset driving state interval, the driving safety level corresponding to that interval is taken as the driving safety level of the intelligent connected vehicle.

[0109] Including but not limited to the following examples:

[0110] Driving status interval 1: [0.0, 1.0]:

[0111] Safety level: Extremely high risk level, indicating that the area is severely obstructed, obstacles are close, and the vehicle's posture is violently disturbed, making it unsuitable to continue driving and requiring emergency braking or autonomous stopping.

[0112] Driving state interval 2: (1.0, 2.0]:

[0113] Safety level: Medium to high risk level, indicating that there is localized instability or an approaching obstacle. The speed should be reduced and the sensing frequency increased.

[0114] Driving range 3: (2.0, 3.5)

[0115] Safety level: Medium risk level, indicating that the road is generally passable, but there are minor uncertainties and disturbances, so people should remain vigilant.

[0116] Driving range 4: (3.5, 5.0)

[0117] Safety level: Low risk level, indicating a good integration index, allowing the current driving strategy to continue.

[0118] Driving status range 5: (5.0, 6.0]:

[0119] Safety level: The safety level indicates high passability, stable attitude, no abnormal obstacles, and normal cruise operation.

[0120] In this implementation plan, the driving safety level determination mechanism establishes a well-structured and responsive state classification system by integrating dynamic feature indicators from multiple dimensions. Compared with the traditional method of risk identification that relies solely on obstacle distance or single sensor data, this method introduces composite indicators such as regional connectivity index, relative displacement index, and attitude response index. This allows for a more comprehensive reflection of the current environmental traffic conditions, obstacle approach trends, and the vehicle's own attitude stability. By dividing the driving state fusion index into multiple continuous risk intervals, it not only clarifies the driving suggestions (such as deceleration, stopping, and holding) corresponding to each risk level, but also gives the system a higher level of scenario adaptability. This mechanism is particularly suitable for dealing with uncertain driving scenarios such as urban congestion, complex intersections, or low visibility, and can provide executable decision support for intelligent connected vehicles, thereby improving overall driving safety and driving intelligence.

[0121] Please see Figure 3This invention provides a technical solution: a safety detection system for intelligent connected vehicles based on multimodal sensor fusion, comprising: a multi-source data acquisition unit, used to acquire multi-source perception data of the intelligent connected vehicle during driving based on a set sampling period, the multi-source perception data including road image sequences, obstacle perception time-series data, and vehicle attitude time-series data; a multimodal feature construction unit, used to construct a multimodal driving feature set of the intelligent connected vehicle based on the multi-source perception data, the multimodal driving feature set including image connectivity features, proximity evolution features, and attitude fluctuation features; a state analysis unit, used to analyze a driving state index set of the intelligent connected vehicle based on the multimodal driving feature set, the driving state index set including regional connectivity index, relative displacement index, and attitude response index; a fusion analysis unit, used to analyze a driving state fusion index of the intelligent connected vehicle based on the driving state index set; and a safety level determination unit, used to determine the driving safety level of the intelligent connected vehicle based on the driving state fusion index.

[0122] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0123] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A safety detection method for intelligent connected vehicles based on multimodal sensor fusion, characterized in that, Includes the following steps: Based on a set sampling period, multi-source perception data of intelligent connected vehicles during driving is acquired. The multi-source perception data includes road image sequences, obstacle perception time-series data, and vehicle attitude time-series data. A multimodal driving feature set for intelligent connected vehicles is constructed based on multi-source perception data. The multimodal driving feature set includes image connectivity features, proximity evolution features, and attitude fluctuation features. Based on the multimodal driving feature set, the driving state index set of intelligent connected vehicles is analyzed. The driving state index set includes regional connectivity index, relative displacement index, and attitude response index. Based on the driving state index set, the driving state fusion index of intelligent connected vehicles is analyzed. The driving safety level of intelligent connected vehicles is determined based on the driving status fusion index.

2. The safety detection method for intelligent connected vehicles based on multimodal sensor fusion according to claim 1, characterized in that, The road image sequence includes road images at multiple time points. Each road image consists of several pixels, and each pixel corresponds to a pixel brightness value. The obstacle perception time series data includes obstacle recording data at multiple time points. The obstacle recording data includes the three-dimensional spatial coordinates, relative velocity estimate, acceleration estimate, and obstacle detection confidence score for each obstacle. The vehicle attitude time series data includes yaw angle, pitch angle, roll angle, lateral acceleration, longitudinal acceleration, and yaw angular velocity values ​​at multiple time points.

3. The safety detection method for intelligent connected vehicles based on multimodal sensor fusion according to claim 1, characterized in that, The specific steps for constructing a multimodal driving feature set for intelligent connected vehicles based on multi-source sensing data are as follows: Read multi-source perception data of intelligent connected vehicles during driving; Multi-source perception data is input into a pre-trained multimodal driving feature analysis model to extract image connectivity features, proximity evolution features, and attitude fluctuation features of intelligent connected vehicles, and to construct a multimodal driving feature set. The multimodal driving feature analysis model includes a road feature extraction subnetwork, an obstacle sensing feature extraction subnetwork, and an attitude feature extraction subnetwork.

4. The safety detection method for intelligent connected vehicles based on multimodal sensor fusion according to claim 3, characterized in that, The specific steps for extracting image connectivity features, proximity evolution features, and attitude fluctuation features of intelligent connected vehicles are as follows: In the road feature extraction subnetwork of the multimodal driving feature analysis model, connectivity and occlusion analysis are performed on the road image sequence to obtain the image connectivity features of intelligent connected vehicles. In the obstacle sensing feature extraction subnetwork of the multimodal driving feature analysis model, distance and speed evolution analysis is performed on the obstacle sensing time series data to obtain the approach evolution characteristics of intelligent connected vehicles. In the attitude feature extraction subnetwork of the multimodal driving feature analysis model, attitude change analysis is performed on the vehicle attitude time series data to obtain the attitude fluctuation characteristics of intelligent connected vehicles.

5. The safety detection method for intelligent connected vehicles based on multimodal sensor fusion according to claim 1, characterized in that, Image connectivity features include the proportion of passable areas in the image, edge pixel perturbation density, central region occlusion rate, and image grayscale complexity. Proximity trend features include minimum obstacle distance value, confidence proximity clustering intensity, average relative velocity change rate, acceleration direction variation rate, and obstacle orientation concentration. Attitude fluctuation features include yaw angle standard deviation value, pitch angle change rate, acceleration direction switching frequency, and heading angle abrupt change value.

6. The safety detection method for intelligent connected vehicles based on multimodal sensor fusion according to claim 5, characterized in that, The specific steps for analyzing the regional connectivity index of intelligent connected vehicles are as follows: Read the edge pixel perturbation density and image grayscale complexity of the intelligent connected vehicle, and perform normalization processing; By combining the normalized edge pixel perturbation density and image grayscale complexity with the corresponding passable area ratio and central area occlusion rate, a comprehensive analysis is performed to obtain the regional connectivity index of intelligent connected vehicles.

7. The safety detection method for intelligent connected vehicles based on multimodal sensor fusion according to claim 5, characterized in that, The specific steps for analyzing the relative displacement index of intelligent connected vehicles are as follows: Read the proximity trend characteristics of intelligent connected vehicles and perform normalization processing; By comprehensively analyzing the proximity trend characteristics after normalization, the relative displacement index of intelligent connected vehicles is obtained.

8. The safety detection method for intelligent connected vehicles based on multimodal sensor fusion according to claim 5, characterized in that, The specific steps for analyzing the attitude response index of intelligent connected vehicles are as follows: Read the attitude fluctuation characteristics of intelligent connected vehicles and perform normalization processing; By comprehensively analyzing the normalized attitude fluctuation characteristics, the attitude response index of intelligent connected vehicles is obtained.

9. The safety detection method for intelligent connected vehicles based on multimodal sensor fusion according to claim 1, characterized in that, The specific steps for determining the driving safety level of intelligent connected vehicles based on the driving state fusion index are as follows: Read the driving status fusion index of intelligent connected vehicles and compare it with multiple preset driving status intervals. Each driving status interval corresponds to a driving safety level. When the driving state fusion index falls into a preset driving state range, the driving safety level corresponding to that range is taken as the driving safety level of the intelligent connected vehicle.

10. A safety detection system for intelligent connected vehicles based on multimodal sensor fusion, employing the safety detection method for intelligent connected vehicles based on multimodal sensor fusion as described in any one of claims 1-9, characterized in that, include: The multi-source data acquisition unit is used to acquire multi-source perception data of intelligent connected vehicles during driving based on a set sampling period. The multi-source perception data includes road image sequences, obstacle perception time-series data, and vehicle attitude time-series data. The multimodal feature construction unit is used to construct a multimodal driving feature set for intelligent connected vehicles based on multi-source perception data. The multimodal driving feature set includes image connectivity features, proximity evolution features, and attitude fluctuation features. The state analysis unit is used to analyze the driving state index set of intelligent connected vehicles based on the multimodal driving feature set. The driving state index set includes regional connectivity index, relative displacement index, and attitude response index. The fusion analysis unit is used to analyze the fusion index of driving status of intelligent connected vehicles based on the driving status index set. The safety level determination unit is used to determine the driving safety level of intelligent connected vehicles based on the driving state fusion index.

Images