Driving behavior analysis method and system based on multi-modal sensor fusion

Abstract

The invention provides a driving behavior analysis method and system based on multi-modal sensor fusion, and the method comprises the steps: firstly obtaining a multi-source sensor data set of a target vehicle containing at least two different sensor source driving environment perception data, and carrying out the time-space alignment processing of the multi-source sensor data set, and generating a multi-source fusion data set; then, through a feature extraction layer of a preset behavior analysis network, a driving behavior feature set containing vehicle operation, environment interaction and behavior continuity features is extracted from the multi-source fusion data set, an anomaly recognition layer is used for carrying out anomaly recognition on the driving behavior feature set, and an anomaly recognition result is generated; and finally, generating driving strategy optimization data according to an abnormal recognition result, and feeding back the driving strategy optimization data to a vehicle control system to adjust a driving strategy. According to the method, driving behaviors can be comprehensively analyzed, abnormity can be timely found, the driving strategy is optimized, and the driving safety and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent assisted driving, and more particularly, to a driving behavior analysis method and system based on multi-modal sensor fusion. Background Art

[0002] With the rapid development of the automotive industry and the continuous progress of intelligent driving technology, the need for accurate analysis and optimization of driving behavior has become increasingly urgent. At present, in the field of driving behavior analysis, there are many limitations in the existing technologies.

[0003] Some traditional methods often only focus on the basic operation characteristics of the vehicle, such as acceleration, deceleration, steering, etc., while ignoring the interaction characteristics between the vehicle and the surrounding environment and the continuity characteristics of driving behavior. Driving behavior is a complex process, which is not only related to the vehicle's own operations, but also closely related to the surrounding environment. At the same time, driving behavior has a certain continuity and coherence. The lack of comprehensive consideration of these characteristics makes the analysis of driving behavior insufficiently comprehensive and in-depth, and it is impossible to accurately identify potential abnormal driving behaviors.

[0004] For the identification of abnormal driving behaviors, most of the existing technologies are based on simple rule judgments. For example, a fixed speed threshold is set to judge speeding behaviors. This simple rule judgment method cannot adapt to complex and changeable driving scenarios, and it is difficult to effectively identify some atypical abnormal behaviors, resulting in the inability to timely discover potential risks during driving. Summary of the Invention

[0005] In view of the problems mentioned above, in combination with the first aspect of the present invention, embodiments of the present invention provide a driving behavior analysis method based on multi-modal sensor fusion, and the method includes: Obtain a multi-source sensor data set of a target vehicle, where the multi-source sensor data set includes driving environment perception data from at least two different sensor sources; Perform spatio-temporal alignment processing on the multi-source sensor data set to generate a multi-source fusion data set; Based on the feature extraction layer of a preset behavior analysis network, perform behavior feature extraction processing on the multi-source fusion data set to obtain a driving behavior feature set, where the driving behavior feature set includes vehicle operation features, environment interaction features, and behavior continuity features; Based on the abnormal recognition layer of a preset behavior analysis network, perform abnormal behavior recognition processing on the driving behavior feature set to generate a driving behavior abnormal recognition result; Generate driving strategy optimization data according to the driving behavior abnormal recognition result, and feedback the driving strategy optimization data to the vehicle control system to trigger a driving strategy adjustment operation.

[0006] In another aspect, an embodiment of the present invention further provides a driving behavior analysis system based on multi-modal sensor fusion, including a processor and a machine-readable storage medium. The machine-readable storage medium is connected to the processor. The machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0007] Based on the above aspects, the present invention obtains a multi-source sensor data set including driving environment perception data from at least two different sensor sources, and performs spatio-temporal alignment processing on the multi-source sensor data set to generate a multi-source fusion data set, effectively solving the problem of inconsistency of different sensor data in time and space, so that the fused data can accurately reflect the driving scene information at the same moment and space, improving the availability and reliability of the data. Using the feature extraction layer of the preset behavior analysis network to extract behavior features from the multi-source fusion data set, a driving behavior feature set including vehicle operation features, environment interaction features and behavior continuity features is obtained, realizing in-depth mining and comprehensive capture of driving behavior features, and accurately depicting driving behavior from multiple aspects. Based on the abnormal recognition layer of the preset behavior analysis network, abnormal behavior recognition processing is performed on the driving behavior feature set to generate a driving behavior abnormal recognition result, which can timely and accurately detect abnormal behaviors during driving, providing a strong guarantee for driving safety. Finally, driving strategy optimization data is generated according to the driving behavior abnormal recognition result and fed back to the vehicle control system to trigger driving strategy adjustment operations, realizing dynamic optimization and real-time adjustment of driving strategies, and effectively improving the safety, comfort and efficiency of driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a schematic execution flow diagram of a driving behavior analysis method based on multi-modal sensor fusion provided by an embodiment of the present invention.

[0009] Figure 2 is a schematic diagram of exemplary hardware and software components of a driving behavior analysis system based on multi-modal sensor fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0010] The present invention will be specifically described below with reference to the accompanying drawings of the specification. Figure 1 is a schematic flow diagram of a driving behavior analysis method based on multi-modal sensor fusion provided by an embodiment of the present invention. The driving behavior analysis method based on multi-modal sensor fusion will be introduced in detail below.

[0011] Step S110: Obtain a multi-source sensor data set of a target vehicle, where the multi-source sensor data set includes driving environment perception data from at least two different sensor sources.

[0012] In an actual application scenario, the target vehicle is equipped with a variety of different types of sensors to collect relevant information about the driving environment. Assume that these sensors are mainly divided into types A, B, C, etc. Sensor A may be a vision sensor, such as a camera, which can collect image data in areas such as in front of and on the side of the vehicle. These image data contain visual feature information of targets such as road signs, other vehicles, and pedestrians. Sensor B can be a radar sensor, which can measure information such as the distance, relative speed, and angle between the target object and the vehicle. These data are output in the form of electrical signals and are converted into specific numerical information after conversion. Sensor C may be an inertial measurement unit, which can sense motion state information such as the acceleration and angular velocity of the vehicle, reflecting the dynamic changes during the vehicle's driving. The data from these different sensors together constitute a multi-source sensor data set, and the data of each sensor has its unique properties and dimensions. For example, the image data collected by sensor A may be a multi-dimensional pixel matrix, the measurement data of sensor B may be a one-dimensional combination of distance, speed, and angle values, and the data of sensor C may be a multi-dimensional vector regarding acceleration and angular velocity.

[0013] Step S120: Perform spatio-temporal alignment processing on the multi-source sensor data set to generate a multi-source fusion data set.

[0014] Due to the different working characteristics of different sensors, there are often differences in the time and space benchmarks of the data they collect. In order to effectively fuse this data, spatio-temporal alignment processing is required, which specifically includes the following sub-steps: Step S121: Perform timestamp alignment processing on each type of source data in the multi-source sensor data set to generate a time-synchronized data set. Among them, the timestamp alignment processing includes interpolating and resampling the source data with different sampling frequencies to achieve a unified time benchmark.

[0015] The sampling frequencies of different sensors are different, which results in inconsistent time intervals for the data they collect. Assume that the sampling frequency of sensor A is fA, the sampling frequency of sensor B is fB, and fA is not equal to fB. In order to achieve timestamp alignment, it is necessary to first determine a unified time benchmark T. For the data collected by each sensor, it is necessary to process it according to this unified time benchmark.

[0016] For the source data of off-frequency sampling, an interpolation resampling method is adopted. Taking the data of sensor A and sensor B as an example, assume that the data collected by sensor A at time points t1, t2, t3... are DA1, DA2, DA3... respectively, and the data collected by sensor B at time points t1', t2', t3'... are DB1, DB2, DB3... respectively. Under the unified time reference T, for a specific time point Tk, if sensor A does not directly collect data at the moment of Tk, the data value at this moment needs to be estimated by interpolation. Common interpolation methods include linear interpolation. Assume that Tk is between ti and t_(i + 1), then the data DAk at the moment of Tk can be estimated according to the data DAi and DA_(i + 1) at the moments of ti and t_(i + 1) through the linear interpolation formula. The same applies to the data of sensor B. After such processing, the data of all sensors are under the unified time reference T, forming a time-synchronized data set.

[0017] Step S122: Perform spatial coordinate transformation processing on the time-synchronized data set to generate a spatially aligned data set, where the spatial coordinate transformation processing includes mapping the coordinate systems of different sensors to a unified vehicle coordinate system.

[0018] Different sensors are installed at different positions on the vehicle, each with its own independent coordinate system. To unify these data spatially, it is necessary to map their coordinate systems to a unified vehicle coordinate system. Assume that the coordinate system of sensor A is SA, the coordinate system of sensor B is SB, and the unified vehicle coordinate system is SV.

[0019] First, it is necessary to determine the transformation relationship of each sensor coordinate system relative to the unified vehicle coordinate system. This can be obtained by measuring the installation position and attitude of the sensor. Assume that the rotation matrix of sensor A relative to the unified vehicle coordinate system is RA and the translation vector is TA, and the rotation matrix of sensor B relative to the unified vehicle coordinate system is RB and the translation vector is TB. For the spatial data point PA (in the coordinate system SA) collected by sensor A, to transform it to the unified vehicle coordinate system SV, the following transformation is required: first rotate PA by the rotation matrix RA to obtain the rotated vector PA', and then add the translation vector TA to obtain the coordinate PV_A in the unified vehicle coordinate system. For the data PB (in the coordinate system SB) of sensor B, similarly rotate it by the rotation matrix RB to obtain PB', and then add the translation vector TB to obtain the coordinate PV_B in the unified vehicle coordinate system. Through such coordinate transformation, the data of all sensors are unified to the vehicle coordinate system, forming a spatially aligned data set.

[0020] Step S123: Based on a preset source fusion rule, perform feature-level fusion processing on the data from different sensors in the spatially aligned data set to generate the multi-source fusion data set. Each data unit in the multi-source fusion data set contains a fusion feature vector of at least two types of source data.

[0021] The preset source fusion rule defines how to fuse data from different sensors. In this step, a feature-level fusion method is adopted. Assume that after the previous spatio-temporal alignment processing, the data of sensor A is FA and the data of sensor B is FB, both of which are multi-dimensional feature vectors.

[0022] First, it is necessary to extract features from the data of different sensors. For the data FA of sensor A, the feature vector VA is extracted through the feature extraction algorithm EA; for the data FB of sensor B, the feature vector VB is extracted through the feature extraction algorithm EB. Then, according to the preset source fusion rule, the feature vectors VA and VB are fused. A common fusion method is weighted concatenation. Weights wA and wB (wA + wB = 1) are assigned to the feature vectors VA and VB respectively, and the weighted feature vectors are concatenated to obtain the fusion feature vector VF. For example, VF = [wA * VA; wB * VB], where ";" represents the concatenation operation. In this way, each data unit contains a fusion feature vector of at least two types of source data, forming a multi-source fusion data set.

[0023] Step S130: Based on the feature extraction layer of a preset behavior analysis network, perform behavior feature extraction processing on the multi-source fusion data set to obtain a driving behavior feature set, where the driving behavior feature set includes vehicle operation features, environmental interaction features, and behavior continuity features.

[0024] The feature extraction layer of the preset behavior analysis network is used to extract features related to driving behavior from the multi-source fusion data set, specifically through the following branches: Step S131: Through the vehicle operation feature extraction branch of the feature extraction layer, perform normalization processing and multi-scale convolution feature extraction on the vehicle control signals in the multi-source fusion data set to generate a standardized vehicle operation feature vector. Among them, the normalization processing includes mapping the steering wheel angle, throttle depth, and brake pressure to a unified numerical interval, and the multi-scale convolution feature extraction uses convolution kernels of different scales to capture local patterns of direction control, acceleration, and braking behaviors respectively.

[0025] The multi-source fusion data set contains control signals of the vehicle, such as steering wheel angle, throttle depth, and brake pressure. First, normalize these control signals. For the throttle depth and brake pressure, linear transformation can be used to map them to the interval [0, 1]. For example, for the throttle depth β, its normalized value β' = (β - β_min) / (β_max - β_min), where β_min and β_max are the minimum and maximum values of the throttle depth respectively; the same applies to the brake pressure γ.

[0026] For the steering wheel angle α, in order to retain its direction information, a symmetric normalization method is used to map it to the interval [-1, 1]. Let the maximum absolute value of the steering wheel angle be α_max_abs, and the normalized steering wheel angle α' = α / α_max_abs. In this way, the steering wheel angle for a left turn is negative, and for a right turn is positive, retaining the direction information.

[0027] Then, perform multi-scale convolution feature extraction. Use convolution kernels of different scales, such as convolution kernels K1, K2, and K3, to perform convolution operations on the normalized vehicle control signals respectively. Convolution kernel K1 can be used to capture local patterns of direction control behavior, convolution kernel K2 is used to capture local patterns of acceleration behavior, and convolution kernel K3 is used to capture local patterns of braking behavior. Through the convolution operation, feature maps of different scales are obtained, and pooling operations, such as max pooling, are performed on these feature maps to compress the dimension of the feature maps, and finally a standardized vehicle operation feature vector V_op is generated.

[0028] Step S132: Through the environmental interaction feature extraction branch of the feature extraction layer, perform coordinate system alignment and dynamic target association analysis on the environmental perception signals in the multi-source fusion data set to generate an environmental interaction feature vector, where the coordinate system alignment includes converting the three-dimensional coordinates of the camera detection target and the radar speed measurement data to the same vehicle coordinate system, and the dynamic target association analysis includes calculating the lateral distance deviation between the host vehicle and the target vehicle, the predicted collision time value, and the signal light state encoding.

[0029] The environmental perception signals in the multi-source fusion data set contain target information detected by the camera and radar speed measurement data. First, perform coordinate system alignment to convert the three-dimensional coordinates of the camera detection target to the same vehicle coordinate system as the radar data. Assume that the coordinates of the target detected by the camera in its own coordinate system are PC, and through the coordinate conversion method mentioned above, convert it to the coordinates PV_C in the unified vehicle coordinate system.

[0030] Then, dynamic target association analysis is performed. For the detected target vehicle, the lateral distance deviation Δd between the host vehicle and the target vehicle is calculated, that is, the distance difference between the host vehicle and the target vehicle in the lateral direction. The original formula for the predicted time to collision TTC is TTC = d_rel / v_rel. However, when the relative speed v_rel approaches zero, TTC tends to infinity and cannot be calculated in practice. To solve this problem, a velocity smoothing term ε is introduced, and the modified formula is TTC = d_rel / (v_rel + ε), where ε is a very small positive number that can be set according to the actual situation, such as 0.01 m / s. This can avoid invalid values of TTC, ensure the effectiveness of TTC in the environmental interaction feature vector, and prevent outliers from damaging the model stability. The signal light state encoding can determine the state of the signal light (such as red, green, yellow) through an image recognition algorithm based on the image information collected by the camera, and encode it as a numerical value. The information calculated above is combined into the environmental interaction feature vector V_env.

[0031] Step S133: Through the behavior continuity feature extraction branch of the feature extraction layer, calculate the sliding window statistics and long short-term dependence modeling for the time series signals in the multi-source fusion data set, and generate a behavior continuity feature vector. Among them, the sliding window statistics calculation includes the extraction of normalized indexes for the volatility of the steering wheel angle, the change frequency of the throttle, and the duration of braking. The long short-term dependence modeling uses a recurrent neural network to extract the temporal behavior pattern.

[0032] The time series signals in the multi-source fusion data set contain information on vehicle operations and environmental perception that changes over time. First, the sliding window statistics are calculated. For the steering wheel angle, calculate its volatility within a sliding window, for example, by calculating the standard deviation of the steering wheel angle within the window to represent the volatility. For the throttle depth, calculate its change frequency within the sliding window, such as counting the number of times the throttle depth changes. For the braking pressure, calculate its duration. Normalize these statistics and map them to a unified numerical interval.

[0033] In terms of long short-term dependence modeling, the original method used a recurrent neural network (such as LSTM) to process the time series signals. However, the recurrent neural network has a high processing delay for long sequences and is difficult to meet the requirements of real-time driving analysis. To solve this problem, a lightweight temporal model TCN (Temporal Convolutional Network) is used to replace the recurrent neural network. TCN has the efficient computing characteristics of a convolutional network and can reduce the computing delay when processing long sequences. Input the time series signals that have undergone sliding window statistics calculation and normalization into the TCN. The network will learn the long short-term dependence relationships in the signals, extract the temporal behavior patterns, and finally generate the behavior continuity feature vector V_cont.

[0034] Step S134: Through the fusion module of the feature extraction layer, perform dimension alignment and weighted fusion on the normalized vehicle operation feature vector, environmental interaction feature vector, and behavior continuity feature vector to generate the driving behavior feature set, where the dimension alignment includes projecting each feature vector into a unified dimension space using a fully connected layer, and the weighted fusion includes dynamically allocating weights to the vehicle operation feature and environmental interaction feature based on the attention mechanism, and applying a time decay factor to the behavior continuity feature to strengthen the influence of recent behaviors.

[0035] First, perform dimension alignment. Use a fully connected layer to process the normalized vehicle operation feature vector V_op, environmental interaction feature vector V_env, and behavior continuity feature vector V_cont. Let the weight matrices of the fully connected layer be W_op, W_env, and W_cont respectively, and the bias vectors be b_op, b_env, and b_cont respectively. Perform a linear transformation on each feature vector through the fully connected layer, that is, V_op' = W_op * V_op + b_op, V_env' = W_env * V_env + b_env, V_cont' = W_cont * V_cont + b_cont, so that they are projected into a unified dimension space.

[0036] Then, perform weighted fusion. Based on the attention mechanism, dynamically allocate weights to the vehicle operation feature vector V_op' and environmental interaction feature vector V_env'. Let the weights calculated by the attention mechanism be w_op_att and w_env_att respectively, and w_op_att + w_env_att = 1. For the behavior continuity feature vector V_cont', apply a time decay factor λ, and the more recent the behavior, the greater its influence on the result. Finally, splice the weighted feature vectors to generate the driving behavior feature set V_drive = [w_op_att * V_op'; w_env_att * V_env'; λ * V_cont'].

[0037] Step S140: Based on the anomaly recognition layer of the preset behavior analysis network, perform anomaly behavior recognition processing on the driving behavior feature set to generate a driving behavior anomaly recognition result.

[0038] The anomaly recognition layer of the preset behavior analysis network is used to analyze the driving behavior feature set and identify the anomaly behaviors therein, specifically including the following steps: Step S141: Input the vehicle operation feature vector, environmental interaction feature vector, and behavior continuity feature vector in the driving behavior feature set into the feature association module of the anomaly recognition layer. Generate a first association weight matrix between the vehicle operation feature vector and the environmental interaction feature vector, and a second association weight matrix between the behavior continuity feature vector and the vehicle operation feature vector through the multi-head attention mechanism.

[0039] Input the vehicle operation feature vector V_op, environmental interaction feature vector V_env, and behavior continuity feature vector V_cont in the driving behavior feature set into the feature association module of the anomaly recognition layer. Using the multi-head attention mechanism, for the vehicle operation feature vector V_op and the environmental interaction feature vector V_env, calculate the association weights between them through multiple attention heads. Let the number of attention heads be n. For the i-th attention head, calculate the association weight matrix A_op_env_i between the vehicle operation feature vector V_op and the environmental interaction feature vector V_env. Concatenate the association weight matrices calculated by all attention heads to obtain the first association weight matrix A_op_env. Similarly, calculate the second association weight matrix A_cont_op between the behavior continuity feature vector V_cont and the vehicle operation feature vector V_op.

[0040] Step S142: Based on the first association weight matrix, perform weighted aggregation on the environmental interaction feature vector to generate a first fusion feature vector aligned with the dimension of the vehicle operation feature vector. At the same time, based on the second association weight matrix, perform weighted aggregation on the behavior continuity feature vector to generate a second fusion feature vector aligned with the dimension of the vehicle operation feature vector.

[0041] Perform weighted aggregation on the environmental interaction feature vector V_env according to the first association weight matrix A_op_env. Let each row of A_op_env represent the association weight between an element of the vehicle operation feature vector and each element of the environmental interaction feature vector. Multiply these weights by the corresponding elements of the environmental interaction feature vector and sum them up to obtain the first fusion feature vector V_fusion1 aligned with the dimension of the vehicle operation feature vector. Similarly, perform weighted aggregation on the behavior continuity feature vector V_cont according to the second association weight matrix A_cont_op to obtain the second fusion feature vector V_fusion2 aligned with the dimension of the vehicle operation feature vector.

[0042] Step S143: Concatenate the vehicle operation feature vector, the first fusion feature vector, and the second fusion feature vector along the channel dimension, and input them into the multi-scale temporal convolutional module of the anomaly recognition layer. Extract driving behavior patterns at different time scales through parallel dilated convolutional layers, and perform max pooling and channel fusion on the temporal features output by each dilated convolutional layer to generate a multi-scale fusion feature tensor.

[0043] Concatenate the vehicle operation feature vector V_op, the first fusion feature vector V_fusion1, and the second fusion feature vector V_fusion2 along the channel dimension to obtain the concatenated feature vector V_concat = [V_op; V_fusion1; V_fusion2]. Input V_concat into the multi-scale temporal convolutional module of the anomaly recognition layer. This module contains multiple parallel dilated convolutional layers, each with a different dilation rate, for extracting driving behavior patterns at different time scales. Let the number of dilated convolutional layers be m. For the j-th dilated convolutional layer, its dilation rate is r_j. Perform a convolution operation on V_concat to obtain the temporal feature T_j. Perform a max pooling operation on the temporal feature T_j output by each dilated convolutional layer to compress its dimension, and then perform channel fusion on all pooled temporal features to generate the multi-scale fusion feature tensor T_fusion.

[0044] Step S144: Input the multi-scale fusion feature tensor into the dynamic scoring module of the anomaly recognition layer, and map it to the anomaly probability value of the current time window through a fully connected layer. Among them, the weight parameters of the dynamic scoring module are adaptively adjusted according to the fluctuation amplitude of the vehicle operation feature vector and the target density of the environmental interaction feature vector, and the output layer uses a non-linear activation function to generate an anomaly score.

[0045] Input the multi-scale fusion feature tensor T_fusion into the dynamic scoring module of the anomaly recognition layer. This module contains a fully connected layer, and maps the multi-scale fusion feature tensor T_fusion to the anomaly probability value of the current time window through the fully connected layer. The weight parameters of the dynamic scoring module will be adaptively adjusted according to the fluctuation amplitude of the vehicle operation feature vector and the target density of the environmental interaction feature vector. Let the fluctuation amplitude of the vehicle operation feature vector be σ_α, and the target density of the environmental interaction feature vector be ρ_env. Since the numerical ranges of σ_α and ρ_env may vary greatly, in order to avoid the weight bias towards a certain feature, σ_α and ρ_env are normalized before calculating the weight.

[0046] Let the mean of σ_α be μ_σ_α and the standard deviation be σ_σ_α. After standardization, σ_α_std = (σ_α - μ_σ_α) / σ_σ_α. Let the mean of ρ_env be μ_ρ_env and the standard deviation be σ_ρ_env. After standardization, ρ_env_std = (ρ_env - μ_ρ_env) / σ_ρ_env.

[0047] Adjust the weight matrix W_score and bias vector b_score of the fully connected layer according to the standardized σ_α_std and ρ_env_std. The output layer uses a non-linear activation function (such as the Sigmoid function) to process the output of the fully connected layer to generate the anomaly score S.

[0048] Step S145: Input the number of dynamic targets, relative speed variance in the environmental interaction feature vector, and the standard deviation of the steering wheel angle in the vehicle operation feature vector into the threshold generation sub-network of the anomaly recognition layer, and calculate the dynamic anomaly determination threshold related to the scenario through the fully connected layer.

[0049] Input the number of dynamic targets N, relative speed variance σ_rel in the environmental interaction feature vector, and the standard deviation of the steering wheel angle σ_α in the vehicle operation feature vector into the threshold generation sub-network of the anomaly recognition layer. Since the dimensions of these three input parameters are not unified, the number of dynamic targets N is a dimensionless count, the dimension of the relative speed variance σ_rel is the square of the speed unit (such as (m / s)²), and the dimension of the standard deviation of the steering wheel angle σ_α is the angle unit (such as °). Directly inputting them into the fully connected layer without standardization will lead to an imbalance in parameter weight distribution.

[0050] Therefore, before inputting into the threshold generation sub-network, standardize N, σ_rel, and σ_α. Let the mean of N be μ_N and the standard deviation be σ_N. After standardization, N_std = (N - μ_N) / σ_N. Let the mean of σ_rel be μ_σ_rel and the standard deviation be σ_σ_rel. After standardization, σ_rel_std = (σ_rel - μ_σ_rel) / σ_σ_rel. Let the mean of σ_α be μ_σ_α and the standard deviation be σ_σ_α. After standardization, σ_α_std = (σ_α - μ_σ_α) / σ_σ_α.

[0051] This sub-network contains a fully connected layer. Through the fully connected layer, a linear transformation is performed on the standardized input parameters [N_std; σ_rel_std; σ_α_std]. Let the weight matrix of the fully connected layer be W_thresh and the bias vector be b_thresh. Then the dynamic anomaly determination threshold T_thresh = W_thresh * [N_std; σ_rel_std; σ_α_std] + b_thresh.

[0052] Step S146: Perform a sliding window cumulative calculation on the anomaly scores of consecutive time windows. When the cumulative score exceeds the dynamic anomaly determination threshold and the duration meets the preset conditions, trigger an anomaly behavior flag and generate an initial anomaly determination sequence.

[0053] Perform a sliding window cumulative calculation on the anomaly score S of consecutive time windows. Assume the size of the sliding window is k. In each time window, accumulate the anomaly scores within the current window to obtain the cumulative score S_sum. When S_sum exceeds the dynamic anomaly determination threshold T_thresh, and this situation persists for a duration that meets the preset conditions (such as the duration being greater than t_0), trigger an anomaly behavior flag and generate an initial anomaly determination sequence S_init, where the elements in S_init are 0 or 1, indicating whether there is an anomaly behavior in this time window.

[0054] Step S147: Input the initial anomaly determination sequence into the temporal correction module of the anomaly recognition layer. Capture the time dependence of historical anomaly flags through a gated recurrent unit network, and combine the time decay factor in the behavior continuity feature vector to perform probability correction on the anomaly flag of the current time window, and output the smoothed driving behavior anomaly recognition result.

[0055] Input the initial anomaly determination sequence S_init into the temporal correction module of the anomaly recognition layer. This temporal correction module uses a gated recurrent unit network (such as GRU) to capture the time dependence of historical anomaly flags. The gated recurrent unit network can effectively handle the long-range dependence relationships in sequence data through its internal gating mechanism. In this scenario, the changing pattern of anomaly behaviors over time can be learned based on the historical anomaly flag information in the initial anomaly determination sequence S_init.

[0056] At each time step of the gated recurrent unit network, receive the initial anomaly flag of the current time window and the hidden state of the previous time step. Assume the current time step is t, the value of the initial anomaly determination sequence S_init at time step t is S_init_t, and the hidden state of the previous time step is h_{t - 1}. The gated recurrent unit network will update the hidden state h_t of the current time step through a series of calculations based on these two inputs. These calculations include the calculation of the reset gate, update gate, and candidate hidden state. The reset gate is used to determine how much information of the hidden state h_{t - 1} of the previous time step needs to be reset, the update gate is used to determine how much information of the hidden state h_{t - 1} of the previous time step needs to be retained to the current time step, and the candidate hidden state is calculated based on the current input and the reset hidden state of the previous time step.

[0057] Meanwhile, the time decay factor λ in the behavior continuity feature vector is incorporated. The time decay factor reflects the importance of recent behaviors for current anomaly determination. The more recent the behavior, the greater its impact on the current anomaly determination. The time decay factor λ is combined with the hidden state h_t output by the gated recurrent unit network to correct the probability of the anomaly flag for the current time window.

[0058] Specifically, let the hidden state h_t output by the gated recurrent unit network be processed through a fully connected layer and a non-linear activation function to obtain a preliminary corrected probability P_pre_t. Then, the preliminary corrected probability P_pre_t and the time decay factor λ are combined with weights. Assuming the weighting coefficients are w_λ and w_pre (w_λ + w_pre = 1), the corrected anomaly probability P_corrected_t = w_λ * λ + w_pre * P_pre_t.

[0059] Finally, the anomaly flag for the current time window is updated according to the corrected anomaly probability P_corrected_t. A threshold T_corrected can be set. If P_corrected_t is greater than T_corrected, it is considered that there is an abnormal behavior in the current time window, and the corresponding anomaly flag is set to 1; otherwise, it is set to 0. Through such processing, the smoothed driving behavior anomaly recognition result S_smoothed is output. This result takes into account the time dependence of historical anomaly flags and the behavior continuity feature, and can more accurately reflect the abnormal conditions in driving behavior.

[0060] Step S150: Generate driving strategy optimization data based on the driving behavior anomaly recognition result, and feedback the driving strategy optimization data to the vehicle control system to trigger a driving strategy adjustment operation.

[0061] Based on the driving behavior anomaly recognition result, it is necessary to generate corresponding driving strategy optimization data to adjust the driving strategy of the vehicle. For example, it specifically includes the following steps: Step S151: Based on the anomaly behavior flag in the driving behavior anomaly recognition result, match the corresponding anomaly correction strategy template from the preset driving strategy optimization rule library. The anomaly correction strategy template includes steering wheel angle correction parameters, desired speed adjustment coefficients, and safe vehicle distance thresholds.

[0062] A preset driving strategy optimization rule library stores abnormal correction strategy templates corresponding to various abnormal behaviors. After obtaining the driving behavior abnormality recognition result S_smoothed, according to the abnormal behavior identifier therein, a match is made in the rule library. Suppose the abnormal behavior identifiers are different category numbers, such as ID_1, ID_2, etc., and each number corresponds to a specific abnormal behavior. For each abnormal behavior category, there is a corresponding abnormal correction strategy template in the rule library. For example, when the abnormal behavior identifier is ID_1, the corresponding abnormal correction strategy template T_template_1 includes the steering wheel angle correction parameter Δα_template_1, the desired speed adjustment coefficient k_v_template_1, and the safe vehicle distance threshold d_safe_template_1. Through this matching operation, the abnormal correction strategy template corresponding to the current abnormal behavior is found.

[0063] Step S152: Extract the vehicle operation feature vector and the environment interaction feature vector from the driving behavior feature set, normalize the standard deviation of the steering wheel angle, the mean throttle depth, and the brake pressure integral in the vehicle operation feature vector to generate an operation state evaluation vector, and at the same time standardize the lateral distance deviation, the predicted collision time, and the relative speed variance of the target vehicle in the environment interaction feature vector to generate an environment risk evaluation vector.

[0064] Extract the vehicle operation feature vector V_op and the environment interaction feature vector V_env from the driving behavior feature set V_drive. For the vehicle operation feature vector V_op, extract the standard deviation of the steering wheel angle σ_α, the mean throttle depth μ_β, and the brake pressure integral I_γ therein. Normalize these parameters. For example, for the standard deviation of the steering wheel angle σ_α, map it to the interval [0, 1] through linear transformation to obtain the normalized standard deviation of the steering wheel angle σ_α_norm. Similarly, normalize the mean throttle depth μ_β and the brake pressure integral I_γ to obtain μ_β_norm and I_γ_norm. Combine these normalized parameters into an operation state evaluation vector V_op_eval = [σ_α_norm; μ_β_norm; I_γ_norm].

[0065] For the environmental interaction feature vector V_env, extract the lateral distance deviation Δd, the predicted collision time TTC, and the relative speed variance σ_rel of the target vehicle from it. Standardize these parameters. For example, by calculating the mean and standard deviation of each parameter, convert them into a standard normal distribution with a mean of 0 and a standard deviation of 1. Let the mean of the lateral distance deviation Δd be μ_Δd and the standard deviation be σ_Δd, then the standardized lateral distance deviation Δd_std = (Δd - μ_Δd) / σ_Δd. Similarly, standardize the predicted collision time TTC and the relative speed variance σ_rel of the target vehicle to obtain TTC_std and σ_rel_std. Combine these standardized parameters into the environmental risk assessment vector V_env_eval = [Δd_std; TTC_std; σ_rel_std].

[0066] Step S153: Input the operation state evaluation vector and the environmental risk assessment vector into the linear weighting module of the anomaly correction policy template, and calculate the dynamic weight of the steering wheel angle correction parameter and the influence factor of the environmental risk assessment vector, where the dynamic weight is adaptively adjusted according to the ratio of the standard deviation of the steering wheel angle to the lateral distance deviation, and the influence factor is generated based on the product of the reciprocal of the predicted collision time and the relative speed variance of the target vehicle.

[0067] Input the operation state evaluation vector V_op_eval and the environmental risk assessment vector V_env_eval into the linear weighting module of the anomaly correction policy template. For the dynamic weight w_Δα of the steering wheel angle correction parameter, it is adaptively adjusted according to the ratio of the standard deviation of the steering wheel angle σ_α_norm in the operation state evaluation vector to the lateral distance deviation Δd_std in the environmental risk assessment vector. For example, w_Δα = f(σ_α_norm / Δd_std), where f is a predefined function that can be designed according to specific requirements to achieve reasonable adjustment of the dynamic weight.

[0068] For the influence factor f_env of the environmental risk assessment vector, it is generated based on the product of the reciprocal of the predicted collision time TTC_std in the environmental risk assessment vector and the relative speed variance σ_rel_std of the target vehicle. That is, f_env = 1 / TTC_std * σ_rel_std. Through such calculations, the dynamic weight w_Δα of the steering wheel angle correction parameter and the influence factor f_env of the environmental risk assessment vector are obtained.

[0069] Step S154: Based on the dynamic weight and the influence factor, perform proportional-integral correction on the steering wheel angle correction parameter to generate an initial angle correction amount. At the same time, calculate the attenuation gradient of the desired speed adjustment coefficient according to the difference between the average throttle depth and the safe distance threshold, and generate a dynamic brake response curve in combination with the time derivative of the brake pressure integral.

[0070] Perform proportional-integral correction on the steering wheel angle correction parameter Δα_template in the abnormal correction strategy template based on the dynamic weight w_Δα and the influence factor f_env. The proportional-integral correction includes a proportional term and an integral term. The proportional term adjusts Δα_template according to the dynamic weight and the influence factor, and the integral term accumulates the historical deviation. Let the proportional coefficient be k_p and the integral coefficient be k_i. The initial angle correction amount Δα_init can be calculated as follows: first calculate the proportional term k_p*w_Δα*f_env*Δα_template, then calculate the integral term k_i*∫(w_Δα*f_env*Δα_template)dt, and add the proportional term and the integral term to obtain the initial angle correction amount Δα_init.

[0071] For the desired speed adjustment coefficient k_v_template, calculate the attenuation gradient according to the difference between the average throttle depth μ_β_norm in the operation state evaluation vector and the safe distance threshold d_safe_template in the abnormal correction strategy template. Let the attenuation gradient be g_kv, g_kv = h(μ_β_norm - d_safe_template), where h is a predefined function.

[0072] Define the brake pressure integral as the cumulative value of the brake pressure within a time window. Let the brake pressure be P_brake(t), and the brake pressure integral I_γ = ∫P_brake(t)dt. The time derivative of the brake pressure integral is the instantaneous brake pressure P_brake(t). Generate a dynamic brake response curve in combination with the instantaneous brake pressure P_brake(t). The dynamic brake response curve describes the change of the brake pressure over time and can be designed according to specific driving scenarios and requirements. For example, it can be represented by a linear or non-linear function.

[0073] Step S155: Check the vehicle dynamics constraints for the initial angle correction amount, limit the initial angle correction amount according to the current vehicle speed and the maximum allowable range of the tire sideslip angle to generate a safe angle correction instruction. At the same time, perform longitudinal acceleration smoothing filtering on the desired speed adjustment coefficient to generate a smoothed speed adjustment instruction.

[0074] Perform vehicle dynamics constraint verification on the initial steering angle correction Δα_init. The steering operation of the vehicle is restricted by the current vehicle speed v and the maximum allowable range δ_max of the tire sideslip angle. According to the vehicle dynamics model, the maximum sideslip angle that the tire can withstand is different at different vehicle speeds. Let the maximum allowable steering angle correction calculated based on the current vehicle speed v be Δα_max. If the initial steering angle correction Δα_init is greater than Δα_max, it is limited to Δα_max; if Δα_init is less than -Δα_max, it is limited to -Δα_max. After the limiting process, a safe steering angle correction command Δα_safe is generated.

[0075] For the desired speed adjustment coefficient k_v_template, perform longitudinal acceleration smoothing filtering. Sudden changes in longitudinal acceleration will affect the ride comfort and driving stability of the vehicle, so it is necessary to smooth the desired speed adjustment coefficient. A low-pass filter can be used to filter k_v_template, such as a first-order low-pass filter, and its output is the smoothed desired speed adjustment coefficient k_v_smooth. A smoothed speed adjustment command is generated according to k_v_smooth, and this command can make the speed adjustment of the vehicle smoother.

[0076] Step S156: Calculate the feedforward control term and feedback compensation term of the brake pressure correction amount based on the dynamic brake response curve and the relative speed variance of the target vehicle in the environmental risk assessment vector, where the feedforward control term is generated according to the deviation ratio between the safe distance threshold and the actual distance, and the feedback compensation term is dynamically adjusted based on the exponential decay function of the predicted time to collision value.

[0077] Based on the dynamic brake response curve and the relative speed variance σ_rel_std of the target vehicle in the environmental risk assessment vector, calculate the feedforward control term and feedback compensation term of the brake pressure correction amount. The feedforward control term is generated according to the deviation ratio between the safe distance threshold d_safe_template in the anomaly correction strategy template and the actual distance d_actual. Let the feedforward control coefficient be k_ff, and the feedforward control term ΔP_ff = k_ff * (d_safe_template - d_actual) / d_safe_template.

[0078] The feedback compensation term is dynamically adjusted based on the exponential decay function of the predicted collision time TTC_std in the environmental risk assessment vector. Let the feedback compensation coefficient be k_fb, and the exponential decay function be exp(-a*TTC_std), where a is a preset parameter. The feedback compensation term ΔP_fb = k_fb*σ_rel_std*exp(-a*TTC_std). Add the feedforward control term and the feedback compensation term to obtain the brake pressure correction ΔP_brake.

[0079] Step S157: Perform control instruction fusion on the safety corner correction instruction, smooth speed adjustment instruction, and brake pressure correction amount to generate multi-dimensional driving strategy optimization data. During the fusion process, a priority arbitration mechanism is used to sort the execution order of the steering instruction and the braking instruction, and phase compensation is performed on the instruction timing based on the response delay parameter of the vehicle control system.

[0080] Fuse the safety corner correction instruction Δα_safe, the smooth speed adjustment instruction k_v_smooth, and the brake pressure correction amount ΔP_brake. During the fusion process, a priority arbitration mechanism is used to sort the execution order of the steering instruction and the braking instruction. Generally, the priority of the braking instruction may be higher than that of the steering instruction, but in an emergency, synchronous adjustment may be required (such as steering + braking when avoiding obstacles). The original step did not define a conflict resolution rule, which may lead to control chaos.

[0081] To solve this problem, a dynamic priority allocation mechanism is introduced. The priorities of the steering instruction and the braking instruction are adjusted in real time according to the collision risk. Let the collision risk assessment value be R_collision, which can be calculated from factors such as the predicted collision time TTC in the environmental interaction feature vector, e.g., R_collision = 1 / TTC. When R_collision is below a certain threshold, according to the normal priority, the priority of the braking instruction is higher than that of the steering instruction; when R_collision is higher than this threshold, it means that the collision risk is high, and both the steering and braking instructions need to be executed simultaneously. At this time, a cooperative control strategy can be adopted to coordinately adjust the steering instruction and the braking instruction to avoid instruction conflicts.

[0082] Meanwhile, based on the response delay parameter τ of the vehicle control system, phase compensation is performed on the instruction timing. Since the vehicle control system takes a certain amount of time to respond after receiving an instruction, the transmission time of the instruction needs to be adjusted. For example, for the brake pressure correction instruction, its transmission time is advanced by τ time to ensure that the vehicle can respond in a timely manner. After priority arbitration and phase compensation, the safe corner correction instruction, smooth speed adjustment instruction, and brake pressure correction amount are concatenated to generate the multi-dimensional driving strategy optimization data D_optimize = [Δα_safe; k_v_smooth; ΔP_brake].

[0083] Step S158: Package the multi-dimensional driving strategy optimization data into a standardized control message according to a preset vehicle control bus protocol, and send it to the steering motor controller, throttle servo unit, and electronic brake control module respectively through the actuator interface of the vehicle control system to trigger real-time driving strategy adjustment operations. At the same time, the adjusted vehicle operation feature vector is sent back to the feature extraction layer of the behavior analysis network for closed-loop feedback verification.

[0084] Package the multi-dimensional driving strategy optimization data D_optimize according to a preset vehicle control bus protocol to generate a standardized control message M_control. The preset vehicle control bus protocol stipulates the data format, transmission rate, communication rules, etc. Through the actuator interface of the vehicle control system, the standardized control message M_control is sent to the steering motor controller, throttle servo unit, and electronic brake control module respectively. The steering motor controller adjusts the steering wheel angle according to the received safe corner correction instruction Δα_safe; the throttle servo unit adjusts the throttle opening according to the smooth speed adjustment instruction k_v_smooth; the electronic brake control module adjusts the brake pressure according to the brake pressure correction amount ΔP_brake, thereby triggering real-time driving strategy adjustment operations.

[0085] Meanwhile, the adjusted vehicle operation feature vector V_op_updated is sent back to the feature extraction layer of the behavior analysis network. The behavior analysis network will perform operations such as behavior feature extraction and abnormal behavior recognition again based on the sent-back adjusted vehicle operation feature vector to evaluate the effect of the driving strategy adjustment and achieve closed-loop feedback verification. If it is found that there are still abnormal behaviors or the effect of the driving strategy adjustment is not ideal, the driving strategy will be optimized and adjusted again until a satisfactory driving effect is achieved.

[0086] Step S210: Obtain a sample multi-source sensor data set and a basic behavior analysis network. The sample multi-source sensor data set includes a multi-source fusion data set that has completed spatio-temporal alignment processing, and the sample multi-source sensor data set is labeled with corresponding driving behavior category labels.

[0087] To train the basic behavior analysis network, it is necessary to obtain a sample multi-source sensor data set and the basic behavior analysis network. The process of obtaining the sample multi-source sensor data set is similar to the previous steps S110 - S120. First, the raw data of different sensors is collected, and then spatio-temporal alignment processing is performed to generate a multi-source fusion data set. The difference is that the sample multi-source sensor data set is also labeled with corresponding driving behavior category labels, such as "abnormal following strategy under complex road conditions". In urban congested road conditions, vehicles need to start and stop frequently and adjust the following distance. Normal following behavior needs to dynamically adjust its own speed and spacing according to the speed, acceleration of the vehicle in front and the distance from the vehicle in front. When there are situations such as inappropriate maintenance of the following distance, frequent and unnecessary acceleration and deceleration, it can be determined as an abnormal following strategy. Another example is "abnormal driving trajectory in a curve". When driving in a curve, the vehicle should maintain a reasonable driving trajectory according to the curvature, slope of the curve and its own speed. If the vehicle shows excessive deviation from the center of the lane or a too tortuous driving trajectory in the curve, it belongs to an abnormal driving trajectory in the curve. Another example is "improper timing and manner of lane change on the highway". When changing lanes on the highway, factors such as the speed and distance of the vehicles in front and behind and the acceleration ability of its own vehicle need to be considered. If a forced lane change is made under unsafe conditions, or the speed changes abnormally during the lane change process, it can be determined as improper timing and manner of lane change. These labels can be obtained through manual annotation or other reliable annotation methods and are used to represent the driving behavior category corresponding to each data sample. The basic behavior analysis network is an initial neural network model, and its structure and parameters still need to be optimized through training.

[0088] Step S220: Load the sample multi-source sensor data set into the feature extraction layer of the basic behavior analysis network to generate a sample driving behavior feature set.

[0089] Input the sample multi-source sensor data set into the feature extraction layer of the basic behavior analysis network. The structure and function of the feature extraction layer are the same as those described in step S130, including a vehicle operation feature extraction branch, an environmental interaction feature extraction branch, a behavior continuity feature extraction branch, and a fusion module. Through the processing of these branches and modules, a sample driving behavior feature set is extracted from the sample multi-source sensor data set, which also includes vehicle operation features, environmental interaction features, and behavior continuity features.

[0090] Step S230: Load the sample driving behavior feature set into the anomaly recognition layer of the basic behavior analysis network to generate a sample anomaly recognition prediction result.

[0091] Input the set of sample driving behavior features into the anomaly recognition layer of the basic behavior analysis network. The structure and function of the anomaly recognition layer are the same as those described in step S140, including a feature association module, a multi-scale temporal convolution module, a dynamic scoring module, a threshold generation sub-network, and a temporal correction module. Through the processing of these modules, perform anomaly behavior recognition on the set of sample driving behavior features to generate sample anomaly recognition prediction results. This result represents the anomaly behavior prediction of the basic behavior analysis network for each sample data.

[0092] Step S240: Calculate the first network learning cost based on the sample anomaly recognition prediction results and the driving behavior category labels.

[0093] Calculate the first network learning cost according to the sample anomaly recognition prediction results and the driving behavior category labels annotated in the sample multi-source sensor data set. Methods such as the cross-entropy loss function can be used to calculate the learning cost. Let the sample anomaly recognition prediction result be P_pred, the driving behavior category label be P_true, and the cross-entropy loss function be L_ce. For each sample data, calculate its cross-entropy loss L_ce_i, and then sum or average the cross-entropy losses of all sample data to obtain the first network learning cost L_first. The first network learning cost reflects the degree of difference between the prediction result of the basic behavior analysis network and the true label. The smaller the learning cost, the more accurate the network's prediction.

[0094] Step S250: Jointly train the feature extraction layer and the anomaly recognition layer of the basic behavior analysis network based on the first network learning cost to generate an optimized behavior analysis network.

[0095] Based on the first network learning cost L_first, jointly train the feature extraction layer and the anomaly recognition layer of the basic behavior analysis network. Use the backpropagation algorithm to calculate the gradient according to the first network learning cost, and then update the parameters of the feature extraction layer and the anomaly recognition layer according to the gradient. During the training process, continuously adjust the parameters of the network to gradually reduce the first network learning cost. After multiple iterative trainings, an optimized behavior analysis network is obtained. The feature extraction layer and the anomaly recognition layer of this network achieve the generalization extraction of driving behavior features and the anomaly pattern recognition ability through parameter optimization.

[0096] Step S260: Use the optimized behavior analysis network as the preset behavior analysis network, where the feature extraction layer and the anomaly recognition layer of the preset behavior analysis network achieve the generalization extraction of driving behavior features and the anomaly pattern recognition ability through parameter optimization.

[0097] Use the optimized behavior analysis network obtained through training as the preset behavior analysis network. This network can more accurately extract driving behavior-related features from multi-source sensor data in the feature extraction layer, including vehicle operation features, environmental interaction features, and behavior continuity features. In the anomaly recognition layer, it can more accurately identify abnormal patterns in driving behavior and generate accurate anomaly recognition results. In this way, the accuracy and reliability of the entire driving behavior analysis method are improved.

[0098] Step S310: Obtain an augmented example multi-source sensor data set, which is generated by performing data augmentation processing on the example multi-source sensor data set. The data augmentation processing includes at least one of the following operations: random time jitter, feature channel noise injection, and local feature masking.

[0099] To further improve the performance and generalization ability of the preset behavior analysis network, it is necessary to perform data augmentation processing on the existing example multi-source sensor data set to generate an augmented example multi-source sensor data set. The purpose of data augmentation processing is to increase the diversity of data so that the network can learn feature patterns in more different scenarios.

[0100] First, perform the random time jitter operation. Select a target data segment from the example multi-source sensor data set. Assume that the target data segment contains data from different sensors, such as data DA of sensor A, data DB of sensor B, etc. For the data DA of sensor A, perform random time jitter processing. The time offset of the random time jitter needs to be controlled within the range not exceeding the preset time threshold, and to avoid spatio-temporal misalignment of multi-sensor data, the jitter range is limited to be less than the sensor sampling interval. Assume that the preset time threshold is T_thresh and the sensor sampling interval is T_sample. Randomly generate a time offset Δt, and -min(T_thresh, T_sample) ≤ Δt ≤ min(T_thresh, T_sample).

[0101] Translate the data DA of sensor A according to the time offset Δt to obtain the time-augmented data DA_time. Since time jitter may affect the accuracy of spatial coordinate transformation, spatial coordinate transformation needs to be performed again after time jitter. Assume that the coordinate system of sensor A is SA, the unified vehicle coordinate system is SV, the original rotation matrix of sensor A relative to the unified vehicle coordinate system is RA, and the translation vector is TA. After time jitter, according to the new timestamp and the motion state of the sensor, recalculate the rotation matrix RA' and the translation vector TA', and transform the time-augmented data DA_time from the coordinate system SA to the unified vehicle coordinate system SV.

[0102] Next, a feature channel noise injection operation is performed. For the data DB of sensor B in the target data segment, a feature channel noise injection process is carried out. The amplitude of the injected noise should not exceed a preset ratio of the original feature value. Let the preset ratio be r_noise. For each feature channel in the data DB, a noise vector N is randomly generated, and the amplitude of each element in the noise vector N does not exceed r_noise times the original feature value of that feature channel. The noise vector N is added to the data DB to obtain the noise-augmented data DB_noise. This can simulate the possible noise interference in the actual environment and make the network more robust to noise.

[0103] In addition, a local feature masking operation can also be performed. In the target data segment, some local regions are randomly selected for feature masking. For example, for the data DC of sensor C, one or more consecutive time steps and feature channels are randomly selected, and the data values in these regions are set to zero. Through the local feature masking operation, the network can learn more important feature information in the data and avoid over-reliance on certain local features.

[0104] Through at least one of the above data augmentation operations, each data segment in the sample multi-source sensor data set is processed, and finally an augmented sample multi-source sensor data set is generated. The data in this set not only contains the information of the original sample multi-source sensor data set but also adds more different variant data samples, providing richer training materials for subsequent network training.

[0105] Step S320: Load the augmented sample multi-source sensor data set into the preset behavior analysis network to generate an augmented driving behavior feature set.

[0106] The generated augmented sample multi-source sensor data set is input into the preset behavior analysis network. The feature extraction layer of the preset behavior analysis network will process the augmented sample multi-source sensor data set, and its processing process is similar to that of step S130.

[0107] In the vehicle operation feature extraction branch, the vehicle control signals in the augmented sample multi-source sensor data set are normalized and multi-scale convolutional feature extraction is performed. For the vehicle control signals after random time jitter, feature channel noise injection, or local feature masking, they are still normalized according to the previous method, and the steering wheel angle, throttle depth, and brake pressure are mapped to a unified numerical interval. Then, convolutional kernels of different scales are used to capture the local patterns of direction control, acceleration, and braking behaviors respectively to generate a standardized vehicle operation feature vector.

[0108] In the environmental interaction feature extraction branch, coordinate system alignment and dynamic target association analysis are performed on the environmental perception signals in the augmented sample multi-source sensor data set. Even though the environmental perception signals have undergone data augmentation processing, the three-dimensional coordinates of the camera-detected targets and the radar speed measurement data still need to be converted to the same vehicle coordinate system, and the lateral distance deviation between the host vehicle and the target vehicle, the predicted collision time, and the signal light status encoding are calculated to generate an environmental interaction feature vector.

[0109] In the behavior continuity feature extraction branch, sliding window statistic calculation and long short-term dependence modeling are performed on the time series signals in the augmented sample multi-source sensor data set. For the time series signals processed by time jitter and other methods, the normalized indexes of the steering wheel angle volatility, the throttle change frequency, and the brake duration are calculated, and a recurrent neural network is used to extract the temporal behavior pattern to generate a behavior continuity feature vector.

[0110] Finally, through the fusion module of the feature extraction layer, dimension alignment and weighted fusion are performed on the generated standardized vehicle operation feature vector, environmental interaction feature vector, and behavior continuity feature vector to generate an augmented driving behavior feature set. This set reflects the driving behavior features corresponding to the augmented sample multi-source sensor data set and contains more different feature patterns generated due to data augmentation.

[0111] Step S330: Calculate the second network learning cost based on the augmented anomaly recognition prediction result corresponding to the augmented driving behavior feature set and the driving behavior category label of the original sample multi-source sensor data set.

[0112] Input the augmented driving behavior feature set into the anomaly recognition layer of the preset behavior analysis network. The processing process of this layer is the same as that in step S140. Through the processing of the feature association module, multi-scale temporal convolutional module, dynamic scoring module, threshold generation sub-network, and temporal correction module, anomaly behavior recognition is performed on the augmented driving behavior feature set to generate an augmented anomaly recognition prediction result.

[0113] Then, calculate the second network learning cost according to the augmented anomaly recognition prediction result and the driving behavior category label of the original sample multi-source sensor data set. The cross-entropy loss function and other methods can also be used for calculation. Let the augmented anomaly recognition prediction result be P_pred_aug, and the driving behavior category label of the original sample multi-source sensor data set be P_true_ori. For each augmented data sample, calculate its cross-entropy loss L_ce_aug_i, and then sum or average the cross-entropy losses of all augmented data samples to obtain the second network learning cost L_second. The second network learning cost reflects the difference degree between the prediction result of the preset behavior analysis network when processing augmented data and the original true label, and it is used to evaluate the performance of the network when facing more diverse data.

[0114] Step S340: Perform incremental training on the preset behavior analysis network based on the second network learning cost to generate an updated behavior analysis network.

[0115] Based on the calculated second network learning cost L_second, perform incremental training on the preset behavior analysis network. The purpose of incremental training is to further optimize the parameters of the network on the basis of the existing preset behavior analysis network so that it can better process various data patterns in the augmented sample multi-source sensor data set.

[0116] Adopt the backpropagation algorithm to calculate the gradient according to the second network learning cost. The gradient represents the change direction and degree of the network parameters under the current training data. According to the calculated gradient, update the parameters of the feature extraction layer and the anomaly recognition layer of the preset behavior analysis network. During the update process, parameters such as the learning rate can be used to control the step size of parameter update to avoid too fast or too slow parameter update.

[0117] During the incremental training process, continuously adjust the parameters of the network to gradually reduce the second network learning cost. After multiple iterative trainings, the network can learn new feature patterns and anomaly patterns in the augmented data, thereby improving its recognition ability of driving behaviors in different scenarios. Finally, an updated behavior analysis network is generated, which has higher accuracy and generalization ability when processing multi-source sensor data for driving behavior analysis and anomaly recognition.

[0118] Step S410: The specific process of obtaining the sample multi-source sensor data set includes collecting the original multi-source sensor data stream, where the original multi-source sensor data stream includes a first source data stream and a second source data stream, and the original multi-source sensor data stream is labeled with driving behavior category labels.

[0119] When obtaining the sample multi-source sensor data set, first collect the original multi-source sensor data stream. Assume that the first source data stream comes from sensor X and the second source data stream comes from sensor Y. Sensor X may be a sensor for detecting the vehicle's surrounding environment, such as a lidar, which can collect information such as the distance and angle of objects around the vehicle in real time to form the first source data stream. Sensor Y may be a sensor for monitoring the vehicle's own state, such as an in-vehicle CAN bus sensor, which can obtain information such as the vehicle's speed, acceleration, and steering wheel angle to form the second source data stream.

[0120] While collecting these original data streams, it is necessary to label them with driving behavior category tags. These driving behavior category tags are more complex and difficult to directly judge than the previous examples. For example, "abnormal driving behavior at complex intersections". At complex multi-lane intersections, vehicles need to perform reasonable passing operations according to traffic lights, traffic signs, and the dynamics of other vehicles and pedestrians. If there are situations such as driving in the wrong lane or hesitating at the intersection causing traffic jams, it can be labeled as abnormal driving behavior at complex intersections. Another example is "abnormal driving stability in bad weather". Under bad weather conditions such as rainy days and snowy days, the road surface friction decreases, and the vehicle's handling performance will be affected. When driving normally, it is necessary to control the vehicle speed, brakes, and steering more carefully. If the vehicle experiences excessive skidding, sudden braking resulting in tire locking, etc., it belongs to abnormal driving stability in bad weather. Another example is "violation of parking and driving regulations in the parking lot". In the parking lot, vehicles need to drive and park according to the specified routes. If there are situations such as driving in reverse or randomly occupying the fire lane for parking, it can be determined as a violation of parking and driving regulations in the parking lot. Through such labeling, clear target tags are provided for subsequent model training, enabling the model to learn the sensor data characteristics corresponding to different complex driving behaviors.

[0121] Step S420: Perform timestamp alignment processing on the first source data stream and the second source data stream to generate a time-synchronized data stream.

[0122] Since the sampling frequencies and time bases of sensor X and sensor Y may be different, it is necessary to perform timestamp alignment processing on the first source data stream and the second source data stream. Let the sampling frequency of sensor X be fX, the sampling frequency of sensor Y be fY, and fX is not equal to fY.

[0123] First, determine a unified time base T. For each data point in the first source data stream, record its original timestamp tX_i. Similarly, for each data point in the second source data stream, record its original timestamp tY_j. Then, according to the unified time base T, perform interpolation resampling on these data points.

[0124] For the first source data stream, if there is no directly corresponding sampling data point at a certain unified time point Tk, interpolation methods need to be used to estimate the data value at this time point. Linear interpolation can be adopted. Assume that Tk is between tX_m and tX_(m + 1), and according to the data values at tX_m and tX_(m + 1), estimate the data value at Tk using the linear interpolation formula. The same applies to the second source data stream.

[0125] After such interpolation resampling processing, the first source data stream and the second source data stream have the same time step under the unified time reference T, thereby generating a time-synchronized data stream. Each data point in this data stream corresponds to the same time point, providing temporal consistency for subsequent spatial coordinate transformation and data fusion.

[0126] Step S430: Perform spatial coordinate transformation processing on the time-synchronized data stream to generate a spatially aligned data stream.

[0127] The installation positions and coordinate systems of different sensors may be different. Therefore, it is necessary to perform spatial coordinate transformation processing on the time-synchronized data stream to unify it into the same coordinate system. Assume that the coordinate system of sensor X is SX, the coordinate system of sensor Y is SY, and the unified vehicle coordinate system is SV.

[0128] First, it is necessary to determine the transformation relationship of sensor X and sensor Y relative to the unified vehicle coordinate system. By measuring the installation position and attitude of the sensors, the rotation matrix RX and translation vector TX of sensor X relative to the unified vehicle coordinate system, and the rotation matrix RY and translation vector TY of sensor Y relative to the unified vehicle coordinate system can be obtained.

[0129] For the data point PX of sensor X in the time-synchronized data stream (in coordinate system SX), to transform it to the unified vehicle coordinate system SV, first rotate PX by the rotation matrix RX to obtain the rotated vector PX', and then add the translation vector TX to obtain the coordinate PV_X in the unified vehicle coordinate system. For the data point PY of sensor Y (in coordinate system SY), similarly rotate it by the rotation matrix RY to obtain PY', and then add the translation vector TY to obtain the coordinate PV_Y in the unified vehicle coordinate system.

[0130] Through such coordinate transformation, all data points in the time-synchronized data stream are unified into the vehicle coordinate system, generating a spatially aligned data stream. The data in this data stream is spatially consistent, facilitating subsequent feature-level fusion processing.

[0131] Step S440: Perform feature-level fusion processing on the spatially aligned data stream based on the source fusion strategy to generate a multi-source fusion data stream.

[0132] After obtaining the spatially aligned data stream, it is necessary to perform feature-level fusion processing on it based on the source fusion strategy. The source fusion strategy defines how to fuse the data of sensor X and sensor Y to make full use of the information of different sensors.

[0133] First, extract features from the data of sensor X and sensor Y in the spatially aligned data stream. For the data of sensor X, extract the feature vector VX through the feature extraction algorithm EX; for the data of sensor Y, extract the feature vector VY through the feature extraction algorithm EY.

[0134] Then, fuse the feature vectors VX and VY according to the source fusion strategy. A common fusion method is weighted concatenation. Assign weights wX and wY (wX + wY = 1) to the feature vectors VX and VY respectively, and concatenate the weighted feature vectors to obtain the fused feature vector VF. For example, VF = [wX * VX; wY * VY], where ";" represents the concatenation operation.

[0135] Through such feature-level fusion processing, the data of sensor X and sensor Y are fused together to generate a multi-source fusion data stream, and each data point in it contains the fused feature information of sensor X and sensor Y.

[0136] Step S450: Perform time window segmentation processing on the multi-source fusion data stream to generate the example multi-source sensor data set, where each data segment of the time window corresponds to the driving behavior category label one by one.

[0137] To convert the multi-source fusion data stream into an example multi-source sensor data set suitable for model training, time window segmentation processing needs to be performed on it. The specific steps are as follows: Step S451: Determine the behavior start time point and behavior end time point corresponding to the driving behavior category label.

[0138] For more complex driving behavior category labels, determining the behavior start and end time points requires more detailed analysis and avoiding problems caused by subjective annotation. Taking "abnormal driving trajectory on a curve" as an example, objective indicators are used to determine the start and end times. Set the lane departure distance threshold D_thresh and the duration threshold N seconds. When the vehicle enters the curve, continuously monitor the distance D between the vehicle's driving trajectory and the center of the lane. When D is greater than D_thresh and the duration exceeds N seconds, it is considered that the abnormal behavior of the driving trajectory on the curve starts. When the vehicle exits the curve and D is less than D_thresh and lasts for a period of time (such as M seconds), it is considered that the abnormal behavior ends.

[0139] For "abnormal following vehicle strategy under complex road conditions", factors such as the relative distance and speed difference between the host vehicle and the leading vehicle can be comprehensively considered. Set a relative distance threshold d_thresh and a speed difference threshold v_thresh. When the relative distance between the host vehicle and the leading vehicle continuously remains less than d_thresh and the speed difference continuously remains greater than v_thresh for a certain period of time (such as P seconds), it is considered that the abnormal following vehicle strategy behavior starts. When the relative distance recovers above d_thresh and the speed difference recovers within v_thresh and lasts for a period of time (such as Q seconds), it is considered that the abnormal behavior ends.

[0140] For "improper timing and manner of lane change on highway", factors such as the turn signal of the vehicle, the speed change before and after the lane change, and the safe distance from the front and rear vehicles are combined. When the vehicle turns on the turn signal, if it starts to move under the condition of not meeting the safe lane change condition (such as the safe distance from the front and rear vehicles is less than the safety threshold s_thresh) and lasts for a period of time (such as R seconds), it is considered that the abnormality starts. When the vehicle completes the lane change and the relative position and speed relationship with the surrounding vehicles return to normal (such as the safe distance from the front and rear vehicles is greater than s_thresh) and lasts for a period of time (such as S seconds), it is considered that the abnormality ends.

[0141] Step S452: Taking the behavior start time point and the behavior end time point as boundaries, intercept the data segment that matches the driving behavior category label from the multi-source fusion data stream.

[0142] Perform normalization processing on the intercepted data segment to improve the quality and consistency of the data. Taking the data segment of "abnormal following vehicle strategy under complex road conditions" as an example, first perform denoising processing to remove possible noise interference in the sensor data. Filtering algorithms such as moving average filtering and median filtering can be used to smooth the data such as the speed, acceleration of the host vehicle, and the relative distance from the leading vehicle, and remove high-frequency noise. Then perform normalization processing to map each eigenvalue in the data to a unified numerical interval. For example, for the vehicle speed eigenvalue v, it can be normalized to the [0, 1] interval through linear transformation, that is, v_norm = (v - v_min) / (v_max - v_min), where v_min and v_max are the minimum and maximum values of the speed eigenvalue in the data segment respectively. For other features such as acceleration and relative distance, the same normalization processing is performed. Finally, perform feature scaling. According to the importance and variation range of different features, the eigenvalues are appropriately scaled. For example, for the relative distance feature, since its variation range may be large, it can be scaled to have a similar scale to features such as speed and acceleration. Through these normalization processes, a normalized behavior data segment is generated. For the data segments of behaviors such as "abnormal driving trajectory on a curve" and "improper timing and manner of lane change on highway", the same denoising, normalization, and feature scaling processes are performed.

[0143] Step S453: Perform normalization processing on the data segment to generate a normalized behavior data segment, where the normalization processing includes denoising, normalization, and feature scaling.

[0144] Perform normalization processing on the intercepted data segment to improve the quality and consistency of the data. First, perform denoising processing to remove possible noise interference in the data. Filtering algorithms such as moving average filtering and median filtering can be used to smooth the data and remove high-frequency noise.

[0145] Then perform normalization processing to map each feature value in the data to a unified numerical interval. For example, for a certain feature value x in the data segment, it can be normalized to the [0, 1] interval through linear transformation, that is, x_norm = (x - x_min) / (x_max - x_min), where x_min and x_max are the minimum and maximum values of the feature value in the data segment, respectively.

[0146] Finally, perform feature scaling. According to the importance and variation range of different features, appropriately scale the feature values. For example, for some features with a large variation range, scale them to have a similar scale to other features. Through these normalization processes, a normalized behavior data segment is generated.

[0147] Step S454: Associate and store the normalized behavior data segment with the corresponding driving behavior category label to generate the sample multi-source sensor data set.

[0148] Associate the generated normalized behavior data segment with the corresponding driving behavior category label. Data structures such as dictionaries or database records can be used to store the normalized behavior data segment as the value in the key-value pair and the corresponding driving behavior category label as the key. For example, associate and store the normalized behavior data segment corresponding to "abnormal following strategy under complex road conditions" with this label. Through such associated storage, a sample multi-source sensor data set is formed. Each data segment in this sample multi-source sensor data set corresponds to a clear driving behavior category label, providing accurate training data and labels for subsequent model training.

[0149] Figure 2 FIG. shows a schematic diagram of exemplary hardware and software components of a driving behavior analysis system 100 based on multi-modal sensor fusion that can implement the idea of the present application provided by some embodiments of the present application. For example, the processor 120 can be used on the driving behavior analysis system 100 based on multi-modal sensor fusion and is used to execute the functions in the present application.

[0150] The driving behavior analysis system 100 based on multi-modal sensor fusion can be a general-purpose server or a special-purpose server, both of which can be used to implement the driving behavior analysis method based on multi-modal sensor fusion of the present application. Although only one server is shown in the present application, for convenience, the functions described in the present application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0151] For example, the driving behavior analysis system 100 based on multi-modal sensor fusion may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and different forms of storage media 140, such as disks, ROM, or RAM, or any combination thereof. Exemplarily, the driving behavior analysis system 100 based on multi-modal sensor fusion may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The driving behavior analysis system 100 based on multi-modal sensor fusion also includes an I / O interface 150 between the computer and other input / output devices.

[0152] For ease of explanation, only one processor is described in the driving behavior analysis system 100 based on multi-modal sensor fusion. However, it should be noted that the driving behavior analysis system 100 based on multi-modal sensor fusion in the present application may also include multiple processors. Therefore, the steps executed by one processor described in the present application can also be jointly executed or separately executed by multiple processors. For example, if the processor of the driving behavior analysis system 100 based on multi-modal sensor fusion executes step A and step B, it should be understood that step A and step B can also be jointly executed by two different processors or separately executed in one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor jointly execute steps A and B.

[0153] In addition, the embodiments of the present invention also provide a readable storage medium, in which computer-executable instructions are preset. When the processor executes the computer-executable instructions, the driving behavior analysis method based on multi-modal sensor fusion as described above is implemented.

[0154] It should be noted that, in order to simplify the description of the present invention disclosure and thus help the understanding of one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, sometimes multiple features are merged into one embodiment, drawing, or description thereof.

Claims

1. A driving behavior analysis method based on multimodal sensor fusion, characterized in that: The method comprises: Acquire a multi-source sensor data set of a target vehicle, wherein the multi-source sensor data set includes driving environment perception data from at least two different sensor sources; Performing spatiotemporal alignment processing on the multi-source sensor data set to generate a multi-source fusion data set; Based on the feature extraction layer of the preset behavior analysis network, the multi-source fusion data set is subjected to behavior feature extraction processing to obtain a driving behavior feature set, wherein the driving behavior feature set includes vehicle operation features, environment interaction features, and behavior continuity features; Based on the abnormal recognition layer of the preset behavior analysis network, the driving behavior feature set is subjected to abnormal behavior recognition processing to generate a driving behavior abnormality recognition result; Driving strategy optimization data is generated according to the abnormal driving behavior identification result, and the driving strategy optimization data is fed back to the vehicle control system to trigger a driving strategy adjustment operation.

2. The driving behavior analysis method based on multimodal sensor fusion according to claim 1 is characterized in that: The performing spatiotemporal alignment processing on the multi-source sensor data set to generate a multi-source fusion data set includes: Performing a timestamp alignment process on each source data in the multi-source sensor data set to generate a time synchronization data set, wherein the timestamp alignment process includes performing interpolation resampling on the source data sampled at different frequencies to achieve a unified time reference; Performing spatial coordinate conversion processing on the time synchronization data set to generate a spatial alignment data set, wherein the spatial coordinate conversion processing includes mapping the coordinate systems of different sensors to a unified vehicle coordinate system; Based on a preset source fusion rule, feature-level fusion processing is performed on data from different sensors in the spatial alignment data set to generate the multi-source fusion data set, wherein each data unit in the multi-source fusion data set contains a fusion feature vector of at least two source data.

3. The driving behavior analysis method based on multimodal sensor fusion according to claim 2 is characterized in that: The method further comprises: Acquire a sample multi-source sensor data set and a basic behavior analysis network, wherein the sample multi-source sensor data set includes a multi-source fusion data set that has completed spatiotemporal alignment processing, and the sample multi-source sensor data set is annotated with a corresponding driving behavior category label; Loading the sample multi-source sensor data set into the feature extraction layer of the basic behavior analysis network to generate a sample driving behavior feature set; Loading the sample driving behavior feature set into the anomaly recognition layer of the basic behavior analysis network to generate a sample anomaly recognition prediction result; Calculating a first network learning cost based on the sample abnormality recognition prediction result and the driving behavior category label; Based on the first network learning cost, jointly train the feature extraction layer and the abnormality recognition layer of the basic behavior analysis network to generate an optimized behavior analysis network; The optimized behavior analysis network is used as the preset behavior analysis network, wherein the feature extraction layer and the abnormality recognition layer of the preset behavior analysis network achieve generalized extraction of driving behavior features and abnormal pattern recognition capabilities through parameter optimization.

4. The driving behavior analysis method based on multimodal sensor fusion according to claim 3 is characterized in that: The step of obtaining a sample multi-source sensor data set includes: Collecting an original multi-source sensor data stream, wherein the original multi-source sensor data stream includes a first source data stream and a second source data stream, and the original multi-source sensor data stream is annotated with a driving behavior category label; Performing timestamp alignment processing on the first source data stream and the second source data stream to generate a time synchronization data stream; Performing spatial coordinate conversion processing on the time synchronization data stream to generate a spatial alignment data stream; Based on the source fusion strategy, the spatially aligned data stream is subjected to feature-level fusion processing to generate a multi-source fused data stream; The multi-source fusion data stream is segmented into time windows to generate the sample multi-source sensor data set, wherein the data segment of each time window corresponds one-to-one to the driving behavior category label.

5. The driving behavior analysis method based on multimodal sensor fusion according to claim 4 is characterized in that: The step of performing time window segmentation processing on the multi-source fusion data stream to generate the sample multi-source sensor data set includes: Determine the behavior start time point and behavior end time point corresponding to the driving behavior category label; Taking the behavior start time point and the behavior end time point as boundaries, extracting data segments matching the driving behavior category label from the multi-source fusion data stream; Performing standardization processing on the data segments to generate standardized behavior data segments, wherein the standardization processing includes denoising, normalization, and feature scaling; The standardized behavior data segments are associated with corresponding driving behavior category labels and stored to generate the sample multi-source sensor data set.

6. The driving behavior analysis method based on multimodal sensor fusion according to claim 3 is characterized in that: The method further comprises: Acquire an augmented sample multi-source sensor data set, wherein the augmented sample multi-source sensor data set is generated by performing data augmentation processing on the sample multi-source sensor data set, wherein the data augmentation processing includes at least one of the following operations: random time jitter, feature channel noise injection, and local feature shielding; Loading the augmented sample multi-source sensor data set into the preset behavior analysis network to generate an augmented driving behavior feature set; Calculating a second network learning cost based on the augmented anomaly recognition prediction result corresponding to the augmented driving behavior feature set and the driving behavior category label of the original sample multi-source sensor data set; Incrementally train the preset behavior analysis network based on the second network learning cost to generate an updated behavior analysis network.

7. The driving behavior analysis method based on multimodal sensor fusion according to claim 6 is characterized in that: The performing data augmentation processing on the sample multi-source sensor data set includes: Randomly selecting a target data segment from the sample multi-source sensor data set; Performing random time jitter processing on the first source data in the target data segment to generate time augmented data, wherein the time offset of the random time jitter processing does not exceed a preset time threshold; Performing feature channel noise injection processing on the second source data in the target data segment to generate noise augmented data, wherein the injected noise amplitude does not exceed a preset proportion of the original feature value; Performing feature fusion on the time augmented data and the noise augmented data to generate an augmented data segment; The augmented data segment is associated with the original driving behavior category label and then added to the sample multi-source sensor data set to generate the augmented sample multi-source sensor data set.

8. The driving behavior analysis method based on multimodal sensor fusion according to claim 1, characterized in that: The feature extraction layer based on the preset behavior analysis network performs behavior feature extraction processing on the multi-source fusion data set to obtain a driving behavior feature set, including: The vehicle control signal in the multi-source fusion data set is normalized and multi-scale convolution feature extracted by the vehicle operation feature extraction branch of the feature extraction layer to generate a standardized vehicle operation feature vector, wherein the normalization includes mapping the steering wheel angle, throttle depth and brake pressure to a uniform numerical interval, and the multi-scale convolution feature extraction uses convolution kernels of different scales to capture local patterns of direction control, acceleration and braking behaviors respectively; Through the environmental interaction feature extraction branch of the feature extraction layer, coordinate system alignment and dynamic target association analysis are performed on the environmental perception signals in the multi-source fusion data set to generate an environmental interaction feature vector, wherein the coordinate system alignment includes converting the three-dimensional coordinates of the camera detection target and the radar speed measurement data into the same vehicle coordinate system, and the dynamic target association analysis includes calculating the lateral distance deviation between the host vehicle and the target vehicle, the collision time prediction value and the signal light state coding; Through the behavior continuity feature extraction branch of the feature extraction layer, the time series signal in the multi-source fusion data set is subjected to sliding window statistics calculation and long-term and short-term dependency modeling to generate a behavior continuity feature vector, wherein the sliding window statistics calculation includes extraction of normalized indicators of steering wheel angle volatility, throttle change frequency and braking duration, and the long-term and short-term dependency modeling uses a recurrent neural network to extract time series behavior patterns; The standardized vehicle operation feature vector, the environment interaction feature vector and the behavior continuity feature vector are dimensionally aligned and weightedly fused through the fusion module of the feature extraction layer to generate the driving behavior feature set, wherein the dimension alignment includes projecting each feature vector into a unified dimensional space using a fully connected layer, and the weighted fusion includes dynamically allocating weights to the vehicle operation features and the environment interaction features based on the attention mechanism, and applying a time decay factor to the behavior continuity features to strengthen the impact of recent behavior.

9. The driving behavior analysis method based on multimodal sensor fusion according to claim 8, characterized in that: The abnormality recognition layer based on the preset behavior analysis network performs abnormal behavior recognition processing on the driving behavior feature set to generate a driving behavior abnormality recognition result, including: Inputting the vehicle operation feature vector, the environment interaction feature vector and the behavior continuity feature vector in the driving behavior feature set into the feature association module of the abnormality recognition layer, and generating a first association weight matrix between the vehicle operation feature vector and the environment interaction feature vector, and a second association weight matrix between the behavior continuity feature vector and the vehicle operation feature vector through a multi-head attention mechanism; Performing weighted aggregation on the environment interaction feature vector based on the first association weight matrix to generate a first fused feature vector dimensionally aligned with the vehicle operation feature vector, and performing weighted aggregation on the behavior continuity feature vector based on the second association weight matrix to generate a second fused feature vector dimensionally aligned with the vehicle operation feature vector; The vehicle operation feature vector, the first fused feature vector and the second fused feature vector are spliced ​​according to the channel dimension, and input into the multi-scale temporal convolution module of the abnormal recognition layer, and the driving behavior patterns of different time scales are extracted through the parallel expansion convolution layers, and the temporal features output by each expansion convolution layer are subjected to maximum pooling and channel fusion to generate a multi-scale fusion feature tensor; The multi-scale fusion feature tensor is input into the dynamic scoring module of the anomaly recognition layer, and is mapped into an anomaly probability value of the current time window through a fully connected layer, wherein the weight parameter of the dynamic scoring module is adaptively adjusted according to the fluctuation amplitude of the vehicle operation feature vector and the target density of the environment interaction feature vector, and the output layer uses a nonlinear activation function to generate an anomaly score; The number of dynamic targets and the relative speed variance in the environmental interaction feature vector and the standard deviation of the steering wheel angle in the vehicle operation feature vector are input into the threshold generation subnetwork of the anomaly recognition layer, and the scene-related dynamic anomaly determination threshold is calculated through the fully connected layer; Perform sliding window cumulative calculation on the abnormality scores of continuous time windows, and when the cumulative score exceeds the dynamic abnormality determination threshold and the duration meets the preset conditions, trigger the abnormal behavior identification and generate an initial abnormality determination sequence; The initial abnormality judgment sequence is input into the timing correction module of the abnormality recognition layer, the time dependency of historical abnormality identification is captured through the gated recurrent unit network, and the abnormality identification of the current time window is probability corrected in combination with the time decay factor in the behavior continuity feature vector, and the smoothed driving behavior abnormality recognition result is output.

10. A driving behavior analysis system based on multimodal sensor fusion, characterized in that: It includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the driving behavior analysis method based on multimodal sensor fusion as described in any one of claims 1 to 9.

Images