A Multimodal Vehicle Driving Style Recognition Method Based on Unified Fusion Features

Abstract

This invention discloses a multimodal vehicle driving style recognition method based on unified fusion features; it includes the following steps: (1) preprocessing the multimodal data to facilitate the next feature extraction stage; the preprocessing operations include adding random noise, normalization, wavelet denoising, and sliding window segmentation; (2) dividing the dataset into training and testing sets, extracting unified statistical, temporal, spatial, and features through three feature extraction subnetworks, and then using attention to efficiently fuse the three features; (3) calculating the joint loss of classification and optimizing the network parameters through backpropagation to obtain the optimal driving style recognition model. This invention only requires a smartphone to realize vehicle driving style detection, and has low cost, high flexibility, and interactivity; this invention can simultaneously extract key features based on spatiotemporal and statistical information and has low sensitivity to parameters in the module.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and intelligent transportation technology. Specifically, it relates to a multimodal vehicle driving style recognition method based on unified fusion features. Background Technology

[0002] Advances in intelligent sensing and wireless communication technologies have enhanced the ability of various embedded devices to collect vehicle driving data. Compared to methods using fixed in-vehicle sensors (such as CAN-Bus) or dedicated equipment (such as high-definition cameras), data collection devices primarily using multi-sensor embedded in smartphones and other portable embedded devices offer advantages such as convenience, interoperability, application upgradeability, rich data types, and efficient data processing, and have received widespread research attention. Driving style reflects a driver's fine-grained driving habits during vehicle movement, and this research benefits numerous intelligent applications, such as driving assessment and assistance, vehicle interaction, and car insurance assessment. Compared to previous driving style recognition methods based on vehicle GPS spatiotemporal trajectory data, multimodal driving data generated by multiple sensors can more accurately characterize vehicle driving behavior. Multimodal driving style detection methods are mainly divided into two categories: feature engineering-based methods and deep learning-based methods. Feature engineering-based methods mainly design meaningful features based on the inherent patterns of the data, such as mean, variance, maximum, and minimum values. These methods are limited by domain knowledge, time-consuming, and resource-intensive, and tend to overlook higher-order statistical features. Correspondingly, deep learning-based methods typically consider the temporal or spatial relationships of data to design neural networks, enabling the automatic extraction of spatiotemporal features from data, but they cannot explore statistical features.

[0003] Previous driving style recognition methods (such as LSTM and CNN) are good at extracting spatiotemporal features but ignore statistical features. Furthermore, traditional statistical feature extraction methods (such as k-means, fuzzy inference systems, random forests, etc.) require manual feature selection, which is time-consuming and labor-intensive. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention aims to propose a method for driving style recognition using unified fusion features. This invention automatically learns a robust feature to characterize the complex distribution and spatiotemporal patterns of driving styles in multimodal perception. The feature is extracted end-to-end through three carefully designed networks, integrating three different types of features representing data characteristics into a unified fusion feature, exhibiting strong robustness. Furthermore, this invention introduces the concept of kernel embedding into the autoencoder, enabling the statistical subnet to extract high-order statistical features within the mapped Hilbit space. Simultaneously, carefully designed spatial and temporal subnets address the spatiotemporal relationships in the data, allowing the extracted features to enable the proposed model to learn representations beyond the inherent features of the data itself. More importantly, the proposed model does not require processing data segments of different driving styles; instead, it achieves perfect segmentation through simple windowing, making it more closely aligned with practical applications.

[0005] This invention first introduces k-kernel mean embedding technology to extract high-order statistical features. Then, it uses three sub-networks—statistical, temporal, and spatial—to extract corresponding features to form a unified feature set, and uses attention to measure the importance of this unified feature set. Finally, a joint loss function is used to optimize the distribution, reconstruction, and classification performance of the three sub-networks to train an efficient driving style recognition model. The technical solution of this invention is described in detail below.

[0006] A multimodal vehicle driving style recognition method based on unified fusion features includes the following steps:

[0007] (1) Preprocess the multimodal data to facilitate feature extraction in the next step; the preprocessing operations include adding random noise, normalization, wavelet denoising and sliding window segmentation;

[0008] (2) Divide the dataset into training set and test set, extract unified statistical features, temporal features and spatial features through three feature extraction subnets, and then fuse the three features;

[0009] (3) Calculate the driving style classification for the unified features of the fusion, calculate the joint loss of the driving style classification, and optimize the network parameters through backpropagation to obtain the optimal driving style recognition model.

[0010] In this invention, in step (1), the multimodal driving data is selected from several of the following: acceleration, angular velocity, magnetic sensing data, GPS data, fuel consumption data, electroencephalogram (EEG), electrocardiogram (ECG), or electromyography (EMG).

[0011] In this invention, in step (2), the three feature extraction subnets are respectively the statistical subnet f st Time subnet f te Space subnet f sp Statistical subnet f stExtracting distribution-based statistical features and temporal subnet f using an Autoencoder te Temporal features are extracted using GRU+CNN, and the spatial subnetwork is f. sp Spatial features are extracted using GRU.

[0012] In this invention, in step (2), the statistical subnet f is... st Using matrix X as input, the aim is to pass through neural network f st Automatically learned statistical features are represented as follows:

[0013]

[0014] Where F st Indicates statistical characteristics. This is the feature mapping function, which extracts high-dimensional or even infinite-dimensional features from the mapped Hilbert space distribution. Specifically, the mean of the feature mapping for each input sample is calculated as the vector input of the kernel mean, as expressed in the following formula:

[0015]

[0016] Where ws is the sliding window length and M is the set of driving style types. To make the statistical subnet f... st It can automatically select from different feature kernels The optimal kernel is learned by using k-kernel mean embedding to learn the maximum mapping mean from multiple kernel functions, as shown in the following formula:

[0017]

[0018] in This represents the optimal feature mapping. Since the k-kernel mean embedding mapping is not necessarily invertible, f... st The invertibility of feature mapping is guaranteed by introducing an encoder (Encoder) and a decoder (Decoder) in the Autoencoder, i.e.:

[0019]

[0020] Where f st (·)and They represent feature mapping and inverse mapping, respectively. and These represent the encoder and decoder, respectively.

[0021] Time subnet f te It consists of a parallel 2-layer GRU network and a 4-layer 1D CNN network, both used to extract temporal relationships from multimodal driving data. The input form of the CNN network is a column vector. Temporal patterns are extracted from the data along the time dimension using 1D convolution; the input form of the GRU network is a row vector. This is used to learn the temporal relationships within the data. T represents transpose, and d is the dimension composed of all modal channels.

[0022] Space subnet f sp A two-layer GRU network is used to extract the spatial relationships between each mode; its input is a column vector. The dependencies between them form spatial feature vectors, establishing spatial relationships between modes.

[0023] In this invention, in step (2), the different types of features extracted from the three subnets are linked to form a unified feature, and the importance of different features is measured by attention. The formula is calculated as follows:

[0024] F un =[F st ;F te ;F sp (10)

[0025] a f =Sigmoid(W s (ReLU(W r F un +b r ))+b s (11)

[0026]

[0027] Where F un Indicates statistical characteristics F st Time characteristics F te Spatial features F sp The unified feature formed after connection, a f Indicates that the attention network is F un The generated weight vector, F f The unified fusion feature formed after attention fusion, Representing element-wise multiplication, Sigmoid(·) and ReLU(·) represent the Sigmoid function and the ReLU function, respectively. s W r ,b r ,b s} represents the learnable parameters during the training process.

[0028] In this invention, in step (3), the loss function of the multimodal driving style recognition network includes three parts: distribution loss, reconstruction loss, and classification loss; the maximum mean difference distance is used. The calculation metric for the statistical subnet distribution embedding is as follows:

[0029]

[0030] Where x i and x j This represents two styles of input line data, where N and M represent the lengths of the two input samples, based on... Distributed loss The calculation formula is:

[0031]

[0032] The calculation approximates the average vector difference between the input and output of the Autoencoder. The reconstruction loss is calculated using the mean squared error function, as shown in the following formula:

[0033]

[0034] The driving style classification loss uses the cross-entropy loss function, and the calculation formula is as follows:

[0035]

[0036] in For driving style categories, y q This indicates that the label of the current input sample is q (0 or 1). To predict driving style using the Softmax function The probability is q. In summary, the embodiments of the present invention calculate the joint loss function using the following formula:

[0037]

[0038] Where λ and γ are balancing parameters, and B is the batch size.

[0039] In this invention, in step (3), the driving style classification is calculated by using a fully connected layer and a Softmax function to fuse the unified features.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] 1. Compared with the previous manual feature design method, the model proposed in this invention can extract high-dimensional or even infinite-dimensional statistical features from driving style sequence data through kernel mean embedding technology;

[0042] 2. Compared with the technical solution of fixed equipment, the model proposed in this invention can realize the driving style detection of vehicles using only a smartphone, which has the advantages of low cost, high flexibility and interactivity;

[0043] 3. Previous spatiotemporal feature extraction methods could not extract statistical information. This method can extract key features based on both spatiotemporal and statistical information and has low sensitivity to parameters in the module.

[0044] 4. This method uses only fixed sliding window data segments as input data, instead of applying different sliding windows to different driving styles, making the model more adaptable to real-world applications;

[0045] 5. Unlike detection methods based on video data, this method only requires multiple sensors built into the mobile phone to achieve efficient detection, with an average prediction execution time of 1-3ms and a sliding window of 3-5s. Attached Figure Description

[0046] Figure 1 : A flowchart of the vehicle driving style recognition method based on unified fusion features in this embodiment of the invention; arrows indicate the data flow direction.

[0047] Figure 2 : A schematic diagram of the network process for training the driving style recognition model in this embodiment of the invention; arrows indicate the data flow direction.

[0048] Figure 3 This is a schematic diagram of the sub-network structure for extracting unified features in this embodiment of the invention. Arrows indicate data flow. Conv1D represents a 1D convolutional layer, MeanPool1D represents a pooling layer with a 1D mean pooling kernel, ReLU() and Tanh() are activation functions, and FC represents a fully connected layer. ConvB represents the CNN network block constituting (c) the temporal sub-network, and FCB represents the network block constituting (e) the statistical sub-network.

[0049] Figure 4 The proposed model in this invention is visualized as a clustering result during iterative training at the 0th, 30th, 50th and 100th iterations. Detailed Implementation

[0050] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0051] This invention introduces a unified fusion feature strategy into the feature extraction method for multimodal driving style recognition. It maps arbitrary distributions to Hilbert space using k-kernel mean embedding technology, enabling a statistical subnetwork to extract high-order statistical features from multimodal driving data. The statistical subnetwork is used not only for reconstructing the original data but also for reconstructing the distribution embedding. A GRU network is used to extract spatial features of the multimodal driving sequence; a CNN+GRU network is used to extract temporal features of the multimodal driving sequence; unified features are extracted from the statistical, temporal, and spatial subnetworks using different data input formats, and an attention network is used to measure the importance of these unified features; a joint loss function optimizes the model's distribution, reconstruction, and classification performance.

[0052] Figure 1 This is a flowchart of a vehicle driving style recognition method based on unified fusion features proposed in this invention. The process can be divided into four stages, including data preprocessing S1, feature extraction S2, loss calculation S3, and driving style classification S4.

[0053] This invention can collect multimodal driving style data through various sensors embedded in smartphones, such as accelerometers, gyroscopes, magnetometers, and GPS. Each sensor collects data corresponding to one modality, forming realistic multimodal driving style data such as acceleration, angular velocity, and GPS. The general process of this method is as follows: First, the multimodal data is preprocessed to facilitate feature extraction in the next step, including adding random noise, normalization, wavelet denoising, and sliding window segmentation. Next, the dataset is divided into a training set (including a validation set) and a test set in a 7:3 ratio. Unified temporal, spatial, and statistical features are extracted through three feature extraction subnetworks, and attention is used to efficiently fuse the three features. Finally, the joint loss for classification is calculated, and the network parameters are optimized through backpropagation to obtain the optimal driving style recognition model.

[0054] Each input multimodal driving sequence can be represented as (X,y), where X=[x1,.x2,y3,y4,y5,y6,y7,y8,y9 ... m ,..,x M ], m∈[1,M], M represents M modes, This represents a set of driving styles, such as aggressive driving, drowsy driving, and normal driving. The input data for modality m is... ws is the length of the sliding window, d m This represents the number of channels for mode m. Furthermore, it can also represent the input single-timestep data for mode m. i∈[1,ws]. To enhance the network model's robustness against data interference, we add random Gaussian noise to the original data, as shown in the following formula:

[0055]

[0056] in n represents the noise-free raw data. k They are independent with a mean of 0 and a variance of σ. 2 Gaussian random variables.

[0057] The principle of wavelet transform is to decompose the original signal into a superposition of a series of wavelet functions. It can not only obtain the frequency of the signal but also locate the time. The formula is expressed as follows:

[0058]

[0059] Where a and t represent the scaling and translation amounts, respectively. Wavelet denoising is particularly suitable for processing non-stationary and nonlinear signals. Different wavelet functions have different denoising effects. In this embodiment of the invention, wavelet coefficients are estimated using the db4 series wavelet and soft thresholding method. When the absolute value of the wavelet coefficient |ω| is greater than the threshold λ, λ is subtracted; otherwise, it is 0. The formula is as follows:

[0060]

[0061] Subsequently, the data needs to be normalized using a max-min method to unify the range of all data within [-1, 1]. This embodiment of the invention uses the following formula for normalization:

[0062]

[0063] Where x, x′, x min and x max These are the original data, the normalized data, and the minimum and maximum values ​​in the original data, respectively.

[0064] To explore the spatiotemporal and statistical characteristics of multimodal driving style data, we designed three feature extraction subnetworks, a statistical subnetwork, and a statistical subnetwork. st Time subnet f te and spatial subnet f sp To extract statistical, temporal, and spatial features. To better explore the inherent patterns in the data, the input formats for different subnets are different. Statistical subnet f st Using matrix X as input, the aim is to pass through neural network f st Automatically learned statistical features are represented as follows:

[0065]

[0066] We introduce kernel mean embedding to map each instance to a Hilbert space. This is the feature mapping function, which extracts high-dimensional or even infinite-dimensional features from the mapped Hilbert space distribution. Specifically, the mean of the feature mapping for each input sample is calculated as the vector input of the kernel mean, as expressed in the following formula:

[0067]

[0068] Our goal is to make the statistical subnet f st It can automatically select from different feature kernels The optimal kernel is learned by using k-kernel mean embedding to learn the maximum mapping mean from multiple kernel functions, as shown in the following formula:

[0069]

[0070] Because fst (·) Using k-kernel mean embedding mapping is not necessarily invertible, so we introduce an encoder (Encoder) and decoder (Decoder) from the Autoencoder to ensure the invertibility of the feature mapping, i.e.:

[0071]

[0072] Where f st (·)and They represent feature mapping and inverse mapping, respectively. and These represent Encoder and Decoder, respectively.

[0073] Time subnet f te It consists of a parallel 2-layer GRU subnetwork and a 4-layer 1D CNN subnetwork, and its network structure is shown in [reference needed]. Figure 3 (d) and Figure 3 (c) Both are used to extract temporal relationships from multimodal driving data; the input to the CNN subnet is a column vector. Temporal patterns are extracted from the data along the time dimension using 1D convolution, where ws is the sliding window length; the input to the GRU subnet is a row vector. This is used to learn the temporal relationships in the data, where d represents the dimension composed of all modal channels. GRU is a simpler gated recurrent neural network than LSTM. Each GRU unit contains multiple gate components, and its calculation formula is as follows:

[0074]

[0075] Where r t It is a reset door z t It's an update door. h represents the candidate state at the current moment. t Indicates the current state. t Defines how to combine a new input with the previous memory location, z t It specifies the amount of memory to reserve for the previous memory unit.

[0076] Unlike previous methods that used 1D CNNs to convolve data along the time dimension to extract spatial patterns, the spatial subnetwork f proposed in this embodiment of the invention... sp Spatial relationships between modal data are extracted using a stacked GRU network; its network structure is shown below. Figure 3 (d) Input format is column vector By learning the spatial feature vectors formed by the dependencies between each modality, spatial associations of each modality are established.

[0077] We then concatenate the different types of features extracted from the three subnets to form a unified feature, and measure the importance of different features using attention. The formulas for these two parts are as follows:

[0078] F un =[F st ;F te ;F sp (10)

[0079] a f =Sigmoid(W s (ReLU(W r F un +b r ))+b s (11)

[0080]

[0081] Where F un Indicates statistical characteristics F st Time characteristics F te Spatial features F sp The unified feature formed after connection, a f Let F represent the attention network. un The generated weight vector, F f Features after attention fusion This represents element-wise multiplication. Sigmoid(·) and ReLU(·) represent the Sigmoid function and the ReLU function, respectively. s W r ,b r ,b s} represents the learning parameters during the training process.

[0082] The loss function of the multimodal driving style recognition network in this invention comprises three parts: distribution loss, reconstruction loss, and classification loss. We use the maximum mean difference distance. The calculation metric for the statistical subnet distribution embedding is as follows:

[0083]

[0084] Where x i and x j This represents two styles of input line data, where N and M represent the lengths of the two input samples. Based on The formula for calculating distributed loss is:

[0085]

[0086] The reconstruction loss is calculated using the mean squared error function, as shown in the following formula:

[0087]

[0088] The driving style classification loss uses the cross-entropy loss function, and the calculation formula is as follows:

[0089]

[0090] in For driving style categories, y q This indicates that the label of the current input sample is q (0 or 1). To predict driving style using the Softmax function The probability is q. In summary, the embodiments of the present invention calculate the joint loss function using the following formula:

[0091]

[0092] Where λ and γ are balancing parameters, and B is the batch size.

[0093] Figure 2 This is a schematic diagram of the neural network structure for training the driving style recognition model proposed in this invention. The network structure comprises five parts: multimodal driving sequence N1, data preprocessing N2, unified feature extraction N3, feature fusion N4, and driving style classification N5. Wherein:

[0094] N1 includes multiple modal data, such as acceleration, orientation, magnetometer, GPS, etc., and the duration of a single sliding window input is 5 seconds.

[0095] N2 performs preprocessing operations on the input data to facilitate neural network processing. These operations include adding Gaussian random noise to enhance the model's ability to resist noise interference, minimization normalization to improve computational efficiency, wavelet denoising to suppress high-frequency noise interference, and sliding window segmentation to divide equal-length driving style sample data.

[0096] N3 consists of three feature extraction subnetworks, whose functions are to extract distribution-based statistical features through an Autoencoder, extract spatial features through a GRU, and extract temporal features through a GRU+CNN, respectively. The network structure of each subnetwork is as follows (…). Figure 3 There are different data input formats, and the extracted output features are linked together to form a unified feature;

[0097] N4 can measure the importance of uniform features through attention networks, further improving the network model's ability to identify data features;

[0098] N5 calculates driving style classification using fully connected layers and a Softmax function on the fused unified features.

[0099] Figure 3 The diagram illustrates the sub-network structure for extracting uniform features, including (a) the CNN network block ConvB, (b) the Autoencoder network block FCB, (c) the CNN temporal subnetwork, (d) the GRU temporal / spatial subnetwork, and (e) the Autoencoder statistical subnetwork. Wherein:

[0100] (a) shows the network structure of each layer in the temporal subnet of the CNN in (c), which consists of a 1D convolutional layer with a kernel of 3, a ReLU activation function layer, a 1D mean pooling layer with a kernel of 2, and a 20% random dropout layer.

[0101] (b) is the network structure of the convolutional layer in the Autoencoder statistical subnet of (e), which consists of a fully connected layer, a ReLU activation function layer, and a 10% random dropout layer.

[0102] (c) shows the network structure of the CNN temporal subnet, consisting of an input layer and three (a) CNN block layers. The size of the input layer is...

[0103]

[0104] (d) shows the network structure for both the temporal and spatial subnets, employing stacked GRU networks with 128 hidden states per layer. The input layer size of the GRU temporal subnet is... The size of the input layer of the GRU spatial subnet is

[0105] (e) is the network structure of the statistical subnet, which is divided into an encoder on the left and a decoder on the right. The encoder consists of an input layer and four (b) Autoencoder network blocks. The input size is the matrix X obtained by the sliding window. The decoder is after the encoder and consists of four (b) Autoencoder network blocks and a Tan activation function.

[0106] In this invention, the kernel function of the statistical subnet can be replaced by a multinomial kernel function, a sigmoid kernel function, a Gaussian kernel function, a Laplace kernel function, etc.; the multimodal driving data in this invention can be increased or decreased according to the type of sensor of the acquisition device, and the data types are not limited to acceleration, angular velocity, magnetometer, GPS, fuel consumption data, electroencephalogram (EEG), electrocardiogram (ECG), electromyogram (EMG), etc.; the CNN used to extract spatial features in this invention can be replaced by other types of convolutional neural networks such as ResNet, Inception, and DenseNet; the GRU network used to extract temporal features in this invention can be replaced by other types of recurrent neural networks such as RNN and LSTM; the Autoencoder network used to extract statistical features in this invention can be replaced by other autoencoder variants such as Sparse Autoencoder and Variational Autoencoder; in addition to extracting statistical features from the input sample matrix, it is also possible to (1) add additional information such as driving context, road, weather, and map data as metadata to jointly extract statistical features; (2) use spatiotemporal features as input vectors for secondary statistical feature extraction after feature embedding.

[0107] Schematic diagram of neural network structure in this invention ( Figure 2 The feature fusion layer N4 in the ) can be replaced with a simple connection or a variant of the attention network such as key-value attention, multi-head attention, structured attention, or Top-k attention. Alternatively, PCA can be applied for principal component analysis.

[0108] In this invention, the integration of three types of features makes the model less sensitive to the parameters of any single subnet, such as...

[0109] (1) The performance of the encoder / decoder in the statistical subnet fluctuates only about 3 points in the number of layers {1, 2, 3, 4, 5, 6}.

[0110] (2) The latent representation of the last layer of the encoder in the statistical subnet varies very little in the range of 1D to 6D (D is the input dimension);

[0111] (3) The performance of GRU in the temporal and spatial subnets does not change much when the implicit representation {32,64,128,256,512} is used.

[0112] (4) The performance does not change much when the number of convolutional layers in the temporal subnet is {2, 4, 6, 8, 10}.

[0113] Table 1 shows the ablation results of key parameters of the proposed model on the publicly available driving behavior detection dataset UAH-Driveset.

[0114] Table 1 shows the parameter ablation results of the proposed model.

[0115]

[0116] Figure 4 This is a visualization of the clustering results of the proposed model during iterative training at stages 0, 30, 50, and 100. Figure 4 It is evident that the proposed model has essentially formed a clustering prototype after 30 iterations; and after 50 iterations, it almost achieves the same good clustering effect as after 100 iterations.

Claims

1. A multimodal vehicle driving style recognition method based on unified fusion features, characterized in that, Includes the following steps: (1) Preprocess the multimodal data to facilitate feature extraction in the next step; the preprocessing operations include adding random noise, normalization, wavelet denoising and sliding window segmentation; (2) Divide the dataset into training set and test set, extract unified statistical features, temporal features and spatial features through three feature extraction subnets, and then fuse the three features; (3) Calculate the driving style classification for the fused unified features, calculate the joint loss of the driving style classification, and optimize the network parameters through backpropagation to obtain the optimal driving style recognition model; where: In step (2), the three feature extraction subnets are statistical subnets. Time subnet and space subnet Statistical Subnet Extracting distribution-based statistical features and temporal subnets using an Autoencoder Temporal features are extracted using GRU+CNN, and a spatial subnetwork is used. Spatial features are extracted using GRU; Statistical Subnet With matrix As input, it is intended to be used by neural networks. Automatically learned statistical features are represented as follows: （5） in Indicates statistical characteristics, This is the feature mapping function, which extracts high-dimensional or even infinite-dimensional features from the mapped Hilbert space distribution. Specifically, the mean of the feature mapping for each input sample is calculated as the vector input of the kernel mean, as expressed in the following formula: （6) in It is the length of the sliding window. It is a collection of driving style types; to make the statistical subnet It can automatically select from different feature kernels The best core for learning in China pass Kernel mean embedding learns the maximum mapping mean from multiple kernel functions, as shown in the following formula: (7) in This represents the optimal feature mapping, because Kernel mean embedding mapping is not necessarily invertible, therefore The invertibility of feature mapping is guaranteed by introducing an autoencoder and decoder, i.e.: (8) in, and They represent feature mapping and inverse mapping, respectively. and These represent the encoder and decoder, respectively.

2. The multimodal vehicle driving style recognition method according to claim 1, characterized in that, In step (1), the multimodal driving data is selected from several of the following: acceleration, angular velocity, magnetic sensing data, GPS data, fuel consumption data, electroencephalogram (EEG), electrocardiogram (ECG), or electromyography (EMG).

3. The multimodal vehicle driving style recognition method according to claim 1, characterized in that, In step (2), Time Subnet It consists of a parallel 2-layer GRU network and a 4-layer 1D CNN network, both used to extract temporal relationships from multimodal driving data; the input form of the CNN network is a column vector. The GRU network extracts temporal patterns from the data along the time dimension using 1D convolution; the input form of the GRU network is a row vector. In order to learn the temporal relationships in the data; Indicates transpose. The dimension composed of all modal channels; Space subnet A two-layer GRU network is used to extract the spatial relationships between each mode; its input is a column vector. By learning the dependencies between all modal channels to form spatial feature vectors, spatial correlations between modalities are established.

4. The multimodal vehicle driving style recognition method according to claim 1, characterized in that, Statistical Subnet The network structure adopts an encoder and decoder structure, where the encoder includes an input layer and four Autoencoder network blocks. The decoder is an antisymmetric structure similar to the encoder and is designed after the encoder output. It includes four Autoencoder network blocks and a Tan activation function. The Autoencoder network blocks are composed of fully connected layers, ReLU activation function layers and 10% random dropout layers connected in sequence. Time Subnet The GRU network in the text uses stacked GRU network blocks, with 128 hidden states in each layer; the CNN network includes an input layer and three CNN block layers. Each CNN block layer consists of a 1D convolutional layer with a kernel of 3, a ReLU activation function layer, a 1D mean pooling layer with a kernel of 2, and a 20% random dropout layer connected in sequence. Space subnet The same stacked GRU network is used, with 128 hidden states in each layer.

5. The multimodal vehicle driving style recognition method according to claim 1, characterized in that, In step (2), the different types of features extracted from the three subnets are concatenated to form a unified feature, and the importance of different features is measured by attention. The formula is as follows: (10) (11) (12) in Indicates statistical characteristics Time characteristics and spatial features The unified characteristics formed after connection This indicates that through the attention network... The generated weight vector, The unified fusion feature formed after attention fusion, Represents element-wise multiplication. They represent functions and function,{ } represents the parameters that can be learned during the training process.

6. The multimodal vehicle driving style recognition method according to claim 1, characterized in that, In step (3), the loss function of the multimodal driving style recognition network consists of three parts: distribution loss, reconstruction loss, and classification loss; the maximum mean difference distance is used. The calculation metric for the statistical subnet distribution embedding is as follows: (13) in and This represents two styles of input line data. and Indicates the length of the two types of input samples, based on Distribution loss The calculation formula is: (14) The calculation approximates the average vector difference between the input and output of the Autoencoder; the reconstruction loss is calculated using the mean squared error function, as shown in the following formula: (15) The classification loss uses the cross-entropy loss function, and the calculation formula is as follows: (16) in As to the types of driving styles, This indicates that the label of the current input sample is The label is either 0 or 1. To predict driving style using the Softmax function for The probability of; The joint loss function is calculated using the following formula: (17) in and It is a balance parameter.

7. The multimodal vehicle driving style recognition method according to claim 1, characterized in that, In step (3), the driving style classification is calculated by using a fully connected layer and the Softmax function to calculate the unified features fused together.

Images