Driving style detection method and system based on mobile phone sensor

Abstract

The invention discloses a driving style detection method and system based on a mobile phone sensor, and relates to the technical field of driving style detection, and the method comprises the steps: collecting various mobile phone sensor data when a user drives; the method comprises the following steps: respectively inputting various mobile phone sensor data into a full convolutional network FCN and a time sequence model TimesNet, capturing one-dimensional features of signals changing along with time in the input mobile phone sensor data through the FCN, expanding a one-dimensional time sequence in the mobile phone sensor data to a two-dimensional space through the TimesNet to obtain two-dimensional features, and merging the one-dimensional features and the two-dimensional features to obtain two-dimensional features; and performing category division to obtain scores of each mobile phone sensor on different classification categories, and obtaining a driving style through the scores. According to the method, the one-dimensional features extracted by the FCN and the two-dimensional features extracted by the TimesNet are combined to be used in the DSD task, so that the obtaining effect of the two-dimensional features is improved, and the accuracy of driving style detection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of driving detection, and in particular to a driving style detection method and system based on a mobile phone sensor. Background Art

[0002] Driving Style Detection (DSD) can not only improve driving safety, but also has broad application prospects in energy conservation and emission reduction, driver mental health testing, driving training, insurance risk warning and other fields. It can also lay a research foundation for the future study of human-like driving style of intelligent autonomous driving and the customization of intelligent vehicles.

[0003] At present, DSD is mainly divided into methods based on driver dynamics and methods based on vehicle dynamics. The driver dynamics-based method mainly judges the driver's driving style through the driver's head or body image features or the driver's physiological signals (eye movement data, EEG, ECG, skin potential, etc.). However, this type of method is costly and highly invasive, which can easily affect the driver's normal driving. In addition, this method can only indirectly analyze the vehicle's driving status, resulting in poor detection of some categories of driving styles. In addition, the computer vision-based methods in this type of research are also difficult to be accepted by drivers due to environmental influences and privacy issues. In comparison, the vehicle dynamics-based method mainly uses GPS, inertial sensors, etc. to capture the vehicle's speed, acceleration, steering angle and other feature data, and performs DSD based on this. This type of method has the advantages of low invasiveness, privacy insensitivity, and not easily affected by environmental factors, and is becoming increasingly important.

[0004] Although some vehicles are currently equipped with advanced sensors and advanced driver assistance systems (ADAS), these in-vehicle systems are not applicable to all vehicles due to cost constraints. The computing, storage and communication capabilities of current smartphones are rapidly improving. Compared with the internal CAN bus method of the vehicle, this method can provide more abundant vehicle raw data and feature information at a lower cost. Therefore, it has significant advantages in cost, deployment and interaction, bringing new ways for DSD for low-income groups.

[0005] When the existing technology performs DSD through mobile phone sensor data, the one-dimensional time series obtained has limitations in representation capabilities and needs to be supplemented by extracting two-dimensional features. However, the methods adopted by the existing technology may cause some one-dimensional feature details to be blurred or lost when extracting two-dimensional features, and the noise may also be amplified during the conversion process, resulting in inaccurate final DSD results. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and provide a driving style detection method and system based on a mobile phone sensor, so as to solve the problem that when extracting two-dimensional features in the prior art, some one-dimensional feature details may be blurred or lost, and noise may also be amplified during the conversion process, resulting in inaccurate final DSD results.

[0007] The present invention specifically provides the following technical solutions:

[0008] A driving style detection method based on a mobile phone sensor comprises the following steps:

[0009] Inputting the plurality of mobile phone sensor data into the pre-trained fully convolutional network FCN and the time series model TimesNet respectively, capturing the one-dimensional features of the signal changing over time in the input mobile phone sensor data through FCN, expanding the one-dimensional time series in the mobile phone sensor data into a two-dimensional space through TimesNet to obtain two-dimensional features, and merging the one-dimensional features with the two-dimensional features;

[0010] The scores of the combined features on different driving style categories are extracted, and the scores are converted into probability distributions, and the driving style category with the highest probability is selected as the predicted driving style of the user.

[0011] Preferably, before collecting the various mobile phone sensor data of the user while driving, the method further includes:

[0012] Resampling the multiple historical mobile phone sensor data, and fusing the resampled data;

[0013] The fused data is redirected, Kalman filtered, filled with null values, extracted, and segmented in a sliding time window to obtain the processed training set, validation set, and test set, and the training set, validation set, and test set are standardized;

[0014] The fully convolutional network FCN and the time series model TimesN are trained using standardized training and test sets. e t for pre-training.

[0015] Preferably, the method of capturing the one-dimensional features of the signal changing over time in the input mobile phone sensor data by FCN, and expanding the one-dimensional time series in the mobile phone sensor data to a two-dimensional space by TimesNet to obtain the two-dimensional features includes:

[0016] Input a variety of mobile phone sensor data into the TimesNet model, decompose the time series changes in the mobile phone sensor data into different periods, and transform the one-dimensional time series of different periods into two-dimensional space; the specific expression is:

[0017] A=Avg(Amp(FFT(X 1D )));

[0018] f1,…,f k =arg Topk(A);

[0019]

[0020]

[0021] in, represents the intensity of each frequency component in X1D, and the k frequencies with the largest intensity {f1,…,f k} corresponds to the most significant k period lengths {p1,…,p k}, a one-dimensional time series with a time length of T and a channel dimension of C The above process is simplified as follows:

[0022] A,{f1,...,f k},{P1,...,p k}=Period(X 1D );

[0023] The original one-dimensional time series is folded based on the selected period. The process can be formalized as:

[0024]

[0025] Where A is the intensity of each frequency component, and Padding(·) is to add 0 at the end of the sequence so that the sequence length is p i The function of integer division, Corresponding to the period p i Dominant two-dimensional temporal changes.

[0026] Preferably, the one-dimensional time series in the mobile phone sensor data is expanded to a two-dimensional space through TimesNet to obtain two-dimensional features, and further includes:

[0027] TimesNet is obtained by stacking the normalized TimesBlock in a residual manner. The specific expression is:

[0028]

[0029] Among them, TimesBlock contains the following sub-processes:

[0030]

[0031] in, It is a two-dimensional feature The 2D tensor after the i-th transformation in ; Inception is a parameter-efficient convolution module; is the learned two-dimensional feature. Convert back to one-dimensional space Perform aggregations where a function is used to truncate the padded sequence;

[0032] k different one-dimensional representations The weighted summation of the intensity of the corresponding frequency is used to obtain the final output. The specific expression is:

[0033]

[0034] Among them, d model is the dimension of the embedding vector, Softmax is the activation function, is the weight of each cycle, The final output of the TimesNet module is obtained by aggregating each frequency.

[0035] Preferably, the fully convolutional network FCN comprises three stacked convolutional blocks, each convolutional block consisting of a one-dimensional convolutional filter set accompanied by batch normalization and a linear rectification function ReLu.

[0036] Preferably, merging the one-dimensional feature with the two-dimensional feature comprises the following steps:

[0037] Use Gaussian error linear unit to nonlinearly transform the two-dimensional features obtained by TimesNet, and flatten the two-dimensional features after nonlinear transformation into one-dimensional features;

[0038] The one-dimensional features extracted by FCN are passed through the global average pooling layer to generate one-dimensional features with global information;

[0039] Merge the one-dimensional features of the global average pooling layer and the one-dimensional features flattened by TimesNet.

[0040] Preferably, extracting the scores of the combined features on different driving style categories, converting the scores into probability distribution, and selecting the driving style category with the highest probability as the predicted driving style of the user, comprises:

[0041] The merged features are processed and output through a fully connected layer, and the output dimension is used as the number of driving style categories of the task. The scores of the merged features in different driving style categories are extracted.

[0042] The softmax function is used to convert the score into a probability distribution, and the argmax operation is used to select the driving style category with the largest probability, and the largest driving style category is used as the user's predicted driving style; the specific expression is:

[0043] f s =[X TimesNet ; E FCN ];

[0044] z i =W y f s +b y ;

[0045]

[0046] Among them, X TimesNet It is the result of reshaping the high-level features extracted by TimesNet. FCN It is the feature extracted by FCN after global average pooling, f s is the merged feature, which is used as the input of the fully connected layer. y , b y are the weight matrix and bias vector of the fully connected layer, Z i is the unnormalized score vector of the i-th sample output by the fully connected layer, with dimensions C, Z i,j represents the score of the i-th sample as type j, k is the type within the dimension range, P i,j is the probability that the i-th sample is of type j calculated by the Softmax function, is the type predicted by the i-th sample, P i is the probability of any type in the i-th sample.

[0047] The present invention provides a driving style detection system based on a mobile phone sensor, comprising:

[0048] The acquisition module is used to collect various mobile phone sensor data while the user is driving;

[0049] A feature capture module, which is used to input the multiple mobile phone sensor data into the pre-trained fully convolutional network FCN and the time series model TimesNet, respectively, to capture the one-dimensional features of the signal changing over time in the input mobile phone sensor data through FCN, to expand the one-dimensional time series in the mobile phone sensor data to a two-dimensional space through TimesNet, to obtain two-dimensional features, and to merge the one-dimensional features with the two-dimensional features;

[0050] The style detection module is used to extract the scores of the combined features on different driving style categories, convert the scores into probability distribution, and select the driving style category with the highest probability as the user's predicted driving style.

[0051] The present invention provides a computer device, comprising a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of the above-mentioned driving style detection method based on a mobile phone sensor.

[0052] The present invention provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned driving style detection method based on a mobile phone sensor are implemented.

[0053] Compared with the prior art, the present invention has the following significant advantages:

[0054] The present invention sets a fully convolutional network FCN and a time series model TimesNet, and uses the one-dimensional features extracted by FCN to expand the one-dimensional time series in the mobile phone sensor data to the two-dimensional space through TimesNet to obtain the two-dimensional features, thereby making up for the deficiency that FCN can only extract features from the one-dimensional space sequence, so that the model can capture more complex time dependencies, and performs driving style detection by fusing the one-dimensional features with the two-dimensional features, thereby simultaneously considering the changes of the time series in the one-dimensional and two-dimensional spaces, reducing the influence of the one-dimensional feature details when extracting the two-dimensional features, and avoiding the amplification of noise in the conversion process without directly converting the one-dimensional features with the two-dimensional features, directly capturing complex patterns and structures from the original data, and improving the acquisition efficiency and effect of driving style detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of a driving style detection method based on a mobile phone sensor of the present invention;

[0056] Figure 2 is a redirection method diagram of the present invention; wherein, Figure 2 (a) is the coordinate system diagram of the mobile phone. Figure 2 (b) is the Northeastern Universe (ENU) coordinate system, Figure 2 (c) is the North East Earth (NED) coordinate system, Figure 2 (d) is the vehicle coordinate system;

[0057] Figure 3 It is a sliding time window diagram of the present invention;

[0058] Figure 4 This is the TimesNet-FCN model diagram of the present invention. DETAILED DESCRIPTION

[0059] The following is a clear and complete description of the technical solutions of the embodiments of the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0060] like Figure 1 As shown, a driving style detection method based on a mobile phone sensor of the present invention specifically includes the following steps:

[0061] Step S1: Collect various mobile phone sensor data of the user while driving.

[0062] Before collecting various mobile phone sensor data while the user is driving, it also includes:

[0063] The natural driving data collected by smartphone sensors may contain noise and inaccurate measurements, and there are problems with timestamps and sampling rates being out of sync between multiple sensors. Collect multiple historical mobile phone sensor data while the user is driving, and preprocess the multiple historical mobile phone sensor data. The preprocessing specifically involves resampling the multiple mobile phone sensor data and fusing the resampled data to obtain a multivariate time series signal Trip of N trips. N After coordinate system redirection, Kalman filtering and null value filling, clean data is obtained.

[0064] The fused data is feature extracted, and the (sliding time window) signal is split into overlapping windows, and the overlapping windows are split into training sets. After maximum and minimum standardization processing, multiple multivariate time windows are obtained. The significance of preprocessing is to improve data quality and consistency, reduce noise and interference, so that subsequent feature extraction and classification are more accurate and reliable.

[0065] Next, combine Figure 1 The data preprocessing is further explained as shown below.

[0066] 1. Resampling and data fusion:

[0067] The raw data is obtained by capturing signals from the speedometer, magnetometer, gyroscope, direction sensor and GPS. During data collection, due to the scheduling problem of the mobile phone operating system or the hardware limitation of the sensor, the sampling rate of each sensor is not consistent, and the acquisition rate may fluctuate, resulting in inconsistent sampling time intervals. After reading the data (accelerometer signal (10Hz, unequal time interval), magnetometer signal (10Hz, unequal time interval), gyroscope signal (10Hz, unequal time interval), direction sensor signal (10Hz, unequal time interval) and GPS signal (1Hz, unequal time interval)), the signals collected by each sensor are first resampled using linear interpolation, and the sensor signal with a low sampling rate (for example: GPS1Hz) is upsampled to a high sampling rate (10Hz), and the time series with unequal time intervals is converted to equal time intervals. The linear interpolation formula is as follows:

[0068]

[0069] Where t is the new time point to be interpolated. a and t b are the known time points on both sides of the new time point t, and t a <t<t b . s(t a ) and s(t b ) is the time t a and t b The known value at time t. s′(t) is the interpolation result at time t.

[0070] In the collected data set, the signals of each built-in sensor of the smartphone are multiple separate time series. Since the time when each sensor receives the recording instruction is inconsistent, even if the sensor signals are resampled to the same sampling rate and equal time interval, the timestamps of each sensor cannot completely correspond.

[0071] Use timestamp-based inexact merging to fuse data from sensors with inconsistent timestamps. If the timestamps are exactly the same, merge them directly; if they are different, find the closest timestamp to merge. This is very useful when processing sensor data with different sampling rates or incomplete synchronization.

[0072] 2. Coordinate system redirection:

[0073] like Figure 2 As shown, first, a set of vectors in the mobile phone coordinate system is converted to the East-North-Up (ENU) coordinate system using the quaternion representing the mobile phone direction obtained from the smartphone direction sensor at time t, that is:

[0074] v=[x v,y v ,z v ] T ;

[0075] p phone =0+x v i+y v j+z v k;

[0076] q=q w +q x i+q y j+q z k;

[0077] v ENU =q·p phone ·q -1 ;

[0078] Where v is a rotated vector in the smartphone coordinate system at time t, and p phone It is a pure quaternion converted from a three-dimensional vector v, where the real part is 0 and the three components x of the vector v are v ,y v , z v As the imaginary part of the quaternion, q is the quaternion representing the direction of the mobile phone obtained from the smartphone orientation sensor at time t. -1 is the inverse quaternion of q, v ENU After the quaternion is rotated, the vector part of the rotated quaternion is extracted (ignoring the real part).

[0079] Then use a fixed rotation matrix to transform the data in the ENU coordinate system into the North-East-Down (NED) coordinate system, that is:

[0080]

[0081] v NED =R ENU to NED ·v ENU ;

[0082] Among them, R ENU to NED is the rotation matrix that transforms the ENU coordinate system to the NED coordinate system, v NED is the sensor vector in the NED coordinate system after transformation.

[0083] Finally, after obtaining the vehicle's yaw angle through GPS, the sensor vector in the NED coordinate system is converted to the vehicle coordinate system. That is:

[0084] q car =Q(yaw cor ,0,0);

[0085]

[0086]

[0087] Among them, yaw car The car heading angle is obtained by the clockwise angle of the GPS device relative to geographic north. car It is the quaternion converted from Euler angle by function Q, p NED is the sensor three-dimensional vector v in the NED coordinate system NED The converted pure quaternion, in which the real part is 0, the vector v NED The three components As the imaginary part of the quaternion, Yes car The inverse quaternion, v NED It is the final redirection result after conversion, that is, the vector in the vehicle coordinate system at time t.

[0088] 3. Other preprocessing:

[0089] The collected natural driving data will be processed by Kalman filtering after resampling, data fusion, and coordinate system redirection. Since mobile phone sensors (such as accelerometers and gyroscopes) may be affected by various interferences and noises (such as vibration, mechanical noise, and temperature changes) when collecting data, Kalman filtering can help improve data quality and improve the accuracy and reliability of driving behavior recognition.

[0090] After Kalman filtering, the features of the input model are extracted, and the sliding time window segmentation and data set division are performed. Figure 3 As shown, the i-th multivariate time series itinerary x i,t is divided into sliding time windows. For each window W k , which consists of the following time steps:

[0091] W k =[x i，t , x i，t+1 , ..., x i，t+d-1 ];

[0092]

[0093] Among them, k represents the index of the window, d represents the size of the time window, and the window W k Tags It is directly inherited from itinerary x i,t Label y i , t represents the time step at the start of the time window, which is incremented in the following way:

[0094] t k+1= t k + c;

[0095] Among them, c is the step size, which determines the overlapping degree between time windows. When c < d, there is an overlap between windows; when c = d, there is no overlap between windows, which can significantly increase the quantity and diversity of training samples to help reduce overfitting during training. The data utilization rate is improved through the overlapping part between windows, and adjacent windows share some data to maintain the temporal correlation between windows.

[0096] After the sliding time window segmentation, all windows will be randomly divided into a training set, a validation set, and a test set according to a ratio, and then the training set, the validation set, and the test set will be subjected to Min - Max Scaling to scale the feature values to between 0 and 1. This method is achieved by subtracting the minimum value of each feature from it and dividing by the range of the feature (the difference between the maximum value and the minimum value). The specific mathematical formula is:

[0097]

[0098] Among them, X is the original data, X min and X max are respectively the minimum and maximum values of this feature in the training set, and X norm is the normalized value. Each feature in the dataset usually has different dimensions and ranges. Using Min - Max Scaling, these data with different dimensions can be converted to a unified scale, which helps to improve the effectiveness of data processing and the performance of the model, and optimization algorithms such as gradient descent can also converge faster and more smoothly.

[0099] Step S2: Use the time series model TimesNet combined with the fully convolutional network FCN to construct the Times Net - FCN model.

[0100] Pre - train the fully convolutional network FCN and the time series model TimesNet in the TimesNet - FCN model using the standardized training set and test set.

[0101] Step S3: Input various mobile phone sensor data during driving into the pre - trained fully convolutional network FCN and the time series model TimesNet respectively. Through the FCN, capture the one - dimensional features of the signal changing with time in the input mobile phone sensor data. Through TimesNet, expand the one - dimensional time series in the mobile phone sensor data to a two - dimensional space to obtain two - dimensional features, and merge the one - dimensional features and the two - dimensional features.

[0102] TimesNet and FCN are combined for end-to-end feature extraction, automatically learning and extracting high-level features from natural driving data from multiple dimensions. Through this end-to-end approach, the model can capture complex patterns and structures directly from raw data, improving the efficiency and effectiveness of feature extraction without the need for manual design and selection of features.

[0103] Use TimesNet and FCN to extract features from the preprocessed data. FCN has been proven to be an effective learning model for time series classification problems and is usually used as a feature extractor to capture the one-dimensional features of the input signal that change over time. TimesNet converts the one-dimensional time series into two-dimensional space, extracts two-dimensional time series changes from multiple cycles, and then performs adaptive fusion. This makes up for the deficiency that FCN can only extract features from one-dimensional space sequences, allowing the model to capture more complex time dependencies.

[0104] like Figure 4 As shown in the figure, FCN is used to capture the one-dimensional features of the input signal that change over time, and TimesNet is used to expand the one-dimensional features to two-dimensional space and extract two-dimensional features, including:

[0105] The mobile phone sensor data (after embedding the preprocessed natural driving data window) is input into the TimeNet model, and the time series changes are decomposed into different periods through the modular structure. By converting the one-dimensional time series into two-dimensional space, the periodicity can be calculated by the fast Fourier transform (FTT) of the time dimension; the specific expression is:

[0106] A=Avg(Amp(FFT(X 1D )));

[0107] f1,...,f k =arg Topk(A);

[0108]

[0109]

[0110] in, Represents X 1D The intensity of each frequency component in the k frequencies with the largest intensity {f1,…,f k} corresponds to the most significant k period lengths {p1,…,p k}, a one-dimensional time series with a time length of T and a channel dimension of C The above process is simplified as

[0111] A,{f1,...,f k}{p1,...,p k}=Period(X 1D ).

[0112] The original one-dimensional time series is folded based on the selected period, and the input sequence is embedded in the layer to obtain two-dimensional features. The specific expression is:

[0113]

[0114] Where Padding(·) is to add 0 at the end of the sequence so that the sequence length can be p i Function of integer division; through the above operations, a set of two-dimensional tensors is obtained Corresponding to the period p i For the above two-dimensional vector, each column and each row correspond to adjacent moments and adjacent periods, and adjacent moments and periods often contain similar timing changes. Therefore, the above two-dimensional tensor will show two-dimensional locality, so it is easy to capture information through 2D convolution.

[0115] Before merging the one-dimensional features with the two-dimensional features, it also includes:

[0116] TimesNet is obtained by stacking the normalized TimesBlocks in a residual manner. The input sequence is embedded in the layer to obtain two-dimensional features. For the lth layer TimesBlock, its input Then, 2D convolution is used to extract the two-dimensional time series changes. The specific expression is:

[0117]

[0118] Specifically, TimesBlock contains the following sub-processes:

[0119]

[0120] in, It is a two-dimensional feature After this transformation, the 2D tensor is processed by a parameter-efficient Inception module, which uses a multi-scale 2D core and is one of the most recognized visual foundation models. Then, the learned 2D representation Convert back to one-dimensional space For aggregation, the Trunc(·) function is used to truncate the padded sequence and convert the length p i ×f i Truncated to original length T.

[0121] k different one-dimensional representations The final output is obtained by weighted summing of the intensities of the corresponding frequencies. Similar to the design in Autoformer, the intensity A of each frequency component can reflect the relative importance of the selected frequency and period, and is represented in one dimension using intensity aggregation. The specific expression is:

[0122]

[0123] Among them, A is the intensity of each frequency component. Through the above design, TimesNet completes the temporal change modeling process of "extracting two-dimensional temporal changes in multiple cycles and then adaptively fusing them".

[0124] Fully Convolutional Network:

[0125] The mobile phone sensor data is input into the fully convolutional network (FCN) model, and the time series data is processed through one-dimensional convolution to capture the characteristics of the input signal changing over time. Each time window can be regarded as a one-dimensional time series with a time length of T and a channel dimension of C.

[0126] Among them, the fully convolutional network consists of three stacked convolutional blocks, each of which consists of a one-dimensional convolutional filter set accompanied by batch normalization and linear rectification function ReLu.

[0127] Suppose we add L convolutional layers to the model. A set of one-dimensional convolutional filters is applied to each layer. Unlike two-dimensional convolution, one-dimensional convolution mainly processes time series data and captures the characteristics of the input signal changing over time. and bias To define the one-dimensional convolution filter of each layer, where l∈{1,…,L} represents the index of each layer, d is the duration of the filter (the size of the convolution kernel), and F represents the range covered by each filter in the time dimension. l It represents the feature dimension of layer l. For layer l, the unnormalized activation value of the i-th component is the normalized activation matrix from the previous layer The calculated formula is as follows:

[0128]

[0129] Where f(·) represents the activation function. t′ is the offset index of the convolution kernel in the time dimension. In the convolution operation, by combining the values ​​of the current time step and multiple time steps before and after it, the model can effectively extract dynamic features in time series data.

[0130] The sizes of the one-dimensional convolution filters are fixed to 128, 256, and 128, respectively, and the convolution kernel sizes are set to 8, 5, and 3.

[0131] Merge one-dimensional features with two-dimensional features, including:

[0132] After the preprocessed data is extracted by TimesNet with high-level features of two-dimensional time series changes, the two-dimensional features obtained by TimesNet are nonlinearly transformed using Gaussian error linear units, and dropout operations are performed to enhance the generalization ability of the model and prevent overfitting, and the two-dimensional features that have undergone nonlinear transformation are flattened into one-dimensional features.

[0133] The one-dimensional features extracted by FCN are passed through a global average pooling layer to generate one-dimensional features with global information.

[0134] The one-dimensional features of the global average pooling layer and the one-dimensional features flattened by TimesNet are merged and input into the fully connected layer.

[0135] That is, linear transformation and dropout operations are performed on the features extracted by TimesNet to enhance the generalization ability of the model. At the same time, global average pooling is performed on the features extracted by FCN. Then, the features of these two parts are aggregated and input into the fully connected layer.

[0136] Step S4: extract the scores of the merged features on different driving style categories, convert the scores into probability distribution, and select the driving style category with the highest probability as the user's predicted driving style.

[0137] The predicted driving style results are obtained through the softmax function and argmax function.

[0138] Extract the scores of the merged features on different driving style categories, convert the scores into probability distribution, and select the driving style category with the highest probability as the user's predicted driving style, including:

[0139] The merged features are processed and output through a fully connected layer, and the output dimension of the fully connected layer is used as the number of driving style categories of the task, and the scores of the merged features in different driving style categories are extracted.

[0140] The softmax function is used to convert the score into a probability distribution, and the argmax operation is used to select the driving style category with the largest probability, and the largest driving style category is used as the user's predicted driving style; the specific expression is:

[0141] f s =[X TimesNet ; E FCN ];

[0142] z i =W y f s +b y ;

[0143]

[0144]

[0145] Among them, X TimesNet It is the result of reshaping the high-level features extracted by TimesNet. FCN It is the feature extracted by FCN after global average pooling; f s is the merged feature, which is used as the input of the fully connected layer; W y , b y are the weight matrix and bias vector of the fully connected layer respectively; z i is the unnormalized score vector of the i-th sample output by the fully connected layer, whose dimension is C (i.e., the number of label types in the DSD task), z i,j represents the score of the i-th sample being of type j; P i,j is the probability that the i-th sample is of type j calculated by the Softmax function; is the type predicted by the i-th sample (category labels include normal, aggressive, and sleepy).

[0146] The above method combines the one-dimensional time series features and the two-dimensional time series features of the input data captured in the feature extraction stage, thereby improving the classification performance of the model.

[0147] The present invention will be further described below through experiments:

[0148] 1. Dataset:

[0149] The experiment used two natural driving datasets, including the public dataset UAH-Driveset and the self-collected dataset Sensorlogger-Driveset.

[0150] (1)UAH-Driveset:

[0151] UAH-Driveset is the most commonly used natural driving public dataset in the work related to driving style detection based on smartphones. The dataset was collected by 6 drivers using an application called DriveSafe. The detailed information is shown in Table 1.

[0152] Table 1 Naturalistic driving public datasets

[0153]

[0154] A total of 40 trips totaling 500 minutes of natural driving raw data and processed data were collected. They fixed the mobile phone on the front windshield of the vehicle. The trip data contains accelerometer, GPS and mobile phone posture information, and the front camera is also used to record the environmental information of the road during driving. In the dataset, the road type (highway and secondary road) of each trip is marked, and the trip is also marked as normal, aggressive, and drowsy. In addition, since the mobile phone posture in this dataset is fixed and Kalman filtering has been performed, only some preprocessing methods are applied to this dataset.

[0155] (2)Sensorlogger-Driveset:

[0156] The phone was fixed on the center console of the car using a phone holder, and the phone could be pointed at the driver's face in any posture. The natural driving data was collected using Sens or Logger, developed by Kelvin, Tsz Hei Choi and still being updated. This is an easy-to-use and beautifully designed data logging application that can record readings of common sensors on Android phones, iPhones, iPads, Apple Watches, and AirPods, including accelerometers, gyroscopes, GPS, audio, cameras, and any nearby Bluetooth devices. It can also record device properties such as screen brightness, battery level, and network status. The inertial sensor sampling rate was set to 10Hz and the GPS sampling rate was 1Hz. At the same time, the driver's facial information was captured using the front camera of the phone for subsequent research.

[0157] Sensorlogger-Driveset collected a total of 34 trips of 3 drivers, totaling about 1,000 minutes of natural driving data. These data include driving routes in Lanzhou, Zhengzhou and other places for both iOS and Android devices. In the natural driving test, the driver is completely undisturbed, so the collected data is highly reliable and can intuitively reflect the driver's driving style and driving habits in a natural state. Many studies use the Driving Behavior Questionnaire (DBQ) or Driving Anger Scale (DAS) to label the driver's driving style, but it is found that DBQ and DAS may not be applicable to the Chinese driving environment and are easily affected by the driver's memory. The driving style is controlled by informing the driver of the driving style type in advance to label the driving style as normal and aggressive. In addition, since most trip routes may cover very complex road types, the road type is not marked.

[0158] (3) Dataset differences:

[0159] All data preprocessing methods will be used to process the raw data in Sensorlogger-Driveset. Through these preprocessing methods, the raw sensor data will be converted into a form more suitable for model training and driving style detection. In the preprocessed UAH-Driveset, only part of the preprocessing process is used because the Kalman filter and redirected data are already provided. As shown in Table 2, 9 features are selected in UAH-Driveset, and 14 features are selected for input into the model in Sensorlogger-Driveset. In order to avoid data leakage, the latitude and longitude and altitude information in GPS are not selected. In addition, in order to verify the effectiveness of the proposed TimesNet-FCN model, the preprocessed data set of Saleh et al. in their work is also used, which contains 9 features including accelerometer, GPS, direction sensor, and road information extracted from the front camera (Distance to ahead vehicle, Number of detected vehicles).

[0160] Table 2 Different feature selection schemes

[0161]

[0162] 2. Experimental setup and training details:

[0163] The experimental platform is built on NVIDIA GeForce RTX 4070Ti, whose video memory size is 12GB. The PyTorch framework is used to train and test the model. For specific experimental environment data, please refer to Table 3. The AdamW optimizer is used, which improves the performance and generalization ability of the model through more reasonable weight decay processing and is easier to use. The learning rate is set to 0.005, and the weight decay is fixed to 0.0025. The Batch Size and Epoch of the experiment in UAH-Driveset are set to 1500 and 1500 respectively, and the Batch Size and Epoch of the experiment in Sensorlogger-Driveset are set to 3000 and 1500 respectively. The weighted cross entropy loss is also used as the loss function to solve the problem of imbalance in the number of samples. The formula is as follows:

[0164]

[0165] Where N is the total number of samples, w j is the weight of category j to which sample i belongs. The ratio of the number of samples in each category in the training set will be calculated as the weight before the data is input into the model.i,j is the predicted probability that sample i belongs to category j.

[0166] The sampling rate of each data set after resampling is set to 10Hz. After preprocessing, each trip is divided into a time window with a duration of 6.4 seconds and an overlap time of 3.2 seconds through a sliding time window. Each data set is divided into training, validation, and test sets in a ratio of 7:1:2.

[0167] Table 3 Detailed information of the experimental platform

[0168]

[0169] Evaluation Metrics:

[0170] Accuracy, precision, recall and F1-measure are used as evaluation indicators of the model, and macro-average is used to aggregate multi-category indicators. These indicators can comprehensively measure the performance of the model in the DSD task from different perspectives and are widely used in related work. The calculation formulas of the four indicators are as follows:

[0171]

[0172] Where N is the number of categories. i Indicates the number of samples correctly classified into category i; FP i Indicates the number of samples misclassified as category i; FN i Indicates the number of samples that are actually of category i but are classified as other categories; TN i It represents the number of samples that are not actually of category i and are correctly classified as other categories.

[0173] 3. Comparative experiment:

[0174] First, the proposed TimesNet-FCN is compared with some existing methods, including classical machine learning methods and deep learning methods, on the preprocessed UAH-Driveset and Sensorlogger-Driveset. Because most existing methods are not open source or implemented in TensorFlow, most methods are re-implemented in Pytorch according to the description in the literature. The comparison results are shown in Table 4. Among the classical machine learning methods, RandomForest and Adaboost also perform well on some datasets. The performance of deep learning-based methods is generally better than that of classical machine learning-based methods, among which FCN-LSTM and the proposed TimesNet-FCN perform very well on all datasets. The proposed TimesNet-FCN achieved 93.09%, 93.19%, 93.07% and 93.13% accuracy, precision, recall and F1 in the processed UAH-Driveset, and 98.91%, 98.02%, 98.48% and 98.25% in the Sensorlogger-Driveset, which are better than the other compared methods.

[0175] It was also found that in Sensorlogger-Driveset, most methods achieved good performance. In the processed UAH-Driveset, the performance was generally low. It is believed that this is due to the smaller data volume of UAH-Driveset compared to Sensorlogger-Driveset and the smaller number of selected features. Therefore, TimesNet-FCN was verified in the UAH-Driveset dataset preprocessed by Saleh et al., which is used in many works. It was found that the proposed TimesNet-FCN model still achieved an F1 performance of 96.66%, which is slightly better than the F1 performance of CNN-LSTM by 0.72%.

[0176] Table 4 Comparison results of two different feature selections in UAH-Driveset and other methods in Sensorlogger-Driveset in four indicators

[0177]

[0178] TimesNet hyperparameter comparison:

[0179] By fixing the hyperparameters of the FCN part and adjusting and comparing the hyperparameters of the TimesNet part, we can understand the impact of TimesNet hyperparameter settings on the performance of the proposed TimesNet-FCN model. Among them, referring to the settings in the study by Wu et al., the number of frequencies with the highest intensity, Top-k, is fixed to 3, and the Dropout after TimesNet is set to 0.3. The number of layers of TimesNet is set to {1,2,3}, the dimension of the embedding vector (d_model) and the intermediate feature dimension (d_ff) of the Inception module are set to {8,16,32.64}, and the number of convolution kernels (num_kernels) is set to {3,6,9}.

[0180] Through grid search, the best performance F1: 93.13% was achieved in the processed UAH-Driveset when layers=2, d_model=16, d_ff=16, and num_kernels=6. In the Sensorlogger-Driveset, the best performance F1: 98.25% was achieved when layers=1, d_model=16, d_ff=16, and num_kernels=6. In the UAH-Driveset processed by Saleh et al., the best performance F1: 96.66% was achieved when layers=2, d_model=16, d_ff=64, and num_kernels=6.

[0181] During the study, other hyperparameters that achieve the best performance are kept fixed, and only one hyperparameter is adjusted within a range to observe the impact of each hyperparameter on the F1 score of the model in the three datasets. Through observation, it is concluded that in Sensorlogger-Driveset, the performance of the model is less sensitive to the hyperparameter settings. On the one hand, this may be because the FCN part may contribute more to the performance of the model on the Sensorlogger-Driveset dataset than the TimesNet part. On the other hand, in the Sensorlogger-Driveset dataset, the model may have achieved a higher performance level.

[0182] At this point, small adjustments to the hyperparameters have limited effect on improving model performance, resulting in low sensitivity. In the UAH-Driveset preprocessed by the two methods, the F1 score increased significantly when d_ff and d_model increased within a certain range, while too high or too low values ​​may lead to performance degradation. For layers and num_kernels, too many or too few layers and cores will affect model performance, and the optimal value varies depending on the dataset. These results show that each dataset has different sensitivities to hyperparameters, so when tuning hyperparameters, it is necessary to combine the specific dataset characteristics to ensure the optimization of model performance.

[0183] 4. Ablation experiment:

[0184] (1) Model validity study:

[0185] In order to study the efficacy of each module in the proposed TimesNet-FCN model, the model is split into independent TimesNet and FCN, and their respective performance changes are observed. Specifically, we hope to understand the performance of each module on different datasets and analyze their contribution to the overall model performance through this split method. Table 5 shows the performance of each independent module in three datasets, including the preprocessed UAH-Driveset, Sensorlogger-Driveset, and UAH-Driveset preprocessed by Saleh et al.

[0186] Table 5 shows the performance of each independent module in three datasets. The results show that TimesNet performs poorly in all datasets, especially in the processed UAH-Driveset dataset. By further observing other indicators during the training process, it is found that this is caused by overfitting in the late stage of training. In contrast, FCN significantly outperforms TimesNet in all indicators on all datasets, especially in the Sensorlogger-Driveset dataset.

[0187] In addition, it was found that the combination of TimesNet and FCN significantly improved the performance of the model on various data sets, which shows that the two complement each other and the combined model can better capture the complex features in the data. It can effectively improve the classification performance. Overall, the ablation experiment verified the contribution of each component to the overall model performance, especially the contribution of FCN. However, TimesNet also captured the advanced features that FCN could not capture in one-dimensional space by extracting two-dimensional time series features, further improving the overall performance of the model and its ability to process complex data.

[0188] Table 5 Comparison of four indicators on different feature selection of two datasets after splitting the model into separate TimesNet and FCN

[0189]

[0190] (2) Redirection effectiveness study:

[0191] As shown in Table 6, in order to verify the effectiveness of the redirection method proposed in the preprocessing method, experiments were conducted in the Sensorlogger-Driveset dataset. In the experiment, the redirection method in the preprocessing was cancelled, and the TimesNet-FCN model was used for the DSD task, and the results were compared with the results of the redirection method that was not cancelled. The experimental results show that after canceling the redirection method, the accuracy of the model decreased by 0.13%, the precision decreased by 0.24%, the recall rate decreased by 0.19%, and the F1 score decreased by 0.22%. This proves that the proposed redirection method can improve the accuracy and robustness of the model. Especially when the mobile phone is placed differently in the car, or the direction and position change, this redirection method can effectively improve the robustness of the method in real driving scenarios.

[0192] Table 6 Comparison of the impact of redirection on the final result in Sensorlogger-DriveSet

[0193]

[0194] 5. Classification visualization:

[0195] In order to better display the results of feature extraction and classification to facilitate analysis, the feature extraction and classification effects of the proposed TimesNet-FCN model in three datasets were visualized through t-SNE and confusion matrix.

[0196] It can be seen that before feature extraction, the distribution of sample points in the three data sets in the low-dimensional space has no obvious category boundaries, showing strong hybridity, reflecting the limitations of the original features in distinguishing different driving styles. After feature extraction by the TimesNet-FCN model, a feature representation with higher discrimination is generated, making the distribution of samples of different categories in the low-dimensional space more separable, forming a clear clustering structure. Specifically, in the processed UAH-Driveset, most of the samples after feature extraction can be correctly classified, among which the aggressive driving samples form obvious clusters in the feature space and are well separated from samples of other categories. However, the separation of normal driving and drowsy driving samples in the feature space is low, and some samples still overlap to a certain extent. In the UAH-Driveset and Sensorlogger-Driveset preprocessed by Saleh et al., except for a small number of samples that are incorrectly displayed in the areas of other categories, most of the samples form obvious clustering areas. The confusion matrix further verifies the statement.

[0197] Based on the above method, the present invention provides a driving style detection system based on a mobile phone sensor, comprising: an acquisition module, a feature capture module and a style detection module.

[0198] Among them, the acquisition module is used to collect various mobile phone sensor data when the user is driving; the feature capture module is used to input various mobile phone sensor data into the pre-trained fully convolutional network FCN and time series model TimesNet respectively, and capture the one-dimensional features of the signal changing over time in the input mobile phone sensor data through FCN, and expand the one-dimensional time series in the mobile phone sensor data to two-dimensional space through TimesNet to obtain two-dimensional features, and merge the one-dimensional features with the two-dimensional features; the style detection module is used to extract the scores of the merged features in different driving style categories, and convert the scores into probability distributions, and select the driving style category with the largest probability as the user's predicted driving style.

[0199] The present invention also provides a computer device, including a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of a driving style detection method based on a mobile phone sensor.

[0200] In accordance with the disclosed embodiments, a computing device may communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth communications, etc.), or with any device (e.g., routers, modems, etc.) that enables a computing device to communicate with one or more other computing devices.

[0201] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of a driving style detection method based on a mobile phone sensor are implemented.

[0202] According to the disclosed embodiments, the storage medium may be a non-volatile computer-readable storage medium, such as but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0203] The above content is a further detailed description of the present invention in combination with a specific preferred embodiment. For technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A driving style detection method based on a mobile phone sensor, characterized in that: The steps include: Collect various mobile phone sensor data while users are driving; Inputting the plurality of mobile phone sensor data into the pre-trained fully convolutional network FCN and the time series model TimesNet respectively, capturing the one-dimensional features of the signal changing over time in the input mobile phone sensor data through FCN, expanding the one-dimensional time series in the mobile phone sensor data into a two-dimensional space through TimesNet to obtain two-dimensional features, and merging the one-dimensional features with the two-dimensional features; The scores of the combined features on different driving style categories are extracted, and the scores are converted into probability distributions, and the driving style category with the highest probability is selected as the predicted driving style of the user.

2. The driving style detection method based on a mobile phone sensor as claimed in claim 1, characterized in that: Before collecting the various mobile phone sensor data of the user while driving, the method also includes: Resampling the multiple historical mobile phone sensor data, and fusing the resampled data; The fused data is redirected, Kalman filtered, filled with null values, extracted, and segmented in a sliding time window to obtain the processed training set, validation set, and test set, and the training set, validation set, and test set are standardized; Standardized training sets and test sets are used to pre-train the fully convolutional network FCN and time series model TimesNet.

3. The driving style detection method based on mobile phone sensor as claimed in claim 1, characterized in that: The one-dimensional features of the signal changing over time in the input mobile phone sensor data are captured by FCN, and the one-dimensional time series in the mobile phone sensor data is expanded to a two-dimensional space by TimesNet to obtain two-dimensional features, including: Input a variety of mobile phone sensor data into the TimesNet model, decompose the time series changes in the mobile phone sensor data into different periods, and transform the one-dimensional time series of different periods into two-dimensional space; the specific expression is: A=Avg(Amp(FFT(X 1D ))); in, Represents X 1D The intensity of each frequency component in the k frequencies with the largest intensity {f1,…,f k } corresponds to the most significant k period lengths {p1,…,p k }, a one-dimensional time series with a time length of T and a channel dimension of c The above process is simplified as follows: A,{f1,…,f k },{p1,…,p k }=Period(X 1D ); The original one-dimensional time series is folded based on the selected period. The process can be formalized as: Where A is the intensity of each frequency component, and Padding(·) is to add 0 at the end of the sequence so that the sequence length is p i The function of integer division, Corresponding to the period p i Dominant two-dimensional temporal changes.

4. The driving style detection method based on a mobile phone sensor as claimed in claim 3, characterized in that: The method of expanding the one-dimensional time series in the mobile phone sensor data to a two-dimensional space through TimesNet to obtain two-dimensional features also includes: TimesNet is obtained by stacking the normalized TimesBlock in a residual manner. The specific expression is: Among them, TimesBlock contains the following sub-processes: in, It is a two-dimensional feature The 2D tensor after the i-th transformation in ; Inception is a parameter-efficient convolution module; is the learned two-dimensional feature. Convert back to one-dimensional space Aggregation is performed, where the Trunc(·) function is used to truncate the padded sequence; k different one-dimensional representations The weighted summation of the intensity of the corresponding frequency is used to obtain the final output. The specific expression is: Among them, d model is the dimension of the embedding vector, Softmax is the activation function, is the weight of each cycle, The final output of the TimesNet module is obtained by aggregating each frequency.

5. The driving style detection method based on mobile phone sensor as claimed in claim 1, characterized in that: The fully convolutional network FCN consists of three stacked convolutional blocks, each of which consists of a one-dimensional convolutional filter set accompanied by batch normalization and a linear rectification function ReLu.

6. The driving style detection method based on mobile phone sensor as claimed in claim 1, characterized in that: The one-dimensional feature and the two-dimensional feature are merged, comprising the following steps: Use Gaussian error linear unit to nonlinearly transform the two-dimensional features obtained by TimesNet, and flatten the nonlinearly transformed two-dimensional features into one-dimensional features; The one-dimensional features extracted by FCN are passed through the global average pooling layer to generate one-dimensional features with global information; Merge the one-dimensional features of the global average pooling layer and the one-dimensional features flattened by TimesNet.

7. The driving style detection method based on mobile phone sensor as claimed in claim 1, characterized in that: The scores of the combined features on different driving style categories are extracted, and the scores are converted into probability distributions, and the driving style category with the highest probability is selected as the predicted driving style of the user, including: The merged features are processed and output through a fully connected layer, and the output dimension is used as the number of driving style categories of the task. The scores of the merged features in different driving style categories are extracted. The softmax function is used to convert the score into a probability distribution, and the argmax operation is used to select the driving style category with the largest probability, and the largest driving style category is used as the user's predicted driving style; the specific expression is: f s =[X TimesNet ;E FCN ]; z i =W y f s +b y ; Among them, X TimesNet It is the result of reshaping the high-level features extracted by TimesNet. FCN It is the feature extracted by FCN after global average pooling, f s is the merged feature, which is used as the input of the fully connected layer. y , b y are the weight matrix and bias vector of the fully connected layer, z i is the unnormalized score vector of the i-th sample output by the fully connected layer, with dimensions C, z i,j represents the score of the i-th sample as type j, k is the type within the dimension range, P i,j is the probability that the i-th sample is of type j calculated by the Softmax function, is the type predicted by the i-th sample, P i is the probability of any type in the i-th sample.

8. A driving style detection system based on mobile phone sensors, characterized in that: include: The acquisition module is used to collect various mobile phone sensor data while the user is driving; A feature capture module, which is used to input the multiple mobile phone sensor data into the pre-trained fully convolutional network FCN and the time series model TimesNet, respectively, to capture the one-dimensional features of the signal changing over time in the input mobile phone sensor data through FCN, to expand the one-dimensional time series in the mobile phone sensor data to a two-dimensional space through TimesNet, to obtain two-dimensional features, and to merge the one-dimensional features with the two-dimensional features; The style detection module is used to extract the scores of the combined features on different driving style categories, convert the scores into probability distribution, and select the driving style category with the highest probability as the user's predicted driving style.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of a driving style detection method based on a mobile phone sensor as claimed in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a driving style detection method based on a mobile phone sensor according to any one of claims 1 to 7 are implemented.

Images