Intelligent driving vehicle riding comfort evaluation method based on multi-modal data

Abstract

The invention belongs to the field of automatic driving technology and man-machine interaction, and discloses a multi-modal data-based intelligent driving vehicle riding comfort evaluation method, which comprises the following steps of: acquiring vehicle dynamic performance data, vehicle driving behavior information and passenger human body physiological indexes through a sensor, and collecting subjective scores of passengers at the same time; and preprocessing the collected data, eliminating invalid physiological reactions, carrying out subjective score consistency verification, carrying out association analysis and screening valid data, and carrying out image conversion to form a final training sample. A systematized automatic driving vehicle comfort evaluation model is constructed, and vehicle dynamic performance, physiological response and subjective feeling are comprehensively considered. According to the systematized evaluation model for the riding comfort of the autonomous vehicle, vehicle behaviors and objective responses and subjective feelings of passengers are integrated into an evaluation system, and the systematized evaluation model is more accurate and reliable.

Description

Technical Field

[0001] The present invention belongs to the fields of autonomous driving technology and human-computer interaction, and relates to a method for evaluating the ride comfort of intelligent driving vehicles based on multimodal data. Background Art

[0002] In recent years, with the popularization of electric vehicles and the improvement of the level of autonomous driving technology, high-level autonomous driving vehicles are gradually transforming the driver role into a passenger. Although this transformation relieves the driving burden of the passenger, it increases the risk of discomfort such as motion sickness. In order to improve the risk of discomfort, it is first necessary to accurately evaluate the comfort of the passenger.

[0003] However, the current evaluation of the comfort of intelligent driving vehicles mostly relies on dynamic indicators such as acceleration or subjective scoring through questionnaires for analysis. However, these methods have problems such as the lack of physiological signal response modeling, single data modality, poor scoring consistency, and insufficient model generality and intelligence. Therefore, there is an urgent need to construct an intelligent evaluation method that integrates multi-source data and has deep sensing capabilities to improve the objectivity, accuracy, and real-time performance of comfort analysis.

[0004] The patent with the application publication number CN112353392B discloses a method for evaluating the comfort of occupants of intelligent driving vehicles, which obtains the physical sign information of the measured occupant in a stationary state and during the driving of the intelligent driving vehicle, as well as the subjective comfort evaluation index of the measured occupant. Through the objective comfort evaluation model based on the physical sign information, the objective comfort evaluation index based on the physical sign signal is calculated; through the occupant comfort prediction model based on vehicle dynamics, the occupant comfort prediction evaluation index based on vehicle dynamics information is obtained; finally, an occupant comfort comprehensive evaluation model is constructed, and based on the occupant comfort comprehensive evaluation index predicted by the occupant comfort comprehensive evaluation model and the vehicle three-degree-of-freedom model, a vehicle dynamics control domain based on occupant comfort is established to ensure ride comfort. However, this method tends to be applied and deployed, and its modeling method in terms of ride comfort evaluation is relatively traditional, with limited ability to express complex non-linear characteristics, and the output is a comprehensive index K, without fine-grained distinction in scoring.

[0005] Based on the above background, in order to overcome the deficiencies of the existing evaluation methods, it is necessary to propose a new method for evaluating the ride comfort of autonomous driving vehicles to improve the accuracy and practicality of the evaluation. Summary of the Invention

[0006] In order to solve the above technical problems, the present invention provides a method for evaluating the ride comfort of intelligent driving vehicles based on multimodal data.

[0007] The technical solution of the present invention:

[0008] An intelligent driving vehicle ride comfort evaluation method based on multi-modal data, the steps are as follows:

[0009] S1. Collect target data;

[0010] S11. Determine test conditions: clarify the test environment and conditions, including vehicle type, road conditions, and driving speed;

[0011] S12. Arrange sensors: Arrange sensors on the vehicle and the human body. The sensors include triaxial acceleration sensors, galvanic skin response (GSR) acquisition electrodes and GSR meters, electrocardiogram (ECG) acquisition electrodes and ECG meters. The computer is connected to the vehicle OBD-II interface through a USB-CAN interface device; among them, the triaxial acceleration sensors are used to collect vehicle dynamic performance data, and the computer reads driving behavior parameters in the CAN bus by connecting to the vehicle OBD-II interface through a USB-CAN interface device. The GSR acquisition electrodes and GSR meters, and the ECG acquisition electrodes and ECG meters are used to collect human physiological indicators;

[0012] Collect vehicle dynamic performance data, which is obtained by triaxial acceleration sensors installed under the co-pilot seat; the triaxial acceleration sensors collect the dynamic accelerations of the vehicle along the forward direction X-axis, the lateral Y-axis perpendicular to the forward direction on the horizontal plane, and the vertical Z-axis perpendicular to the forward direction on the vertical plane;

[0013] Collect driving behavior parameters, and the computer is connected to the vehicle OBD-II interface through a USB-CAN interface device and reads them through the CAN bus; the driving behavior parameters include vehicle speed, engine speed, throttle opening, accelerator pedal opening, and driving operations such as lane change, acceleration, and overtaking manually recorded and synchronously marked on the time stamp. The duration of the operation is calculated through the start and end time stamps of the action;

[0014] Collect human physiological indicators. The GSR acquisition electrodes are fixed on the thenar muscles of both palms to measure the human GSR signal and are connected to the GSR meter through a wireless transmission device; the ECG acquisition electrodes are arranged at the chest, left arm, and right arm positions to measure the human ECG signal and are connected to the ECG meter through a wireless transmission device; the GSR meter and the ECG meter are connected to the computer through their microcomputer interfaces and transmit the collected signals to the computer; among them, the ECG acquisition electrodes are three-lead patch electrodes for ECG acquisition;

[0015] S13. Collect the required information: The vehicle dynamic performance data and driving behavior parameters under different conditions recorded by the vehicle OBD are used to evaluate the vehicle dynamic performance and its impact on comfort; synchronously record the human physiological indicators to quantify the human physiological response to vehicle movement; collect the subjective scores of passengers or drivers to evaluate the comfort based on their actual feelings. The evaluation range is [1, 10] points, and a 0.5-point interval is introduced. 1 point means extremely uncomfortable, and 10 points means very comfortable;

[0016] S2. Perform association analysis to screen valid data;

[0017] S21. Data preprocessing: Systematically preprocess the raw data collected in step S1. First, perform interval filtering on the subjective scores. Second, judge the physiological response amplitude. If the standard deviation of heart rate variability < 2 bpm or the change rate of skin conductance amplitude < 5%, it is regarded as an invalid response and the data is excluded;

[0018] The preprocessing method is as follows: Systematically preprocess the raw data collected in step S1. First, perform interval filtering on the subjective scores and retain the subjective scores between [3, 8]. Second, judge the physiological response amplitude. For the electrocardiogram signal, calculate the standard deviation of heart rate variability SDNN within a 5s window before and after the driving operations of lane change, acceleration, and overtaking recorded manually:

[0019]

[0020] where n is the total number of RR intervals, is the i-th RR interval, is the average value of all RR intervals within this window; where the RR interval is the time interval between two consecutive R waves in an electrocardiogram within the window;

[0021] If the standard deviation of heart rate variability SDNN < 2 bpm, it is regarded as an invalid response and the corresponding data is excluded;

[0022] For the skin conductance signal, calculate the change rate of skin conductance amplitude within a 5s window before the driving operations of lane change, acceleration, and overtaking recorded manually and a 10s window after the driving operations:

[0023] First, calculate the average skin conductance values within the 5s window before the driving operation and the 10s window after the driving operation respectively:

[0024]

[0025] where N is the number of sampling points within the window, is the skin conductance value at the a-th moment;

[0026] Then the change rate of skin conductance amplitude is:

[0027]

[0028] where, is the average skin conductance value within the 10s window after the operation, is the average skin conductance value within the 5s window before the operation;

[0029] If < 5%, it is regarded as an invalid response and the corresponding data is excluded;

[0030] S22. Subjective scoring consistency verification: Use the coefficient of consistency analysis to analyze the preprocessed data, exclude samples with large differences in scores of the same driving operation by multiple evaluators, and samples with large differences in scores before and after for the same driving operation by a single evaluator; perform segmented equalization processing on the subjective scores to avoid sample bias in the training set;

[0031] The method for subjective scoring consistency verification is: Use Kendall's W coefficient of concordance to analyze the scoring coordination of multiple evaluators for the same driving operation. First, calculate the total sum of squared rank differences :

[0032]

[0033] Among them, is the total rank sum of the b-th sample, that is, the sum of the scores of all evaluators for this sample, is the average value of the rank sums of all samples;

[0034] Then Kendall's W coefficient of concordance is:

[0035]

[0036] Among them, is the number of evaluators, is the number of samples evaluated by each evaluator, is the total sum of squared rank differences;

[0037] If <0.7, it is regarded as large differences in scores of the same driving operation by multiple evaluators, and the corresponding data is excluded;

[0038] Use Cronbach's α coefficient of internal consistency to measure the consistency of a single evaluator's scores for the same driving operation multiple times:

[0039]

[0040] Among them, is the number of items, that is, the number of the same driving operations scored by a single evaluator, is the variance of the score of the c-th same driving operation, is the overall variance of the scores of the evaluator in all the same driving operations;

[0041] If <0.75, it is regarded as large differences in scores before and after for the same driving operation multiple times by a single evaluator, and the corresponding data is excluded;

[0042] Segment the subjective score \(y\in[3, 8]\), use piecewise random downsampling to balance the number of samples, divide the subjective score into several sub - intervals, take the number of samples in the score segment with the least samples as the upper limit, and randomly downsample the remaining score segments to this number to balance the samples in the low, medium, and high score segments;

[0043] S23. Image conversion of data: Extract vehicle dynamic performance data and human physiological index data through time windows, perform time - frequency conversion to generate frequency - domain images. The frequency - domain images include acceleration images and physiological images; Construct a structured driving behavior vector from driving behavior parameters; The subjective score is used as the label of the training target value of the subsequent convolutional neural network - based evaluation model for the ride comfort of autonomous vehicles; The acceleration image, physiological image, and structured driving behavior vector are used as the inputs of the convolutional neural network - based evaluation model for the ride comfort of autonomous vehicles; Use the acceleration image, physiological image, and structured driving behavior vector to construct a training set and bind the subjective score as the label. Each sample consists of {acceleration image, physiological image, structured driving behavior vector, subjective score label};

[0044] After screening, perform image conversion on the sample data retained in S22. Convert the dynamic accelerations of the X - axis, Y - axis, and Z - axis of the vehicle dynamic performance data into Mel spectrograms through short - time Fourier transform and Mel filtering, map the Mel spectrograms of the X - axis, Y - axis, and Z - axis to the R, G, and B channels respectively, and combine them into a three - channel color image to form the acceleration image; Convert the electrocardiogram signal and galvanic skin signal of the human physiological index data into spectrograms using CWT and Mel transform respectively to form the physiological image; The driving behavior parameters are not converted into images, but a structured driving behavior vector is constructed; The sizes of the acceleration image and physiological image are unified to \(224\times224\times3\), and all inputs are bound with the score label to form the final training samples:

[0045] ;

[0046] Among them, is the acceleration image, is the physiological image, is the structured driving behavior vector, is the subjective score label value;

[0047] S3. Construct a convolutional neural network - based evaluation model for the ride comfort of autonomous vehicles: Input the three - channel information of vehicle dynamic performance data, human physiological indexes, and driving behavior parameters into the constructed convolutional neural network - based evaluation model for the ride comfort of autonomous vehicles;

[0048] The convolutional neural network - based evaluation model for the ride comfort of autonomous vehicles is:

[0049] Construct a three - way neural network system:

[0050] The convolutional neural network 1 branch extracts the acceleration image features: ;

[0051] The convolutional neural network 2 branch extracts the image features of the electrocardiogram signal and the galvanic skin signal: ;

[0052] The multi-layer perceptron branch processes the driving behavior vector: ;

[0053] The three-way neural network system outputs the feature fusion and then performs a non-linear mapping:

[0054] ;

[0055] Among them, ;

[0056] ;

[0057] Among them, is the weight matrix of the first fully connected layer; is the bias term of the first layer; is the rectified linear unit activation function: ReLU(x) = max(0, x); is the output weight of the second layer; is the bias term of the output layer; is the predicted value of the comfort score output by the comfort evaluation model of the autonomous driving vehicle;

[0058] Train the network with the mean squared error loss function:

[0059] ;

[0060] Among them, is the total number of samples in the training set, is the true subjective score label value of the i-th sample, is the score value predicted by the model for the i-th sample, is the loss of the model training target, that is, the mean squared error loss;

[0061] S4. Output the comfort evaluation result.

[0062] Advantages of the present invention: The acceleration information and physiological index information adopted by the present invention have strong visualization after being input as images, and the driving behavior information is input as a behavior vector, which is convenient for interpretation and visual tuning; the deep model adopted by the present invention has strong expressive power, especially high ability to capture non-linear physiological changes and adaptive behavior characteristics; the comfort evaluation method of the present invention is not limited to certain working conditions, and any working condition as long as the relevant driving behavior information is collected and input can use this method for evaluation. Brief Description of the Drawings

[0063] Figure 1 This is the overall flowchart of an intelligent driving vehicle ride comfort evaluation method based on multi-modal data according to the present invention;

[0064] Figure 2 This is the flowchart of the evaluation model of an intelligent driving vehicle ride comfort evaluation method based on multi-modal data according to the present invention;

[0065] Figure 3 This is the multi-modal evaluation index of an intelligent driving vehicle ride comfort evaluation method based on multi-modal data according to the present invention. Detailed Implementation Modes

[0066] The following further describes the detailed implementation modes of the present invention in combination with the drawings and technical solutions

[0067] As Figure 1 shown, this embodiment exemplarily shows an intelligent driving vehicle ride comfort evaluation method based on multi-modal data, including the following steps:

[0068] 1) Determine the test conditions: First, the vehicle type should have typical intelligent driving capabilities and stable power response characteristics. Second, the test road environment is selected to include a mixed traffic scenario of urban arterial roads and expressway sections, with various typical operating conditions such as lane changing, acceleration, braking, and ramps, to ensure that the test covers diverse driving conditions. In terms of setting the driving speed, to avoid the weakening of operations at too low speeds or uncontrollable data fluctuations at high speeds, the test limits the vehicle to run within the range of 30 - 80 km / h and monitors it in real time through GPS speed measurement. At each set scenario, the operation trigger time point is calibrated by preset signs, voice prompts, or the tester's manual recording method to ensure the consistency and time series alignability of behavior events and collected signals. In addition, to avoid environmental factors interfering with the physiological state of the occupants, the entire experiment is carried out under relatively constant outdoor temperature and humidity conditions, and the temperature inside the vehicle is adjusted between 22°C ± 1°C to avoid errors in skin conductance response caused by thermal and cold stimuli;

[0069] 2) Arrange sensors: For collecting vehicle dynamic performance data, a three-axis accelerometer is used. The sensor is fixed to the bottom of the co-pilot seat through double-sided tape and a shock-proof bracket, ensuring tight connection to the rigid connection points of the vehicle body structure to avoid interference of the seat's flexible structure on vibration signals. For collecting physiological index data of the occupants, a physiological signal collection device such as the BIOPAC MP36 multi-channel physiological signal collection system is used. The electrocardiogram signal is collected using the ECG100C module, and the three-lead electrodes are attached to the chest, left arm, and right arm positions; the skin conductance signal is collected through the EDA100C module, and the electrodes are fixed to the thenar eminence areas of both hands to improve conductivity and reduce motion artifacts.

[0070] To ensure stable electrode adhesion and excellent signal channel quality, before the formal experiment, all the test subjects need to perform the maximum voluntary contraction test action. The action process includes: the subject actively moves the arms slightly, and at the same time opens and closes the palms in sequence, simulating the displacement interference that may occur in the experiment. During this period, the instrument synchronously records the changes in skin conductance and electrocardiogram signals, and detects whether there is electrode detachment, signal loss, severe noise or pseudo-abnormal differences. This process also records the response peak value of the skin conductance under the maximum tension state, as a reference for the individual physiological response amplitude, which helps with later data normalization and response sensitivity analysis.

[0071] 3) Collect the required data: First, the computer accesses the CAN bus through the vehicle OBD interface, and uses the Bluetooth adapter to record multiple driving behavior parameters during the vehicle operation in real time, including vehicle speed, throttle opening, braking state, engine speed, etc., and synchronously extracts the dynamic accelerations of the vehicle along the forward direction X-axis, the lateral Y-axis perpendicular to the forward direction on the horizontal plane, and the vertical Z-axis perpendicular to the forward direction on the vertical plane under typical operation scenarios such as lane change, acceleration, and overtaking.

[0072] Secondly, collect the electrocardiogram and skin conductance signals synchronously recorded during the experiment. Set a fixed time window before and after each driving operation. The electrocardiogram extraction window is 5 s before and after the driving operation, and the skin conductance extraction window is 5 s before the driving operation and 10 s after the driving operation.

[0073] Finally, after each driving operation is completed, the occupant needs to subjectively rate the experience according to their personal real feelings. The rating standard uses an equal-interval scoring method from 1 to 10 points, where 1 point represents extreme discomfort and 10 points represents complete comfort, and a 0.5-point interval is introduced. The rating is completed by the occupant through the tablet interface or voice input method, and is automatically bound to the operation number, time stamp, and corresponding multi-modal acquisition data. The subjective evaluation form of comfort is shown in Table 1.

[0074] Table 1 Subjective evaluation form of comfort 4)

[0076] 5) Correlation analysis to screen valid data:

[0077] Data preprocessing: Systematically preprocess the collected raw data. First, filter the subjective rating interval, and retain the subjective ratings between [3, 8] points; secondly, judge the physiological response amplitude, extract the corresponding physiological waveform segments, and calculate the standard deviation of heart rate variability SDNN within the 5 s window before and after the lane change, acceleration, and overtaking driving operations recorded manually for the electrocardiogram signal:

[0078]

[0079] where \(n\) is the total number of RR intervals, is the \(i\)-th RR interval, is the average value of all RR intervals within the window; where the RR interval is the time interval between consecutive R waves in an electrocardiogram within the window;

[0080] If the standard deviation of heart rate variability SDNN < 2 bpm, it is regarded as an invalid response, and the corresponding data is excluded;

[0081] The electrodermal signal is to calculate the change rate of electrodermal amplitude in the 5s window before lane change, acceleration, and overtaking driving operations and the 10s window after driving operations recorded manually:

[0082] First, calculate the average skin conductance values within the 5s window before driving operations and the 10s window after driving operations respectively:

[0083]

[0084] where \(N\) is the number of sampling points within the window, is the skin conductance value at the \(a\)-th moment;

[0085] Then the change rate of electrodermal amplitude is:

[0086]

[0087] where, is the average skin conductance value of the 10s window after operation, is the average skin conductance value of the 5s window before operation;

[0088] If < 5%, it is regarded as an invalid response, and the corresponding data is excluded.

[0089] Subjective score consistency verification: Use Kendall's W coefficient of concordance to analyze the scoring coordination of multiple evaluators for the same driving operation. First, calculate the total sum of squared rank differences :

[0090]

[0091] where, is the sum of ranks of the \(b\)-th sample, that is, the sum of scores of all evaluators for this sample, is the average value of the sum of ranks of all samples;

[0092] Then Kendall's W coefficient of concordance is:

[0093]

[0094] where, is the number of evaluators, is the number of samples evaluated by each evaluator, is the total sum of squared rank differences;

[0095] If < 0.7, it is considered that there are large differences in the scores of the same driving operation by multiple evaluators, and the corresponding data is excluded;

[0096] The Cronbach's α internal consistency coefficient is used to measure the consistency of a single evaluator's scores for multiple identical driving operations:

[0097]

[0098] Among them, is the number of items, that is, the number of identical driving operations scored by a single evaluator, is the variance of the score of the c-th identical driving operation, is the overall variance of the evaluator's scores in all identical driving operations;

[0099] If < 0.75, it is considered that there are large differences in the scores of a single evaluator for multiple identical driving operations before and after, and the corresponding data is excluded;

[0100] After excluding abnormal samples, considering that the original score labels show an obvious skewed distribution and the samples are highly concentrated in the middle score interval, there is a risk of prediction bias in the model training process. To ensure that the samples in each score interval are fully learned during training, in this paper, the score labels are divided into three segments: low [3, 4.5], middle [5, 6.5], and high [7, 8]. Taking the number of samples in the score segment with the least samples as the upper limit, the remaining score segments are randomly downsampled to this number to balance the samples in the low, middle, and high score segments, so as to construct a balanced training set.

[0101] Image conversion of data: For the sample data retained after screening, image conversion is performed. The dynamic accelerations of the X-axis, Y-axis, and Z-axis of the vehicle dynamic performance data are converted into Mel spectrograms through short-time Fourier transform and Mel filtering. The Mel spectrograms of the X-axis, Y-axis, and Z-axis are respectively mapped to the R, G, and B channels and combined into a three-channel color image to form an acceleration image; the electrocardiogram signal and galvanic skin signal of the physiological index data of the occupant's body are respectively converted into spectrograms using CWT and Mel transform to form a physiological image; the driving behavior parameters are not converted into images, and a structured driving behavior vector is constructed; the sizes of the acceleration image and the physiological image are unified to 224×224×3, and all inputs are bound to the subjective score labels to form the final training samples:

[0102] ;

[0103] Among them, is the acceleration image, is a physiological image, is a structured driving behavior vector, is the subjective score label value.

[0104] 6) Construct an evaluation model for the ride comfort of autonomous vehicles based on a convolutional neural network:

[0105] Construct a three-way neural network system:

[0106] The convolutional neural network branch 1 extracts the acceleration image features: ;

[0107] The convolutional neural network branch 2 extracts the physiological image features: ;

[0108] The multi-layer perceptron branch processes the driving behavior vector: ;

[0109] Nonlinear mapping is performed after fusing the three-way output features:

[0110] ;

[0111] Among them,

[0112] ;

[0113] Among them, is the weight matrix of the first fully connected layer, is the bias term of the first layer, is the rectified linear unit activation function: ReLU(x)=max(0,x), is the output weight of the second layer, is the bias term of the output layer, is the predicted value of the comfort score output by the model.

[0114] Train the network with the mean squared error loss function:

[0115] ;

[0116] Among them, is the total number of samples in the training set, is the true subjective score value label of the i-th sample, is the score value predicted by the model for the i-th sample, is the loss of the model training target, that is, the mean squared error loss.

[0117] Finally, output the evaluation result of the ride comfort of intelligent driving vehicles; Figure 2 is the flowchart of the evaluation model of a method for evaluating the ride comfort of intelligent driving vehicles based on multi-modal data according to the present invention. Figure 3The multi-modal evaluation index of an intelligent driving vehicle ride comfort evaluation method based on multi-modal data according to the present invention.

[0118] The technical solution of the present invention is not limited to the restrictions of the above specific embodiments, and any technical deformation made according to the technical solution of the present invention falls within the protection scope of the present invention.

Claims

1. An intelligent driving vehicle ride comfort evaluation method based on multi-modal data, characterized in that, The steps are as follows: S1. Collect target data; S11. Determine test conditions: Define the test environment and conditions, including vehicle type, road conditions, and driving speed; S12. Arranging sensors: Arrange sensors on the vehicle and the occupant's body. The sensors include a triaxial acceleration sensor, a galvanic skin response (GSR) acquisition electrode and a GSR meter, an electrocardiogram (ECG) acquisition electrode and an ECG meter. The computer is connected to the vehicle through an interface device interface; among them, the triaxial acceleration sensor is used to collect vehicle dynamic performance data, and the computer reads the driving behavior parameters in the bus through an interface device connected to the vehicle interface, and the GSR acquisition electrode and GSR meter, and the ECG acquisition electrode and ECG meter are used to collect the physiological indexes of the human body. S13. Collect required information: Through the vehicle Record the vehicle dynamic performance data and driving behavior parameters under different conditions, which are used to evaluate the vehicle dynamic performance and its impact on comfort; simultaneously record the physiological indexes of the occupant's body to quantify the physiological response of the human body to vehicle movement; collect the subjective scores of the occupants, evaluate the comfort based on their actual feelings, the evaluation range is [1, 10] points, and introduce score intervals, divided into extremely uncomfortable, divided into very comfortable; S2. Conduct correlation analysis to screen valid data; S21. Data preprocessing: Systematically preprocess the raw data collected in step S1. First, perform interval filtering on the subjective scores. Second, judge the physiological response amplitude. If the standard deviation of heart rate variability < 2 bpm or the change rate of skin conductance amplitude < 5%, it is regarded as an invalid response and the data is excluded; S22. Subjective score consistency verification: Analyze the preprocessed data using the consistency analysis coefficient, and exclude samples with large differences in scores given by multiple evaluators for the same driving operation, as well as samples with large differences in scores given by a single evaluator for multiple identical driving operations before and after; Perform segmented equalization processing on the subjective scores to avoid sample bias in the training set; S23. Image conversion of data: Generate frequency-domain images including acceleration images and physiological images by window extraction and time-frequency conversion of vehicle dynamic performance data and physiological index data of the vehicle occupants. The frequency-domain images include acceleration images and physiological images; Construct a structured driving behavior vector from driving behavior parameters; The subjective score is used as the label of the training target value for the subsequent convolutional neural network-based evaluation model of the ride comfort of autonomous vehicles; The acceleration image, physiological image, and structured driving behavior vector are used as the inputs of the convolutional neural network-based evaluation model of the ride comfort of autonomous vehicles; Use the acceleration image, physiological image, and structured driving behavior vector to construct a training set and bind the subjective score as the label. Each sample consists of {acceleration image, physiological image, structured driving behavior vector, subjective score label}; After screening, perform image conversion on the sample data retained in S22. Convert the dynamic accelerations of the X-axis, Y-axis, and Z-axis of the vehicle dynamic performance data into Mel spectrograms through short-time Fourier transform and Mel filtering. Map the Mel spectrograms of the X-axis, Y-axis, and Z-axis to the R, G, and B channels respectively and combine them into a three-channel color image to form an acceleration image; Convert the electrocardiogram signal and skin conductance signal of the physiological index data of the vehicle occupants into spectrograms using CWT and Mel transform respectively to form a physiological image; The driving behavior parameters are not converted into images, but a structured driving behavior vector is constructed; The sizes of the acceleration image and physiological image are unified to 224×224×3. All inputs are bound to the subjective score label to form the final training samples: ； Among them, is the acceleration image, is the physiological image, is the structured driving behavior vector, is the subjective scoring label value; S3. Construct a convolutional neural network-based evaluation model for the ride comfort of autonomous vehicles: Input the three-channel information of vehicle dynamic performance data, physiological index data of vehicle occupants, and driving behavior parameters into the constructed convolutional neural network-based evaluation model for the ride comfort of autonomous vehicles; S4. Output the ride comfort evaluation result.

2. The intelligent driving vehicle ride comfort evaluation method based on multi-modal data according to claim 1, characterized in that In step S12, Collect vehicle dynamic performance data, which is obtained by a three-axis acceleration sensor installed under the co-pilot seat; The three-axis acceleration sensor collects the dynamic accelerations of the vehicle along the forward direction X-axis, the lateral Y-axis perpendicular to the forward direction on the horizontal plane, and the vertical Z-axis perpendicular to the forward direction on the vertical plane; Collect driving behavior parameters. The computer connects to the vehicle through the interface device interface and reads through the bus; The driving behavior parameters include vehicle speed, engine speed, throttle opening, accelerator pedal opening, and lane change, acceleration, and overtaking driving operations manually recorded and synchronously marked on the time stamp. The duration of the operation is calculated by the start and end time stamps of the action; Collect the physiological indexes of the occupant's body. The skin conductance acquisition electrodes are fixed on the surfaces of the thenar muscles of both palms to measure the skin conductance signal of the occupant's body and are connected to the skin conductance meter through a wireless transmission device; the electrocardiogram acquisition electrodes are arranged at the chest, left arm, and right arm positions to measure the electrocardiogram signal of the occupant's body and are connected to the electrocardiograph through a wireless transmission device. The skin conductance meter and the electrocardiograph are connected to the computer through their microcomputer interfaces, and the collected skin conductance signals and electrocardiogram signals are transmitted to the computer; among them, the electrocardiogram acquisition electrodes are three-lead patch electrodes for electrocardiogram acquisition.

3. The intelligent driving vehicle ride comfort evaluation method based on multi-modal data according to claim 2, wherein In step S21, The preprocessing method is as follows: systematically preprocess the original data collected in step S1. First, perform interval filtering on the subjective scores, and retain the subjective scores between 3 and 8 points. Second, judge the physiological response amplitude. For the electrocardiogram signal, calculate the standard deviation of heart rate variability SDNN within the window before and after the driving operations of lane change, acceleration, and overtaking recorded manually respectively: , where is the total number of interphases, is the i-th interphase, is the average value of all interphases within this window; where the interphase is the time interval between one wave and the next wave in an electrocardiogram within the window; If the standard deviation of heart rate variability , it is regarded as an invalid response, and the corresponding data is excluded; The galvanic skin signal is for calculating the change rate of the galvanic skin amplitude before the lane change, acceleration, and overtaking driving operations recorded manually and after the window of the driving operation: First, calculate the average skin conductance values within the window before and after driving operations respectively: before the driving operation and within the window after the driving operation: , where is the number of sampling points within the window, is the skin conductance value at the Then the skin conductance amplitude change rate is: , where is the average skin conductance value of the window after the operation, and is the average skin conductance value of the window before the operation; ​ If , it is regarded as an invalid reaction and the corresponding data is excluded.

4. The intelligent driving vehicle ride comfort evaluation method based on multi-modal data according to claim 3, wherein In step S22, The method for subjective score consistency verification is as follows: Use Kendall's coefficient of concordance to analyze the scoring coordination of multiple appraisers for the same driving operation. First, calculate the total sum of squared rank differences : , where is the sum of the ranks of the th sample, that is, the sum of the scores given by all evaluators to this sample, is the average of the rank sums of all samples; Then Kendall's coefficient of concordance is: , where is the number of evaluators, is the number of samples evaluated by each evaluator, is the total sum of squared rank differences; If , it is considered that there are significant differences in the scores of the same driving operation by multiple evaluators, and the corresponding data are excluded; Use Cronbach's α internal consistency coefficient to measure the consistency of a single evaluator's scores for multiple identical driving operations: , where is the number of items, i.e., the number of identical driving operations scored by a single appraiser, is the -th variance of the scores of identical driving operations, is the overall variance of the scores of the appraiser in all identical driving operations; If <0.75, it is considered that there is a large difference in the scores before and after a single evaluator's multiple identical driving operations, and the corresponding data is excluded; Subjective scoring Segmentation: Use segmented random downsampling to balance the number of samples. Divide the subjective scoring into several sub-intervals. Set the number of samples in the segment with the fewest samples as the upper limit, and randomly downsample the remaining segments to this number to balance the samples in the low, medium, and high scoring segments.

5. The intelligent driving vehicle ride comfort evaluation method based on multi-modal data according to claim 4, characterized in that, In step S3, The evaluation model for the ride comfort of an autonomous vehicle based on a convolutional neural network is: Construct a three-way neural network system: The convolutional neural network 1 branch extracts the acceleration image features: ; Two branches of convolutional neural network extract physiological image features: ; The multi-layer perceptron branch processes the driving behavior vector: ; After the output features of the three-way neural network system are fused, a non-linear mapping is performed: ； Among them, ; ； Among them, is the weight matrix of the first fully connected layer; is the bias term of the first layer; is the rectified linear unit activation function: ReLU(x) = max(0, x); is the output weight of the second layer; is the bias term of the output layer; is the predicted value of the comfort score output by the comfort evaluation model of the autonomous vehicle; Train the evaluation model for the ride comfort of an autonomous vehicle with a mean square error loss function: ； Among them, is the total number of samples in the training set, is the true subjective score label value of the i-th sample, is the score value predicted by the model for the i-th sample, is the loss of the model training objective, namely the mean square error loss.

Images