A highly integrated real vehicle and virtual environment man-machine co-driving online evaluation system and method

Abstract

The application discloses a kind of real vehicle and virtual environment highly fused man-machine co-pilot online evaluation system and method, and online evaluation system includes simulation scene digital twin module based on real road, intelligent driving simulation module, man-machine co-pilot control module, real vehicle module and online evaluation and model parameter self-optimization module, and the evaluation method is: first, simulation scene test based on digital twin and acceleration test;Second, driving person and intelligent driving system decision synchronism test;Third, driving person driving behavior consistency test;Beneficial effects: through simulation scene digital twin technology based on real road and acceleration test technology, the authenticity and test efficiency of man-machine co-pilot test are improved, the verification and calibration of driving person and intelligent driving system decision signal synchronism are realized through signal time synchronization technology, avoid man-machine conflict, and the algorithm performance of man-machine collaborative control strategy is improved.

Description

Technical Field

[0001] This invention relates to an online evaluation system and method for human-machine co-driving, specifically to an online evaluation system and method for human-machine co-driving that highly integrates real vehicle and virtual environment. Background Technology

[0002] Currently, due to the technological limitations and social dilemmas of fully autonomous driving, coupled with low public acceptance, human-machine co-driving will become a long-term form of intelligent driving in the future. Before being put on the road, human-machine co-driving systems must undergo thorough testing. The main problems in human-machine co-driving testing include the following:

[0003] The authenticity and efficiency of the test scenarios were not considered. Currently, human-machine co-driving tests mainly rely on scenario-based simulation tests, where the types and content of test scenarios are artificially set, lacking authenticity. Recent research has integrated digital twin technology into autonomous driving tests, such as Chinese patents 202211723189.8, 202210917996.7, and 202210466439.8. By collecting and replaying real road data, these tests serve as test scenarios for autonomous driving, improving realism. However, this method has drawbacks: firstly, directly generating test scenarios from real road data suffers from low efficiency due to the scarcity of challenging scenarios; secondly, there has been a lack of research applying this real-road-scenario-based digital twin technology to human-machine co-driving tests. While Chinese patent 202110090864.7 attempts to apply digital twin technology to human-machine co-driving tests, it does not involve creating digital twins of real road scenarios, resulting in a lack of authenticity in the simulation scenarios.

[0004] No relevant research has focused on testing the synchronization of decision-making between drivers and intelligent driving systems. Current research on human-machine co-driving tests mainly concentrates on testing the decision-making algorithms of intelligent driving systems and human-machine collaborative control strategies based on driver driving states, failing to consider the synchronization of decision-making between drivers and intelligent driving systems. Drivers and intelligent driving systems inherently exhibit asynchronous decision-making, stemming from their different decision-making and control mechanisms. Assuming both use control signal output as the starting point for decision-making, when facing the same driving situation, the intelligent driving system perceives the surrounding environment through various sensors and directly outputs control commands through intelligent decision-making and control algorithms, making decisions rapidly. However, due to reaction time and mechanical gaps when manipulating the steering wheel and pedals, the driver's decision-making may lag behind that of the intelligent driving system. Human-machine co-driving achieves collaborative control by assigning driving weight factors to both the driver and the intelligent driving system simultaneously. Therefore, the synchronization of decision-making between drivers and intelligent systems is crucial for human-machine co-driving testing and directly affects the performance of human-machine collaborative control strategies. In online testing of human-machine co-driving, the synchronization of the driver's and intelligent system's decisions should be verified in real time, and adjustment methods should be designed to synchronize the two when there is a lack of synchronization.

[0005] The current research on human-machine co-driving systems fails to consider changes in driver aggression during the driving process. Most current performance testing studies focus on safety, while a few studies on comfort consider the impact of driver habits, but all assume the same driver maintains the same level of aggression throughout the process. However, in real life, a driver's aggression can vary due to various circumstances. For example, when approaching a traffic light, a driver might suddenly increase aggression to avoid waiting; an urgent task requiring a quick arrival at a destination might also increase aggression; and in-car music or making phone calls while driving can also alter a driver's aggression. Failing to consider these changes in driver aggression reduces the driver's comfort experience, consequently decreasing their acceptance and trust in human-machine co-driving systems. Summary of the Invention

[0006] The purpose of this invention is to solve many problems existing in the current human-machine co-driving test process, and to provide a human-machine co-driving online evaluation system and method that highly integrates real vehicle and virtual environment.

[0007] The human-machine co-driving online evaluation system that highly integrates real vehicles and virtual environments provided by this invention includes a digital twin module based on a real road simulation scenario, an intelligent driving simulation module, a human-machine co-driving control module, a real vehicle module, and an online evaluation and model parameter self-optimization module. These modules are arranged in parallel, and their structural composition is as follows:

[0008] The digital twin module for simulated scenarios based on real roads collects, processes, and classifies real road data. It uses digital twin technology to generate simulated scenarios for human-machine co-driving tests and selects key scenarios based on safety and comfort testing for acceleration testing. This module consists of four sub-modules: a real road data collection and processing module, a simulated scenario space construction module for human-machine co-driving tests, a key scenario selection module based on safety and comfort, and a simulated scenario information injection module. The simulated scenario space construction module discretizes scenario elements with high importance weights to construct the simulated scenario space. The key scenario selection module establishes key criteria based on safety and comfort functions, uses a global optimization algorithm to select key scenarios in the scenario space, and injects the generated key scenarios into the online testing and evaluation module for human-machine co-driving tests via the simulated scenario information injection module for safety and comfort acceleration testing.

[0009] The intelligent driving simulation module provides a simulated driving environment for drivers and collects their status and operational actions. The intelligent driving simulation module consists of four sub-modules: a simulation scene rendering and display module, a driving somatosensory simulation module, a driver module, and a driving simulation cabin module. The driver status information output by the driver module and the driver decision signals output by the driving simulation cabin module are input into the online evaluation and model parameter self-optimization module for evaluating the synchronization of driver and intelligent driving system decisions and the consistency of driver driving behavior.

[0010] The human-machine co-driving control module realizes human-machine co-driving by formulating a collaborative control strategy between the intelligent driving system and the driver. The human-machine co-driving control module consists of two sub-modules: the intelligent driving system control module and the human-machine collaborative control module. The intelligent driving system decision signals output by the intelligent driving system are input into the online evaluation and model parameter self-optimization module for evaluating the synchronization of driver and intelligent driving system decisions.

[0011] The real vehicle module involves driving a real vehicle on the test track, executing a human-machine collaborative control strategy, and collecting the motion status of the real vehicle. The real vehicle module consists of a vehicle module, a real vehicle status acquisition module, and a real vehicle status injection module. The real vehicle status information collected by the real vehicle status acquisition module will be injected into the online evaluation and model parameter self-optimization module through the real vehicle status injection module for safety and comfort acceleration testing.

[0012] The online evaluation and model parameter self-optimization module verifies the safety and comfort of the human-machine co-driving system through scenario-based accelerated testing; verifies the synchronization of driver and intelligent driving system decision signal outputs through driver and intelligent driving system decision synchronization testing, and can perform synchronization calibration in asynchronous situations; and verifies changes in driver aggression during driving through driver behavior consistency testing, thereby optimizing the model parameters of the intelligent driving system control module and improving driver comfort. The online evaluation and model parameter self-optimization module consists of three sub-modules: an accelerated testing-based safety and comfort evaluation module, a driver and intelligent driving system decision synchronization evaluation module, and a driver behavior consistency evaluation module.

[0013] The present invention provides a human-machine co-driving online evaluation method that highly integrates real vehicle and virtual environment, the method comprising the following steps:

[0014] The first step, simulation scenario testing based on digital twins and accelerated testing, is as follows:

[0015] Step 1: Real-world road data collection and processing. Various sensors are installed on real vehicles to build a real-world road data collection platform. To ensure the representativeness of the collected scene data, large-scale data collection is carried out under traffic conditions of different road types, weather types, and lighting types. After the data collection is completed, the data is first preprocessed to remove data segments with errors and missing data. Then, the scene data collected by each sensor is synchronized in time and space, and the data is formatted to meet the data structure required for scene construction. Finally, the data is classified according to the scene type.

[0016] Step 2: Constructing a simulation scenario space for human-machine co-driving testing. Based on the characteristics of human-machine co-driving and the performance to be tested, design the type of simulation scenario, list the scenario elements, determine the importance weight values ​​of the scenario elements based on the group decision fuzzy network hierarchical analysis method F-ANP, select scenario elements with high importance weight values ​​for discretization, and construct the simulation scenario space.

[0017] In the construction of the simulation scene space, the scene elements are first listed. Static scene elements include the number of lanes, lane width, lane curvature, weather and lighting. Dynamic scene elements include the initial position, initial speed, driving speed and driving acceleration of the main vehicle and the target vehicle.

[0018] Then, the importance weights of scene elements are determined using the group decision-making fuzzy network hierarchical analysis method (F-ANP). The relationships between scene elements are analyzed, and the scene elements are organized and hierarchically structured to construct an ordered, progressively layered structural model. A judgment matrix is ​​constructed based on the relative importance ratios between scene elements. The relative ratios between scene elements are determined by the influence transmission model, which reflects the importance of scene elements to the human-machine co-driving system. The influence transmission model sets three assumptions, as follows:

[0019] 1) The impact of scene elements on the human-machine co-driving system will be passed down level by level with the human-machine co-driving system, and this impact will not decrease with the passing down of levels;

[0020] 2) Different types of scene elements have the same impact on the human-machine co-driving system;

[0021] 3) The degree of influence of scene elements on the human-machine co-driving system is represented by the number of times scene elements are transmitted between the levels of the human-machine co-driving system. The more times they are transmitted, the greater the degree of influence, and the degree of influence is linearly related to the number of transmissions.

[0022] The number of times scene elements in the influence transmission model are transmitted through each level of the human-machine co-driving system is calculated using the following formula:

[0023] (1);

[0024] In the formula, P(n) represents the number of times the influence of a certain scene element is transmitted through each level of the human-machine co-driving system, n represents the number of element attributes in that scene element, and E i This represents the number of times the influence of the i-th element's attribute in the scene is propagated across all levels of the human-machine co-driving system.

[0025] Find the largest eigenvalue of the judgment matrix and its corresponding eigenvector, and normalize the eigenvector corresponding to the largest eigenvalue of the judgment matrix to obtain the importance weight of the scene element.

[0026] To ensure the rationality of the importance weighting index, the matrix must undergo a consistency check. The formula for calculating the consistency check coefficient CR is as follows:

[0027] (2);

[0028] In the formula, RI is the standard value of the hierarchical overall ranking average random consistency index, and CI is the hierarchical overall ranking consistency index, the relevant calculation formula of which is:

[0029] (3);

[0030] (4);

[0031] (5);

[0032] In the formula, λ max Let n be the order of the judgment matrix, A be the judgment matrix, and ω be the largest eigenvalue. * To determine the eigenvectors of the judgment matrix, if CR < 0.1, it indicates that the judgment matrix has good consistency and meets the consistency requirements.

[0033] After obtaining the importance weights of the scene elements, the scene elements with high importance weights are selected according to the dimensions of the scene space to be constructed, and the discrete range and discrete length are determined, thus constructing the simulation scene space.

[0034] Step 3: Screening of key scenarios based on safety and comfort: Selecting scenario challenge and scenario occurrence probability as the screening principles for key scenarios, constructing a criticality function, and using optimization algorithms to screen key scenarios that meet the criticality threshold, thereby realizing digital twins of simulation scenarios based on real roads and accelerating testing.

[0035] In security testing, a key security function is constructed using the scenario's hazard level and the probability of its occurrence, as shown in the following formula:

[0036] (6);

[0037] In the formula, V s (x) is the security critical function, R s (x) represents the scene's danger level, P s (x) represents the probability of the scenario occurring, R0 s (x) By enhancing the collision time ETTC characterization, ETTC is calculated as follows:

[0038] (7);

[0039] In the formula, R represents the longitudinal distance between the main vehicle and the target vehicle, ∆v represents the relative velocity between the main vehicle and the target vehicle, and ∆a represents the relative acceleration between the main vehicle and the target vehicle;

[0040] In comfort testing, a key comfort function is constructed using scenario comfort and scenario occurrence probability, as shown in the following equation:

[0041] (8);

[0042] In the formula, V c (x) is the key comfort function, C c (x) represents scene comfort, P c (x) represents the probability of the scenario occurring, C c (x) is calculated using the jerk and the root mean square value of acceleration, as shown in the following formula:

[0043] (9);

[0044] In the formula, J lon (x) represents the longitudinal abrupt change, a lon (x) Root mean square value of longitudinal acceleration, α c With β c α is the weighted value of the longitudinal jerk and the root mean square value of the longitudinal acceleration. c With β c Take 0.5 for all;

[0045] The probability of scenario occurrence P in the safety critical function and the comfort critical function s (x) and P c (x) By using the convex combination method, the scene captured from the real road is transformed into a scene in the scene space, thus obtaining: A1(R) in the 3D scene space. a1 , v a1 , a a1 ), A2(R a2 , v a2 , a a2 ), A3(R a3 , v a3 , a a3 ), A4(R a4 , v a4 , a a4 ), A5(R a5 , v a5 , a a5 ), A6(R a6 , v a6 , a a6 ), A7(R) a7 , v a7 , a a7 ), A8(R a8 , v a8 , a a8 Let B be eight uniformly discrete scenes in the scene space, and let B be a scene recorded from real road data, and B(R) = ... b , v b , a b Inside the cube formed by A1, A2, A3, A4, A5, A6, A7, and A8, L 11 L is the Euclidean distance from B to the plane containing A2, A4, A6, and A8. 12 L is the Euclidean distance from B to the plane containing A1, A3, A5, and A7. 21 L is the Euclidean distance from B to the plane containing A3, A4, A7, and A8. 22 L is the Euclidean distance from B to the plane containing A1, A2, A5, and A6. 31 L is the Euclidean distance from B to the plane containing A5, A6, A7, and A8. 32 Let B be the Euclidean distance from B to the planes containing A1, A2, A3, and A4. Then, the real scene B can be represented by scenes A1, A2, A3, A4, A5, A6, A7, and A8 in scene space using the convex combination method as follows:

[0046] (10)

[0047] In the formula, ω1, ω2, ω3, ω4, ω5, ω6, ω7, and ω8 are the weight coefficients assigned to A1, A2, A3, A4, A5, A6, A7, and A8, respectively, and their calculation formula is as follows:

[0048] (10);

[0049] (11);

[0050] (12);

[0051] (13);

[0052] (14);

[0053] (15);

[0054] (16);

[0055] (17);

[0056] After transforming real road scene data into scenes in scene space, the probability of occurrence of simulated scenes in scene space is obtained. Then, the safety key function value and comfort key function value of the simulated scene are obtained. Safety key threshold and comfort key threshold are set respectively. Using a global optimization algorithm, key scenes for human-machine co-driving safety test and comfort test are obtained.

[0057] In key scenarios, the safety and comfort of human-machine co-driving were tested by sampling scenarios. Safety was evaluated by the accident rate, and comfort was evaluated by the driver's expected comfort level. The driver's expected comfort level was obtained in advance by recording the vehicle's state data when the driver was driving alone.

[0058] The second step is to test the synchronization of driver and intelligent driving system decisions, and the specific process is as follows:

[0059] Step 1: Test the synchronization of driver and intelligent driving system decisions. Sample the driver's decision signals and the intelligent driving system's decision signals at the same time. The decision signals include steering wheel angle signals, accelerator pedal signals and brake pedal signals. Determine whether the two decisions are synchronized by comparing the deviation between the driver's and intelligent driving system's decision signals.

[0060] The sampling frequency f and sampling time T are set to simultaneously collect decision signals from both the driver and the intelligent driving system. The driver's decision signal is as follows:

[0061] ;

[0062] In the formula, p d-sw,tn p is the driver's steering wheel angle signal. d-ap,tn p is the driver's accelerator pedal signal. d-bp,tn For the driver's brake pedal signal, n=fT;

[0063] The decision signals of an intelligent driving system are:

[0064] ;

[0065] In the formula, p a-sw,tn For the steering wheel angle signal of the intelligent driving system, p a-ap,tn For the accelerator pedal signal of the intelligent driving system, p a-bp,tn For the brake pedal signal of the intelligent driving system;

[0066] Calculate the deviation L of the steering wheel angle signal between the driver and the intelligent driving system during the sampling time. sw As shown in the following formula:

[0067] ;

[0068] Calculate the deviation L of the accelerator pedal signal between the driver and the intelligent driving system during the sampling time. ap As shown in the following formula:

[0069] ;

[0070] Calculate the deviation L of the brake pedal signal between the driver and the intelligent driving system during the sampling time. bp As shown in the following formula:

[0071] ;

[0072] Steering wheel angle signal deviation setting threshold ε sw The threshold for accelerator pedal signal deviation is set to ε. ap The brake pedal signal deviation setting threshold is ε. bp When the deviations of all three decision signals are less than the deviation threshold, it is considered that the driver and the intelligent driving system have good decision synchronization; when the deviation of one or more decision signals is greater than the deviation threshold, it is considered that the driver and the intelligent driving system have poor decision synchronization and calibration is required.

[0073] Step 2: Calibration of driver and intelligent driving system decision synchronization. Curve fitting is used to calibrate the decision signals of the driver and intelligent driving system. Spline fitting based on least squares method is performed with the decision signal of intelligent driving system as the benchmark to minimize the cost function of driver decision signal and intelligent driving system decision curve, thereby achieving the calibration of driver and intelligent driving system decision synchronization.

[0074] A cubic spline interpolation function s(t) is constructed from the steering wheel angle signal. The spline function fitting based on least squares is performed in the spline function space S. k Inside, find the value for p a-sw (t) Best approximation of the norm s * (t), as shown in the following formula:

[0075] ;

[0076] s * (t) represents the fitted curve of the steering wheel angle signal of the intelligent driving system. To facilitate calibration, a calibration curve s for the steering wheel angle signal of the intelligent driving system is constructed. * (t+b) allows the fitting curve of the steering wheel angle signal of the intelligent driving system to be shifted along the time axis, where b is the shift length, constructing a curve that modulates the sampled driver steering wheel angle signal with s. * The cost function J(b) for (t+b) is shown in the following equation:

[0077] ;

[0078] The gradient descent method is used to minimize the cost function J(b) to achieve synchronization calibration of driver and intelligent driving system decision-making.

[0079] The third step is a test of the consistency of the driver's driving behavior, the specific process of which is as follows:

[0080] Step 1: Identify driver aggression based on CNN-BP neural network. Collect the driving state and driving operation quantities of drivers with known driving aggression, and obtain the driver aggression model through CNN-BP neural network training.

[0081] To analyze the consistency of driver behavior in real time during driving, a driver aggression model is established in advance before conducting human-machine co-driving tests, and the driving aggression level D is defined. agg The discrete levels are divided into five categories, as shown in the following formula:

[0082] (1);

[0083] Ten drivers with more than five years of driving experience were selected for each level of driving aggression. Their driving state and driving manipulation quantities under normal driving conditions were collected. Among them, the driving state quantity X S The composition is shown in the following formula:

[0084] (2);

[0085] In the formula, x he For head movement signals, x ey For eye movement signals, x mi For electrocardiogram (ECG) signals, x sk For electrodermal signals, x mu For electromyography (EMG) signals, x br EEG signal, driving control quantity X C The composition is shown in the following formula:

[0086] (3);

[0087] In the formula, x sw For steering wheel angle signal, x ap For accelerator pedal position signal, x bp This is the brake pedal position signal;

[0088] X S With driving control amount X C The input matrix X, which serves as the input layer of the neural network, is shown in the following equation:

[0089] (4);

[0090] Based on the BP neural network, convolutional and pooling layers are added to optimize the feature values. The CNN-BP neural network is used to train the driver's driving aggression model, and the feature values ​​are normalized as shown in the following formula:

[0091] (5);

[0092] In the formula, x i Let x be the eigenvalues ​​of each type in the input matrix. ni These are the normalized eigenvalues;

[0093] The normalized matrix X n The input is fed into a convolutional layer for convolution. The convolutional layer extracts and optimizes the normalized feature values ​​using the parameters in the convolution kernel, i.e., the weights of the locally connected neural network. The convolutional layer sets two 1×1 convolution kernels to X. n Each data point is concatenated, and two 2×2 convolutional kernels are applied to X. n Partially connect the 2×2 local components, and then perform two 3×3 convolution kernels on X. n By performing a fully connected layer, the input feature matrix can be analyzed from multiple perspectives and ranges, resulting in the convolutional matrix M, as shown in the following equation:

[0094] (6);

[0095] In the formula, F is the kernel size, P is the padding size, and S is the stride. After passing through a convolutional layer with S=1, two 3×3 feature matrices, two 2×2 feature matrices, and two eigenvalues ​​are obtained.

[0096] The matrix M output by the convolutional layer is input into the pooling layer for feature dimensionality reduction. The pooling layer consists of two 2×2 pooling kernels and two 3×3 pooling kernels, and finally obtains six optimized feature values, which are input into the next layer of the BP neural network. The BP neural network layer is set with twenty neurons, and the output layer has five neurons, which correspond to five driving aggression levels.

[0097] The CNN-BP neural network is trained using the backpropagation algorithm for both the convolutional layers and the BP neural network. The training method for the BP neural network model is as follows:

[0098] First, the derivative of the connection weights between the output layer and the hidden layer is calculated, as shown in the following equation:

[0099] ;

[0100] In the formula, E is the error function, and θ 2ij y represents the connection weights between the input layer and the hidden layer. k For output layer output, u2k The activation values ​​between the output layer and the hidden layer;

[0101] The derivative of the connection weights between the input layer and the hidden layer is shown in the following equation:

[0102] ;

[0103] In the formula, θ 1ij q represents the connection weights between the input layer and the hidden layer, and q is the number of neurons in the output layer.

[0104] The adjustment value of the connection weights between the input layer and the hidden layer is calculated using the following formula:

[0105] ;

[0106] In the formula, z j For the hidden layer, u 1k The activation values ​​between the input layer and the hidden layer;

[0107] The gradient descent algorithm is used to train the CNN-BP neural network until the error function E converges, thus obtaining the driver's driving aggression model.

[0108] Step 2: Real-time test of driver aggression. In the intelligent driving simulator, head movement and eye movement signals of the driver are collected in real time through head movement and eye movement devices worn by the driver. ECG, skin conductance, electromyography, and electroencephalography signals of the driver are collected in real time through ECG, skin conductance, electromyography, and electroencephalography devices worn by the driver are collected in real time. Steering wheel angle signal, accelerator pedal position signal, and brake pedal position signal are collected in real time. The above signals are used as input to the driver aggression model obtained in Step 1 to obtain the driver's real-time aggression.

[0109] Step 3: Optimize the parameters of the human-machine co-driving model based on driving aggression. Adjust the decision model parameters of the intelligent driving system in the human-machine co-driving system according to the driver's real-time driving aggression to make the driving aggression of the intelligent driving system match the driving aggression of the driver, thereby improving the driver's comfort. At the same time, set a threshold for the aggression of the intelligent driving system to prevent situations where both the driver's and the intelligent driving system's driving aggression are too high, thus affecting driving safety.

[0110] The beneficial effects of this invention are:

[0111] This invention provides an online human-machine co-driving evaluation system and method that highly integrates real vehicle and virtual environments, with the following specific benefits:

[0112] 1. This invention provides a method for constructing a simulation scene space for human-machine co-driving testing. It designs test scene types based on the characteristics of human-machine co-driving and the performance to be tested, determines the importance weight values ​​of scene elements based on the group decision fuzzy network hierarchical analysis method (F-ANP), selects scene elements with high importance weight values ​​for discretization, and realizes the construction of simulation scene space, laying the foundation for generating test scenes.

[0113] 2. This invention provides a key scenario screening method based on safety and comfort. By constructing key safety functions and key comfort functions, key scenario screening criteria are established. A global optimization algorithm is used to screen key scenarios in the scenario space, providing a source of test scenarios for accelerating safety and comfort testing.

[0114] 3. This invention provides a method for testing the synchronization of driver and intelligent driving system decision-making. By sampling and analyzing the decision signals of the driver and intelligent driving system, the time synchronization of driver and intelligent driving system decision-making is verified.

[0115] 4. This invention provides a method for calibrating the synchronization of driver and intelligent driving system decision-making. By using a spline fitting method based on the least squares method to calibrate the decision signals of the driver and intelligent driving system, it avoids human-machine conflict caused by the time asynchrony of driver and intelligent driving system decision-making and improves the algorithm performance of human-machine cooperative control strategy.

[0116] 5. This invention provides a method for identifying driver aggression based on neural networks. By constructing a driver aggression model through a CNN-BP neural network, a real-time identification of driver aggression can be achieved, providing a basis for improving driver comfort.

[0117] 6. This invention provides a method for optimizing the parameters of a human-machine co-driving model based on driving aggression. By optimizing the model parameters of the intelligent driving system control module through the driver's real-time driving aggression, the driving style of the intelligent driving system is consistent with that of the driver, thereby improving the driver's driving comfort.

[0118] In summary, the human-machine co-driving online evaluation system and method that highly integrates real vehicle and virtual environment provided by this invention improves the authenticity and efficiency of human-machine co-driving tests through digital twin technology and accelerated testing technology based on simulated scenarios of real roads. It realizes the verification and calibration of the synchronization of decision signals between the driver and the intelligent driving system through signal time synchronization technology, avoids human-machine conflicts, and improves the algorithm performance of human-machine collaborative control strategies. It improves the driving comfort of the human-machine co-driving system through driver aggression identification and model parameter optimization technology based on neural networks. Attached Figure Description

[0119] Figure 1 This is a structural block diagram of the human-machine co-driving online evaluation system described in this invention.

[0120] Figure 2 This is a schematic diagram illustrating the overall steps of the online human-machine co-driving evaluation method described in this invention.

[0121] Figure 3 This is a schematic diagram of the technical route of the human-machine co-driving online evaluation method described in this invention.

[0122] Figure 4 This is a schematic diagram of the influence transfer model architecture for step two of the first step described in this invention.

[0123] Figure 5 This is a schematic diagram of the three-dimensional scene space convex combination method in step three of the first step of the present invention.

[0124] Figure 6 This is a schematic diagram of the CNN-BP neural network architecture in step one of the third step of the present invention. Detailed Implementation

[0125] Please see Figures 1 to 6 As shown:

[0126] The human-machine co-driving online evaluation system that highly integrates real vehicles and virtual environments provided by this invention includes a digital twin module based on a real road simulation scenario, an intelligent driving simulation module, a human-machine co-driving control module, a real vehicle module, and an online evaluation and model parameter self-optimization module. These modules are arranged in parallel, and their structural composition is as follows:

[0127] The digital twin module for simulated scenarios based on real roads collects, processes, and classifies real road data. It uses digital twin technology to generate simulated scenarios for human-machine co-driving tests and selects key scenarios based on safety and comfort testing for acceleration testing. This module consists of four sub-modules: a real road data collection and processing module, a simulated scenario space construction module for human-machine co-driving tests, a key scenario selection module based on safety and comfort, and a simulated scenario information injection module. The simulated scenario space construction module discretizes scenario elements with high importance weights to construct the simulated scenario space. The key scenario selection module establishes key criteria based on safety and comfort functions, uses a global optimization algorithm to select key scenarios in the scenario space, and injects the generated key scenarios into the online testing and evaluation module for human-machine co-driving tests via the simulated scenario information injection module for safety and comfort acceleration testing.

[0128] The intelligent driving simulation module provides a simulated driving environment for drivers and collects their status and operational actions. The intelligent driving simulation module consists of four sub-modules: a simulation scene rendering and display module, a driving somatosensory simulation module, a driver module, and a driving simulation cabin module. The driver status information output by the driver module and the driver decision signals output by the driving simulation cabin module are input into the online evaluation and model parameter self-optimization module for evaluating the synchronization of driver and intelligent driving system decisions and the consistency of driver driving behavior.

[0129] The human-machine co-driving control module realizes human-machine co-driving by formulating a collaborative control strategy between the intelligent driving system and the driver. The human-machine co-driving control module consists of two sub-modules: the intelligent driving system control module and the human-machine collaborative control module. The intelligent driving system decision signals output by the intelligent driving system are input into the online evaluation and model parameter self-optimization module for evaluating the synchronization of driver and intelligent driving system decisions.

[0130] The real vehicle module involves driving a real vehicle on the test track, executing a human-machine collaborative control strategy, and collecting the motion status of the real vehicle. The real vehicle module consists of a vehicle module, a real vehicle status acquisition module, and a real vehicle status injection module. The real vehicle status information collected by the real vehicle status acquisition module will be injected into the online evaluation and model parameter self-optimization module through the real vehicle status injection module for safety and comfort acceleration testing.

[0131] The online evaluation and model parameter self-optimization module verifies the safety and comfort of the human-machine co-driving system through scenario-based accelerated testing; verifies the synchronization of driver and intelligent driving system decision signal outputs through driver and intelligent driving system decision synchronization testing, and can perform synchronization calibration in asynchronous situations; and verifies changes in driver aggression during driving through driver behavior consistency testing, thereby optimizing the model parameters of the intelligent driving system control module and improving driver comfort. The online evaluation and model parameter self-optimization module consists of three sub-modules: an accelerated testing-based safety and comfort evaluation module, a driver and intelligent driving system decision synchronization evaluation module, and a driver behavior consistency evaluation module.

[0132] The human-machine co-driving online evaluation method that highly integrates real vehicle and virtual environment provided by this invention specifically includes the following steps:

[0133] Step 1: Simulation scenario testing based on digital twins and accelerated testing;

[0134] The second step is to test the synchronization of driver and intelligent driving system decisions.

[0135] The third step is a test of the consistency of the driver's driving behavior.

[0136] The simulation scenario testing process based on digital twins and accelerated testing in the first step is as follows:

[0137] Step 1: Real-world road data acquisition and processing. Various sensors are installed on actual vehicles to build a real-world road data acquisition platform. To ensure the representativeness of the collected scene data, large-scale data collection is conducted under different road types, weather conditions, and lighting conditions. After data acquisition, the data is first preprocessed, removing erroneous and missing data segments. Then, the scene data collected by each sensor is synchronized in time and space, and the data is formatted to meet the data structure requirements for scene construction. Finally, the data is classified according to scene type.

[0138] Step 2: Construction of the simulation scenario space for human-machine co-driving testing. Based on the characteristics of human-machine co-driving and the performance to be tested, the types of simulation scenarios are designed, scenario elements are listed, and the importance weight values ​​of scenario elements are determined based on the group decision fuzzy network hierarchical analysis method (F-ANP). Scenario elements with high importance weight values ​​are selected for discretization to construct the simulation scenario space.

[0139] Currently, autonomous driving tests primarily focus on safety. However, in human-machine co-driving, the driver's experience is also crucial due to the addition of a human driver. Therefore, this patent selects safety and comfort as the performance aspects to be tested in human-machine co-driving. For the performance aspects to be tested in human-machine co-driving, considering the attributes of each scenario type and their frequency of occurrence on real roads, following, lane-changing, and overtaking scenarios are selected as simulation test scenarios. These three scenario types have a high frequency of occurrence on real roads and are prone to inconsistencies between human and machine decisions. The following scenario is mainly used for longitudinal safety and comfort testing in human-machine co-driving; the lane-changing scenario is mainly used for lateral safety and comfort testing; and the overtaking scenario is used for safety and comfort testing of the lateral and longitudinal coupling in human-machine co-driving.

[0140] In the construction of the simulation scene space, the scene elements are first listed. Static scene elements include the number of lanes, lane width, lane curvature, weather and lighting, while dynamic scene elements include the initial position, initial speed, driving speed and driving acceleration of the main vehicle and the target vehicle.

[0141] Then, the importance weights of scene elements are determined using the group-based fuzzy network hierarchical analysis method (F-ANP). The relationships between scene elements are analyzed, and the elements are organized and hierarchically structured to construct an ordered, progressively layered structural model. A judgment matrix is ​​constructed based on the relative importance ratios between scene elements. These relative ratios are determined by the influence transmission model, which reflects the importance of scene elements to the human-machine co-driving system. The influence transmission model is attached. Figure 4 As shown, there are three assumptions about the influence transmission model:

[0142] 1) The impact of scene elements on the human-machine co-driving system will be passed down level by level with the human-machine co-driving system, and this impact will not decrease with the passing down of levels;

[0143] 2) Different types of scene elements have the same impact on the human-machine co-driving system;

[0144] 3) The degree of influence of scene elements on the human-machine co-driving system can be represented by the number of times scene elements are transmitted between the levels of the human-machine co-driving system. The more times they are transmitted, the greater the degree of influence, and the degree of influence is linearly related to the number of transmissions.

[0145] The number of times scene elements in the influence transmission model are transmitted through each level of the human-machine co-driving system can be calculated using the following formula:

[0146] (1)

[0147] In the formula, P(n) represents the number of times the influence of a certain scene element is transmitted through each level of the human-machine co-driving system, n represents the number of element attributes in that scene element, and Ei This represents the number of times the influence of the i-th element attribute in the scene is propagated across all levels of the human-machine co-driving system.

[0148] The table below shows the correspondence between the difference in the number of times the influence of scene elements is transmitted and the relative ratio in the 1-9 scale method:

[0149]

[0150] This allows us to construct a judgment matrix. By solving for the largest eigenvalue of the judgment matrix and its corresponding eigenvector, and normalizing the eigenvector corresponding to the largest eigenvalue of the judgment matrix, we can obtain the importance weights of the scene elements.

[0151] To ensure the rationality of the importance weighting index, the matrix must undergo a consistency check. The formula for calculating the consistency check coefficient CR is as follows:

[0152] (2)

[0153] In the formula, RI is the standard value of the hierarchical overall ranking average random consistency index, and CI is the hierarchical overall ranking consistency index, the relevant calculation formula of which is:

[0154] (3)

[0155] (4)

[0156] (5);

[0157] In the formula, λ max Let n be the order of the judgment matrix, A be the judgment matrix, and ω be the largest eigenvalue. * To determine the eigenvectors of the judgment matrix, if CR < 0.1, it indicates that the judgment matrix has good consistency and meets the consistency requirements.

[0158] After obtaining the importance weights of the scene elements, select the scene elements with high importance weights according to the dimensions of the scene space to be constructed, determine the discrete range and discrete length, and then construct the simulation scene space.

[0159] Step 3: Key Scenarios Selection Based on Safety and Comfort. The selection criteria for key scenarios are based on both scenario challenge and probability of occurrence. A criticality function is constructed, and an optimization algorithm is used to select key scenarios that meet the criticality threshold, thereby achieving digital twin creation and accelerated testing of simulated scenarios based on real roads.

[0160] In security testing, a key security function is constructed using the scenario's hazard level and the probability of its occurrence, as shown in the following formula:

[0161] (6)

[0162] In the formula, V s (x) is the security critical function, R s (x) represents the scene's hazard level, P s (x) represents the probability of scenario R occurring. s (x) can be characterized by Enhanced Time to Collision (ETTC), which is calculated as follows:

[0163] (7)

[0164] In the formula, R represents the longitudinal distance between the main vehicle and the target vehicle, ∆v represents the relative velocity between the main vehicle and the target vehicle, and ∆a represents the relative acceleration between the main vehicle and the target vehicle.

[0165] In comfort testing, a key comfort function is constructed using scenario comfort and scenario occurrence probability, as shown in the following equation:

[0166] (8)

[0167] In the formula, V c (x) is the key comfort function, C c (x) represents scene comfort, P c (x) represents the probability of scenario C occurring. c (x) can be calculated from the jerkiness and the root mean square value of the acceleration, as shown in the following formula:

[0168] (9)

[0169] In the formula, J lon (x) represents the longitudinal abrupt change, a lon (x) Root mean square value of longitudinal acceleration, α c With β c α is the weighted value of the longitudinal jerk and the root mean square value of the longitudinal acceleration. c With β c You can take 0.5 for all of them.

[0170] The probability of scenario occurrence P in the safety critical function and the comfort critical function s (x) and P c (x) By using the convex combination method, the scene captured from the real road is transformed into a scene in the scene space, thus obtaining: A1(R) in the 3D scene space. a1 , v a1 , a a1 ), A2(R a2 , v a2 , a a2 ), A3(R a3 , v a3 , a a3 ), A4(R a4 , v a4 , a a4 ), A5(R a5 , v a5 , a a5 ), A6(R a6 , v a6 , a a6 ), A7(R) a7 , v a7 , a a7 ), A8(R a8 , v a8 , a a8 Let B be eight uniformly discrete scenes in the scene space, and let B be a scene recorded from real road data, and B(R) = ... b , v b , a b Inside the cube formed by A1, A2, A3, A4, A5, A6, A7, and A8, L 11 L is the Euclidean distance from B to the plane containing A2, A4, A6, and A8. 12 L is the Euclidean distance from B to the plane containing A1, A3, A5, and A7. 21 L is the Euclidean distance from B to the plane containing A3, A4, A7, and A8. 22 L is the Euclidean distance from B to the plane containing A1, A2, A5, and A6. 31 L is the Euclidean distance from B to the plane containing A5, A6, A7, and A8. 32 Let B be the Euclidean distance from B to the planes containing A1, A2, A3, and A4. Then, the real scene B can be represented by scenes A1, A2, A3, A4, A5, A6, A7, and A8 in scene space using the convex combination method as follows:

[0171] (10)

[0172] In the formula, ω1, ω2, ω3, ω4, ω5, ω6, ω7, and ω8 are the weight coefficients assigned to A1, A2, A3, A4, A5, A6, A7, and A8, respectively, and their calculation formula is as follows:

[0173] (10)

[0174] (11)

[0175] (12)

[0176] (13)

[0177] (14)

[0178] (15)

[0179] (16)

[0180] (17)

[0181] After transforming real-world road scene data into scenarios within a scene space, the probability of occurrence of simulated scenarios in that space can be obtained. Furthermore, the key safety and comfort function values ​​for these simulated scenarios can be calculated. By setting key safety and comfort thresholds respectively and employing a global optimization algorithm, key scenarios for human-machine co-driving safety and comfort testing can be derived.

[0182] The safety and comfort of human-machine co-driving were tested in key scenarios. Safety was evaluated using the accident rate, while comfort was evaluated using the driver's expected comfort level, which could be obtained in advance by recording the vehicle's state data when the driver was driving alone. Because the key scenarios were taken from real roads, they had a high degree of realism. Furthermore, because they posed a high challenge to the performance being tested, acceleration testing of human-machine co-driving was achieved.

[0183] The second step, the test process for the synchronization of driver and intelligent driving system decision-making, is as follows:

[0184] Step 1: Synchronization Test of Driver and Intelligent Driving System Decisions. Driver and intelligent driving system decision signals are sampled at the same time. These signals include steering wheel angle, accelerator pedal, and brake pedal signals. The synchronization of decisions is determined by comparing the deviations between the driver's and the intelligent driving system's decision signals.

[0185] The sampling frequency f and sampling time T are set to simultaneously collect decision signals from both the driver and the intelligent driving system. The driver's decision signal is as follows:

[0186]

[0187] In the formula, p d-sw,tn p is the driver's steering wheel angle signal. d-ap,tn p is the driver's accelerator pedal signal. d-bp,tn For the driver's brake pedal signal, n=fT.

[0188] The decision signals of an intelligent driving system are:

[0189]

[0190] In the formula, p a-sw,tn For the steering wheel angle signal of the intelligent driving system, p a-ap,tn For the accelerator pedal signal of the intelligent driving system, p a-bp,tn This is the brake pedal signal for the intelligent driving system.

[0191] Calculate the deviation L of the steering wheel angle signal between the driver and the intelligent driving system during the sampling time. sw As shown in the following formula:

[0192]

[0193] Calculate the deviation L of the accelerator pedal signal between the driver and the intelligent driving system during the sampling time. ap As shown in the following formula:

[0194]

[0195] Calculate the deviation L of the brake pedal signal between the driver and the intelligent driving system during the sampling time. bp As shown in the following formula:

[0196]

[0197] Faced with the same driving scenario, assuming the driver's operation is correct, the decision signals from the driver and the intelligent driving system may vary slightly due to fluctuations in the driver's driving state, but the differences will not be significant. Therefore, a threshold of ε is set for the steering wheel angle signal deviation. sw Set a threshold value of ε for the accelerator pedal signal deviation. ap Set a threshold value of ε for brake pedal signal deviation. bp When the deviations of all three decision signals are less than the deviation threshold, the driver and the intelligent driving system can be considered to have good decision-making synchronization; when one or more decision signal deviations are greater than the deviation threshold, the driver and the intelligent driving system can be considered to have poor decision-making synchronization, and calibration is required.

[0198] Step 2: Calibration of driver and intelligent driving system decision synchronization. A curve fitting method is used to calibrate the decision signals of the driver and the intelligent driving system. Using the decision signal of the intelligent driving system as a benchmark, spline fitting based on the least squares method is performed to minimize the cost function between the driver's decision signal and the intelligent driving system's decision curve, thus achieving the calibration of driver and intelligent driving system decision synchronization.

[0199] Taking the steering wheel angle signal as an example, a cubic spline interpolation function s(t) is constructed. The spline function fitting based on least squares is performed in the spline function space S. k Inside, find the value for p a-sw (t) Best approximation of the norm s * (t), as shown in the following formula:

[0200]

[0201] s * (t) represents the fitted curve of the steering wheel angle signal of the intelligent driving system. To facilitate calibration, a calibration curve s for the steering wheel angle signal of the intelligent driving system is constructed. * (t+b) allows the fitting curve of the steering wheel angle signal of the intelligent driving system to be shifted along the time axis, where b is the shift length. This constructs the relationship between the sampled driver steering wheel angle signal and s. * The cost function J(b) for (t+b) is shown in the following equation:

[0202]

[0203] By minimizing the cost function J(b) using the gradient descent method, the synchronization calibration of the driver's and intelligent driving system's decisions can be achieved.

[0204] The process of testing driver consistency in driving behavior in the third step is as follows:

[0205] Step 1: Driver Aggression Identification Based on CNN-BP Neural Network. Driver state parameters and control parameters of drivers with known driving aggression are collected and trained using a CNN-BP neural network to obtain a driver aggression model.

[0206] To analyze the consistency of driver behavior in real time during driving, a driver aggression model needs to be established before conducting human-machine co-driving tests. The driving aggression level D... agg The discrete levels are divided into five categories, as shown in the following formula:

[0207]

[0208] Ten drivers with over five years of driving experience were selected for each level of driving aggression. Their driving state and operational parameters under normal driving conditions were collected. The driving state parameter X... S The composition is shown in the following formula:

[0209]

[0210] In the formula, x he For head movement signals, x ey For eye movement signals, x mi For electrocardiogram (ECG) signals, x sk For electrodermal signals, x mu For electromyography (EMG) signals, x br This is an electroencephalogram (EEG) signal. Driving control input X C The composition is shown in the following formula:

[0211]

[0212] In the formula, x sw For steering wheel angle signal, x ap For accelerator pedal position signal, x bp This is the brake pedal position signal.

[0213] X S With driving control amount X C The input matrix X, which serves as the input layer of the neural network, is shown in the following equation:

[0214]

[0215] Due to the large number of feature values, convolutional and pooling layers are added to the BP neural network to optimize the feature values. Therefore, a CNN-BP neural network is used to train the driver's aggressiveness model. The structure of the CNN-BP neural network is shown in the attached figure. Figure 6 As shown. Considering the inconsistency in the dimensions and values ​​of the eigenvalues ​​in X, the eigenvalues ​​are first normalized, as shown in the following formula:

[0216]

[0217] In the formula, x i Let x be the eigenvalues ​​of each type in the input matrix. ni These are the normalized eigenvalues.

[0218] The normalized matrix X n The input is fed into a convolutional layer for convolution. The convolutional layer extracts and optimizes the normalized feature values ​​using the parameters in the convolution kernel, i.e., the weights of the locally connected neural network. The convolutional layer sets two (1×1) convolution kernels to X. nEach data point is concatenated, and two (2×2) convolutional kernels are applied to X. n Partially connect the 2×2 local regions, and use two (3×3) convolution kernels to connect X. n A fully connected layer is used to analyze the input feature matrix from multiple perspectives and ranges, resulting in the convolutional matrix M, as shown in the following equation:

[0219]

[0220] In the formula, F is the kernel size, P is the padding size, and S is the stride. After passing through a convolutional layer with S=1, two 3×3 feature matrices, two 2×2 feature matrices, and two eigenvalues ​​are obtained.

[0221] The matrix M output from the convolutional layer is input into the pooling layer for feature dimensionality reduction. The pooling layer consists of two (2×2) pooling kernels and two (3×3) pooling kernels, ultimately yielding six optimized feature values, which are then input into the next layer, the backpropagation (BP) neural network. The BP neural network layer has 20 neurons, with 5 neurons in the output layer, corresponding to five levels of driving aggression.

[0222] To obtain an accurate model of driver aggression, the CNN-BP neural network also needs to be trained. Pooling layers do not require training since they have no parameters, while both convolutional layers and the BP neural network are trained using the backpropagation algorithm. The following section uses the BP neural network as an example to introduce the model training method.

[0223] First, the derivative of the connection weights between the output layer and the hidden layer is calculated, as shown in the following equation:

[0224]

[0225] In the formula, E is the error function, and θ 2ij y represents the connection weights between the input layer and the hidden layer. k For output layer output, u 2k This represents the activation value between the output layer and the hidden layer.

[0226] The derivative of the connection weights between the input layer and the hidden layer is shown in the following equation:

[0227]

[0228] In the formula, θ 1ij q represents the connection weights between the input layer and the hidden layer, and q represents the number of neurons in the output layer.

[0229] The adjustment value for the connection weights between the input layer and the hidden layer can be calculated using the following formula:

[0230]

[0231] In the formula, z j For the hidden layer, u 1k This represents the activation value between the input layer and the hidden layer.

[0232] The CNN-BP neural network is trained using the gradient descent algorithm until the error function E converges, thus obtaining the driver's driving aggression model.

[0233] Step 2: Real-time testing of driver aggression. In the intelligent driving simulator, head and eye movement signals of the driver are collected in real time using head and eye trackers worn by the driver. ECG, skin conductance, electromyography (EMG), and electroencephalography (EEG) signals of the driver are also collected in real time using ECG, skin conductance, EMG, and EEG detectors worn by the driver. Steering wheel angle, accelerator pedal position, and brake pedal position signals are also collected in real time. These signals are used as input to the driver aggression model obtained in Step 1 to obtain the driver's real-time aggression.

[0234] Step 3: Optimize the parameters of the human-machine co-driving model based on driving aggression. Adjust the parameters of the intelligent driving system decision model in the human-machine co-driving system according to the driver's real-time driving aggression to match the driver's driving aggression, thereby improving driver comfort. At the same time, set a threshold for the aggression of the intelligent driving system to prevent situations where both the driver's and the intelligent driving system's driving aggression are too high, thus affecting driving safety.

Claims

1. A human-machine co-driving online evaluation method that highly integrates real vehicle and virtual environment, characterized in that: The method includes the following steps: The first step, simulation scenario testing based on digital twins and accelerated testing, is as follows: Step 1: Real-world road data collection and processing. Various sensors are installed on real vehicles to build a real-world road data collection platform. To ensure the representativeness of the collected scene data, large-scale data collection is carried out under traffic conditions of different road types, weather types, and lighting types. After the data collection is completed, the data is first preprocessed to remove data segments with errors and missing data. Then, the scene data collected by each sensor is synchronized in time and space, and the data is formatted to meet the data structure required for scene construction. Finally, the data is classified according to the scene type. Step 2: Constructing a simulation scenario space for human-machine co-driving testing. Based on the characteristics of human-machine co-driving and the performance to be tested, design the type of simulation scenario, list the scenario elements, determine the importance weight values ​​of the scenario elements based on the group decision fuzzy network hierarchical analysis method F-ANP, select scenario elements with high importance weight values ​​for discretization, and construct the simulation scenario space. In the construction of the simulation scene space, the scene elements are first listed. Static scene elements include the number of lanes, lane width, lane curvature, weather and lighting. Dynamic scene elements include the initial position, initial speed, driving speed and driving acceleration of the main vehicle and the target vehicle. Then, the importance weights of scene elements are determined using the group decision-making fuzzy network hierarchical analysis method (F-ANP). The relationships between scene elements are analyzed, and the scene elements are organized and hierarchically structured to construct an ordered, progressively layered structural model. A judgment matrix is ​​constructed based on the relative importance ratios between scene elements. The relative ratios between scene elements are determined by the influence transmission model, which reflects the importance of scene elements to the human-machine co-driving system. The influence transmission model sets three assumptions, as follows: 1) The impact of scene elements on the human-machine co-driving system will be passed down level by level with the human-machine co-driving system, and this impact will not decrease with the passing down of levels; 2) Different types of scene elements have the same impact on the human-machine co-driving system; 3) The degree of influence of scene elements on the human-machine co-driving system is represented by the number of times scene elements are transmitted between the levels of the human-machine co-driving system. The more times they are transmitted, the greater the degree of influence, and the degree of influence is linearly related to the number of transmissions. The number of times scene elements in the influence transmission model are transmitted through each level of the human-machine co-driving system is calculated using the following formula: (1)； In the formula, P(n) represents the number of times the influence of a certain scene element is transmitted through each level of the human-machine co-driving system, n represents the number of element attributes in that scene element, and E i This represents the number of times the influence of the i-th element's attribute in the scene is propagated across all levels of the human-machine co-driving system. Find the largest eigenvalue of the judgment matrix and its corresponding eigenvector, and normalize the eigenvector corresponding to the largest eigenvalue of the judgment matrix to obtain the importance weight of the scene element. To ensure the rationality of the importance weighting index, the matrix must undergo a consistency check. The formula for calculating the consistency check coefficient CR is as follows: (2)； In the formula, RI is the standard value of the hierarchical overall ranking average random consistency index, and CI is the hierarchical overall ranking consistency index, the relevant calculation formula of which is: (3)； (4)； (5)； In the formula, λ max Let n be the order of the judgment matrix, A be the judgment matrix, and ω be the largest eigenvalue. * To determine the eigenvectors of the judgment matrix, if CR < 0.1, it indicates that the judgment matrix has good consistency and meets the consistency requirements. After obtaining the importance weights of the scene elements, the scene elements with high importance weights are selected according to the dimensions of the scene space to be constructed, and the discrete range and discrete length are determined, thus constructing the simulation scene space. Step 3: Screening of key scenarios based on safety and comfort: Selecting scenario challenge and scenario occurrence probability as the screening principles for key scenarios, constructing a criticality function, and using optimization algorithms to screen key scenarios that meet the criticality threshold, thereby realizing digital twins of simulation scenarios based on real roads and accelerating testing. In security testing, a key security function is constructed using the scenario's hazard level and the probability of its occurrence, as shown in the following formula: (6)； In the formula, V s (x) is the security critical function, R s (x) represents the scene's danger level, P s (x) represents the probability of the scenario occurring, R0 s (x) By enhancing the collision time ETTC characterization, ETTC is calculated as follows: (7)； In the formula, R represents the relative longitudinal distance between the main vehicle and the target vehicle. v represents the relative speed between the vehicle and the target vehicle. 'a' represents the relative acceleration between the main vehicle and the target vehicle; In comfort testing, a key comfort function is constructed using scenario comfort and scenario occurrence probability, as shown in the following equation: (8)； In the formula, V c (x) is the key comfort function, C c (x) represents scene comfort, P c (x) represents the probability of the scenario occurring, C c (x) is calculated using the jerk and the root mean square value of acceleration, as shown in the following formula: (9)； In the formula, J lon (x) represents the longitudinal abrupt change, a lon (x) Root mean square value of longitudinal acceleration, α c With β c α is the weighted value of the longitudinal jerk and the root mean square value of the longitudinal acceleration. c With β c Take 0.5 for all; The probability of scenario occurrence P in the safety critical function and the comfort critical function s (x) and P c (x) By using the convex combination method, the scene captured from the real road is transformed into a scene in the scene space, thus obtaining: A1(R) in the 3D scene space. a1 , v a1 , a a1 ), A2(R a2 , v a2 , a a2 ), A3(R a3 , v a3 , a a3 ), A4(R a4 , v a4 , a a4 ), A5(R a5 , v a5 , a a5 ), A6(R a6 , v a6 , a a6 ), A7(R) a7 , v a7 , a a7 ), A8(R a8 , v a8 , a a8 Let B be eight uniformly discrete scenes in the scene space, and let B be a scene recorded from real road data, and B(R) = ... b , v b , a b Inside the cube formed by A1, A2, A3, A4, A5, A6, A7, and A8, L 11 L is the Euclidean distance from B to the plane containing A2, A4, A6, and A8. 12 L is the Euclidean distance from B to the plane containing A1, A3, A5, and A7. 21 L is the Euclidean distance from B to the plane containing A3, A4, A7, and A8. 22 L is the Euclidean distance from B to the plane containing A1, A2, A5, and A6. 31 L is the Euclidean distance from B to the plane containing A5, A6, A7, and A8. 32 Let B be the Euclidean distance from B to the planes containing A1, A2, A3, and A4. Then, the real scene B can be represented by scenes A1, A2, A3, A4, A5, A6, A7, and A8 in scene space using the convex combination method as follows: (10) In the formula, ω1, ω2, ω3, ω4, ω5, ω6, ω7, and ω8 are the weight coefficients assigned to A1, A2, A3, A4, A5, A6, A7, and A8, respectively, and their calculation formula is as follows: (10)； (11)； (12)； (13)； (14)； (15)； (16)； (17)； After transforming real road scene data into scenes in scene space, the probability of occurrence of simulated scenes in scene space is obtained. Then, the safety key function value and comfort key function value of the simulated scene are obtained. Safety key threshold and comfort key threshold are set respectively. Using a global optimization algorithm, key scenes for human-machine co-driving safety test and comfort test are obtained. In key scenarios, the safety and comfort of human-machine co-driving were tested by sampling scenarios. Safety was evaluated by the accident rate, and comfort was evaluated by the driver's expected comfort level. The driver's expected comfort level was obtained in advance by recording the vehicle's state data when the driver was driving alone. The second step is to test the synchronization of driver and intelligent driving system decisions, and the specific process is as follows: Step 1: Test the synchronization of driver and intelligent driving system decisions. Sample the driver's decision signals and the intelligent driving system's decision signals at the same time. The decision signals include steering wheel angle signals, accelerator pedal signals and brake pedal signals. Determine whether the two decisions are synchronized by comparing the deviation between the driver's and intelligent driving system's decision signals. The sampling frequency f and sampling time T are set to simultaneously collect decision signals from both the driver and the intelligent driving system. The driver's decision signal is as follows: ； In the formula, p d-sw,tn p is the driver's steering wheel angle signal. d-ap,tn p is the driver's accelerator pedal signal. d-bp,tn For the driver's brake pedal signal, n=fT; The decision signals of an intelligent driving system are: ； In the formula, p a-sw,tn For the steering wheel angle signal of the intelligent driving system, p a-ap,tn For the accelerator pedal signal of the intelligent driving system, p a-bp,tn For the brake pedal signal of the intelligent driving system; Calculate the deviation L of the steering wheel angle signal between the driver and the intelligent driving system during the sampling time. sw As shown in the following formula: ； Calculate the deviation L of the accelerator pedal signal between the driver and the intelligent driving system during the sampling time. ap As shown in the following formula: ； Calculate the deviation L of the brake pedal signal between the driver and the intelligent driving system during the sampling time. bp As shown in the following formula: ； Steering wheel angle signal deviation setting threshold ε sw The threshold for accelerator pedal signal deviation is set to ε. ap The brake pedal signal deviation setting threshold is ε. bp When the deviations of all three decision signals are less than the deviation threshold, it is considered that the driver and the intelligent driving system have good decision synchronization; when the deviation of one or more decision signals is greater than the deviation threshold, it is considered that the driver and the intelligent driving system have poor decision synchronization and calibration is required. Step 2: Calibration of driver and intelligent driving system decision synchronization. The decision signals of the driver and intelligent driving system are calibrated by curve fitting. Spline fitting based on least squares method is performed with the decision signal of intelligent driving system as the benchmark to minimize the cost function of driver decision signal and intelligent driving system decision curve, so as to realize the calibration of driver and intelligent driving system decision synchronization. A cubic spline interpolation function s(t) is constructed from the steering wheel angle signal. The spline function fitting based on least squares is performed in the spline function space S. k Inside, find the value for p a-sw (t) Best approximation of the norm s * (t), as shown in the following formula: ； s * (t) represents the fitted curve of the steering wheel angle signal of the intelligent driving system. To facilitate calibration, a calibration curve s for the steering wheel angle signal of the intelligent driving system is constructed. * (t+b) allows the fitting curve of the steering wheel angle signal of the intelligent driving system to be shifted along the time axis, where b is the shift length, constructing a curve that modulates the sampled driver steering wheel angle signal with s. * The cost function J(b) for (t+b) is shown in the following equation: ； The gradient descent method is used to minimize the cost function J(b) to achieve synchronization calibration of driver and intelligent driving system decision-making. The third step is a test of the consistency of the driver's driving behavior, the specific process of which is as follows: Step 1: Identify driver aggression based on CNN-BP neural network. Collect the driving state and driving operation quantities of drivers with known driving aggression, and obtain the driver aggression model through CNN-BP neural network training. To analyze the consistency of driver behavior in real time during driving, a driver aggression model is established in advance before conducting human-machine co-driving tests, and the driving aggression level D is defined. agg The discrete levels are divided into five categories, as shown in the following formula: (1)； Ten drivers with more than five years of driving experience were selected for each level of driving aggression. Their driving state and driving manipulation quantities under normal driving conditions were collected. Among them, the driving state quantity X S The composition is shown in the following formula: (2)； In the formula, x he For head movement signals, x ey For eye movement signals, x mi For electrocardiogram (ECG) signals, x sk For electrodermal signals, x mu For electromyography (EMG) signals, x br EEG signal, driving control quantity X C The composition is shown in the following formula: (3)； In the formula, x sw For steering wheel angle signal, x ap For accelerator pedal position signal, x bp This is the brake pedal position signal; X S With driving control amount X C The input matrix X, which serves as the input layer of the neural network, is shown in the following equation: (4)； Based on the BP neural network, convolutional and pooling layers are added to optimize the feature values. The CNN-BP neural network is used to train the driver's driving aggression model, and the feature values ​​are normalized as shown in the following formula: (5)； In the formula, x i Let x be the eigenvalues ​​of each type in the input matrix. ni These are the normalized eigenvalues; The normalized matrix X n The input is fed into a convolutional layer for convolution. The convolutional layer extracts and optimizes the normalized feature values ​​using the parameters in the convolution kernel, i.e., the weights of the locally connected neural network. The convolutional layer sets two 1×1 convolution kernels to X. n Each data point is concatenated, and two 2×2 convolutional kernels are applied to X. n Partially connect the 2×2 local components, and then perform two 3×3 convolution kernels on X. n By performing a fully connected layer, the input feature matrix can be analyzed from multiple perspectives and ranges, resulting in the convolutional matrix M, as shown in the following equation: (6)； In the formula, F is the kernel size, P is the padding size, and S is the stride. After passing through a convolutional layer with S=1, two 3×3 feature matrices, two 2×2 feature matrices, and two eigenvalues ​​are obtained. The matrix M output by the convolutional layer is input into the pooling layer for feature dimensionality reduction. The pooling layer consists of two 2×2 pooling kernels and two 3×3 pooling kernels, and finally obtains six optimized feature values, which are input into the next layer of the BP neural network. The BP neural network layer is set with twenty neurons, and the output layer has five neurons, which correspond to five driving aggression levels. The CNN-BP neural network is trained using the backpropagation algorithm for both the convolutional layers and the BP neural network. The training method for the BP neural network model is as follows: First, the derivative of the connection weights between the output layer and the hidden layer is calculated, as shown in the following equation: ； In the formula, E is the error function, and θ 2ij y represents the connection weights between the input layer and the hidden layer. k For output layer output, u 2k The activation values ​​between the output layer and the hidden layer; The derivative of the connection weights between the input layer and the hidden layer is shown in the following equation: ； In the formula, θ 1ij q represents the connection weights between the input layer and the hidden layer, and q is the number of neurons in the output layer. The adjustment value of the connection weights between the input layer and the hidden layer is calculated using the following formula: ； In the formula, z j For the hidden layer, u 1k The activation values ​​between the input layer and the hidden layer; The gradient descent algorithm is used to train the CNN-BP neural network until the error function E converges, thus obtaining the driver's driving aggression model. Step 2: Real-time test of driver aggression. In the intelligent driving simulator, head movement and eye movement signals of the driver are collected in real time through head movement and eye movement devices worn by the driver. ECG, skin conductance, electromyography, and electroencephalography signals of the driver are collected in real time through ECG, skin conductance, electromyography, and electroencephalography devices worn by the driver are collected in real time. Steering wheel angle signal, accelerator pedal position signal, and brake pedal position signal are collected in real time. The above signals are used as input to the driver aggression model obtained in Step 1 to obtain the driver's real-time aggression. Step 3: Optimize the parameters of the human-machine co-driving model based on driving aggression. Adjust the decision model parameters of the intelligent driving system in the human-machine co-driving system according to the driver's real-time driving aggression to make the driving aggression of the intelligent driving system match the driving aggression of the driver, thereby improving the driver's comfort. At the same time, set a threshold for the aggression of the intelligent driving system to prevent situations where both the driver's and the intelligent driving system's driving aggression are too high, thus affecting driving safety.

2. A human-machine co-driving online evaluation system that highly integrates real vehicle and virtual environment, wherein the online evaluation system uses the human-machine co-driving online evaluation method that highly integrates real vehicle and virtual environment as described in claim 1, characterized in that: The aforementioned human-machine co-driving online evaluation system, which highly integrates real vehicles and virtual environments, includes a digital twin module based on a real road simulation scenario, an intelligent driving simulation module, a human-machine co-driving control module, a real vehicle module, and an online evaluation and model parameter self-optimization module. These modules are arranged in parallel, and their structural composition is as follows: The digital twin module for simulation scenarios based on real roads collects, processes, and classifies real road data. It uses digital twin technology to generate simulation scenarios for human-machine co-driving tests and selects key scenarios based on safety and comfort tests for human-machine co-driving acceleration tests. The digital twin module for simulation scenarios based on real roads consists of four sub-modules: a real road data collection and processing module, a simulation scenario space construction module for human-machine co-driving tests, a key scenario selection module based on safety and comfort, and a simulation scenario information injection module. The simulation scenario space construction module for human-machine co-driving tests constructs the scenario space of the simulation scenario by discretizing scenario elements with high importance weight values. The key scenario screening module based on safety and comfort establishes key scenario screening criteria by establishing key safety functions and key comfort functions. It uses a global optimization algorithm to screen key scenarios in the scenario space and injects the generated key scenarios into the human-machine co-driving online testing and evaluation module through the simulation scenario information injection module for accelerated testing of safety and comfort. The intelligent driving simulation module provides a simulated driving environment for drivers and collects their status and operational actions. The intelligent driving simulation module consists of four sub-modules: a simulation scene rendering and display module, a driving somatosensory simulation module, a driver module, and a driving simulation cabin module. The driver status information output by the driver module and the driver decision signals output by the driving simulation cabin module are input into the online evaluation and model parameter self-optimization module for evaluating the synchronization of driver and intelligent driving system decisions and the consistency of driver driving behavior. The human-machine co-driving control module realizes human-machine co-driving by formulating a collaborative control strategy between the intelligent driving system and the driver. The human-machine co-driving control module consists of two sub-modules: the intelligent driving system control module and the human-machine collaborative control module. The intelligent driving system decision signals output by the intelligent driving system are input into the online evaluation and model parameter self-optimization module for evaluating the synchronization of driver and intelligent driving system decisions. The real vehicle module involves driving a real vehicle on the test track, executing a human-machine collaborative control strategy, and collecting the motion status of the real vehicle. The real vehicle module consists of a vehicle module, a real vehicle status acquisition module, and a real vehicle status injection module. The real vehicle status information collected by the real vehicle status acquisition module will be injected into the online evaluation and model parameter self-optimization module through the real vehicle status injection module for safety and comfort acceleration testing. The online evaluation and model parameter self-optimization module verifies the safety and comfort of the human-machine co-driving system through scenario-based accelerated testing; verifies the synchronization of driver and intelligent driving system decision signal outputs through driver and intelligent driving system decision synchronization testing, and can perform synchronization calibration in asynchronous situations; and verifies changes in driver aggression during driving through driver behavior consistency testing, thereby optimizing the model parameters of the intelligent driving system control module and improving driver comfort. The online evaluation and model parameter self-optimization module consists of three sub-modules: an accelerated testing-based safety and comfort evaluation module, a driver and intelligent driving system decision synchronization evaluation module, and a driver behavior consistency evaluation module.

Images