A method for dynamically adjusting the strength of a takeover request in alignment with the cognitive state of the driver

Abstract

The application relates to a kind of alignment driving cognitive state takeover request intensity dynamic adjustment method, acquisition multivariate takeover data, the dynamic adjustment result of the takeover request intensity for current driver is obtained by reinforcement learning network;The training steps of reinforcement learning network include: obtaining cognitive state value by driver attribute data, context category is constructed;Calculate the strategy preference probability of each strategy under each context category, as process layer preference soft label;Vehicle traffic state is discretized, and local state is obtained, under the same context category and local state, the time step layer preference soft label is calculated;Based on process layer preference soft label and time step layer preference soft label, train multi-head reward;According to the instant reward calculated by trained multi-head reward, reinforcement learning network is trained based on instant reward.Compared with prior art, the application has the advantages of high individualization level, real-time optimization and high efficiency.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method for dynamically adjusting the intensity of takeover requests in accordance with the driver's cognitive state. Background Technology

[0002] During the operation of an autonomous driving system, when encountering complex or sudden traffic scenarios beyond its designed operating range, the system must promptly hand over vehicle control to the human driver via a takeover request. The safety and smoothness of this takeover process directly affect the overall reliability of the autonomous driving system and the safety of its occupants, and is one of the key challenges in the practical application of autonomous driving technology.

[0003] Studies have shown that individual differences among drivers, such as age and driving experience, significantly affect their takeover performance and reaction time. This indicates that during takeover, drivers exhibit significant individualized characteristics in their cognitive processing of warning information, attention recovery, and operational preparation. Furthermore, cognitive science indicates that a driver's real-time cognitive state, such as workload, situational awareness, and fatigue level, is closely related to their task performance. If the triggering and presentation of takeover requests fail to adequately consider the driver's real-time cognitive state, it may lead to insufficient or overloaded cognitive resources, resulting in delayed takeover, operational errors, or even safety accidents.

[0004] However, current mainstream autonomous driving takeover request strategies generally lack deep integration and dynamic matching with the driver's cognitive state at the design and implementation levels. Common static takeover strategies typically trigger takeover requests based on fixed thresholds, such as time margin (TTC), or preset rules, making it difficult to adapt to dynamically changing traffic environments and driver states. Although some dynamic or phased strategies attempt to attract the driver's attention by gradually increasing the intensity of prompts, such as audiovisual alarm levels, their design logic still focuses on the bottom-line safety goal of "avoiding takeover failure." Essentially, it is a one-way, progressively stronger warning mechanism that fails to establish a closed-loop adjustment capability based on the driver's real-time cognitive state. When facing complex and ever-changing real-world takeover scenarios, such strategies lack adaptability and flexibility, making it difficult to achieve efficient, stable, and human-centered takeover support. From an algorithmic and theoretical perspective, existing takeover request intensity designs mostly remain at the level of static mapping based on instantaneous states or phased rule control, failing to model the takeover process as a sequential optimal control problem under a Markov decision process. Therefore, they cannot leverage the theoretical advantages of reinforcement learning in handling state evolution uncertainty, long-term cumulative effects of actions, and online adaptive optimization of strategies. Meanwhile, existing methods typically rely on sparse terminal indicators or manually constructed reward functions, lacking identifiable dense reward representations that can characterize the marginal utility of actions at each time step under different driver contexts and cognitive states. This leads to instability in the reinforcement learning-based policy learning process, sensitivity to initialization, and difficulty in forming reproducible and generalizable optimal control strategies. Therefore, how to dynamically and finely align the takeover request strength with the driver's real-time cognitive state after an autonomous vehicle issues a takeover request, thereby shortening the takeover time and improving the safety and stability of the takeover process, is a technical problem that needs to be solved. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method for dynamically adjusting the intensity of takeover requests in line with the driver's cognitive state. By decomposing takeover preferences into whole-process layer and time-step layer preferences and combining them with a context-driven multi-head reward model, it achieves stable learning of personalized dense rewards from sparse and fluctuating takeover performance data, thereby driving a reinforcement learning strategy to achieve dynamic adjustment of takeover request intensity in line with the driver's cognitive state in real time.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] According to one aspect of the present invention, a method for dynamically adjusting the intensity of a takeover request to align with the driver's cognitive state is provided. The method acquires multivariate takeover data and employs a reinforcement learning network to obtain a dynamic adjustment result for the intensity of the takeover request for the current driver. The training steps of the reinforcement learning network include:

[0008] S1. Collect historical multi-dimensional takeover data, including historical vehicle traffic status data, driver physiological signal data, driver attribute data, and takeover request intensity.

[0009] S2. Obtain cognitive state values ​​through driver physiological signal data and construct context categories; in each context category, conduct repeated trials of each preset strategy and collect takeover trajectories; by comparing the takeover trajectories of each strategy, obtain the strategy preference probability of each strategy under each context category, which serves as a process-level preference soft label.

[0010] S3. Discretize the vehicle traffic state to obtain local states. Under the same context category and local state, filter trajectory pairs from the takeover trajectory data and calculate the takeover time difference of each trajectory pair. Calculate the time step layer preference soft label based on the takeover time difference.

[0011] S4. Based on the process layer preference soft label and the time step layer preference soft label, obtain personalized dense reward, and train the multi-head reward based on context category;

[0012] S5. Construct a reinforcement learning network, calculate immediate rewards based on the trained multi-head reward system, and train the reinforcement learning network based on the immediate rewards.

[0013] Furthermore, the vehicle traffic status data includes the collision time between the vehicle and the obstacle in front; the driver attribute data includes the driver's age and driving experience; and the driver physiological signal data includes continuous time-step physiological signals of head pitch angle, head orientation, pupil diameter, and eye opening / closing ratio.

[0014] Furthermore, the construction of the context categories includes: dividing the driver's age and driving experience into two levels, high and low, and combining them according to the levels to obtain four context categories, including: high age and high driving experience, high age and low driving experience, low age and high driving experience, and low age and low driving experience.

[0015] Furthermore, the construction of the process layer preference soft label includes:

[0016] Within the same context category, multiple repeated trials are conducted for each preset strategy to collect takeover trajectories. These trajectories are then divided into several comparison blocks, each containing one takeover trajectory for each strategy. Within each comparison block, all takeover trajectories are compared pairwise. If the corresponding strategy... The takeover time of the takeover trajectory is shorter than that of the strategy. then strategy It won in this comparison; the statistical strategy was effective. The corresponding takeover trajectory wins the most times in all comparison blocks and the total number of comparisons. The strategy preference probability is calculated and Laplace smoothing is performed to obtain the process layer preference soft label.

[0017] Furthermore, the construction of the time step layer preference soft labels includes:

[0018] The takeover request intensity and collision time in traffic conditions are discretized into several local intervals, each containing a discretized segment of the takeover request intensity and collision time in traffic conditions. Within the same context category and the same local interval, takeover trajectories implementing different strategies are paired to obtain trajectory pairs. In each trajectory pair, one takeover trajectory performs a takeover request intensity enhancement action in the current local interval, while the other performs a takeover request intensity maintenance action. The takeover time difference between each trajectory pair is calculated, and the takeover time weight is calculated based on the magnitude of the takeover time difference. The time step layer preference soft label is obtained by accumulating the weighted winning count and total weighted count of all trajectory pairs in each context category and each local interval.

[0019] Furthermore, the state of the reinforcement learning network includes current vehicle traffic state data, takeover request strength, driver takeover time and context category; actions include increasing takeover request strength and maintaining takeover request strength.

[0020] Furthermore, the multi-head reward includes a shared feature encoder and several reward prediction heads corresponding to context categories; the training process of the multi-head reward includes:

[0021] At the process layer, the cumulative return difference of each policy pair is calculated, and the predicted policy preference probability is obtained through the Sigmoid function; the predicted policy preference probability is made to approximate the process layer preference soft label by minimizing the cross-entropy loss function.

[0022] At the time step level, the time step level preference soft label is converted into log odds form; the reward difference between the takeover request strength enhancement action and the takeover request strength maintenance action under the same context category is calculated; the reward difference is approximated to the log odds label of the time step level preference soft label by minimizing the mean squared error loss function.

[0023] Multi-head reward training is conducted using personalized dense rewards derived from predicted policy preference probabilities and reward differences.

[0024] Furthermore, in S5, at each training time step, the current action is calculated using the multi-head reward based on the current state, context category, and actual action performed. Predicted rewards and alternative actions Predicted rewards The difference between the predicted reward of the current action and the predicted reward of the alternative actions is used as the immediate reward signal. To enhance the state of the learning network, For context category.

[0025] According to a second aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described thereon.

[0026] According to a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] (1) The alignment of takeover request strategy with driver cognitive state is realized, which improves the personalization level of takeover: In the prior art, the prompt intensity is usually set once when the takeover request is triggered based on a fixed threshold or preset rules, or a progressive enhancement strategy with a few fixed stages is adopted. It is difficult to express the differences between different drivers, and it is also difficult to describe the dynamic changes of cognitive state with the takeover process. The present invention introduces driver age and driving experience as context variables, and incorporates the driver's real-time cognitive state, traffic situation state and takeover request prompt intensity state into a unified modeling context Markov decision process. It constructs a personalized takeover modeling framework that can depict the dynamic coupling of "traffic situation - prompt stimulus - cognitive evolution - behavioral response". It overcomes the problem that traditional MDP cannot express individual differences and cognitive evolution. The system can generate differentiated prompt intensity adjustment strategies based on different driver groups, i.e., context categories and their real-time cognitive states. Therefore, the issuance of takeover request is no longer based on static threshold or unified rules, but can be dynamically, continuously and personalized according to the driver's current cognitive ability and traffic situation. Thus, human-machine cognitive alignment is realized in structure, laying the foundation for state representation to improve the adaptability and safety of the takeover process.

[0029] (2) This invention addresses the sparse reward and preference ambiguity issues in policy learning under complex interactive scenarios, achieving stable reward learning: Due to the complex nonlinear coupling relationship between the strength of takeover requests and individual driver differences, traffic urgency, and real-time cognitive state, existing technologies struggle to predetermine quantitative mapping rules for "what kind of driver should use what kind of takeover request strength under what traffic scenario and cognitive state." Therefore, modeling this problem by manually designing reward functions is either impractical in engineering or has extremely poor generalization. Secondly, traditional preference reinforcement learning methods typically only utilize single-layer preference supervision at the trajectory layer, which is insufficient to constrain the solution space, leading to underdetermined problems. The model is highly sensitive to random initialization, and the learned reward structure and policy performance fluctuate significantly. This invention addresses the issues of sparse feedback and performance ambiguity in takeover scenarios. To address the instability of preference learning caused by large dynamic range, this invention proposes a multi-head reward model based on two-layer preferences. By decomposing preferences into overall performance preferences at the entire takeover process layer and local action contribution preferences at the takeover time step layer, and employing joint training with trajectory layer cross-entropy loss and time step layer regression loss, the reward model can stably learn the intrinsic mapping relationship between "traffic scenario—cue intensity—cognitive state—takeover performance" from experimental data. This effectively overcomes the problems of traditional single-layer preference-based reinforcement learning being susceptible to noise and sensitive to initialization. It transforms the originally sparse and lagging takeover time feedback into continuous, dense, and personalized reward signals, providing a stable, reliable, and interpretable learning objective for subsequent reinforcement learning strategy optimization, and ensuring the convergence and generalization ability of strategy training.

[0030] (3) Fine-grained dynamic optimization and closed-loop control of takeover request intensity are realized, improving the overall safety and efficiency of the takeover process: This invention models the adjustment of takeover request intensity as a sequential decision-making process and uses reinforcement learning to train end-to-end policies. By taking personalized dense rewards as optimization objectives, an intelligent decision-making body is constructed with traffic environment state, takeover process state and driver real-time cognitive state as joint inputs and takeover request intensity adjustment as action output. In the entire time window after the takeover request is issued, the prompt intensity is adjusted in real time and continuously according to the changes in traffic urgency and the evolution of driver cognitive state. Therefore, the takeover request is transformed from traditional single-trigger or discrete stage control to closed-loop continuous control oriented to cognitive dynamics, enabling the system to achieve a balance between insufficient wake-up and excessive interference, shortening the takeover time, reducing cognitive load, improving the safety and stability of the takeover process and user experience, and can be extended to other human-machine collaborative control scenarios, providing a set of general technical solutions that can be engineered for human-machine cognitive alignment and dynamic collaboration in autonomous driving systems. Attached Figure Description

[0031] Figure 1A flowchart of the training steps of the reinforcement learning network used in the method for dynamically adjusting the intensity of takeover requests to align with the driver's cognitive state.

[0032] Figure 2 A schematic diagram of the data flow in the reinforcement learning network training steps used in the method for dynamically adjusting the intensity of takeover requests to align with the driver's cognitive state.

[0033] Figure 3 This is a schematic diagram of a multi-reward structure. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] This embodiment provides a method for dynamically adjusting the intensity of takeover requests in accordance with the driver's cognitive state. By collecting multivariate takeover data, a reinforcement learning network is used to obtain a dynamic adjustment result for the intensity of the takeover request for the current driver. Specifically, it includes:

[0036] Real-time collection of multi-dimensional takeover data, including current vehicle traffic status data, current driver physiological signal data, current driver attribute data, and current takeover request intensity; among which, vehicle traffic status data includes the collision time between the vehicle and the obstacle ahead. Driver attribute data includes driver age and driving experience; driver physiological information includes continuous time-step physiological signals of head pitch angle, head orientation, pupil diameter, and eye opening / closing ratio.

[0037] Based on the driver's physiological signals, the driver's current cognitive state value is obtained through cognitive state estimation processing. The values ​​are discretized into level values; at the same time, corresponding context categories are obtained based on the driver's age and driving experience. .

[0038] Construct the state vector at the current time step The input is fed into the trained reinforcement learning network.

[0039] Reinforcement learning networks based on input state vectors and context category Through online action value network Calculate the action value of each action in the current state and select the action with the highest value as the output. The action space is a binary selection, where: This indicates an increase in the strength of the takeover request. This indicates "maintain the current intensity".

[0040] Based on the output action, the corresponding intensity adjustment is performed:

[0041] like Then, the current takeover request strength is increased by a fixed step (e.g., 0.1) to obtain the new takeover request strength. ;like Then keep .

[0042] Updated takeover request strength The output is sent to the vehicle's human-machine interface module to adjust the takeover prompts in real time, such as the volume and intensity level of auditory prompts.

[0043] As the takeover process proceeds, the above steps of data collection, state construction, strategy decision-making, and intensity adjustment are repeated at each time step until the driver completes the takeover or the takeover process ends, thereby achieving dynamic and continuous personalized adjustment of the takeover request intensity based on the driver's real-time cognitive state and traffic situation.

[0044] like Figure 1 As shown, the training steps for the reinforcement learning network include:

[0045] S1. Collect historical multi-dimensional takeover data, including historical vehicle traffic status data, driver physiological signal data, driver attribute data, and takeover request intensity.

[0046] S2. Obtain cognitive state values ​​through driver physiological signal data and construct context categories; in each context category, conduct repeated trials of each preset strategy and collect takeover trajectories; by comparing the takeover trajectories of each strategy, obtain the strategy preference probability of each strategy under each context category, which serves as a process-level preference soft label.

[0047] S3. Discretize the vehicle traffic state to obtain local states. Under the same context category and local states, filter trajectory pairs from the takeover trajectory data and calculate the takeover time difference for each trajectory pair. Calculate the time step layer preference soft label based on the takeover time difference.

[0048] S4. Personalized dense rewards are obtained based on process layer preference soft labels and time step layer preference soft labels, and multi-head rewards based on context category are trained.

[0049] S5. Construct a reinforcement learning network, calculate immediate rewards based on the trained multi-head reward system, and train the reinforcement learning network based on the immediate rewards.

[0050] Through the training steps described above, this embodiment elevates the control of takeover request intensity from single-trigger or phased rule control in existing technologies to a sequential Markov decision process with driver attributes and real-time cognitive states as contextual conditions. It utilizes a multi-headed reward model with joint constraints of process-level preferences and time-step-level preferences to transform the implicit optimal selection patterns of different drivers regarding takeover request intensity under different traffic scenarios and cognitive states into a learnable, personalized, dense reward function. This drives the reinforcement learning network to achieve real-time, continuous, and adaptive optimization of takeover request intensity in actual operation. Compared with existing technologies, the method in this embodiment achieves a transformation from static rules to sequential optimization at the decision modeling level, a transformation from experience-based heuristics to identifiable preference learning at the reward modeling level, and a transformation from offline setting to online closed-loop adjustment at the control level. Therefore, it has advantages such as high personalization, strong stability, and high efficiency.

[0051] like Figure 2 The diagram illustrates the data flow of the reinforcement learning network training process. Specifically, vehicle traffic state data includes the collision time between the vehicle and the obstacle in front; driver attribute data includes the driver's age and driving experience; and driver physiological information includes continuous time-step physiological signals of head pitch angle, head orientation, pupil diameter, and eye opening / closing ratio.

[0052] To characterize the differences in takeover requests among drivers of different ages and driving experiences when constructing context categories, this embodiment categorizes drivers based on two dimensions of driver attribute data, resulting in four context categories. The construction of context categories includes: classifying driver age and driving experience into high and low levels respectively, and combining these levels to obtain four context categories: high age with high driving experience, high age with low driving experience, low age with high driving experience, and low age with low driving experience. In this embodiment, drivers over 24 years old are classified as high age, and drivers 24 years and younger are classified as low age. Drivers with more than 3 years of driving experience are classified as high driving experience, and drivers with 3 years or less of driving experience are classified as low age. Furthermore, for any driver, their corresponding context category remains unchanged throughout the takeover process.

[0053] The entire takeover process is a constrained Markov decision process (cMDP), in which the state of the reinforcement learning network... This includes current vehicle traffic status data, takeover request strength, driver takeover time, and context category. The expression is:

[0054] ,

[0055] This state vector describes the vehicle traffic state data at time step t during the takeover process. and the strength of the takeover request And the driver's real-time reaction ability, i.e., takeover time. .

[0056] Among them, vehicle traffic status data The urgency of a collision is quantified using a normalized collision time-to-crush (TTC) transform, with values ​​ranging from 0 to 10 seconds.

[0057] ;

[0058] TOR stimulus intensity as takeover request intensity , is a discrete index defined as eight equally spaced levels.

[0059]

[0060] This embodiment adjusts the volume of the TOR (Total Torque) to change the TOR stimulation intensity, thereby dynamically optimizing the TOR and improving takeover safety during the takeover process. Therefore, the takeover request intensity in this embodiment... The volume of TOR is 10% higher for each level.

[0061] Finally, the takeover time Reflecting the driver's real-time reaction capability, this value is inferred and further discretized through a three-layer LSTM network. The LSTM network is input to a sliding window of driver physiological data collected over ten consecutive time steps to obtain the predicted takeover time. To reduce the impact of prediction errors and decrease the amount of data required for training, the LSTM output is the predicted takeover time. Discrete to takeover time The three levels:

[0062] .

[0063] The actions include takeover request strength enhancement and takeover request strength preservation, expressed as:

[0064] ,

[0065] Among them, the enhanced action will increase the current takeover request strength. A fixed increment of 0.1 is added, while the hold action maintains the takeover request strength unchanged. In this embodiment, no weakening action is set to ensure the warning strength is not reduced, thereby avoiding potential driver confusion.

[0066] The construction of process-level preference soft labels in S2 includes:

[0067] Within the same context category, multiple repeated trials are conducted for each preset strategy to collect takeover trajectories. These trajectories are then divided into several comparison blocks, each containing one takeover trajectory for each strategy. Within each comparison block, all takeover trajectories are compared pairwise. If the corresponding strategy... The takeover time of the takeover trajectory is shorter than that of the strategy. then strategy It won in this comparison; the statistical strategy was effective. The corresponding takeover trajectory wins the most times in all comparison blocks and the total number of comparisons. The strategy preference probability is calculated and Laplace smoothing is performed to obtain the process layer preference soft label.

[0068] Specifically, by comparing the takeover trajectories generated by the preset strategy in pairs, soft labels reflecting the preferences of different driver groups are constructed. Here, the strategy is the rule for dynamically selecting actions in the action space according to the state space during the takeover process; the trajectory is the state-action sequence generated by the driver in a single takeover process, starting from the initial moment, until the termination condition is met.

[0069] Let the driver feature space be a context category The default strategy set is Repeat for each strategy under each context category. In this experiment, a total of [number] samples were collected. This trajectory. Within the same context... The trajectory is divided into equal parts Each comparison block performs pairwise comparisons of the trajectories within it. The comparison blocks are designed to prevent the overall performance from improving due to driver learning effects as the experiment progresses, thus introducing a bias into preference modeling. For the strategy... and strategy If strategy reaction time Less Than Strategy reaction time then strategy It won in this comparison and was recorded. :

[0070] ,

[0071] in, This is an indicator function; it returns 1 if the condition is true and 0 otherwise, which is the strategy. strategy The number of wins under different contrast blocks. After defining the win statistics, based on the Bradley–Terry model, the soft label for each policy pair under each context is calculated and Laplace smoothed.

[0072] The expression for the obtained policy preference probability is:

[0073] ,

[0074] ,

[0075] in, The Laplace smoothing coefficient is... For policy trajectory pairs ( , In context The number of wins This represents the number of times the two strategies are compared. For the strategy under context category C Relative to strategy The probability of preference, For the strategy under context category C Relative to strategy The probability of preference.

[0076] The construction of time step layer preference soft labels in S3 includes:

[0077] The takeover request intensity and collision time in traffic conditions are discretized to form several local intervals. Each local interval includes discretized segments of takeover request intensity and collision time in traffic conditions. Within the same context category and the same local interval, takeover trajectories implementing different strategies are paired to obtain trajectory pairs. In a trajectory pair, one takeover trajectory performs a takeover request intensity enhancement action in the current local interval, while the other takeover trajectory performs a takeover request intensity maintenance action. The takeover time difference of each trajectory pair is calculated, and the takeover time weight is calculated based on the magnitude of the takeover time difference. The weighted number of wins and the total weighted number of wins of all trajectory pairs in each context category and each local interval are accumulated to obtain the time step layer preference soft label.

[0078] Specifically, when calculating the time step layer preference soft label, the collision time TTC in the state, i.e., the time T required for the vehicle to collide with the obstacle in front, and the takeover request intensity are combined. That is, the volume space is discretized into multiple local intervals with a width of w. ), calculate the effect of actions on performance under different local conditions, including:

[0079] ,

[0080] For the same context category The same comparison block N, including the local interval ( ), and a strategy to strengthen takeover requests. The corresponding trajectory, and the local interval are ( And the strategy of keeping the takeover request strength unchanged. The corresponding trajectory. Calculate the takeover time difference between the trajectory pairs:

[0081] ,

[0082] in, Strategies to enhance the strength of takeover requests The reaction time in the kth comparison block, A strategy to maintain the same takeover request strength The reaction time in the k-th comparison block. (Difference in take-off time) This characterizes the extent to which enhanced actions improve driver performance under the same conditions. To amplify the effectiveness of effective enhanced actions, takeover time is weighted according to the degree of improvement in each comparison. :

[0083] ;

[0084] Calculate the weighted win rate and total number of times for different actions (i.e., enhanced or unchanged takeover requests) within each local interval:

[0085] ,

[0086] ,

[0087] in, For context category Lower local interval ( In the comparison block, the weighted average of the degree to which the enhanced action improves driver performance. For context category Lower local interval ( The comparison block count. If That is, the strategy corresponding to the trajectory that performs the augmentation action. The takeover time is shorter than the strategy corresponding to the trajectory of the execution and maintenance actions. At this point, the enhancing action is determined to win in this comparison, and it is considered that the enhancing action is more effective than the maintaining action in this local state; otherwise, the maintaining action wins; if it equals 0, it is considered a draw.

[0088] Based on the winning statistics and Laplace smoothing, the soft labels of the time step layer preferences are obtained:

[0089] ,

[0090] ,

[0091] in, The Laplace smoothing coefficient is... For context category Below, in a local state interval At the same time, enhance the strategy corresponding to the action. Compared to strategies that maintain the corresponding actions The probability of preference, For context category Below, in a local state interval At the same time, maintain the strategy corresponding to the action. Relative to the strategy of enhancing actions The preference probability reflects the relative advantage of the enhanced action compared to the maintained action in each local interval. The closer it is to 1, the shorter the driver takeover time is in that interval.

[0092] Multi-head reward includes a shared feature encoder and several reward prediction heads corresponding to context categories, such as Figure 3 As shown. The shared portion of the multi-head reward is based on the input vehicle traffic state data. Intensity of takeover request And the driver's real-time reaction ability, i.e., takeover time. Uniform encoding is performed, and the reward header is based on each context category. The output is a differentiated, personalized reward value. Both the feature extractor and the reward head consist of a linear layer, a normalization layer, and an activation function layer.

[0093] The training process for multi-reward training includes:

[0094] At the process layer, the cumulative return difference of each policy pair is calculated, and the predicted policy preference probability is obtained through the Sigmoid function; the predicted policy preference probability is made to approximate the process layer preference soft label by minimizing the cross-entropy loss function.

[0095] At the time step level, the time step level preference soft label is converted into log odds form; the reward difference between the takeover request strength enhancement action and the takeover request strength maintenance action under the same context category is calculated; the reward difference is approximated to the log odds label of the time step level preference soft label by minimizing the mean squared error loss function.

[0096] Multi-head reward training is conducted using personalized dense rewards derived from predicted policy preference probabilities and reward differences.

[0097] The reward for each trajectory The expression is:

[0098] ,

[0099] in, As a discount factor, For the strategy trajectory, These are the network parameters of the reward model. For policy pairs ( , The obtained policy preference probability is:

[0100] ,

[0101] in, The function is a sigmoid function, and the policy preference probability is given by the parameter. The reward model for the strategy strategy The probabilistic representation at the driver preference level. The corresponding minimization of cross-entropy loss function. The expression is:

[0102] ,

[0103] in, To represent the distribution across all trajectory pairs, Indicates the context type and strategy ( , Separate sampling is performed. By minimizing the cross-entropy loss, it is ensured that the reward value output by the reward model fits the driver trajectory preferences of different context categories at the process level.

[0104] Time step layer prefers soft labels Converted to log odds to reflect the strength of preference for enhancement relative to maintenance in a given state, the log odds are:

[0105] ,

[0106] Calculate the output difference of different actions in the same state The parameters obtained are The reward model expresses the relative advantage probability of the reinforcement action in the current local interval and uses the mean squared error loss function. Minimize the differences between them:

[0107] ,

[0108] ,

[0109] in, The empirical sampling distribution is the state distribution. This indicates different context types. Sample the state s separately. By minimizing the error between the step-level output difference and the log-odds label, we ensure that each step can accurately evaluate the effect of using different actions in different states, thereby achieving a higher reward value for the state-action pair output that effectively shortens the driver takeover time.

[0110] Finally, this embodiment obtains a personalized dense reward that has a good fit to the preferences of both the trajectory layer and the time step layer by weighting and summing the loss functions according to a set weight. :

[0111] ,

[0112] in, This is a hyperparameter that controls the weights of trajectory-level and step-level losses.

[0113] In S5, after multi-head reward training, the Dueling DQN architecture is used for reinforcement learning network training, with the following parameters: θ At each training time step, the current action is computed using multi-head reward, based on the current state, context category, and actual action performed. Predicted rewards and alternative actions Predicted rewards The difference between the predicted reward of the current action and the predicted reward of the alternative actions is used as the instant reward signal.

[0114] Received context category The action value function is as follows:

[0115] ,

[0116] Action value function Indicates in context category Below, the state at time step t. Use action Expected returns that can be obtained; In context Below, in state Execute action The instant reward obtained afterward; This is a discount factor used to measure the importance of future rewards relative to immediate rewards; For the next state Next, for all possible actions The action value function takes the maximum value.

[0117] Subsequently, an action-value function is constructed under contextual conditions, and a target-oriented network (with parameters) is used. The parameters are updated using the time-difference method. The parameters are... The reinforcement network is used to learn the policy, and the parameters are... The target network is used as a delayed copy to compute a relatively stable temporal difference objective. During training, environmental interaction data is written to the experience replay buffer and randomly sampled to reduce sequence correlation. The target network uses a soft update approach in each batch to improve learning stability, expressed as:

[0118] ,

[0119] The update parameter m = 0.01 ≪ 1. Actions are selected using a greedy strategy; for each mini-batch, a one-step time difference objective can be calculated. :

[0120] ,

[0121] in, To perform the action at time step t The actual reward received later For target network For the next state All possible actions The Q value is taken as the maximum value.

[0122] Then, the reinforcement learning parameters θ are updated by minimizing the mean square Bellman error:

[0123] ,

[0124] in, It is a one-step time difference objective. This represents the difference between the current expected return and the maximum expected return of the reinforcement learning model, and is used to update the model parameters.

[0125] Through the training process described above, the resulting policy network can adjust the takeover request strength in real time based on dynamic changes in traffic conditions, TOR strength, and cognitive state throughout the entire process from TOR issuance to successful driver takeover. This ultimately achieves fine-grained, dynamic, and personalized takeover request optimization for different driver groups, improving takeover efficiency and safety.

[0126] In the multi-head reward model of this embodiment, the reward function of a single reward head depends on the state. ,action and context categories C Together, we can determine that a parameterized reward function in the form of a neural network can be used, for example... Figure 3 The reward function shown, consisting of a shared feature extractor and a context-conditional reward head, is expressed as follows:

[0127]

[0128] in, These represent the parameters of the multi-reward model. Shared feature generators are used for feature encoding of state-action pairs. For context category C The corresponding multi-head reward output is used to output the corresponding scalar reward value in the feature space, thereby forming a context-conditional single-head reward function.

[0129] This embodiment introduces a contextual Markov decision process (cMDP) to uniformly model the autonomous driving takeover process. Individual attributes such as driver age and experience, along with real-time cognitive state, are used as context variables, and these variables are incorporated into the same decision framework along with the traffic scenario state and the intensity of the takeover request cues for parameterized description. Through this modeling approach, the takeover process is no longer viewed merely as a physical control problem of "environment-action-outcome," but rather as a dynamic system coupled with "traffic situation-cue stimulus-cognitive evolution-behavioral response." Structurally, this solves the problem that traditional MDPs cannot conditionally express individual driver differences or explicitly characterize the dynamic process of cognitive state evolution over time. Therefore, the method in this embodiment can simultaneously perceive changes in traffic urgency, cue intensity, and driver cognitive state within a unified state space, and form different optimal decision mapping relationships based on the driver's context. This enables the modeling capability of different cue intensity adjustment strategies for the same traffic scenario under different user groups and different cognitive stages. It transforms the takeover request from a static threshold trigger or fixed-level control into a continuous modulation process aligned with the driver's cognitive dynamics, providing a personalized and consistent state representation foundation for subsequent strategy optimization.

[0130] Furthermore, this embodiment constructs a two-layer preference reward model adapted to the aforementioned cMDP to learn the implicit optimal criterion for adjusting takeover request intensity from real takeover experimental data. Addressing the lack of analytical formulas or rule models in existing technologies that can quantitatively describe "what level of alert intensity should be used for which type of driver under what traffic conditions and cognitive state," and the problems of takeover performance being affected by fluctuations in human performance leading to unidentifiable preferences and high sensitivity to initialization in traditional single-layer preference-based PbRL reward learning, this embodiment decomposes preferences into overall performance preferences at the entire takeover process layer and local marginal contribution preferences at the takeover time step layer. It trains the reward model through joint loss constraints, making the reward function structurally learnable and reproducible. Therefore, the reward function in this embodiment no longer relies on human experience design but stably recovers the intrinsic mapping relationship between "traffic scenario—alert intensity—cognitive state—takeover performance" from experimental data. This eliminates the multiple solutions and unstable convergence problems caused by performance fluctuations and random initialization, ensuring that the reward structure learned in different training batches remains consistent, thereby inducing a stable and non-volatile takeover request adjustment strategy. In the field of autonomous driving, the fuzzy and sparse takeover optimization criteria, which originally relied solely on empirical rules or single terminal indicators, are transformed into a dense and quantifiable evaluation function output by a reward model. This function continuously assesses the matching degree between traffic scenarios, prompt intensity, and driver cognitive state, thereby providing a stable, learnable, and personalized decision-making basis for the sequential optimization of subsequent takeover request intensity. Furthermore, this data-driven, two-layer preference-identifiable reward modeling paradigm can be extended to other scenarios with strong human-machine interaction, such as forward collision warnings and control handover, providing a stable, reproducible, and engineerable general technical foundation for human-machine cognitive alignment and collaborative control in complex traffic situations.

[0131] This embodiment models the takeover request intensity adjustment as a sequential decision-making process, rather than a traditional one-time triggering or phased rule-based control problem. Based on this, a reinforcement learning agent is constructed, with the takeover request intensity as the action space and the traffic environment state, takeover process state, and driver's real-time cognitive state as joint inputs. Personalized dense rewards learned from a multi-head reward model are used as the optimization objective for end-to-end training of the strategy. In this way, the takeover request is no longer a static decision of "when to issue a prompt," but rather a dynamic modulation process that continuously operates throughout the entire time interval of "prompt issuance—cognitive awakening—attention shift—control takeover." This allows the system to continuously output prompt intensity adjustment actions at each time step based on traffic evolution and changes in driver cognitive state, thus forming a closed-loop human-machine collaborative control strategy. Therefore, the takeover request is upgraded from a traditional single-decision or fixed-phase switching mode to a continuous control problem oriented towards cognitive dynamics, enabling the system to track and respond to the evolution of the driver's cognitive state in real time, rather than setting the intensity only once at the start of the takeover. In the field of autonomous driving, this sequential decision-making reinforcement learning control mechanism achieves true dynamic TOR (Total Disturbance Response)—that is, the intensity of the prompt can gradually increase with changes in traffic urgency and can also adaptively adjust as the driver's real-time cognitive state improves, thus forming a stable equilibrium range between "insufficient arousal" and "excessive disturbance." If TOR is not modeled as a sequential decision-making process but is still regarded as a single or discrete-stage decision, the temporal evolution information of the cognitive state cannot be utilized. The system can only make static responses based on instantaneous observations, and cannot align and control the dynamic process of the cognitive state during the takeover process, nor can it continuously optimize the prompt intensity trajectory throughout the entire takeover time window. Therefore, it is difficult to achieve stable, personalized, and safe optimized takeover request adjustment effects.

[0132] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0133] The electronic device of this invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0134] Multiple components in the device are connected to an I / O interface, including: input units such as a keyboard, mouse, etc.; output units such as various types of displays, speakers, etc.; storage units such as disks, optical disks, etc.; and communication units such as network interface cards, modems, wireless transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks. The processing unit performs the various methods and processes described above, such as the method of the present invention. For example, in some embodiments, the method of the present invention may be implemented as a computer software program tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or the communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the method of the present invention described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute the method of the present invention by any other suitable means (e.g., by means of firmware).

[0135] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0136] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0137] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0138] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for dynamically adjusting the intensity of takeover requests in accordance with the driver's cognitive state, characterized in that, The method collects multi-dimensional takeover data and uses a reinforcement learning network to obtain a dynamic adjustment result for the strength of the current driver's takeover request. The training steps of the reinforcement learning network include: S1. Collect historical multi-dimensional takeover data, including historical vehicle traffic status data, driver physiological signal data, driver attribute data, and takeover request intensity. The vehicle traffic status data includes the collision time between the vehicle and the obstacle in front; the driver attribute data includes the driver's age and driving experience; the driver physiological signal data includes continuous time-step physiological signals of head pitch angle, head orientation, pupil diameter, and eye opening / closing ratio. S2. Obtain cognitive state values ​​through driver physiological signal data and construct context categories; in each context category, conduct repeated trials of each preset strategy and collect takeover trajectories; by comparing the takeover trajectories of each strategy, obtain the strategy preference probability of each strategy under each context category, which serves as a process-level preference soft label. The construction of the context categories includes: dividing the driver's age and driving experience into two levels, high and low, and combining them according to the levels to obtain four context categories, including: high age and high driving experience, high age and low driving experience, low age and high driving experience, and low age and low driving experience. S3. Discretize the vehicle traffic state to obtain local states. Under the same context category and local state, filter trajectory pairs from the takeover trajectory data and calculate the takeover time difference of each trajectory pair. Calculate the time step layer preference soft label based on the takeover time difference. S4. Based on the process layer preference soft label and the time step layer preference soft label, obtain personalized dense reward, and train the multi-head reward based on context category; S5. Construct a reinforcement learning network, calculate the immediate reward based on the trained multi-head reward, and train the reinforcement learning network based on the immediate reward; the state of the reinforcement learning network includes the current vehicle traffic state data, the takeover request strength, the driver's takeover time and context category; the actions include increasing the takeover request strength and maintaining the takeover request strength.

2. The method for dynamically adjusting the intensity of takeover requests to align with the driver's cognitive state according to claim 1, characterized in that, The construction of the process layer preference soft tags includes: Within the same context category, multiple repeated trials are conducted for each preset strategy to collect takeover trajectories. These trajectories are then divided into several comparison blocks, each containing one takeover trajectory for each strategy. Within each comparison block, all takeover trajectories are compared pairwise. If the corresponding strategy... The takeover time of the takeover trajectory is shorter than that of the strategy. then strategy Wins in the current comparison; statistical strategy is effective. The corresponding takeover trajectory wins the most times in all comparison blocks and the total number of comparisons. The strategy preference probability is calculated and Laplace smoothing is performed to obtain the process layer preference soft label.

3. The method for dynamically adjusting the intensity of takeover requests to align with the driver's cognitive state according to claim 1, characterized in that, The construction of the time step layer preference soft labels includes: The takeover request intensity and collision time in traffic conditions are discretized into several local intervals, each containing a discretized segment of the takeover request intensity and collision time in traffic conditions. Within the same context category and the same local interval, takeover trajectories implementing different strategies are paired to obtain trajectory pairs. In each trajectory pair, one takeover trajectory performs a takeover request intensity enhancement action in the current local interval, while the other performs a takeover request intensity maintenance action. The takeover time difference between each trajectory pair is calculated, and the takeover time weight is calculated based on the magnitude of the takeover time difference. The time step layer preference soft label is obtained by accumulating the weighted winning count and total weighted count of all trajectory pairs in each context category and each local interval.

4. The method for dynamically adjusting the intensity of takeover requests to align with the driver's cognitive state according to claim 1, characterized in that, The multi-head reward includes a shared feature encoder and several reward prediction heads corresponding to context categories; The training process for multi-reward training includes: At the process layer, the cumulative return difference of each policy pair is calculated, and the predicted policy preference probability is obtained through the Sigmoid function; the predicted policy preference probability is made to approximate the process layer preference soft label by minimizing the cross-entropy loss function. At the time step level, the time step level preference soft label is converted into log odds form; the reward difference between the takeover request strength enhancement action and the takeover request strength maintenance action under the same context category is calculated; the reward difference is approximated to the log odds label of the time step level preference soft label by minimizing the mean squared error loss function. Multi-head reward training is conducted using personalized dense rewards derived from predicted policy preference probabilities and reward differences.

5. The method for dynamically adjusting the intensity of takeover requests to align with the driver's cognitive state according to claim 1, characterized in that, In step S5, at each training time step, the current action is calculated using the multi-head reward based on the current state, context category, and actual action executed. Predicted rewards and alternative actions Predicted rewards The difference between the predicted reward of the current action and the predicted reward of the alternative actions is used as the immediate reward signal. To enhance the state of the learning network, For context category.

6. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images