Method and system for constructing reinforcement learning reward function based on multi-modal physiological signals

Abstract

The application discloses a reinforcement learning reward function construction method and system based on multi-modal physiological signals, and belongs to the technical fields of automatic driving, artificial intelligence and human-computer interaction. The application solves the problem that the existing reward function design has a fundamental defect, that is, it is impossible to balance the task target and the physiological comfort of a driver or a passenger. The core technical idea lies in a two-stage decoupling architecture. In a first stage (an offline training stage), the purpose of the stage is to train a "physiological reward agent model". In a second stage (a reward generation and deployment stage), in the stage, real-time dependence on all multi-modal physiological signals and their acquisition devices is completely abandoned. Only the "physiological reward agent model" trained in the first stage is used to batch process any data set containing only vehicle state data and driving behavior data, infer the physiological stress level, and calculate the physiological reward according to the physiological stress level.

Description

Technical Field

[0001] This invention belongs to the fields of autonomous driving, artificial intelligence and human-computer interaction technology, and specifically relates to a method and system for constructing a reinforcement learning reward function based on multimodal physiological signals. Background Technology

[0002] As autonomous driving technology evolves to higher levels (L4 / L5), achieving "human-like" intelligent decision-making that passengers can trust has become a research hotspot. Reinforcement learning (RL), as a core method for realizing advanced autonomous driving decision-making, learns control strategies through the interaction between the agent and the environment. In this process, the design of the reward function is crucial, as it directly guides the agent's learning direction and the quality of the final strategy.

[0003] However, existing reinforcement learning methods for autonomous driving still face significant bottlenecks in the design of reward functions.

[0004] (1) In terms of reward function design, existing methods typically focus only on task-oriented objectives, such as lane keeping, speed tracking, or collision avoidance. This design seriously neglects the impact of driving behavior on the real-time physiological and psychological state (such as comfort and stress level) of the driver or passengers. This results in a rigid and mechanical driving style, poor passenger experience, and a lack of "humanity," which in turn leads to a crisis of trust between humans and machines and low passenger acceptance.

[0005] (2) In terms of quantifying physiological state, existing technologies struggle to convert the driver’s real-time physiological state into a stable, reliable, and usable reward signal. On the one hand, real-time acquisition of physiological signals (such as EDA skin conductance and ECG heart rate) requires additional physiological sensors, which results in high deployment costs and inconvenience in practical applications.

[0006] (3) In terms of signal processing and application, the collected physiological signals have inherent defects of high noise and high delay. Current technologies lack the ability to integrate these high-noise, high-delay signals with the real-time vehicle status. Effective correlation methods are difficult to translate into stable and reliable reward signals. This limitation results in low practicality, insufficient accuracy and stability, and poor robustness of physiological state-based reward schemes in engineering practice, which is a technical problem that urgently needs to be solved in this field.

[0007] In summary, the design of existing reward functions has a fundamental flaw: they cannot achieve a balance between task objectives and the physiological comfort of drivers and passengers. This limitation stems from the difficulty in quantifying physiological comfort, particularly the high cost of physiological sensors and the low quality of the signals themselves (high noise, high latency).

[0008] Therefore, in order to meet the need to build more "human-like" and more comfortable autonomous driving strategies, how to overcome the dependence on physiological sensors, reduce deployment costs, and reliably incorporate the driver's physiological state into the reward function has become a key technical challenge that urgently needs to be overcome in the fields of autonomous driving and artificial intelligence. Summary of the Invention

[0009] To address the aforementioned technical problems, this application provides a method and system for constructing a reinforcement learning reward function based on multimodal physiological signals.

[0010] The method includes the following steps:

[0011] S1. Multimodal data acquisition and storage: Real-time acquisition of vehicle status data. Driving behavior data and multimodal physiological signals ;

[0012] S2. Multimodal physiological stress assessment: based on multimodal physiological signals Output physiological stress score ;

[0013] S3. Construction and Training of a Physiological Reward Agent Model: Constructing a model based on vehicle state data. and driving behavior data It can infer the predictive physiological stress score The physiological reward agent model, with training labels as ;

[0014] S4. Physiological Stress Inference: Generating and... using a trained physiological reward proxy model. Corresponding predicted physiological stress score ;

[0015] S5, Physiological Rewards Calculation: The inverse mapping function is used to predict physiological stress scores. Transformed into physiological rewards within a set range ;

[0016] S6, Task Rewards Calculation: Based on vehicle status data Construct a multi-objective weighted reward function to calculate task rewards. ;

[0017] S7, Mixed Rewards: Combining task rewards and physiological rewards The final reward is obtained by performing a weighted summation. ;

[0018] S8. Offline Dataset Construction: This involves constructing the current vehicle status data collected in step S1. Current driving behavior data and vehicle status data at the next moment and the final reward calculated in step S7 Aligned according to time steps, and combined into The quadruple is stored as a dataset file, and the offline reinforcement learning dataset is output.

[0019] S9. Policy Training: Using an offline reinforcement learning dataset, an autonomous driving policy model that incorporates driver physiological preferences is trained using an offline reinforcement learning algorithm.

[0020] Furthermore, vehicle status data Including vehicle speed Longitudinal acceleration lateral acceleration Vehicle center of gravity position, lane centerline position, longitudinal relative distance between the vehicle and the vehicle in front. and the longitudinal relative speed between the vehicle and the vehicle in front Driving behavior data Including accelerator pedal position, brake pedal position, and steering wheel angle; multimodal physiological signals Including attention area signals Seat pressure signal Heart rate variability signal and skin electrical signals .

[0021] Furthermore, based on multimodal physiological signals Output physiological stress score Specifically:

[0022] S31, Data Buffering and Synchronization: Start a timed synchronization analyzer for each signal to extract all data within the time window from each buffer queue;

[0023] S32. Feature Extraction and Standardization: Transform the data sequence within the time window into a low-dimensional and human-readable statistical text description through statistical analysis.

[0024] S33. Construction of structured prompt words: Concatenate all the statistical text descriptions and add system instructions and preset evaluation rules to construct a complete structured prompt word;

[0025] S34, LLM Zero-Shot Inference: The constructed structured cue words are input into a large language model LLM. Using the zero-shot inference capability of LLM, the physiological stress assessment score is inferred in real time based on the evaluation rules and data details in the cue words.

[0026] S35. Score Parsing: The parser extracts physiological stress assessment scores from the LLM output text to obtain standardized physiological stress scores. .

[0027] Furthermore, the physiological reward proxy model is constructed as follows: a long short-term memory network / gated recurrent unit / Transformer network is used as the basic model, and vehicle state data within a time window is used. and driving behavior data Physiological stress scores for the corresponding time window were used as training input. The labels are used as training tags for iterative training until a physiological reward proxy model is obtained.

[0028] Furthermore, the reverse mapping function is specifically as follows:

[0029] ;in, This represents the maximum physiological reward baseline value, which is the highest reward score the agent can obtain when the predicted physiological stress score is 0. This represents the physiological stress penalty coefficient, which determines the extent to which rewards decrease as stress increases.

[0030] Furthermore, calculate task rewards. Specifically:

[0031] ,in, , , and These are the weighting coefficients, For speed rewards, As a comfort reward, Lane keeping reward As a safety reward;

[0032] ,in, For the preset target speed, This is the preset normalization scaling constant;

[0033] ,in, The preset maximum comfort acceleration threshold;

[0034] ,in, To obtain vehicle status data The lateral offset between the vehicle's center of gravity and the lane centerline obtained from the data. This is the preset reward decay coefficient;

[0035] ,in, and The preset penalty coefficient, when hour, ;when hour, Let it be infinity.

[0036] Furthermore, the offline reinforcement learning algorithms are ConservativeQ-Learning algorithm, ImplicitQ-Learning algorithm, and TD3+BC algorithm.

[0037] The system includes a data acquisition module, a label definition module, a model training module, a reward calculation module, a reward mixing module, a dataset construction module, and a policy training module;

[0038] Data acquisition module: Used for multimodal data acquisition and storage, including interfaces for connecting to the CAN bus, the DMS camera, the rPPG sensor, the seat pressure sensor, and the EDA sensor for electrodermatology (EDA), used to acquire vehicle status data. Driving behavior data and multimodal physiological signals ;

[0039] Tag definition module: using multimodal physiological signals Input: Physiological stress score It includes a data caching and synchronization submodule, a feature extraction and standardization submodule, a structured prompt word construction submodule, and an LLM zero-shot inference and parsing submodule;

[0040] Model training module: This module incorporates a Long Short-Term Memory (LSTM) network / gated recurrent unit / Transformer network structure and corresponding training procedures for training physiological reward agent models. , and The input is the physiological reward proxy model, and the output is the trained physiological reward proxy model.

[0041] The reward calculation module loads a trained physiological reward proxy model to perform physiological stress inference and includes a built-in inverse mapping function and a multi-objective weighted reward function; the output is a physiological stress score. Physiological rewards Calculation and task rewards ;

[0042] Reward Mixing Module: Its built-in weighted summation algorithm is used for mixing. and Receive the final reward ;

[0043] Dataset building module: used to build current vehicle status data Current driving behavior data and vehicle status data at the next moment and final reward Package and store it as a standard dataset file;

[0044] Policy Training Module: Built-in offline reinforcement learning algorithm, used to load datasets and train the final autonomous driving policy.

[0045] The beneficial effects of the method described in this invention are as follows:

[0046] This method firstly addresses the challenge of fusing and quantifying multimodal heterogeneous physiological signals, improving the accuracy and robustness of physiological labels. Specifically, existing technologies struggle to handle the heterogeneous (strings, arrays, floating-point numbers) and frequency (20Hz, 15Hz, 50Hz) characteristics of multimodal physiological signals (such as DMS, rPPG, and seat pressure), which are susceptible to noise interference. This method, through the technical solution in step S2, innovatively proposes a "feature extraction-text fusion-LLM zero-shot inference" architecture. This scheme first uses statistical analysis to reduce the dimensionality of high-frequency raw data and translate it into low-dimensional, human-readable statistical text; then, it concatenates this text with pre-defined evaluation rules ("prior knowledge") to form structured prompts; finally, it utilizes the powerful contextual understanding and zero-shot inference capabilities of Large Language Models (LLM) to act as an expert system for scoring. The technical solution in step S2 replaces the traditional complex and error-prone signal processing model, providing a highly robust and flexible way to define high-dimensional physiological pressure labels. .

[0047] The second aspect of this method addresses the high cost, high invasiveness, and low reliability issues associated with the deployment of physiological sensors, offering high practicality, versatility, and deployability. Specifically, existing technologies require the permanent installation of a complete set of expensive and complex physiological sensors (such as DMS cameras, rPPG sensors, and seat pressure matrices) in mass-produced vehicles to introduce physiological signals, which is impractical in terms of cost, hardware, and user acceptance. This method completely solves this problem through a two-stage decoupled architecture in steps S3 and S4. This method only uses sensors in stage one (offline training). A physiological reward proxy model is trained using the first step (S3). In Phase Two (Deployment Phase), Step S4 uses only this proxy model to infer the physiological reward by inputting vehicle state and driving behavior data available to any vehicle. This allows the method to completely eliminate the real-time dependence on all expensive and unstable physiological sensors during final deployment, greatly reducing costs and improving the system's practicality and deployability.

[0048] The third aspect of this method enables human-centric empowerment of existing autonomous driving datasets, significantly expanding the algorithm's application scope. Specifically, existing massive autonomous driving datasets (such as Waymo and nuScenes) and most real-world vehicle data only contain vehicle state and driving behavior data, lacking physiological data. This makes them unsuitable for training human-like, comfortable driving strategies. Due to its two-stage decoupled architecture (the advantage of the second aspect), the physiological reward proxy model trained in this method can be applied to any existing dataset containing only vehicle state and driving behavior data. Step S4 of this method can retrospectively or implicitly generate physiological rewards for these massive existing datasets, greatly expanding the application value of this method and enabling human-centric empowerment of existing datasets.

[0049] The fourth aspect of this method enables autonomous driving strategies to balance task completion and driving comfort, significantly improving human-likeness and user experience. Specifically, existing technologies focus solely on task rewards (such as lane keeping and collision avoidance), resulting in rigid, mechanical driving styles and a poor passenger experience. This method, through steps S7 (reward mixing) and S8 (dataset construction), weighted mixes objective task rewards with a uniquely inferred subjective physiological reward. When the downstream reinforcement learning strategy step S9 is trained using the dataset constructed using this method, it naturally learns to avoid driving behaviors that increase human stress (i.e., low physiological reward) (such as rapid acceleration and sharp turns), thereby training a smoother, more comfortable, and more human-like driving strategy, solving the problems of poor user experience and low trust levels in existing technologies. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the main flow of the method described in the embodiments of the present invention;

[0051] Figure 2 This is a schematic diagram of multimodal physiological stress assessment in an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of the training process of the physiological reward agent model in an embodiment of the present invention;

[0053] Figure 4 This is a structural block diagram of the system described in an embodiment of the present invention. Detailed Implementation

[0054] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0055] Example 1

[0056] This embodiment provides a method for constructing a reinforcement learning reward function based on multimodal physiological signals, such as... Figure 1 As shown, this method is a multi-stage process. The core technical idea of ​​this invention lies in its two-stage decoupling architecture:

[0057] Phase 1 (Offline Training Phase): The sole purpose of this phase is to train a surrogate model. In this phase, the system utilizes the complete set of multimodal physiological signals collected in step S1. (e.g., DMS, rPPG, seat pressure, EDA) and step S2 (multimodal physiological stress assessment module) to define a high-precision, high-dimensional true physiological stress score. Subsequently, the score... These labels are used as supervisory labels to train a physiological reward surrogate model (step S3). The core function of this surrogate model is to learn... arrive The mapping relationship.

[0058] The second stage (reward generation and deployment stage): This stage is where the key advantage of this invention lies. In this stage, the system completely abandons the real-time dependence on all multimodal physiological signals and their acquisition devices (such as cameras, seat sensors, etc.). The system only uses the physiological reward agent model trained in the first stage to batch process any dataset containing only vehicle state data and driving behavior data, infer the physiological stress level, and calculate the physiological reward accordingly.

[0059] This decoupled design, where training is done once and application is everywhere, enables this method to be used for any application lacking [specific features]. The existing datasets of autonomous driving data (and even real-vehicle driving data) empower the system, implicitly injecting human factors engineering rewards into it.

[0060] The purpose of this method is as follows:

[0061] (1) In order to solve the problem that existing reward functions only focus on the task and ignore human subjective feelings, this invention aims to provide a method to quantify the driver's real physiological state (stress level) into a physiological stress label and use it as a supervision signal to construct a physiological reward, so that the reward function of reinforcement learning can take into account both the task goal and human subjective feelings.

[0062] (2) In order to solve the problems of high cost, poor practicality, large signal noise and delay in real-time acquisition of physiological signals, this invention cleverly realizes the mapping from vehicle state and driving behavior to predicted physiological stress level by training a reward prediction model (step S3).

[0063] (3) To improve the deployability, practicality, and robustness of the solution, the purpose of this method is to enable the autonomous driving strategy to completely eliminate its real-time dependence on expensive and unstable physiological sensors during final training and actual deployment by using the above-trained reward prediction model, thereby greatly improving the practicality of the solution. The ultimate goal is to use this reward function to construct an offline dataset on which an autonomous driving strategy with a smoother, more comfortable, and more human-like driving style can be trained.

[0064] Example 2

[0065] This embodiment further defines Embodiment 1 and provides a more detailed description of the method.

[0066] Step S1: Multimodal data acquisition

[0067] Functional purpose: To serve as the original data source for all subsequent steps.

[0068] Technical solution: In real or high-fidelity simulated driving scenarios, the following three types of data are simultaneously collected and stored through onboard sensors and data acquisition equipment:

[0069] (1) Vehicle status data Data is collected via the vehicle's Controller Area Network (CAN) bus. In a preferred embodiment, the... Including: vehicle speed Longitudinal acceleration lateral acceleration Vehicle center of gravity position, lane center line position, and information on obstacles ahead (including relative distance to the vehicle in front). Relative speed to the vehicle in front )wait.

[0070] (2) Driving behavior data The driver's actual control output. In a preferred embodiment, This includes: the position of the accelerator pedal, the position of the brake pedal, and the angle of the steering wheel.

[0071] (3) Multimodal physiological signals ): Including but not limited to:

[0072] Attention area signals : Acquired through the Driver Monitoring System (DMS), which processes camera images and then publishes attention area strings (e.g., "ahead", "left rearview mirror", "phone") at a high frequency (e.g., 20Hz).

[0073] Seat pressure signal : A one-dimensional pressure array (e.g., an array of 108 elements) is collected by a pressure sensor matrix integrated in the driver’s seat and published at a medium frequency (e.g., 15 Hz).

[0074] Heart rate variability signal The instantaneous HRV floating-point values ​​are preferably acquired by a non-contact optical sensor based on remote photoplethysmography (rPPG) and published at a high frequency (e.g., 50 Hz).

[0075] Electrodermal signals Skin conductance (EDA) values ​​are collected at a higher frequency (e.g., 30 Hz) through wearable physiological sensors (such as smart bracelets) or steering wheel-integrated electrodes.

[0076] Step S2: Multimodal physiological stress assessment (label definition)

[0077] Functional Purpose: This is the first core inventive point of this method. Its function is to receive heterogeneous and different frequency signals from step S1. The signal outputs a standardized "physiological stress score". This score is considered the "true label" for subsequent training.

[0078] Technical solutions: such as Figure 2 As shown, it includes the following sub-steps:

[0079] S2-1: Data caching and synchronization: To solve the inter-frequency problem, for each signal ( , , , Set up independent data cache queues (e.g., deque). Start a timed synchronization analyzer (in a preferred embodiment, triggered once every 1 second, with the analysis time window defined as [t-1, t]s), and extract all data within that time window from each cache queue.

[0080] S2-2: Feature Extraction and Standardization (Textualization): The high-frequency raw data sequences within the window are transformed into low-dimensional, human-readable statistical text descriptions through statistical analysis. For example, ... Convert the string sequence into "frequency distribution" text; Convert the array sequence into the text "mean pressure, fluctuation, symmetry"; Convert the numerical sequence into "mean, trend" text. The numerical sequence is decomposed into a “Tonic” signal representing the baseline arousal level and a “Phasic” signal representing transient stress, and its statistical features (e.g., mean Tonic and peak Phasic frequency) are extracted and converted into text.

[0081] The high-frequency raw data sequence within the time window is converted into low-dimensional, human-readable natural language description text through a pre-defined statistical calculation logic. The specific conversion scheme is as follows:

[0082] (1) Attention area signals deal with:

[0083] Count the number of times each area label (such as "Ahead" or "Mobile Phone") appears within the time window, and calculate the percentage of each area.

[0084] Transformation logic: Generate a text description in the form of "Frequency distribution: Front {P_front}%, Phone {P_phone}%, ...", where This represents the percentage value for the corresponding region.

[0085] (2) Seat pressure signal deal with:

[0086] The mean pressure value (Mean) and standard deviation (Std) are calculated for the pressure sensor matrix data. Simultaneously, the absolute value of the difference between the mean pressure values ​​of the left and right halves is calculated as a "symmetry index".

[0087] Transformation logic: Generate a text description in the form of "Seat Pressure: Mean {val_mean}, Fluctuation {val_std}, Symmetry {val_sym}".

[0088] (3) Heart rate variability signal deal with:

[0089] Calculate the mean of HRV values ​​and the linear regression slope (Trend) within this window.

[0090] Transformation logic: Generate a text description in the form of "HRV Status: Average {val_avg}, Trend {val_slope}".

[0091] (4) Electrodermal signals deal with:

[0092] The EDA signal is decomposed using existing open-source signal processing libraries (such as NeuroKit2) to extract the mean of the tonic component and the peak count of the phasic component.

[0093] Transformation logic: Generate a text description in the form of "EDA Features: Tonic Mean {val_tonic}, Phasic Peaks{count}".

[0094] S2-3: Structured Prompt Construction: Concatenate all the "statistical text descriptions" generated in S2-2, and add system instructions and preset evaluation rules (i.e., "prior knowledge") to construct a complete structured prompt. In a preferred embodiment, the "preset evaluation rules" are encoded in the prompt to guide LLM reasoning. These rules preferably include, but are not limited to:

[0095] Attention rule: A high percentage of “ahead” regions (e.g., >0.8) in the statistical text was mapped to a low physiological stress score; a high percentage of distraction regions such as “phone” or “outside the window” (e.g., >0.15) was mapped to a high physiological stress score.

[0096] Seat pressure rules: In the statistical text, a low “stress fluctuation” (indicating rigid or stiff posture) was mapped to a high physiological stress score; a poor “left-right symmetry” (high value) was mapped to a high physiological stress score.

[0097] Heart rate variability rule: A low “mean” (e.g., <40) in the statistical text maps to a high physiological stress score; a declining “trend” (e.g., <-1.0) maps to a high physiological stress score.

[0098] Skin conductance rules: A high “Tonic mean” or high “Phasic peak frequency” in the statistical text corresponds to a high physiological stress score.

[0099] S2-4: LLM Zero-Shot Reasoning: The structured prompts constructed in S2-3 are input into a "Large Language Model (LLM)" (in a preferred embodiment, a GGUF quantized version of models such as Llama or Qwen). Utilizing the zero-shot reasoning capability of the LLM, it acts as an "expert system," reasoning out a score in real time based on the evaluation rules and data details in the prompts.

[0100] S2-5: Score Parsing: Numerical values ​​are extracted from the LLM output text using a parser (e.g., regular expressions) to obtain standardized physiological stress scores. (For example, continuous values ​​in the interval [1, 10]).

[0101] Step S3: Training the physiological reward agent model

[0102] Functional Purpose: This is the second core inventive point of this method. Its function is to train a lightweight "surrogate model" that can "infer" the complex physiological stress defined by S2 using only vehicle data, thereby eliminating the need for surrogate models. Dependence on sensors.

[0103] Technical solutions: such as Figure 3 As shown. The technical solution for this step is to build and train a time series model.

[0104] Training Input Vehicle status data within a time window ( ) and driving behavior data (All from step S1).

[0105] Training labels Physiological stress score output from step S2 .

[0106] Model Implementation: In a preferred embodiment, the model is a Long Short-Term Memory (LST) network, a Gated Recurrent Unit (GRU) network, or a Transformer network. This is achieved by minimizing the model's predicted values ​​with... The mean squared error (MSE) between the two is used for training.

[0107] Output: A trained physiological reward agent model.

[0108] Step S4: Physiological stress assessment

[0109] Functional Purpose: To generate batches of models trained using S3, similar to... Aligned pressure data stream.

[0110] Technical solution: Load the physiological reward agent model trained in step S3, and use all vehicle state data collected in step S1. and driving behavior data Use it as input for batch inference.

[0111] Output: Full-time inferred physiological stress level sequence.

[0112] Step S5: Physiological Reward calculate

[0113] Functional purpose: To transform "stress level" (high score = bad) into "reward" (high score = good).

[0114] Technical solution: A reverse mapping function is used. This function defines a linear negative correlation between physiological stress scores and physiological rewards.

[0115] The specific formula is as follows:

[0116] ;

[0117] in:

[0118] C represents the maximum physiological reward baseline value, which is the highest reward score that the agent can obtain when the predicted physiological stress is 0 (i.e., the ideal state). In a preferred embodiment, C is set to 1.0.

[0119] This represents the physiological stress penalty coefficient, which determines the extent to which rewards decrease as stress increases.

[0120] Specific implementation (implementation of normalization and inversion):

[0121] In a preferred embodiment, if The original interval is [1, 10]. We need to normalize it to the interval [0, 1] and then reverse it (i.e., 1 corresponds to a high reward and 10 corresponds to a low reward). This is essentially a special case of the aforementioned reverse mapping function.

[0122] The specific calculation logic is as follows:

[0123] First, normalize and then invert:

[0124] After mathematical transformation, this formula is equivalent to:

[0125] That is, corresponding to the above general formula , The situation.

[0126] Alternatively, to simplify calculations, the following normalized form is typically used as the specific implementation of the reverse mapping:

[0127] ;

[0128] Maximum physiological stress score: Represents the highest value that physiological stress can reach under the current assessment system (corresponding to the most tense and uncomfortable state).

[0129] Minimum physiological stress score: Represents the lowest physiological stress level that can be achieved under the current assessment system (corresponding to the most relaxed and comfortable state).

[0130] For example: Suppose the LLM scoring system used is set to output range [1, 10], then: Equals 10, It equals 1.

[0131] Output: Physiological reward .

[0132] Step S6: Task Rewards calculate

[0133] Functional Purpose: To execute in parallel with S2. Based on vehicle state. Calculate objective, task-based rewards to ensure the vehicle completes basic driving tasks.

[0134] Technical solution: To construct a multi-objective weighted reward function:

[0135] ;

[0136] in: , , , These are preset sub-reward weighting coefficients used to balance the importance of different task objectives. In a preferred embodiment, each sub-reward is defined as follows:

[0137] Speed ​​Reward Encourage vehicles to track target speeds. .

[0138] ;

[0139] in: To obtain vehicle status data The current vehicle speed obtained from the data; Set a preset target speed (e.g., the current road speed limit); This is the preset normalization scaling constant.

[0140] Comfort Reward Excessive vertical punishment and horizontal Acceleration.

[0141] ;

[0142] in: and To obtain vehicle status data The longitudinal and lateral accelerations obtained in the process; The preset maximum comfort acceleration threshold (e.g.) )

[0143] Lane keeping reward : Penalize the lateral deviation of the vehicle's center of gravity from the lane centerline .

[0144] ;

[0145] in: To obtain vehicle status data The lateral offset (in meters) between the vehicle's center of gravity and the lane centerline, obtained (e.g., through a lane detection or positioning module). This is the preset reward decay coefficient (constant).

[0146] The relative position information of the vehicle's centroid and the lane centerline is obtained based on existing high-precision positioning or visual lane detection technologies. Specifically, this is achieved through onboard cameras combined with mature lane detection algorithms (such as OpenCV and deep learning segmentation models), or through a combination of high-precision maps and GNSS / RTK positioning. This is common knowledge in the field of autonomous driving.

[0147] Safety Rewards Based on the "time-to-collision" time with the vehicle in front. The penalty is determined by the obstacle information ahead. Specifically, the Time Toll Collection (TTC) is first calculated based on the obstacle information. To obtain vehicle status data The collision time (in seconds) with the vehicle in front is calculated.

[0148] when When (i.e., when close to the vehicle in front)

[0149] ;

[0150] in, The longitudinal relative distance to the vehicle in front, This represents the longitudinal relative speed with the vehicle in front (negative when approaching). It is obtained via millimeter-wave radar or onboard cameras (in conjunction with target detection and ranging algorithms). These sensors directly output a list of obstacles, including fields such as relative distance and relative speed; this is a mature, existing technology.

[0151] when When the vehicle is far enough away from or maintains a safe distance from the vehicle in front, the TTC is set to infinity. Then the reward is calculated:

[0152] ;

[0153] in and This is a preset penalty coefficient (constant). Used to define the magnitude of security penalties. Used to define the urgency of security penalties.

[0154] Step S7: Reward Mixing

[0155] Functional purpose: To integrate objective task rewards with subjective physiological rewards.

[0156] Technical solution: The output of step S6 and the output of step S5 Perform a weighted summation:

[0157] ;

[0158] In a preferred embodiment, the weighting coefficient =0.7, =0.3.

[0159] Output: Final Reward .

[0160] Step S8: Offline Dataset Construction

[0161] Functional purpose: To construct the final output for training downstream algorithms.

[0162] Technical solution: The status collected in step S1 ,action Next moment state The final reward calculated in step S7 Aligned according to time steps, and combined into The quadruple is stored as a dataset file (e.g., HDF5 format).

[0163] Output: Offline reinforcement learning dataset.

[0164] Step S9: Policy Training (Downstream Application, Optional)

[0165] Functional objective: To train a "human-like" autonomous driving strategy using the S8 dataset.

[0166] Technical solution: An "Offline Reinforcement Learning (OfflineRL)" algorithm is employed, and the dataset from step S8 is loaded for training. In a preferred embodiment, the algorithm is CQL (Conservative Q-Learning), IQL (Implicit Q-Learning), or TD3+BC.

[0167] Output: An autonomous driving strategy model that incorporates driver physiological preferences.

[0168] Example 3

[0169] This embodiment provides an alternative to the solution in Embodiment 2.

[0170] In step S2, multimodal heterogeneous signal fusion is achieved based on the technical solution of "feature extraction-text fusion-LLM zero-shot inference" (corresponding to S2-2 to S2-4). The alternative solution can achieve fusion modeling through different technical paths.

[0171] In practical applications, an "end-to-end multimodal fusion model" can also be utilized. For example, a large-scale multimodal neural network (LMM) can be constructed, containing independent encoders for different signals (such as those used for...). A 2D convolutional neural network (2D-CNN) for heatmaps, used for The visual Transformer for images, or ViT, is used for... Temporal one-dimensional convolutional networks 1D-CNN, and for Another set of temporal 1D-CNNs or GRUs. The feature vectors output by these encoders are concatenated or fused using a cross-attention mechanism, and then input into a regression head. Supervised learning is used for fine-tuning to directly output physiological stress scores. .

[0172] Alternatively, a "classic machine learning fusion approach" can be adopted. For example, the numerical statistical features extracted in step S2-2 (such as mean pressure, fluctuation, symmetry, mean HRV, and attention frequency) can be used as numerical feature vectors and input into a traditional machine learning regression model, such as Support Vector Regression (SVR), Gradient Boosting Decision Trees (GBDT, such as XGBoost), or a lightweight Multi-Layer Perceptron (MLP), to train the model. The score.

[0173] Step S3 focuses on the implementation schemes of time series models such as Long Short-Term Memory Network (LSTM), Gated Recurrent Unit (GRU), or Transformer. Other regression models that can learn temporal mappings, such as Temporal Convolutional Network (TCN) or Gradient Boosting Decision Tree (GBDT) mentioned above (1.(2)), are also applicable to the training process of the surrogate model proposed in this invention.

[0174] Step S9 (Downstream Application) focuses on loading the dataset constructed in step S8 using offline reinforcement learning (OfflineRL) algorithms (such as CQL, IQL). The output of this method (i.e., the final reward) is... The same applies to other training paradigms (and offline datasets):

[0175] (1) For example, the “physiological reward agent model” (S4) can run in real time in the vehicle computing unit, based on real-time data... and Data, real-time generation of physiological rewards .Should With task rewards When combined, it can provide real-time, human factors-integrated reward signals for "Online Reinforcement Learning (OnlineRL)" algorithms (such as SoftActor-Critic, SAC, or Proximal Policy Optimization, PPO) to guide the real-time optimization of policies in simulated environments or real vehicles.

[0176] (2) Furthermore, the dataset (S8) can also be used for "Imitation Learning (IL)". For example, the final reward calculated in step S7. It can be used as an evaluation criterion to filter out "expert trajectories" with high rewards (i.e., high task performance and high comfort) from the dataset, and then used to train the "Behavioral Cloning (BC)" model.

[0177] Example 4

[0178] This embodiment provides a reinforcement learning reward function construction system based on multimodal physiological signals to implement the method in Embodiment 2. This system can be deployed on an in-vehicle computing unit, a cloud server, or a combination thereof.

[0179] like Figure 4 As shown, the system includes a data acquisition module, a label definition module, a model training module, a reward calculation module, a reward mixing module, a dataset construction module, and a policy training module.

[0180] Data acquisition module (for performing step S1): Technical implementation: It includes interfaces for connecting to the CAN bus, for connecting to the DMS camera, for connecting to the rPPG sensor, for connecting to the seat pressure sensor, and for connecting to the electrodermal absorption (EDA) sensor, for acquiring data. , and data.

[0181] Tag definition module (used to execute step S2): Technical implementation: It integrates the "data caching and synchronization submodule (S2-1)," "feature extraction and standardization submodule (S2-2)," "structured prompt word construction submodule (S2-3)," and "LLM zero-shot inference and parsing submodule (S2-4, S2-5)" described in Embodiment 1. This module uses... Input, Output .

[0182] Model training module (for performing step S3): Technical implementation: It has a built-in LSTM, GRU, or Transformer network structure and corresponding training procedures (such as backpropagation and optimizers) for training the physiological reward surrogate model. This module uses... , and The input is the trained proxy model, and the output is the surrogate model.

[0183] Reward calculation module (for executing steps S4, S5, and S6): Technical implementation: It loads the surrogate model trained in S3 to perform the inference in step S4; and has built-in... The inverse mapping function (step S5) and The multi-objective reward function (step S6).

[0184] Reward Mixing Module (for executing step S7): Technical Implementation: Its built-in weighted summation algorithm is used for mixing. and get .

[0185] Dataset construction module (for performing step S8, preferred implementation): Technical implementation: used to... The data is packaged and stored as a standard dataset file.

[0186] Policy training module (for executing step S9, preferred implementation): Technical implementation: It has a built-in offline reinforcement learning (OfflineRL) algorithm (such as CQL) to load the dataset of S8 and train the final autonomous driving policy.

[0187] In a preferred embodiment, the label definition module and model training module, due to their large computational requirements, can be deployed on a cloud server for execution; while other modules (such as data acquisition and reward calculation) can be deployed in an on-board computing unit for execution.

Claims

1. A method for constructing a reward function based on multi-modal physiological signals, characterized in that, The method includes the following steps: S1, Multimodal data acquisition and storage: Real-time acquisition of vehicle state data , driving behavior data , and multimodal physiological signals ; S2. Multimodal physiological stress assessment: based on multimodal physiological signals Output physiological stress score Specifically, this includes: data caching and synchronization, feature extraction and standardization, structured cue word construction, LLM zero-shot inference and score parsing; S3. Construction and Training of a Physiological Reward Agent Model: Constructing a model based on vehicle state data. and driving behavior data It can infer the predictive physiological stress score The physiological reward agent model, with training labels as ; S4. Physiological Stress Inference: Generating and... using a trained physiological reward proxy model. Corresponding predicted physiological stress score ; S5, Physiological Rewards Calculation: The inverse mapping function is used to predict physiological stress scores. Transformed into physiological rewards within a set range ; S6, Task Rewards Calculation: Based on vehicle status data Construct a multi-objective weighted reward function to calculate task rewards. ; S7, Mixed Rewards: Combining task rewards and physiological rewards The final reward is obtained by performing a weighted summation. ; S8. Offline Dataset Construction: This involves constructing the current vehicle status data collected in step S1. Current driving behavior data and vehicle status data at the next moment and the final reward calculated in step S7 Aligned according to time steps, and combined into The quadruple is stored as a dataset file, and the offline reinforcement learning dataset is output. S9. Policy Training: Using an offline reinforcement learning dataset, an autonomous driving policy model that incorporates driver physiological preferences is trained using an offline reinforcement learning algorithm.

2. The method for constructing a reinforcement learning reward function based on multimodal physiological signals according to claim 1, characterized in that, Vehicle status data Including vehicle speed Longitudinal acceleration lateral acceleration Vehicle center of gravity position, lane centerline position, longitudinal relative distance between the vehicle and the vehicle in front. and the longitudinal relative speed between the vehicle and the vehicle in front Driving behavior data Including accelerator pedal position, brake pedal position, and steering wheel angle; multimodal physiological signals Including attention area signals Seat pressure signal Heart rate variability signal and skin electrical signals .

3. The method for constructing a reinforcement learning reward function based on multimodal physiological signals according to claim 2, characterized in that, Based on multimodal physiological signals Output physiological stress score Specifically: S2-1, Data Buffering and Synchronization: Start a timed synchronization analyzer for each signal to extract all data within the time window from each buffer queue; S2-2, Feature Extraction and Standardization: The data sequence within the time window is converted into a low-dimensional and human-readable statistical text description through statistical analysis; S2-3, Structured Prompt Construction: Concatenate all the statistical text descriptions and add system instructions and preset evaluation rules to construct a complete structured prompt. S2-4, LLM Zero-Shot Inference: The constructed structured cue words are input into a large language model LLM. Using the zero-shot inference capability of LLM, the physiological stress assessment score is inferred in real time based on the evaluation rules and data details in the cue words. S2-5, Score Parsing: The parser extracts the physiological stress assessment score from the LLM output text to obtain a standardized physiological stress score. .

4. The method for constructing a reinforcement learning reward function based on multimodal physiological signals according to claim 3, characterized in that, The physiological reward proxy model is constructed by using a long short-term memory network / gated recurrent unit / Transformer network as the basic model and vehicle state data within a time window. and driving behavior data The physiological stress score for the corresponding time window was used as the training input. The labels are used as training tags for iterative training until a physiological reward proxy model is obtained.

5. The method for constructing a reinforcement learning reward function based on multimodal physiological signals according to claim 4, characterized in that, The reverse mapping function is as follows: ;in, This represents the maximum physiological reward baseline value, which is the highest reward score the agent can obtain when the predicted physiological stress score is 0. This represents the physiological stress penalty coefficient, which determines the extent to which rewards decrease as stress increases.

6. The method for constructing a reinforcement learning reward function based on multimodal physiological signals according to claim 5, characterized in that, Calculate task rewards Specifically: ,in, , , and These are the weighting coefficients, For speed rewards, As a comfort reward, Lane keeping reward As a safety reward; ,in, For the preset target speed, This is the preset normalization scaling constant; ,in, The preset maximum comfort acceleration threshold; ,in, To obtain vehicle status data The lateral offset between the vehicle's center of gravity and the lane centerline obtained from the data. This is the preset reward decay coefficient; ,in, and The preset penalty coefficient, when hour, ;when hour, Let it be infinity.

7. The method for constructing a reinforcement learning reward function based on multimodal physiological signals according to claim 6, characterized in that, The offline reinforcement learning algorithms are ConservativeQ-Learning, ImplicitQ-Learning, and TD3+BC.

8. A reinforcement learning reward function construction system based on multimodal physiological signals, characterized in that, The system includes a data acquisition module, a label definition module, a model training module, a reward calculation module, a reward mixing module, a dataset construction module, and a policy training module; Data acquisition module: Used for multimodal data acquisition and storage, including interfaces for connecting to the CAN bus, for connecting to the driver monitoring system (DMS) camera, for connecting to a non-contact optical sensor based on remote photoplethysmography (rPPG), for connecting to the seat pressure sensor, and for connecting to the skin conductance EDA sensor, used to acquire vehicle status data. Driving behavior data and multimodal physiological signals ; Tag definition module: using multimodal physiological signals Input: Physiological stress score It includes a data caching and synchronization submodule, a feature extraction and standardization submodule, a structured prompt word construction submodule, and an LLM zero-shot inference and parsing submodule; Model training module: This module incorporates a Long Short-Term Memory (LSTM) network / gated recurrent unit / Transformer network structure and corresponding training procedures for training physiological reward agent models. , and The input is the physiological reward proxy model, and the output is the trained physiological reward proxy model. The reward calculation module loads a trained physiological reward proxy model to perform physiological stress inference and includes a built-in inverse mapping function and a multi-objective weighted reward function; the output is a physiological stress score. Physiological rewards Calculation and task rewards ; Reward Mixing Module: Its built-in weighted summation algorithm is used for mixing. and Receive the final reward ; Dataset building module: used to collect current vehicle status data Current driving behavior data and vehicle status data at the next moment and final reward Package and store it as a standard dataset file; Policy Training Module: Built-in offline reinforcement learning algorithm, used to load datasets and train the final autonomous driving policy.

Images