Driving simulator motion prompting algorithm based on deep reinforcement learning

Abstract

The invention discloses a driving simulator motion prompting algorithm based on deep reinforcement learning. According to the method, a motion prompt control problem is modeled into a Markov decision process including a platform physical state, vehicle reference motion sensation and platform motion sensation; a near-end strategy optimization algorithm is adopted, a deep neural network intelligent body is trained to learn an optimal control strategy through a multi-target reward function fusing vestibular perception errors, working space boundaries and action smoothness, and the optimal control strategy is deployed in a real-time control system. Experiments show that the method ensures millisecond-level real-time calculation, meanwhile, the average error reproduced by motion feelings in each axial direction is reduced by about 13.6% compared with a model prediction control method and is reduced by about 42.4% compared with a classic wash-out filtering method, and the method has good universality and expansibility.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of driving simulator control and reinforcement learning, and particularly relates to a motion cueing algorithm for a driving simulator. BACKGROUND

[0002] With the continuous progress of the automotive industry, transportation systems and virtual simulation technology, driving simulators have become an important simulation tool in the fields of road design evaluation, vehicle dynamics testing, and driver behavior training. The core goal is to reproduce real driving scenarios in the laboratory through virtual simulation, thereby overcoming the inherent limitations of high cost, difficult scene control, and poor safety of real vehicle testing. A high-fidelity driving simulator not only relies on visual and auditory feedback, but more importantly, it relies on motion cueing algorithms to accurately reproduce the motion perception of vehicle drivers to ensure the authenticity of user experience and the effectiveness of experimental data. Typical motion cueing algorithms can effectively simulate the motion perception of drivers in typical driving scenarios such as acceleration, braking, and turning by operating the motion platform in pitch, roll, and heave directions.

[0003] However, existing driving simulators still have certain limitations in motion cueing performance, especially in complex driving situations, it is difficult to provide motion perception that is highly consistent with the actual driving environment. Existing motion cueing algorithms usually face the challenge of how to optimize the motion cueing effect within the limited physical space of the simulator. Therefore, designing a motion cueing algorithm that can enhance the perception experience of drivers and efficiently utilize the working space of the simulator has become a hot and difficult topic in this field.

[0004] To address the above problems, various solutions have been proposed by academia and industry. The classic method such as the washout filter decomposes the motion signal by cascading high and low pass filters to achieve basic motion cueing effect with lower computational cost, but it has problems such as conservative use of working space, inherent phase delay, etc., which limit the further improvement of motion cueing accuracy and make it difficult to fully optimize the perception authenticity in actual driving experience. Another type of method based on model predictive control further integrates platform dynamics and human vestibular system model, adjusts the platform motion trajectory dynamically, and optimizes the motion perception effect while considering physical constraints. Although this method performs better in terms of perception accuracy and constraint handling, it strongly depends on the accuracy of the system model and has a large computational burden, and its application in actual systems still faces many challenges.

[0005] In recent years, the rapid development of deep reinforcement learning provides a new idea for the design of motion cueing algorithm. The deep reinforcement learning method represented by proximal policy optimization enables the agent to learn the motion cueing strategy autonomously through closed-loop interaction with the simulation environment, thus showing potential in control accuracy and execution efficiency. Although existing research attempts to introduce deep reinforcement learning into the design of motion cueing algorithm, such as using deep Q network to optimize the time domain and parameters of model predictive control, or proximal policy optimization without considering human vestibular perception, these methods generally have problems such as non-convergent training process, difficulty in extending to any degree of freedom configuration of the simulator, and insufficient consideration of human vestibular system integration in the modeling process.

[0006] To overcome the shortcomings of existing research, the present application proposes a motion cueing algorithm framework based on deep reinforcement learning. The framework takes proximal policy optimization algorithm as the core, continuously trains deep neural network through simulation interaction with the motion platform, gradually improves the perceptual fidelity of motion cueing, and effectively solves the key problems of poor generality and insufficient perceptual fidelity of existing methods by integrating human vestibular system and platform dynamics. SUMMARY

[0007] The present application aims to overcome the shortcomings of existing motion cueing algorithms in generality, perceptual fidelity and computational efficiency, and provides a new end-to-end motion cueing algorithm framework based on deep reinforcement learning. The framework learns the control strategy that maximizes the motion perceptual fidelity under platform physical constraints through autonomous interaction of the agent with the simulation environment, thereby realizing high-fidelity motion cueing adaptive generation independent of accurate system model.

[0008] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0009] A driving simulator motion cueing algorithm based on deep reinforcement learning, characterized in that it comprises the following steps:

[0010] S1, Markov decision process modeling of motion cueing problem: model the motion cueing control problem of the driving simulator as a Markov decision process defined by five tuples (state space S, action space A, state transition function P, reward function R, discount factor γ), which includes the following parts:

[0011] The state space S is used to represent the state characteristics of the system, and the state vector st∈S is composed of three parts of information: the physical state of the motion platform at time t The reference motion sensation calculated by the reference signal generated by the vehicle simulation model through the human vestibular system model And the human perception signal obtained by calculating the platform motion signal through the same vestibular system model , respectively corresponding to the physical state of the platform, the perception target that the agent needs to track, i.e., the reference motion sensation signal, and the actual perception effect produced by the current strategy, i.e., the platform motion sensation.

[0012] The state vector is denoted as , wherein characterizes the physical state of the platform, and at least includes position, velocity, acceleration, attitude angle and angular velocity; is a reference motion sensation signal processed by the vestibular model from the motion signal provided by the vehicle simulation, including a reference perceived specific force and a reference perceived angular velocity; characterizes the motion sensation signal actually perceived by the driver after the platform motion signal is processed by the vestibular model, including an actual perceived specific force and an actual perceived angular velocity.

[0013] The action space A is a continuous control quantity output by the agent, defined as the control instruction of the platform actuator, such as the acceleration and angular velocity instruction of each degree of freedom. The agent selects a continuous action at∈A according to the current state to drive the platform to perform the corresponding action.

[0014] The state transition function P(st+1|st, at): The state transition function describes the process of the system transitioning from the current state to the next state after the action . This process implicitly contains the evolution relationship of the platform dynamics, the human vestibular perception model and the reference signal from the vehicle.

[0015] The reward function rt∈R is in the form of a multi-objective weighted sum, used to guide the learning direction of the agent, and the core is to balance the motion perception fidelity and the platform workspace constraints. The reward function at least includes: a perception fidelity reward item rf and rω in the translation and rotation directions, a constraint violation penalty item rb for punishing the platform motion beyond the physical workspace boundary, an action smoothness reward item rs for suppressing the mutation of the control instruction to improve the motion smoothness, and a positive weight coefficient for balancing each target.

[0016] The discount factor is used to balance the short-term and long-term benefits of the agent, and its value range is [0, 1).

[0017] S2, deep reinforcement learning agent training based on proximal policy optimization: using the proximal policy optimization algorithm, the deep reinforcement learning agent is trained based on the Markov decision process established in step S1; the agent interacts with the simulation environment through the actor-critic architecture to maximize the cumulative reward, iteratively optimizes the parameters of its strategy network and value network, and thus learns the optimal control strategy that can generate high-fidelity motion cues under the physical constraints of the platform.

[0018] In each training cycle, the agent interacts with the environment based on the current policy, samples the state-action-reward sequence, and optimizes the parameters using the shearing policy loss function.

[0019] S3. Deployment and Application of the Trained Model: The policy network trained in S2 is independently deployed in the real-time control system of the driving simulator. At this point, only the forward propagation computation of this network is required, resulting in extremely low computational overhead and meeting the millisecond-level real-time control requirements.

[0020] The system forms the following closed loop:

[0021] 1) Sensors read the physical status of the platform ;

[0022] 2) Obtain reference motion signals from the vehicle model and combine them with a human vestibular system model to calculate reference motion sensations. ;

[0023] 3) Acquire platform motion signals and calculate platform motion sensations using a human vestibular system model. ;

[0024] 4) State vector Input policy network;

[0025] 5) The network instantly outputs the optimal control action 'at' to drive the motion platform;

[0026] 6) Execute in a loop to achieve adaptive, high-fidelity motion cues.

[0027] Specifically, the reward function in step S1 is defined as:

[0028]

[0029] in, As a reward for translational perception fidelity, it is used to reduce the force difference between the perception signals generated by the motion platform and the reference vehicle in the translational direction, and directly drive the intelligent agent to match the linear motion feeling of the human body.

[0030] To enhance rotational perception fidelity, this is used to reduce the difference in perceived angular velocity between the motion platform and the reference vehicle in the direction of rotation, thus matching the human body's angular motion perception.

[0031] The boundary touch reward is used to punish control actions that exceed the physical limits of the motion platform's workspace, ensuring the physical feasibility of the control strategy;

[0032] For smooth motion rewards, large or abrupt changes in control commands are penalized, while encouraging the generation of smooth motion trajectories.

[0033] , , and It is a positive weighting coefficient used to balance the various reward items.

[0034] Specifically, in step S1, the agent's goal is to maximize the cumulative reward of the discount:

[0035]

[0036] in, This is the instant reward at step t+k. This is a discount factor used to balance the agent's emphasis on short-term versus long-term returns.

[0037] Specifically, in step S2, the near-end policy optimization algorithm uses a pruning agent objective function and generalized advantage estimation to improve the stability and sample efficiency of policy updates.

[0038] Specifically, in step S2, both the policy network and the value network are implemented using deep neural networks.

[0039] Specifically, in step S2, the training process employs parallel asynchronous sampling and gradient pruning techniques to accelerate experience collection and ensure training stability.

[0040] Specifically, in steps S1 and S3, the human vestibular system models used include the otolith organ model and the semicircular canal model, which respectively describe translational motion perception and rotational motion perception, and their transfer functions are as follows:

[0041] The general equation for the otolith model is:

[0042]

[0043] in, For the gain of the otolith organs, and For the corresponding time delay, The perceived force signal in the translation direction. It is the input force signal in the translation direction.

[0044] The general equation for the semicircular canal model is:

[0045]

[0046] in, For the gain of the semicircular canals, and The time delay of the transfer function, The angular velocity signal is sensed in the direction of rotation. The input angular velocity signal is in the direction of rotation.

[0047] Compared with the prior art, the beneficial effects of the present invention are: In the present invention,

[0048] 1. Strong versatility and adaptability: Through an end-to-end deep reinforcement learning framework, the agent can learn the optimal strategy autonomously from its interaction with the environment, freeing it from dependence on precise explicit system models and demonstrating strong adaptability to different driving scenarios and platform dynamics.

[0049] 2. High perceptual fidelity: The human vestibular system model is explicitly integrated into the reward function and state space of the Markov decision process, and motion perception error is taken as the core optimization objective, which directly ensures the realism and fidelity of motion cues from the perspective of human perception mechanism.

[0050] 3. Constraint-handling intelligence: By integrating complex workspace constraints into the reward function design, the agent learns autonomously to act within the physical constraints during training, achieving a balance between constraint satisfaction and performance optimization.

[0051] 4. Superior computational efficiency and real-time performance: Compared with model predictive control methods based on rolling optimization, this invention only requires a feedforward neural network for forward computation during the deployment phase, resulting in low computational burden, low latency, and easy deployment in real-time control systems.

[0052] 5. Modular and scalable framework design: The motion cues algorithm framework based on deep reinforcement learning proposed in this invention does not depend on specific platform degree-of-freedom configurations and can be extended to motion platforms with different degrees of freedom and various simulation environments. It facilitates integration with vehicle dynamics models, perception modules, and controller interfaces, providing a unified solution for motion cues systems in intelligent driving simulation and flight simulation. Attached Figure Description

[0053] Figure 1 Framework diagram of a driving simulator based on deep reinforcement learning

[0054] Figure 2 Virtual driving scene in Unity3D

[0055] Figure 3 The image shows the lateral motion perception reproduction effect of a motion cues algorithm for a driving simulator based on deep reinforcement learning proposed in this invention.

[0056] Figure 4 The image shows the longitudinal motion perception reproduction effect of a motion cues algorithm for a driving simulator based on deep reinforcement learning proposed in this invention.

[0057] Figure 5 The vertical motion perception reproduction effect of a motion cues algorithm for a driving simulator based on deep reinforcement learning proposed in this invention is shown in the figure.

[0058] Figure 6 The image shows the pitch motion perception reproduction effect of a motion cues algorithm for a driving simulator based on deep reinforcement learning proposed in this invention.

[0059] Figure 7 The image shows the effect of reproducing the rolling motion sensation of a driving simulator motion cues algorithm based on deep reinforcement learning proposed in this invention.

[0060] Figure 8 The vertical workspace utilization of a motion cues algorithm for a driving simulator based on deep reinforcement learning proposed in this invention.

[0061] Figure 9 The present invention presents a rolling workspace utilization of a motion cues algorithm for a driving simulator based on deep reinforcement learning.

[0062] Figure 10 The pitch workspace utilization of a motion cues algorithm for a driving simulator based on deep reinforcement learning proposed in this invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of this invention.

[0064] This invention provides a motion cues algorithm for driving simulators based on deep reinforcement learning. The core idea of ​​this method is to model the motion cues problem as a Markov decision process and utilize a proximal policy optimization algorithm to allow the agent to autonomously learn a control strategy that achieves an optimal balance between vestibular perception fidelity and platform physical constraints through interaction with the simulation environment. This embodiment uses a typical three-degree-of-freedom motion platform as an example for illustration; however, it should be understood that the method framework provided by this invention is also applicable to motion systems with other degrees of freedom configurations.

[0065] Reference Figure 1 The overall framework of the invention shown includes the following main steps:

[0066] S1. Markov Decision Process Modeling for the Motion Cueing Problem: This step aims to formalize the motion cueing control problem into a Markov decision process model suitable for deep reinforcement learning. First, the state space, action space, state transitions, and reward function are designed. The state space S represents the state characteristics of the system, and its state vector... It consists of three parts of information, and is represented as follows:

[0067]

[0068] in, The physical state of the motion platform, specifically including the platform's attitude angle vector. Angular velocity vector Vertical displacement Vertical velocity and vertical acceleration ,Right now ; To reference motion perception, motion signals generated by the vehicle simulation model are calculated using a human vestibular system model, specifically including perceived contrast. With reference sensing angular velocity ,Right now ; For human motion perception, it is calculated from the platform's motion signals using the same vestibular system model, specifically including actual perceived force. Compared with actual perceived angular velocity ,Right now .

[0069] Action space A represents the continuous control output of the agent. In this embodiment, for the three-degree-of-freedom platform, the action vector output by the agent at each time step t is defined as the control command executable by the platform, specifically... ,in These represent the platform's pitch rate, roll rate, and vertical acceleration, respectively.

[0070] The state transition function P(st+1|st, at) describes the transition of the system from the current state. Through action Then transition to the next state The core of the system's dynamic evolution process is the relationship between platform dynamics, human vestibular perception, and the transfer of reference signals from the vehicle.

[0071] More specifically, the state transition function is based on the dynamic equations of the three-degree-of-freedom motion platform, achieves state updates through Euler forward integration, and combines attitude angle changes and gravity components for coordinate transformation. Its expression is as follows:

[0072]

[0073] in, and These are the platform's pitch and roll angles, respectively. , To correspond to pitch and roll angular velocities, For vertical acceleration, and These represent the platform's displacement and velocity in the vertical direction, respectively. The sampling time step, in this embodiment, is set to maintain consistency with the data frequency of the vehicle simulation model. The time interval was set to 0.016 seconds (corresponding to a sampling frequency of 62.5 Hz). Furthermore, the updated platform attitude information needed to undergo coordinate transformation and be superimposed with a gravity component. Finally, the motion perception of the platform's motion signal was obtained through the aforementioned vestibular system model. Simultaneously, it receives the next-moment motion signal from the vehicle and obtains a reference motion sensation from the vehicle through the same vestibular system model. .

[0074] Specifically, in step S1, the human vestibular system model used includes the otolith organ model and the semicircular canal model, which respectively describe translational motion perception and rotational motion perception, and their transfer functions are as follows:

[0075] The general equation for the otolith model is:

[0076]

[0077] in, For the gain of the otolith organs, and For the corresponding time delay parameters, The perceived force in the translation direction, It is the input force in the translation direction.

[0078] The general equation for the semicircular canal model is:

[0079]

[0080] in, For the gain of the semicircular canals, and For the corresponding time delay parameters, The perceived angular velocity in the direction of rotation. Input angular velocity in the direction of rotation.

[0081] The reward function This is a customized design used to guide the agent to learn while balancing motion perception fidelity with platform workspace constraints. The reward function used in this embodiment... It consists of a weighted sum of four parts: The definitions of each reward item are as follows:

[0082] Translation perception fidelity reward This is used to reduce the force difference between the sensing signals generated by the motion platform and the reference vehicle in the translation direction, and directly drive the intelligent agent to match the linear motion sensation of the human body.

[0083]

[0084] in, Rewards for translational perception fidelity and These are the perception ratios of the simulation platform at time step t and the perception ratio of the reference vehicle at time step t, respectively.

[0085] Rotational perception fidelity reward It is used to reduce the difference in perceived angular velocity in the rotation direction between the motion platform and the reference vehicle, and to match the human body's angular motion perception.

[0086]

[0087] in, As a reward for rotational perception error The angular velocity of the simulation platform at time step t is, For reference, the actual angular velocity of the vehicle;

[0088] Boundary Touch Reward The function is used to punish control actions that cause the motion platform's workspace to exceed physical limits.

[0089]

[0090] in, As a boundary penalty and reward, and These represent the current angle and current position of the motion platform, respectively. Maximum limit to the workspace of the motion platform;

[0091] Motion smoothing rewards It is used to penalize large or abrupt changes in control commands, promoting smooth control. Its function is:

[0092]

[0093] in, For smooth motion rewards, It is the difference between the current action and the previous action. This refers to the control action at time step t.

[0094] , , and It is a positive weighting coefficient used to balance the various reward items.

[0095] In this embodiment, the design of the weighting coefficients is based on multi-objective optimization trade-offs. In principle, the coefficients of the fidelity evaluation term should be determined first. On this basis, Set to a higher value to prioritize and absolutely guarantee system security; while The values ​​are relatively small, intended to fine-tune control quality rather than dominate the learning process. Finally, motion balancing is performed and expert-tuned. An exemplary set of values ​​for each weight coefficient is: =1, =1, =10, =0.1. It can be understood that these weight coefficients can be readjusted and optimized based on the specific platform's dynamic characteristics, workspace limitations, and performance requirements.

[0096] S2. Agent Training Based on Proximal Policy Optimization: This step is based on the Markov decision process model established in S1, and uses the proximal policy optimization algorithm to train the agent. The agent interacts with the simulation environment through an actor-critic architecture, and iteratively optimizes its policy network and value network parameters with the goal of maximizing cumulative reward, thereby learning the optimal control policy that can generate high-fidelity motion cues under the physical constraints of the platform.

[0097] In each training cycle, the agent interacts with the environment based on the current policy, samples the state-action-reward sequence, and optimizes the parameters using the shearing policy loss function.

[0098] More specifically, the near-end policy optimization algorithm balances sample efficiency and training stability by constraining policy updates. Its core is to optimize a stochastic policy approximated by a policy network. Maximize the expected cumulative reward. In each training round, the agent is in state... Input, output continuous action Instant rewards for environmental feedback With the new state The agent maximizes the cumulative reward of the discount. For the goal.

[0099] In this embodiment, the policy gradient theorem provides the basic rules for agent parameter updates, and its expected objective function expression is as follows:

[0100]

[0101] in, For parameter-based The policy function. To reduce the variance of the policy gradient method, this invention introduces a generalized advantage estimate, which is calculated as follows:

[0102]

[0103] in, This is the coefficient that balances the bias and variance. Let be the state value function estimated by the commentator network. Simultaneously, based on GAE, a pruned agent objective function is used to optimize the policy, addressing the high variance advantage estimation and performance degradation issues caused by extreme policy updates. Its expression is as follows:

[0104]

[0105] in, The probability ratio between the old and new strategies. The pruning threshold is used. The final total loss function is weighted by an entropy reward with an entropy coefficient c=0.01. To enhance and encourage exploration, its expression is:

[0106]

[0107] In step S2, the proximal policy optimization algorithm employs two independent neural networks: a policy network that maps states to actions, and a value network that estimates state values. The policy network outputs the mean and standard parameters of a Gaussian distribution over continuous actions. The value network follows the same architecture but terminates at a linear output layer to provide continuous value estimates.

[0108] Specifically, in this invention, both the policy network and the value network are implemented using multi-layer fully connected feedforward neural networks. As a preferred embodiment, each network contains three hidden layers with 128, 128, and 64 nodes respectively. Hyperbolic tangent activation functions are used between the hidden layers to achieve a good balance between the model's non-linear expressiveness and the gradient stability during training. It should be understood that the above network structure configuration is merely an example and can be adaptively adjusted according to the complexity of the specific problem and computational resources.

[0109] The policy network receives state outputs and generates the action mean and log-standard deviation, reflecting a Gaussian distribution. The mean output layer uses a hyperbolic tangent activation function to limit the output to a preset action boundary range; the log-standard deviation output layer directly outputs its value. Furthermore, all fully connected layers use an orthogonal initialization method to initialize the network weights, better preserving the variance characteristics of the data in the early stages of training and promoting stable gradient propagation in the deep neural network.

[0110] S3. Deployment and Application of the Training Model: The optimal policy network trained in step S2 is deployed to the real-time control system of the driving simulator. During actual operation, the agent directly generates control commands based on the received platform physical state, the processed vehicle reference motion sense, and the platform motion sense as input, driving the motion platform to reproduce a high-fidelity driving motion sense within a limited workspace. During operation, the platform state and perception signals are fed back to the policy network in real time, forming a closed-loop control and realizing adaptive motion cues updates.

[0111] More specifically, the trained deep reinforcement learning model is deployed in the motion platform control host and interacts with the vehicle simulation module via a communication interface. The real-time control system periodically collects reference motion signals output by the vehicle model, as well as attitude angle, displacement, and acceleration information returned by the platform sensors. After processing and normalizing the perceived signals, this constitutes the real-time state input of the intelligent agent. The intelligent agent generates platform control commands through forward inference. The commands include continuous control quantities such as pitch angular velocity, roll angular velocity, and vertical acceleration, reproducing a high-fidelity driving motion feel within a limited workspace.

[0112] Before being sent to the actuators, the control commands undergo coordinate transformation and gravity component allocation to match the platform's dynamic characteristics. To ensure smooth execution and equipment safety, the system incorporates motion limiting and slope constraint modules at the output end to constrain the rate of change of commands, preventing actuator saturation and structural impact. Through this deployment method, the model achieves state-input-based execution control while ensuring real-time performance, enabling the driving simulator to accurately reproduce the driver's realistic motion sensations within a limited workspace.

[0113] It should be noted that in this embodiment, a simulation environment consistent with the driving simulator was first constructed, and vehicle motion data under standard driving conditions was generated using a vehicle dynamics model, including longitudinal acceleration, lateral acceleration, angular velocity, and attitude angles. The raw data collected offline was then processed for sensing signal calculation and normalization to ensure that the reference signals were within the platform's achievable operating range.

[0114] The simulated vehicle trajectory should include sufficient maneuverability to support the deployment of the driving simulator. Driving experiments of this invention in Unity3D are as follows: Figure 2 As shown, a long-range trajectory of 62.5 Hz was recorded, including longitudinal, lateral, and vertical maneuvers, their coupling, and stopping. Subsequently, this trajectory was decomposed into short trajectory segments of 20 seconds each to ensure the randomness of the initial state.

[0115] In this embodiment, refer to Figures 3 to 7The paper demonstrates the motion perception reproduction performance of the proposed DRL-MCA method in the lateral, longitudinal, vertical, pitch, and roll axes. It can be seen that this method achieves the highest cross-correlation coefficient and the lowest root mean square error (RMSE) on the two translation-related axes. To quantify the performance evaluation, this example uses the RMSE and cross-correlation coefficient as core indicators for a classic wash filter, and the results are as follows:

[0116] In terms of reproducing translational motion sensation, this method demonstrates significant advantages. On the longitudinal and lateral axes, the root mean square error (RMSE) of this method reaches 0.0314 and 0.0354, respectively, which are approximately 51.6% and 12.6% lower than the 0.0648 and 0.0405 of the model predictive control method and the 0.1873 and 0.0552 of the classical wash-out filtering method. Furthermore, this method achieves the highest cross-correlation coefficient of 0.99 on both axes, indicating that the platform motion generated by this method is not only more accurate in amplitude but also maintains a high degree of consistency with the reference sensation in terms of signal waveform and phase. These results validate the effectiveness of the designed reward function in driving the agent to match human motion sensation.

[0117] In terms of rotational and related motion perception reproduction, on the vertical axis, the root mean square error (RMSE) of our method is 0.0201, falling between 0.0289 for the model predictive control method and 0.0454 for the classical wash-out filtering method, with a cross-correlation coefficient of 0.890, slightly higher than 0.834 for the model predictive control method. On the pitch and roll axes, the classical wash-out filtering method achieves the lowest RMSSE, but at the cost of extremely low workspace utilization and a weak tilt coordination effect, resulting in severely insufficient perception fidelity on the translation axis. In contrast, our method and the model predictive control method actively utilize the platform's pitch and roll motion to simulate continuous longitudinal and lateral forces. Although the absolute error in rotational axis perception increases slightly, it achieves a significant improvement in overall motion perception fidelity. Our method achieves intelligent balancing and optimization of multi-degree-of-freedom coupled motion perception under limited physical constraints.

[0118] Reference Figure 8 , Figure 9 as well as Figure 10Regarding workspace utilization, this method demonstrates its intelligent constraint handling characteristics. In vertical displacement, the method maintains a moderate and sustainable stroke utilization level, avoiding the drastic reverse adjustments after frequent boundary impacts seen in model predictive control methods, and overcoming the motion perception distortion problem caused by overly conservative classical wash-out filtering methods. In pitch and roll angle spaces, the utilization ranges of the three methods are similar, confirming that reasonable utilization of the platform's rotational degrees of freedom for tilt coordination is a common requirement for achieving high-fidelity motion cues. Furthermore, the motion trajectory of this method exhibits small-amplitude, high-frequency oscillations, an inherent characteristic of neural network-based strategies in continuous motion spaces. Although this introduces motion noise to some extent, its amplitude is much smaller than the main motion signal and does not have a decisive negative impact on the core perception fidelity index, indicating that the agent has found an effective balance between noise and performance.

[0119] Regarding computational efficiency, to verify the deployment potential of this method on real-time devices, this example studies the computation time of three motion cueing algorithms at each step from generating the reference input signal to updating the platform action. The results show that this method has a significant advantage in maintaining fast and stable real-time computational efficiency, with an average computation time of 1.040 milliseconds per step. Compared to 1.535 milliseconds for model predictive control and 0.036 milliseconds for classical wash-out filtering, this method demonstrates good real-time performance and computational capability, ensuring its feasibility and efficiency in practical applications.

[0120] In summary, the motion cues algorithm framework for driving simulators based on deep reinforcement learning proposed in this invention achieves excellent overall performance in key dimensions such as the overall fidelity of motion perception reproduction, intelligent utilization of workspace, and computational efficiency after deployment. It not only demonstrates the powerful potential of deep reinforcement learning in addressing motion cues in complex electromechanical systems but also provides a solid technical foundation for its application in higher-degree-of-freedom simulators and online human-machine co-driving simulations.

[0121] The foregoing has shown and described the basic principles, main features, and advantages of this invention. Those skilled in the art will understand that various modifications and substitutions can be made to its form and details without departing from the spirit and essence of this invention. For example, the selection method of state variables, the parameter configuration of the reward function, the number of network layers and activation functions, and the setting of the sampling period and platform degrees of freedom, etc., all constitute equivalent variations and extensions of this invention. The scope of protection of this invention should be determined by the claims.

Claims

1. A motion cueing algorithm based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Markov Decision Process Modeling of Motion Cueing Problem: The motion cueing control problem of driving simulator is modeled as a Markov decision process defined by a quintuple (state space S, action space A, state transition function P, reward function R, and discount factor γ). S2. Agent training based on proximal policy optimization: The deep reinforcement learning agent is trained using the proximal policy optimization algorithm based on the Markov decision process model established in step S1; the agent interacts with the simulation environment through an actor-critic architecture, and iteratively optimizes its policy network and value network parameters with the goal of maximizing cumulative reward. S3. Deployment and application of the training model: Deploy the policy network trained in step S2 to the real-time control system of the driving simulator motion platform; during actual operation, the agent inputs the real-time acquired platform state, vehicle reference motion feeling and platform motion feeling, outputs control commands to drive the platform motion, thereby adaptively and in a closed loop reproduces a realistic driving motion feeling within a limited workspace.

2. The algorithm according to claim 1, characterized in that, The state vector of the state space S in step S1 The physical state of the platform at time t by the motion simulation platform Reference motion sensory signals and the motion sensing signals from the platform It consists of three parts, represented as .

3. The algorithm according to claim 1, characterized in that, The state vector Middle reference motion sensory signal and platform motion sensation It is obtained by processing the motion signals output from the vehicle simulation and the platform motion signals separately through a human vestibular system model.

4. The algorithm according to claim 1, characterized in that, The action space A is the continuous control quantity output by the agent, defined as the control instructions that the platform can execute, such as acceleration and angular velocity instructions for each degree of freedom.

5. The algorithm according to claim 1, characterized in that, The reward function in step S1 is a multi-objective weighted summation form, which includes at least a translation fidelity reward term. Rotation fidelity bonus item Boundary collision reward items and motion smoothing reward items .

6. The algorithm according to claim 1, characterized in that, In step S2, the agent uses a proximal policy optimization algorithm and optimizes the policy through an actor-critic structure. The training process employs parallel asynchronous sampling, gradient pruning, and generalized advantage estimation techniques to improve the stability and sample efficiency of training.

7. The algorithm according to claim 1, characterized in that, In step S2, both the policy network and the value network are implemented using deep neural networks.

8. The algorithm according to claim 1, characterized in that, In step S3, the trained policy network is deployed in the real-time control system of the driving simulator. It receives the platform's physical state, the processed vehicle reference motion sense, and the platform motion sense as input, and generates control commands in real time to drive the motion simulation platform, thereby realizing closed-loop control and adaptive motion prompts.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images