Ambulance suspension intelligent agent based on deep reinforcement learning and optimization method thereof

Abstract

The invention relates to the technical field of automobile dynamics control, in particular to an ambulance suspension intelligent agent based on deep reinforcement learning and an optimization method thereof, and the optimization method comprises the steps: S1, initializing an evaluation network and a control strategy network; s2, adopting a swarm intelligence algorithm to initialize a population; s3, the number M of evolution generations of the controller and the time step T of each generation are set, in each time step, the initial fitness of the controller is set to be 0, and the fitness of the controller is updated according to the reward value and the auxiliary reward of the current time step; s4, after each intergeneration is finished, updating the population according to the fitness by adopting a swarm intelligence algorithm, and training the evaluation network and the control strategy network to update network parameters; and S5, after M generations of evolution are completed, screening out the controller with the highest fitness from the controllers, and setting the network parameters corresponding to the controller in the control strategy network. The invention is at least beneficial to improving the adaptability of the ambulance to different working conditions.

Description

Technical Field

[0001] This invention belongs to the field of automotive dynamics control technology, and in particular relates to an ambulance suspension intelligent agent based on deep reinforcement learning and its optimization method. Background Technology

[0002] Intensive care ambulances are specialized medical vehicles equipped with emergency medical equipment for the urgent transport of critically ill patients, providing emergency care and monitoring of their condition en route. They play a vital role in rapid transport and emergency safety in medical support and disease control. In the face of frequent public health emergencies such as natural disasters, medical treatment of the injured and sick, and sudden accidents, the importance of intensive care ambulances as a crucial link between on-site emergency treatment and hospital care is self-evident. However, in variable driving environments and complex road conditions, while intensive care ambulances expedite transport and ensure critically ill patients receive treatment in the shortest possible time, patient comfort and safety may be reduced. This could lead to adverse reactions such as increased pain and intracranial blood pressure, and even vomiting and aspiration, potentially causing secondary injuries or even death.

[0003] Therefore, optimizing the transport performance of intensive care ambulances and ensuring a comfortable and safe riding environment for critically ill patients during emergency medical missions is a crucial and urgent task for modern medical systems. The physiological and psychological state of patients during transport significantly impacts their subsequent treatment and rehabilitation; thus, improving the riding environment is paramount. Specifically, the key lies in improving the riding comfort and vehicle stability of the ambulance, which primarily depend on the design and performance of the ambulance's controllable suspension system.

[0004] While traditional designs and methods have improved ambulance ride comfort and driving stability to some extent, most studies have only optimized for single operating conditions and have failed to fully explore the potential of intelligent suspension in ambulances. Especially when faced with complex, variable, and uncertain road conditions and the stringent requirements of emergency medical transport, the limitations of traditional intelligent suspension control methods become increasingly apparent, making it difficult to achieve optimal control effects. This not only hinders the improvement of the stability and safety of ambulances during transport but also restricts the development of high-quality ambulance transport technology.

[0005] While deep reinforcement learning has been applied to intelligent suspension control technology, current research largely focuses on generalizing its applications across a wide range of scenarios, with insufficient exploration of its deeper, more specialized applications in suspension control. Specifically, in ambulance intelligent suspension control tasks, the exploration and utilization capabilities of the deep reinforcement learning controller significantly influence whether the control strategy can cover more operating conditions and more complex environments, thus determining the strategy's robustness and generalization ability.

[0006] Furthermore, while existing deep reinforcement learning algorithms for suspension control offer improvements over traditional techniques, they still suffer from the following shortcomings in ambulance scenarios: 1. Insufficient exploration capabilities and incomplete coverage of operating conditions. Specifically, DDPG (Deep Deterministic Policy Gradient) or TD3 (Twin Delayed Deep Deterministic Policy Gradient) explore the environment through "single agent trial and error," which, limited by the exploration scope, cannot cover all operating conditions during ambulance transport. This results in control strategies being effective only in training scenarios, with performance degrading by more than 30% when switching to untrained road conditions or loads. 2. Overestimation bias and low sample efficiency. Specifically, DDPG is prone to overestimation bias due to "maximizing the action value function," and although TD3 alleviates this problem through a dual evaluation network, both rely on "random experience replay." Given the need for rapid convergence in ambulances (the controller needs to quickly adapt to new road conditions in emergency transport scenarios), the training cycle can exceed 10,000 rounds, making it difficult to meet engineering implementation requirements and resulting in low sample utilization efficiency.

[0007] While other related technologies have proposed innovative mechanisms such as "independent experience caching + time logic decay auxiliary rewards" to significantly improve the sample utilization efficiency of deep reinforcement learning algorithms, they still have problems such as lack of global exploration capabilities, limited working condition coverage, inability to collect differentiated data under multiple roads, multiple loads, and multiple speeds on a large scale, insufficient robustness, poor scenario adaptability, and lack of optimization for ambulance needs when it comes to ambulance active suspension control scenarios. Summary of the Invention

[0008] In view of this, the present invention aims to provide an ambulance suspension intelligent agent based on deep reinforcement learning and its optimization method, which is at least beneficial to improving the adaptability of ambulances to different working conditions.

[0009] To achieve the above objectives, the technical solution created by this invention is implemented as follows: This invention provides a method for optimizing ambulance suspension agents based on deep reinforcement learning, comprising: S1: initializing the evaluation network and the control policy network; S2: initializing the population using a swarm intelligence algorithm, wherein the population consists of... The system consists of controllers with different parameters. Each controller has the same network structure as the control strategy network. The swarm intelligence algorithm can be any one of evolutionary strategy, genetic algorithm, or particle swarm optimization. An experience replay cache pool is initialized. as well as A deterministic experience memory cache pool Deterministic experience memory cache pool One-to-one correspondence with controllers within the population; S3: Set the number of generations M for controller evolution and the time step T for each generation. In each time step, perform the following operations for each controller: Set the initial fitness of the controller... Set to 0, extract state information from the ambulance's overall vehicle dynamics system; at each time step t, the control strategy network adjusts the state based on the current time step. Output suspension control force ; to adjust the current suspension control force Reward Value And the state in the next moment. Packaged as primary samples Save to the corresponding deterministic experience memory cache pool Generate deterministic empirical tracking vectors Deterministic experience tracking vector Compared with primary samples Combine to form a new sample And save it to the experience replay cache pool. Based on the reward value of the current time step With auxiliary rewards Update the controller's adaptability S4: After each generation, a swarm intelligence algorithm is used to update the population based on fitness, and the experience replay cache is used. Select group A samples to train the evaluation network and control policy network to update network parameters; S5: After completing the M-generation evolution, from The controller with the highest fitness is selected from the controllers, and its corresponding network parameters are set in the control strategy network to output the suspension control force.

[0010] Furthermore, the evaluation network includes a first evaluation network and a second evaluation network with the same network structure but different network parameters. The first evaluation network outputs a first evaluation result Q1, and the second evaluation network outputs a second evaluation result Q2. The evaluation result output by the evaluation network is min(Q1, Q2). The evaluation result output by the evaluation network is used to guide the control strategy network to optimize the suspension control force. Step S1 also includes: initializing the target evaluation network and the target control strategy network. The parameters of the target evaluation network are consistent with the parameters of the evaluation network, and the parameters of the target control strategy network are consistent with the parameters of the control strategy network. The target evaluation network includes a first target evaluation network and a second target evaluation network. In step S4, after training the evaluation network and the control strategy network, the weights of the target evaluation network and the target control strategy network are also updated synchronously. ', j When =1, The weights of the first evaluation network, The primary objective is to evaluate the network weights. jWhen =2, The weights for the second evaluation network, The weights of the network are evaluated for the second objective; To reduce the weights of the control policy network, The weights of the target control policy network are τ = 0.001.

[0011] Furthermore, in step S4, the experience replay cache pool is... Selecting Group A samples for training the evaluation network and control policy network includes: using samples from the experience replay cache pool. A group of samples is randomly selected, A=128, and the loss function is minimized using the stochastic gradient descent method. Update the weights of the first and second evaluation networks, where E represents the expected value. j When it is 1, This indicates the weights of the first evaluation network. j When it is 2, This indicates the weights of the second evaluation network. , y t Let Q represent the target Q value, and γ be a discount factor, γ = 0.99. The weights of the control policy network are updated using the stochastic gradient ascent method, and the update formula is as follows: , N=A.

[0012] Furthermore, When the swarm intelligence algorithm adopts an evolutionary strategy, in step S2, the parameters of each of the 10 controllers are independently sampled from a Xavier normal distribution with a mean of 0 and a variance of 0.1. In step S4, the population is updated according to fitness using an evolutionary strategy, including: selecting the top 5 elite controllers in terms of fitness, calculating the new mean μ_new and new standard deviation σ_new of the parameters of the 5 elite controllers, and generating 10 new controllers based on the Gaussian distribution corresponding to the new mean μ_new and new standard deviation σ_new, thus forming a new generation of population.

[0013] Furthermore, When the swarm intelligence algorithm is a genetic algorithm, in step S2, the controller weights are flattened into a one-dimensional chromosome vector c, c∈R. dWhere d is the total number of parameters and R represents the set of real numbers; 10 independent chromosome vectors are generated to form a population P={c1, c2, ..., c10}, and the chromosome vectors correspond to the controller. Each parameter value of each chromosome vector is randomly sampled from a uniform distribution in the interval [-0.1, 0.1]. In step S4, the population is updated according to fitness using an evolutionary strategy, including: using a tournament selection strategy to select 10 parent chromosome vectors; performing a single-point crossover operation on the 10 parent chromosome vectors to generate 10 primary offspring chromosome vectors; and performing Gaussian mutation on the chromosome gene loci with a probability pm=0.1, with a mean of 0 and a variance of 0.01, to generate 10 offspring chromosome vectors; retaining the 2 elite chromosome vectors with the highest fitness in the population, and selecting the 10 chromosome vectors with the highest fitness from the 2 elite chromosome vectors and the 10 offspring chromosome vectors to form the new generation population.

[0014] Furthermore, When performing particle swarm optimization using swarm intelligence algorithms, in step S2, the controller is mapped to particles, and the controller weights are mapped to particle positions. The velocity of each particle represents the parameter update direction and magnitude of the particle at the next moment. Ten particles are generated to form a swarm, and the initial position x of each particle is... i 0 The initial velocity v is obtained by random sampling from a uniform distribution in the interval [-0.1, 0.1]. i 0 Random sampling is performed from a uniform distribution in the interval [-0.05, 0.05]; in step S4, for each particle, the position corresponding to the highest fitness in the current generation is obtained as pbest. i The position of the particle with the highest fitness in the entire population is taken as gbest; according to the formula v i t+1 =wv i t +c1r1(pbest i -x i t )+c2r2(gbest-x i t Update the velocity of each particle in the population according to the formula x. i t+1 =x i t +v i t+1 Update the positions of all particles in the population, where w = 0.8, w is the inertia weight, c1 = c2 = 2.0, c1 and c2 are both learning factors, r1 and r2 are both random numbers in the interval [0, 1], and pbest iLet be the individual historical best position of the i-th particle, and gbest be the global best position of the population. If the particle position exceeds the range of [-1, 1], then reflection boundary processing is used. The updated particles constitute the next generation of the population.

[0015] Furthermore, the status information includes: vehicle displacement. Vehicle speed Vehicle acceleration Wheel acceleration Suspension dynamic deflection Suspension dynamic deflection derivative The control force output by the suspension at the previous moment .

[0016] Furthermore, in step S4, ,in, As the core reward, P This is a mandatory penalty item; , , , , , , as well as All are weighting coefficients. , , , , , , , , , , , as well as All are soft reward coefficients. =0.04, =0.4, =4, =0.017, =0.15, =15, P =-500, when the suspension dynamic deflection It is triggered at a certain time, and when triggered, the calculation of the time step of the current generation is forcibly terminated; .

[0017] Furthermore, steps S1 to S5 are used to perform initial optimization on the ambulance suspension agents based on deep reinforcement learning for multiple ambulances; each ambulance is regarded as an individual, and each individual collects and stores operational data in real time during actual transport in a deterministic experience memory cache pool. In this process, operational data includes status information and patient physiological data; each individual undergoes periodic local generational evolution, which includes individual fine-tuning and cloud-based unified optimization. Individual fine-tuning includes updating fitness. Among them, reward value Add health penalty items ,in, The weight of health penalty items, Representing the collected patient heart rates, the control policy network was retrained using local data according to step S4, with a learning rate of 0.0001. Cloud-based unified optimization included: after individual fine-tuning for each network, the locally updated network parameters and key performance indicators were encrypted and uploaded to the cloud server. The cloud server performed unified optimization, where key performance indicators included average fitness and patient physiological data statistics. Unified optimization involved the cloud server using a federated averaging or evolutionary aggregation algorithm to calculate and update weights. Where N is the number of ambulances. Let be the number of samples for the i-th vehicle. The original weights of the control strategy network for vehicle i are used. The updated weights are assigned to the control strategy network of each individual. Based on the global data, the network parameters of the control strategy network of each individual are re-optimized in step S4. The parameters of the control strategy network of the individual with the highest fitness are selected from all individuals and distributed to the control strategy networks of each ambulance.

[0018] In another aspect, this invention provides an ambulance suspension intelligent agent based on deep reinforcement learning. The ambulance suspension intelligent agent based on deep reinforcement learning is obtained based on the above-mentioned optimization method for ambulance suspension intelligent agents based on deep reinforcement learning. During actual transport, the ambulance suspension intelligent agent based on deep reinforcement learning generates suspension control force based on the collected state and applies it to the ambulance active suspension system.

[0019] Compared with existing technologies, the present invention achieves the following beneficial effects: The optimization method for ambulance suspension intelligent agents based on deep reinforcement learning provided by the present invention addresses the shortcomings of related technologies in adapting to the special transport needs of ambulances. It offers a deep reinforcement learning-based ambulance suspension intelligent agent that is adaptable to multiple operating conditions, highly robust, and provides strong comfort. The deep reinforcement learning-based ambulance suspension intelligent agent is obtained through the deep reinforcement learning-based ambulance suspension intelligent agent optimization method. Specifically, on one hand, the present invention constructs a population containing multiple controllers by integrating the "population exploration characteristics" of swarm intelligence algorithms, and generates full-condition data covering "multiple road conditions (Class A to Class C), multiple speeds (20m / s~50m / s), and multiple loads (500kg~1000kg)," ensuring that the control strategy can adapt to the dynamic changes in the transport scenario during ambulance transport, and guaranteeing that the vertical acceleration of the vehicle body can be stably controlled at 0.5m / s² under any road condition / load / speed. 2 The invention addresses the problem of poor condition adaptability in traditional suspension control methods, achieving dynamic adaptive control for ambulances across multiple scenarios. Furthermore, it employs a dual-module synergy of population exploration and deterministic experience tracking: leveraging the global exploration capabilities of swarm intelligence algorithms, the condition coverage is increased to more than three times that of existing DDPG or TD3 algorithms; relying on the "independent experience cache + temporal experience vector" mechanism of deterministic experience tracking, sample utilization efficiency is improved by 80%, and the training cycle is shortened from 10,000 rounds to less than 5,000 rounds; simultaneously, "elite set screening" retains high-quality control strategies, ensuring control robustness. Under extreme conditions without training, the performance degradation rate is less than 5%, resolving the imbalance between exploration and utilization in the DRL algorithm and balancing sample utilization efficiency with control robustness. On the one hand, the swarm intelligence algorithm compensates for the shortcomings of global exploration, ensuring that the control strategy is more adapted to the needs of ambulance transfer, and solving the problems of insufficient exploration and poor scenario adaptation of the DET algorithm in related technologies. In addition, through the design of a lightweight evaluation network and control strategy network and a targeted reward function, this invention focuses on optimizing the vibration suppression of the human body's vertically sensitive frequency band of 4Hz~12.5Hz. The active control force output by the control strategy network can offset the road excitation in this frequency band in real time, increasing the vibration attenuation rate of this frequency band to more than 90%. The reward function assigns the highest weight of 0.3 to the vehicle acceleration (especially the 4Hz~12.5Hz component), guiding the strategy to prioritize reducing vibration in this frequency band, significantly reducing the risk of secondary injury to patients, and solving the problem of insufficient targeting of comfort optimization in related technologies.

[0020] In summary, the ambulance suspension intelligent agent and its optimization method based on deep reinforcement learning provided by this invention, through full-condition exploration, deep experience mining and specific comfort optimization, solves the shortcomings of related technologies in multi-scenario adaptation, robustness and comfort of ambulances. Ultimately, it ensures that intensive care ambulances can provide patients with a low-vibration and highly stable riding environment under any transport conditions, which helps to reduce secondary injuries and ensure transport safety. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 A schematic diagram of the dynamic model of the intelligent suspension system for ambulances as described in an embodiment of the present invention; Figure 2 The architecture diagram of the ambulance suspension intelligent agent optimization method based on deep reinforcement learning, which combines evolutionary strategies and deterministic experience tracking mechanisms, is provided in the embodiments of the present invention. Figure 3 This is a structural diagram of a control strategy network according to an embodiment of the present invention. Detailed Implementation

[0022] Analysis revealed that the deterministic experience tracking mechanism innovatively stores state and operation data in an independent experience memory buffer and cleverly processes this information as auxiliary rewards based on time-based logical decay. This incentivizes the intelligent suspension deep reinforcement learning agent to iterate and improve rapidly. The application of this mechanism significantly accelerates the learning process and improves the optimization efficiency of the strategy, providing strong technical and theoretical support for the practical application of deep reinforcement learning in suspension control and even broader fields. Swarm intelligence algorithms are widely used due to their advantages in global search capabilities and adaptability. They can effectively explore the search space regardless of the nature of the problem. Furthermore, swarm intelligence algorithms possess self-organizing, adaptive, and self-learning characteristics, which are similar to those of deep reinforcement learning, making them somewhat similar and complementary.

[0023] In view of this, the present invention proposes an ambulance suspension intelligent agent and its optimization method based on deep reinforcement learning, which cleverly integrates swarm intelligence algorithm and deterministic experience tracking mechanism, aiming to effectively promote the good performance of ambulance intelligent suspension control strategy under various working conditions, effectively improve the stability and comfort of intensive care ambulances, and reduce secondary harm to patients.

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0025] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0026] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0027] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0028] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] This invention provides a method for optimizing ambulance suspension agents based on deep reinforcement learning, comprising: S1: initializing the evaluation network and the control policy network; S2: initializing the population using a swarm intelligence algorithm, wherein the population consists of... The system consists of controllers with different parameters. Each controller has the same network structure as the control strategy network. The swarm intelligence algorithm can be any one of evolutionary strategy, genetic algorithm, or particle swarm optimization. An experience replay cache pool is initialized. as well as A deterministic experience memory cache pool Deterministic experience memory cache pool One-to-one correspondence with controllers within the population; S3: Set the number of generations M for controller evolution and the time step T for each generation. In each time step, perform the following operations for each controller: Set the initial fitness of the controller... Set to 0, extract state information from the ambulance's overall vehicle dynamics system; at each time step t, the control strategy network adjusts the state based on the current time step. Output suspension control force ; to adjust the current suspension control force Reward Value And the state in the next moment. Packaged as primary samples Save to the corresponding deterministic experience memory cache pool Generate deterministic empirical tracking vectors Deterministic experience tracking vector Compared with primary samples Combine to form a new sample And save it to the experience replay cache pool. Based on the reward value of the current time step With auxiliary rewards Update the controller's adaptability S4: After each generation, a swarm intelligence algorithm is used to update the population based on fitness, and the experience replay cache is used. Select group A samples to train the evaluation network and control policy network to update network parameters; S5: After completing the M-generation evolution, from The controller with the highest fitness is selected from the controllers, and its corresponding network parameters are set in the control strategy network to output the suspension control force.

[0030] In some embodiments, in step S1, the core network parameters of the evaluation network and the core network parameters of the control strategy network can be initially determined by training in a simulation environment or by offline training based on a large amount of historical data. Steps S1 to S5 are optimization training performed on the training path before formal transfer.

[0031] In some embodiments, M can be set to 2000 and T can be set to 1000.

[0032] In some embodiments, extracting state information from the ambulance's vehicle dynamics system includes: obtaining data such as vehicle acceleration, vehicle speed, suspension deflection, and the derivative of suspension deflection from the ambulance's vehicle system via sensors; combining these data into state information for a deep learning system using a state observer and a state estimation algorithm; then, the controller determines the amount of suspension force to be output to the vehicle dynamics system in the current state based on this state information; the vehicle state changes under the action of the suspension force; the ambulance suspension agent based on deep reinforcement learning generates a reward value to evaluate its control actions based on the new system state and the intrinsic reward function; and the ambulance suspension agent based on deep reinforcement learning performs self-iteration and control strategy optimization based on the reward value.

[0033] Furthermore, the evaluation network includes a first evaluation network and a second evaluation network with the same network structure but different network parameters. The first evaluation network outputs a first evaluation result Q1, and the second evaluation network outputs a second evaluation result Q2. The evaluation result output by the evaluation network is min(Q1, Q2). The evaluation result output by the evaluation network is used to guide the control strategy network to optimize the suspension control force. The first evaluation network can be used... This indicates that the second evaluation network can be used This indicates that lightweight policy networks can be used It means that, among them, s Indicates status information, a Indicates control. Represents the network weights. This indicates the weights of the first evaluation network. This indicates the weights of the second evaluation network. This represents the weights of the lightweight strategy network.

[0034] Control a In other words, the design of actions needs to consider the safety requirements of autonomous vehicles and the physical limitations imposed by the power of the actuators, thereby reducing related risks. Therefore, in some embodiments, the control force... a Perform hard constraint processing, that is ,in, , This refers to the range of motion constraints, the value of which can be determined based on the maximum force limit of the actuator.

[0035] In some embodiments, step S1 further includes: initializing a target evaluation network and a target control policy network, wherein the parameters of the target evaluation network are consistent with the parameters of the evaluation network, and the parameters of the target control policy network are consistent with the parameters of the control policy network; the target evaluation network includes a first target evaluation network and a second target evaluation network; in step S4, after training the evaluation network and the control policy network, the weights of the target evaluation network and the weights of the target control policy network are also updated synchronously; wherein, ', j When =1, The weights of the first evaluation network, The primary objective is to evaluate the network weights. j When =2, The weights for the second evaluation network, The weights of the network are evaluated for the second objective; To reduce the weights of the control policy network, Let τ be the weight of the target control policy network; τ = 0.001. This enables soft updates to the target network, ensuring the stability of the updates.

[0036] In some embodiments, reference Figure 3 The control strategy network includes an input layer, a hidden layer, and an output layer. The hidden layer includes a fully connected layer, a ReLU activation layer, a fully connected layer, a ReLU activation layer, a fully connected layer, and a Tanh activation layer connected in sequence. The number of neurons in the fully connected layer connected to the Tanh activation layer is 1, and the number of neurons in the remaining two fully connected layers is the same. The number of neurons in each layer can be 64.

[0037] It should be noted that this invention, targeting the intelligent suspension control task of ambulances, designs a unique lightweight neural network architecture. Without affecting algorithm performance, it significantly reduces the total number of parameters required by the model, decreasing it by more than 80% compared to the traditional DET model, effectively reducing model complexity. This design also greatly reduces the computational burden and improves the algorithm's operating efficiency in actual vehicle systems, enabling the algorithm to run efficiently in resource-constrained in-vehicle environments. After optimization, the control strategy network exhibits excellent computational response speed on a standard PC (Personal Computer) platform, completing the computation response in just 3 milliseconds. This fully meets the stringent real-time control performance requirements of the ambulance intelligent suspension system. This performance advantage ensures that, during actual driving, the ambulance suspension agent based on deep reinforcement learning can adjust the control force in a timely manner according to road conditions and vehicle status, effectively improving the system's reliability and stability.

[0038] Furthermore, in step S4, the experience replay cache pool is... Selecting Group A samples for training the evaluation network and control policy network includes: using samples from the experience replay cache pool. A group of samples is randomly selected, A=128, and the loss function is minimized using the stochastic gradient descent method. Update the weights of the first and second evaluation networks, where E represents the expected value. j When it is 1, This indicates the weights of the first evaluation network. j When it is 2, This indicates the weights of the second evaluation network. , y t Let Q represent the target Q value, and γ be a discount factor, γ = 0.99. The weights of the control policy network are updated using the stochastic gradient ascent method, and the update formula is as follows: N=A, This represents the target gradient of the control policy network. This represents the gradient feedback of the evaluation network on the value of the current action.

[0039] Further reference Figure 2 , When swarm intelligence algorithms employ an evolutionary strategy, they can be based on a Gaussian distribution. The initialization involves a group of 10 controllers, among which... This represents the mean of a Gaussian distribution. This represents the standard deviation of a Gaussian distribution. Figure 2 middle, actor 1 to actor 10 These represent the corresponding controllers. μ 1 to μ 10 These represent the weights of the corresponding controllers.

[0040] In some embodiments, when the swarm intelligence algorithm is an evolutionary strategy, in step S2, the parameters of each of the 10 controllers are independently sampled from a Xavier normal distribution with a mean of 0 and a variance of 0.1; in step S4, updating the population based on fitness using an evolutionary strategy includes: selecting the top 5 elite controllers in terms of fitness, calculating the new mean μ_new and new standard deviation σ_new of the parameters of the 5 elite controllers, and generating 10 new controllers based on the Gaussian distribution corresponding to the new mean μ_new and new standard deviation σ_new to form a new generation of population.

[0041] This invention constructs a population containing 10 controllers by integrating the population exploration characteristics of an evolutionary strategy. It generates comprehensive operating condition data covering multiple roads, speeds, and loads based on Gaussian distribution sampling. The evolutionary strategy uses an elite set to estimate new means and standard deviations, guiding the population towards a better direction. This deep integration of the evolutionary strategy and a deterministic experience tracking mechanism fully utilizes the population characteristics of the evolutionary strategy. Through the diversity and competitive cooperation mechanisms among individuals in the population, large-scale exploration is achieved, widely collecting data under different operating conditions to ensure comprehensive and representative data. Simultaneously, the deterministic experience tracking mechanism deeply mines the intrinsic value of the data, processing deep information in the time-series states, providing strong support for control strategy optimization. The organic combination of these two approaches forms a unique hybrid architecture that effectively balances exploration and utilization.

[0042] In the evolutionary strategy, after obtaining the elite set (the top 5 controllers in terms of fitness), the system does not simply deploy the parameters of the current best controller directly. Instead, it adopts a more robust strategy: re-estimating the Gaussian distribution of the control strategy parameters based on the elite set. This new Gaussian distribution represents the validated high-quality strategy space found in the current evolutionary stage. After completing all the preset generations of evolution, the system selects the controller with the highest fitness from the "elite set" throughout the entire evolutionary history. Its network parameters are determined as the global optimal solution. During the deployment phase, these rigorously tested parameters are solidified and loaded into the control strategy network of the FPGA (Field-Programmable Gate Array) or other onboard computing devices, replacing the initial or old strategies. This ensures that the actual control strategy network inherits the optimal wisdom from the entire population evolution process, ensuring that the ambulance's active suspension can perform the most accurate and efficient control actions under complex and ever-changing real road conditions.

[0043] Furthermore, When the swarm intelligence algorithm is a genetic algorithm, in step S2, the controller weights are flattened into a one-dimensional chromosome vector c, c∈R. dWhere d is the total number of parameters and R represents the set of real numbers; 10 independent chromosome vectors are generated to form a population P={c1, c2, ..., c10}, and the chromosome vectors correspond to the controller. Each parameter value of each chromosome vector is randomly sampled from a uniform distribution in the interval [-0.1, 0.1]. In step S4, the population is updated according to fitness using an evolutionary strategy, including: using a tournament selection strategy to select 10 parent chromosome vectors; performing a single-point crossover operation on the 10 parent chromosome vectors to generate 10 primary offspring chromosome vectors; and performing Gaussian mutation on the chromosome gene loci with a probability pm=0.1, with a mean of 0 and a variance of 0.01, to generate 10 offspring chromosome vectors; retaining the two elite chromosome vectors with the highest fitness in the population, and selecting the 10 chromosome vectors with the highest fitness from the two elite chromosome vectors and the 10 offspring chromosome vectors to form the new generation population. Retaining the two elite chromosome vectors with the highest fitness in the population helps to ensure that excellent genes are not lost.

[0044] In some embodiments, the selection of 10 parent chromosome vectors using a tournament selection strategy includes: randomly selecting 3 individuals from the population each time using a tournament selection strategy, and then selecting the individual with the highest fitness from the 3 individuals to enter the mating pool. This process is repeated 10 times until the mating pool is full.

[0045] In some embodiments, performing a single-point crossover operation on 10 parent chromosome vectors to generate 10 primary offspring chromosome vectors includes: randomly selecting one "crossover point k" (k is a dimension of the chromosome vector, such as the position of the 5th parameter); concatenating the first k parameters of the first parent chromosome vector with the (k+1)th to dth parameters of the second parent chromosome vector to generate one offspring chromosome; similarly, concatenating the first k parameters of the second parent chromosome vector with the last dk parameters of the first parent chromosome vector to generate another offspring chromosome vector, and generating a total of 10 offspring chromosome vectors from 5 pairs of parent chromosome vectors.

[0046] Furthermore, When performing particle swarm optimization using swarm intelligence algorithms, in step S2, the controller is mapped to particles, and the controller weights are mapped to particle positions. The velocity of each particle represents the parameter update direction and magnitude of the particle at the next moment. Ten particles are generated to form a swarm, and the initial position x of each particle is... i 0 The initial velocity v is obtained by random sampling from a uniform distribution in the interval [-0.1, 0.1]. i 0Random sampling is performed from a uniform distribution in the interval [-0.05, 0.05]. The velocity range is smaller than the position range to avoid excessively large initial parameter updates that could lead to controller performance instability. In step S4, for each particle, the position corresponding to the highest fitness in the current generation is obtained as pbest. i The position of the particle with the highest fitness in the entire population is taken as gbest; according to the formula v i t+1 =wv i t +c1r1(pbest i -x i t )+c2r2(gbest-x i t Update the velocity of each particle in the population according to the formula x. i t+1 =x i t +v i t+1 Update the positions of all particles in the population, where w = 0.8, w is the inertia weight, c1 = c2 = 2.0, c1 and c2 are both learning factors, r1 and r2 are both random numbers in the interval [0, 1], and pbest i Let be the individual historical best position of the i-th particle, and gbest be the global best position of the population. If the particle position exceeds the range of [-1, 1], then reflection boundary processing is used. The updated particles constitute the next generation of the population.

[0047] Furthermore, the status information includes: vehicle displacement. Vehicle speed Vehicle acceleration Wheel acceleration Suspension dynamic deflection Suspension dynamic deflection derivative The control force output by the suspension at the previous moment This state information design ensures that the deep reinforcement learning controller can make reasonable decisions based on accurate and effective information, which is a key element in achieving efficient suspension control.

[0048] It should be noted that for deep reinforcement learning, the selection and construction of state information usually has a decisive impact on the decision-making of the deep reinforcement learning controller. Although raw and large amounts of state data can give the agent greater freedom in decision-making, excessively redundant data will greatly interfere with the deep reinforcement learning controller's extraction and integration of truly effective signals. To explore further, based on the actual situation of ambulance intelligent suspension control, some data, although easily obtained in the simulation environment, are often in an unobservable state in real-world applications. Moreover, different state information is often adapted to different control objectives. Therefore, considering the effectiveness and observability of state information, and adapting it to the actual control objectives of ambulance comfort and stability, it is necessary to finely screen and optimize the state information to ensure that the deep reinforcement learning controller can make decisions that meet actual needs based on accurate and effective information. This will enable efficient control of the ambulance intelligent suspension, improve the comfort and stability of the ambulance during operation, and provide more reliable protection for medical rescue work.

[0049] In the status information of this invention, vehicle displacement Vehicle speed Vehicle acceleration The sprung mass displacement, a key indicator for evaluating ambulance ride comfort, directly reflects the vibration and impact on patients and medical equipment during vehicle operation. A smaller sprung mass displacement reduces swaying of passengers and medical equipment, improving ride comfort. It's easy to understand that excessive acceleration increases the inertial forces on patients and medical equipment, leading to discomfort and fatigue. Based on Newton's first law, a detailed analysis of the sprung mass of the patient clarifies other key states, including suspension dynamic deflection. This characterizes the dynamic travel of the suspension, a crucial physical constraint for ambulance suspensions during actual driving. The deep reinforcement learning controller can also learn and understand the current elastic force of the suspension springs through this state. (Suspension dynamic deflection derivative) This effectively characterizes the dynamic response of the suspension system to road excitations, helping the deep reinforcement learning controller accurately assess the suspension's operating state and calculate its damping force accordingly. In addition, The dynamic load changes of the tires are a crucial state directly related to road excitation; however, in practical engineering, unless road conditions are previewed using sensors, it is difficult to obtain an approximate value for this state. Therefore, based on Newton's second law, we choose... and The state combinations are used to reflect the intuitive impact of road stimuli to some extent. It should be noted that while in an ideal theoretical scenario, a deep reinforcement learning controller can make decisions simply by integrating the aforementioned states, given the numerous constraints that may arise in actual ambulance operation, such as actuator constraints and system time delays, it is still necessary to add the intelligent suspension actuator force from the previous moment to ensure system convergence to a certain extent. (This is used as feedback information.)

[0050] Furthermore, in step S4, ,in, As the core reward, P This is a mandatory penalty item; , , , , , , as well as All are weighting coefficients. , , , , , , , , , , , as well as All are soft reward coefficients. =0.04, =0.4, =4, =0.017, =0.15, =15, P =-500, when the suspension dynamic deflection It is triggered at a certain time, and when triggered, the calculation of the time step of the current generation is forcibly terminated; .

[0051] It should be noted that in a natural driving environment, the control of the intelligent suspension system needs to meet multi-objective control performance requirements. Among these, ride comfort is the primary performance indicator; in addition, it must also meet the safety and driving performance requirements of the autonomous vehicle. Based on these considerations, the aforementioned core reward is designed. It is important to note that This indicates the extraction of the vertical acceleration component of the vehicle body in the frequency domain of 4Hz~12.5Hz (which can be calculated in real time using a bandpass filter). Assigned the highest weight, the forced strategy prioritizes suppressing vibrations sensitive to the human body, ultimately achieving an effective reduction in the vehicle's vertical acceleration and effective suppression of vibration acceleration, significantly reducing the risk of secondary injury to patients. The soft reward coefficient can make each part of the reward dimensionless. In the design of the reward function, vehicle displacement... Vehicle speed and vehicle acceleration These factors collectively characterize the ride comfort of an ambulance, which is the most important performance indicator. Suspension dynamic deflection It is an important safety indicator and a real physical constraint in the suspension control process. This characterizes the dynamic load changes of the tires, which is closely related to the handling stability of the ambulance and thus affects the actual driving performance of the ambulance. This ensures that the most suitable and energy-efficient force is used to achieve the best control effect. The integrated reward function is: , hour, ,in, This represents the limit of the suspension's dynamic deflection, which is 0.15m. When the suspension's dynamic deflection exceeds this value, the episode will be forcibly terminated, and the deep reinforcement learning agent will also be severely penalized.

[0052] Furthermore, in step S3, a deterministic empirical tracking vector is generated. This includes: storing primary samples in chronological order into a deterministic experience memory cache. ,Right now , where k is any positive integer; , , Represents the experience processing function. The parameters represent the empirical processing function; .

[0053] Furthermore, steps S1 to S5 are used to perform initial optimization on the ambulance suspension agents based on deep reinforcement learning for multiple ambulances; each ambulance is regarded as an individual, and each individual collects and stores operational data in real time during actual transport in a deterministic experience memory cache pool. In this process, operational data includes status information and patient physiological data; each individual undergoes periodic local generational evolution, which includes individual fine-tuning and cloud-based unified optimization. Individual fine-tuning includes updating fitness. Among them, reward value Add health penalty items ,in, The weight of health penalty items, Representing the collected patient heart rates, the control policy network was retrained using local data according to step S4, with a learning rate of 0.0001. Cloud-based unified optimization included: after individual fine-tuning for each network, the locally updated network parameters and key performance indicators were encrypted and uploaded to the cloud server. The cloud server performed unified optimization, where key performance indicators included average fitness and patient physiological data statistics. Unified optimization involved the cloud server using a federated averaging or evolutionary aggregation algorithm to calculate and update weights. Where N is the number of ambulances. Let be the number of samples for the i-th vehicle. The original weights of the control strategy network for vehicle i are used to update the weights of the control strategy network for each individual vehicle. This helps to cover a wider range of operating conditions. Based on global data, step S4 is used to re-optimize the network parameters of the control strategy network for each individual vehicle. The parameters of the control strategy network of the individual with the highest fitness among all individuals are then distributed to the control strategy networks of each ambulance.

[0054] This process of localized generational evolution is carried out periodically, for example, once a week, to ensure that all vehicles benefit from the collective experience in a synchronized manner.

[0055] Specifically, patient physiological data can include heart rate, blood pressure, etc., and this data can be collected in real time through onboard medical equipment; reward value Adding health penalties helps to further ensure patient safety; reducing the learning rate to 0.0001 helps to avoid overfitting and maintain real-time performance; cloud servers can be deployed based on the hospital's existing infrastructure. Online self-evolution enables the agent to continuously adapt to individual differences (such as patient condition), while cloud aggregation achieves global knowledge sharing, further reducing performance degradation. Simultaneously, sharing only parameters rather than raw data protects patient privacy and aligns with medical ethics.

[0056] Another aspect of this invention provides an ambulance suspension intelligent agent based on deep reinforcement learning. The ambulance suspension intelligent agent based on deep reinforcement learning is obtained based on the above-mentioned optimization method for ambulance suspension intelligent agents based on deep reinforcement learning. During actual transport, the ambulance suspension intelligent agent based on deep reinforcement learning generates suspension control force based on the collected state and applies it to the ambulance active suspension system.

[0057] It should be noted that the ambulance suspension intelligent agent and its optimization method based on deep reinforcement learning provided by this invention are not limited to intelligent suspension control, but are also applicable to semi-intelligent suspension and control systems of all controllable suspensions, and have wide applicability and versatility.

[0058] The hardware system composition of an ambulance provided by the present invention is described below. In some embodiments, the hardware system adopts an architecture composed of an FPGA smart terminal and multiple sensors, specifically including a data acquisition device, a processor, and a memory. The processor is used to run the algorithm corresponding to the ambulance suspension intelligent agent optimization method based on deep reinforcement learning. The memory is used to store network parameters. The data acquisition device is used to collect state data from the ambulance system. The state data is combined into state information adapted to the ambulance suspension intelligent agent optimization method based on deep reinforcement learning through the state observer and state estimation algorithm in the processor.

[0059] The data acquisition device may include a 3-axis accelerometer, a displacement sensor, and an inertial measurement unit. The data acquisition device is used to collect state data and then generate vehicle body displacement data. Vehicle speed Vehicle acceleration Wheel acceleration Suspension dynamic deflection Suspension dynamic deflection derivative The control force output by the suspension at the previous moment Status information, etc.

[0060] In some embodiments, the dynamic model of the intelligent suspension system is as follows: Figure 1 As shown, Figure 1 middle, m b This indicates the sprung mass (i.e., the vehicle body mass). m u Indicates unsprung mass. z b This represents the vertical displacement of the sprung mass (vertical displacement of the vehicle body). z u This represents the vertical displacement of the unsprung mass. k b Indicates suspension stiffness. c b Indicates the damping coefficient. k u Indicates tire stiffness. F This indicates the active control force of the suspension. z r This indicates road surface displacement.

[0061] Simulation data demonstrates that the ambulance suspension intelligent agent and its optimization method based on deep reinforcement learning, as proposed in this invention, have the characteristics of improved control performance, strong robustness and generalization ability, efficient algorithm deployment, and potential for intelligent expansion.

[0062] Regarding the improvement in control performance, the specific results are as follows: Compared with traditional algorithms, the ambulance suspension intelligent agent based on deep reinforcement learning provided by this invention exhibits outstanding performance in the vertical acceleration control of the vehicle body, with a root mean square acceleration value as low as 0.4175 m / s². 2 Compared to the passive suspension of ambulances, ride comfort is improved by over 87%. In comparisons with baseline algorithms such as DDPG, TD3, LQR (Linear Quadratic Regulator), and DET (Deterministic Experience Tracking), performance improvements were achieved by 53.5%, 37.2%, 59.3%, and 29.8%, respectively. This means that vehicle vibration and bumps are significantly suppressed during ambulance travel, providing patients with a smoother and more comfortable transport environment. This effectively reduces discomfort and potential harm caused by vehicle shaking, especially for critically ill patients, helping to maintain their physiological stability and reduce transport risks. Furthermore, this invention also demonstrates significant effectiveness in vehicle displacement and speed control. The RMS (Root Mean Square) value of vehicle displacement is significantly reduced compared to passive suspension, reaching a reduction of up to 92% in some cases. This effectively reduces vehicle sway and improves vehicle posture stability. The RMS value of vehicle speed is also significantly optimized, resulting in a smoother response to road unevenness during driving and indirectly improving overall vehicle ride comfort, thus providing better assurance for the stable operation of in-vehicle medical equipment. Additionally, frequency response characteristic analysis shows that the ambulance suspension intelligence agent based on deep reinforcement learning provided by this invention has a significant suppression effect on vehicle vibration across the entire frequency range, especially performing exceptionally well in the 4Hz-12.5Hz frequency range, where the human body is most sensitive in the vertical direction. It almost completely suppresses or even eliminates the first resonance peak, further demonstrating its enormous potential in improving ride comfort and providing patients with a quieter and smoother riding experience, which helps alleviate patient anxiety during transport.

[0063] Regarding its robustness and generalization ability, the deep reinforcement learning-based ambulance suspension agent provided by this invention exhibits excellent adaptability under different road conditions. Whether on relatively flat Class A roads or challenging Class D roads, it maintains a high level of comfort optimization. It achieves over 85% average comfort optimization on Class C roads and maintains over 80% comfort optimization on Class D roads. This demonstrates that the deep reinforcement learning-based ambulance suspension agent provided by this invention can automatically adjust control force according to different road conditions, effectively responding to changes in road excitation and ensuring that the ambulance provides stable riding comfort for patients under various complex road conditions, greatly improving the applicability of ambulances in actual transport scenarios. Faced with changes in driving speed, the ambulance suspension intelligent agent based on deep reinforcement learning provided by the invention also performed excellently. Specifically, in tests covering 20 different levels of experimental scenarios (10m / s to 50m / s, four road levels: A, B, C, and D), it consistently maintained an optimization effect of over 70%, proving its good adaptability to changes in vehicle speed. It can ensure the riding comfort and driving stability of the ambulance at different driving speeds, enabling the ambulance to provide safe and comfortable transfer conditions for patients when performing emergency tasks, regardless of speed changes. In response to significant load fluctuations in ambulances, the robustness of the ambulance suspension intelligence agent based on deep reinforcement learning provided in this invention has been fully verified. Even when the sprung mass changes by nearly double from unloaded to fully loaded, the key performance indicators only show a slight decrease, and the vehicle acceleration performance degrades by less than 5%, maintaining a comfort optimization level of over 80%. This demonstrates that the ambulance suspension intelligence agent based on deep reinforcement learning provided in this invention can adapt to the impact of load changes, ensuring the control effect of the intelligent suspension system under different load conditions, and effectively guaranteeing the ride comfort and safety of ambulances in actual operation when facing various load situations.

[0064] Regarding the algorithm's efficient deployment and potential for intelligent expansion, the following are specific details: The lightweight neural network architecture significantly reduces the computational resource requirements of the ambulance suspension intelligent agent based on deep reinforcement learning provided by this invention. This not only reduces the hardware requirements for onboard computing equipment and lowers costs, but also improves the ease of deployment of the algorithm in actual vehicle systems. The lower memory usage and computational burden enable the algorithm to load and run more quickly, reducing system response time and improving overall system efficiency, thus providing strong support for the practical application of the ambulance intelligent suspension control system.

[0065] This invention lays a solid foundation for further enhancing the level of intelligence by combining multi-source data. By further integrating multi-source data such as vehicle status, road condition information, and GPS (Global Positioning System) positioning, the ambulance suspension intelligence agent based on deep reinforcement learning provided by this invention is expected to more accurately perceive the driving environment, predict road condition changes in advance, and thus formulate more precise control strategies. This will further improve the control performance of the intelligent suspension system, achieve more personalized and intelligent control, adapt to the future development needs of intelligent transportation and medical transport, and promote the continuous improvement of the intelligence level of special vehicles. For example, it can adjust the suspension stiffness and damping in advance according to road condition information, optimize the vehicle posture before entering bumpy road sections, and improve ride comfort; and combine GPS positioning and vehicle status information to automatically switch the most suitable control mode under different driving areas and operating conditions, thereby improving the overall performance of the ambulance.

[0066] In summary, this invention not only utilizes swarm intelligence algorithms to achieve efficient exploration within a broad search space, but also delves into the intrinsic value of data through a deterministic experience tracking mechanism, thereby promoting iterative optimization of control strategies and effectively improving control performance and robustness.

[0067] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0068] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A deep reinforcement learning based ambulance suspension agent optimization method, characterized in that, Comprising: S1: initialize the evaluation network and the control policy network; S2: initializing a population by a swarm intelligence algorithm, the population being composed of controllers with different parameters, each controller having the same network structure as the control strategy network, the swarm intelligence algorithm being any one of an evolutionary strategy, a genetic algorithm or a particle swarm optimization, initializing an experience replay buffer pool and a deterministic experience memory buffer pool , the deterministic experience memory buffer pool corresponding to the controllers in the population one by one; S3: Set the controller evolution generation number M and the time step T of each generation, and in each time step, perform the following operations on each controller: set the initial fitness of the controller to 0, extract the state information from the ambulance whole vehicle dynamics system; in each time step t, the control strategy network outputs the suspension control force according to the state of the current moment ; the current suspension control force , the reward value and the state of the next moment are packaged into primary samples saved to the corresponding deterministic experience memory buffer pool ;​​ Generating deterministic experience trace vectors , the deterministic experience trace vectors combined with primary samples to form new samples and saved to an experience replay buffer pool , the reward value and auxiliary rewards , updating the fitness of the controller ; S4: after each generation, the swarm intelligence algorithm is used to update the population according to fitness, and the experience replay cache pool is used to update the experience replay cache pool A group of samples are selected to train the evaluation network and the control policy network to update network parameters; S5: After completing the Mth generation evolution, the controller with the highest fitness is selected from the controllers, and the corresponding network parameter of the controller is set in the control strategy network for outputting the suspension control force. The controller with the highest fitness is selected from the controllers, and the corresponding network parameter of the controller is set in the control strategy network for outputting the suspension control force.

2. The deep reinforcement learning-based ambulance suspension agent optimization method of claim 1, wherein, The evaluation network comprises a first evaluation network and a second evaluation network with the same network structure but different network parameters, the first evaluation network outputs a first evaluation result Q1, the second evaluation network outputs a second evaluation result Q2, and the evaluation result output by the evaluation network is min(Q1, Q2), which is used to guide the control policy network to optimize the suspension control force; Step S1 further comprises: initializing a target evaluation network and a target control policy network, the parameters of the target evaluation network are consistent with the parameters of the evaluation network, and the parameters of the target control policy network are consistent with the parameters of the control policy network, and the target evaluation network comprises a first target evaluation network and a second target evaluation network; In step S4, after training the evaluation network and the control policy network, the weights of the target evaluation network and the weights of the target control policy network are also updated synchronously; wherein, j = 1, is a weight of the first evaluation network, is a weight of the first target evaluation network, j = 2, is a weight of the second evaluation network, is a weight of the second target evaluation network; is a weight of the light-weight control policy network, is a weight of the target control policy network; τ = 0.001.​ 3. The deep reinforcement learning-based ambulance suspension agent optimization method of claim 2, wherein, In step S4, from the experience replay buffer pool Training the evaluation network and the control policy network with the A-group samples comprises: randomly select A group of samples from the experience replay cache pool , A = 128, minimize the loss function by using the stochastic gradient descent method update the weights of the first evaluation network and the second evaluation network, wherein E represents the mathematical expectation, j when 1, represents the weights of the first evaluation network, j when 2, represents the weights of the second evaluation network, , y t represents the target Q value, and γ is the discount factor, γ = 0.99; the weights of the control policy network are updated by using the stochastic gradient ascent method, and the update formula is , N = A.

4. The deep reinforcement learning-based ambulance suspension agent optimization method of claim 1, wherein, When the swarm intelligence algorithm is an evolutionary strategy, the parameters of each of the 10 controllers in step S2 are independently sampled from a Xavier normal distribution with a mean of 0 and a variance of 0.

1. In step S4, the evolution strategy is used to update the population according to the fitness, which comprises: selecting the top 5 elite controllers in terms of fitness, calculating the new mean μ_new and the new standard deviation σ_new of the parameters of the 5 elite controllers, generating 10 new controllers based on the Gaussian distribution corresponding to the new mean μ_new and the new standard deviation σ_new to form a new generation population.

5. The deep reinforcement learning-based ambulance suspension agent optimization method of claim 1, wherein, When the swarm intelligence algorithm is a genetic algorithm, the weights of the controller are flattened into a one-dimensional chromosome vector c in step S2, c ∈ R d where d is the total number of parameters, and R represents a real number set; 10 independent chromosome vectors are generated to form a population P = {c1, c2, …, c10}, and each chromosome vector corresponds to a controller; each parameter value of each chromosome vector is randomly sampled from a uniform distribution in the interval [-0.1, 0.1]. In step S4, the evolution strategy is used to update the population according to the fitness, which comprises: using a tournament selection strategy to screen 10 parent chromosome vectors; performing single-point crossover operation on the 10 parent chromosome vectors to generate 10 initial child chromosome vectors; and performing Gaussian mutation with a mean of 0 and a variance of 0.01 on the chromosome gene sites with a probability pm=0.1 to generate 10 child chromosome vectors; retaining the two elite chromosome vectors with the highest fitness in the population, and selecting the 10 chromosome vectors with the highest fitness from the two elite chromosome vectors and the 10 child chromosome vectors to form a new generation population.

6. The deep reinforcement learning-based ambulance suspension agent optimization method of claim 1, wherein, In the step S2, the controller is mapped as a particle, the weight of the controller is mapped as the position of the particle, the speed of each particle represents the parameter updating direction and amplitude of the particle at the next time, 10 particles are generated to constitute a population, the initial position x i 0 Randomly sampled from the uniform distribution of the interval [-0.1, 0.1], the initial speed v i 0 Randomly sampled from the uniform distribution of the interval [-0.05, 0.05] In step S4, for each particle, the position corresponding to the highest fitness in the current generation is obtained as pbest i The position of the particle with the highest fitness in the entire population is obtained as gbest. According to the formula v i t+1 = wv i t + c1r1(pbest i - x i t ) + c2r2(gbest - x i t ) updates the velocity of each particle in the population according to the formula x i t+1 = x i t + v i t+1 updates the position of each particle in the population, wherein w = 0.8, w is an inertia weight, c1 = c2 = 2.0, c1 and c2 are both learning factors, r1 and r2 are both random numbers in the interval [0, 1], pbest i is the individual historical optimal position of the i-th particle, gbest is the global optimal position of the population, wherein if the particle position exceeds the range [-1, 1], a reflective boundary treatment is adopted; the updated particle constitutes a new generation of the population.

7. The deep reinforcement learning-based ambulance suspension agent optimization method of claim 1, wherein, The state information includes: vehicle body displacement , vehicle body speed , vehicle body acceleration , wheel acceleration , suspension dynamic deflection , suspension dynamic deflection derivative , and control force outputted by the suspension at the last time .

8. The deep reinforcement learning-based ambulance suspension agent optimization method according to claim 7, wherein, In step S4, wherein, is a core reward, P is a mandatory penalty term; , , , , , , and are weight coefficients, , , , , , , , , , , , and are soft reward coefficients, = 0.04, = 0.4, = 4, = 0.017, = 0.15, = 15, P = -500, triggered when the suspension deflection is greater than 0.5 m, and, when triggered, mandatorily ending the time step calculation of the current generation. 。 9. The deep reinforcement learning-based ambulance suspension agent optimization method according to claim 8, wherein, The plurality of ambulance suspension agents based on deep reinforcement learning are initialized and optimized by steps S1 to S5. Each ambulance is regarded as an individual, and each individual is real-time collected running data in the actual transfer process and stored in the deterministic experience memory cache pool In which the running data includes state information and patient physiological data; Each individual periodically undergoes local in generation evolution, which includes individual fine-tuning and cloud same optimization, the individual fine-tuning includes: updating the fitness function Wherein, the reward value Increase the health penalty term Wherein, The weight of the health penalty term is, Represent the collected patient heart rate, retrain the control strategy network according to step S4 using local data, the learning rate is 0.0001; The cloud same optimization comprises: after each individual is fine-tuned, the local updated network parameters and key performance indicators are encrypted and uploaded to the cloud server, the cloud server performs unified optimization, wherein the key performance indicators comprise: average fitness and patient physiological data statistics, and the unified optimization comprises: the cloud server adopts a federated average or evolutionary strategy aggregation algorithm to calculate an update weight wherein N is the number of ambulances, is the sample number of the ith vehicle, is the original weight of the control strategy network of the ith vehicle, the update weight is assigned to the control strategy network of each individual, the network parameters of the control strategy network of each individual are re-optimized based on the global data in step S4, and the parameters of the control strategy network of the individual with the highest fitness are selected from all individuals and distributed to the control strategy network of each ambulance.

10. An ambulance suspension agent based on deep reinforcement learning, characterized in that, The deep reinforcement learning-based ambulance suspension agent is obtained based on the deep reinforcement learning-based ambulance suspension agent optimization method in any one of claims 1 to 9, and in the actual transfer process, the deep reinforcement learning-based ambulance suspension agent generates a suspension control force based on the collected state and applies it to the active suspension system of the ambulance.

Images