Team control method and system based on enhanced state-aware soft actor critic

Abstract

A vehicle platoon control method and system based on enhanced state-aware software is proposed. This belongs to the field of intelligent connected vehicle cooperative control technology. It solves the technical problem of lacking a vehicle platoon cooperative control method that balances exploration efficiency, strategy stability, and real-time application capabilities. Specifically, the method involves: a vehicle platoon consisting of three vehicles, with vehicle 1 as the lead vehicle and vehicles 2 and 3 as followers; using the target acceleration or target driving force of vehicle 1 as the control input, establishing the corresponding linear error state equation, and then solving the Riccati equation to obtain the optimal feedback gain matrix, thus obtaining the optimal control law for vehicle 1; defining the state, actions, rewards, and constraints of vehicle 2; defining the state, actions, rewards, and constraints of vehicle 3; training the weight parameters of a reinforcement learning network, using the states of vehicle 2 and vehicle 3 as input, and outputting the corresponding actions of vehicle 2 and vehicle 3 for control of vehicle 2 and vehicle 3.

Description

Technical Field

[0001] This invention belongs to the field of intelligent connected vehicle cooperative control technology, specifically relating to a fleet control method and system based on enhanced state perception software. Background Technology

[0002] With the development of intelligent driving technology, vehicle platoon cooperative control has become an important means to improve road traffic efficiency and driving safety. However, in dynamic traffic environments, speed fluctuations of the vehicle in front are easily propagated and amplified along the platoon, leading to the accumulation of distance and speed errors, affecting platoon stability and robustness. Existing vehicle platoon cooperative control methods mainly include rule-based control methods, optimization-based control methods, and learning-based control methods. Rule-based methods (such as proportional-integral-derivative control and fuzzy control) have simple structures and low computational costs, but their parameters are fixed and their environmental adaptability is weak, making it difficult to balance car-following accuracy and comfort under complex conditions. Optimization-based methods (such as linear quadratic regulators and model predictive control) have strong constraint handling capabilities, but they often rely on accurate models. When the model mismatches or environmental uncertainties increase, the control effect is prone to decline.

[0003] Compared to the methods mentioned above, reinforcement learning methods can adaptively learn control strategies through interaction with the environment, exhibiting better adaptability and robustness in complex dynamic scenarios. Among them, the soft-actor critic algorithm, due to the introduction of the maximum entropy mechanism, possesses strong exploration capabilities and has become an important technical route for continuous control scenarios. However, when the standard soft-actor critic algorithm is directly applied to vehicle platoon cooperative control, it still has the following shortcomings: First, the exploration intensity is difficult to adaptively adjust according to the learning stage and state uncertainty; second, the target entropy setting is usually fixed, making it impossible to perform differentiated updates for different states; third, in the early training stage, the critic network's estimation bias can easily cause policy oscillations, thereby affecting platoon consistency and serial stability.

[0004] Against this backdrop, how to propose a vehicle platoon cooperative control method that balances exploration efficiency, strategy stability, and real-time application capabilities has become a key technical problem that urgently needs to be solved in the field of intelligent connected vehicle cooperative control. Summary of the Invention

[0005] To address the technical problem of lacking a vehicle platoon cooperative control method that balances exploration efficiency, strategy stability, and real-time application capabilities, this invention provides a platoon control method based on an enhanced state-aware soft actor commentator. The method specifically comprises: S1. Establish a vehicle queue model: The vehicle queue consists of three cars, with vehicle 1 as the lead car and vehicles 2 and 3 as the following cars. The vehicle queue model includes a vehicle queue following model, a vehicle queue communication topology model, and a vehicle longitudinal dynamics model. S2. Lead vehicle control based on linear quadratic optimal control: Using the target acceleration or target driving force of vehicle 1 as the control input, after establishing the corresponding linear error state equation, the optimal feedback gain matrix is ​​obtained by solving the Riccati equation, thereby obtaining the optimal control law of vehicle 1. S3. Reinforcement learning-based vehicle following control: Define the state, actions, rewards, and constraints of vehicle 2; define the state, actions, rewards, and constraints of vehicle 3; train the weight parameters of the reinforcement learning network, taking the states of vehicle 2 and vehicle 3 as inputs, and outputting the corresponding actions of vehicle 2 and vehicle 3 for the control of vehicle 2 and vehicle 3.

[0006] Furthermore, the vehicle platoon following model is used to calculate the following vehicle's distance error, speed error, and safety constraints; the vehicle platoon communication topology model is used to determine the state information of the leading vehicle and the preceding vehicle received by the following vehicle; and the vehicle longitudinal dynamics model is used to update the following vehicle's speed, acceleration, and displacement based on reinforcement learning actions.

[0007] Furthermore, the linear error state equation is specifically as follows: ; in, This represents the error state matrix of vehicle 1. This represents the control input matrix for vehicle 1. express Control input for vehicle 1 at any given time. express The augmented state vector of vehicle 1 at time 1.

[0008] Furthermore, the optimal control law for vehicle 1 is as follows: ,in, This is the optimal feedback gain matrix.

[0009] Furthermore, the status, actions, and rewards of vehicle 2 are as follows:

[0010] in, , and They are respectively The status, actions, and rewards of vehicle 2 at any given moment; and They represent The spacing error and speed error of vehicle 2 at any given time; , and They represent The speed, acceleration, and jerk of vehicle 2 at any given moment; and They represent The speed and acceleration of vehicle 1 at time 1; express The acceleration correction amount generated by vehicle 2 based on the acceleration of vehicle 1 at any given time; , , and These represent the weighting coefficients corresponding to the spacing error term, velocity error term, acceleration term, and jerk term, respectively. This represents the global scaling factor.

[0011] Furthermore, the status, actions, and rewards of vehicle 3 are as follows:

[0012] in, , and They are respectively The status, actions, and rewards of vehicle 3 at any given moment; and They represent The spacing error and speed error of vehicle 3 at any given time; , and They represent The speed, acceleration, and jerk of vehicle 3 at any given moment; and They represent The speed and acceleration of vehicle 1 at time 1; express The acceleration correction amount generated by vehicle 3 based on the acceleration of vehicle 1 at any given time; , , and These represent the weighting coefficients corresponding to the spacing error term, velocity error term, acceleration term, and jerk term, respectively. This represents the global scaling factor.

[0013] Furthermore, a soft actor-critic algorithm network architecture was selected for the reinforcement learning network, and the weight parameters of the reinforcement learning network were trained.

[0014] Furthermore, the specific weight parameters for training the reinforcement learning network are as follows: S01. Initialize network parameters: Initialize the policy network parameters as follows: The initialization parameters for the value network and the target value network are as follows: Initialize the experience replay pool as follows: The experience replay pool capacity is ; S02. Introduce a stability enhancement exploration mechanism based on TD error: at the current time... The states, actions, and rewards of vehicles 2 and 3 are set respectively. The target value is constructed based on the sample transfer of the previous moment. The dynamic perturbation coefficient of the current moment is constructed based on the absolute value of the temporal difference error of the previous moment. The dynamic perturbation coefficient is used to perturb the output action of the policy network at the current moment to obtain the final execution action. S03. Sample experience from the experience replay pool: Sample a batch of experience tuples from the experience replay pool in chronological order; S04. Update the critic network: Using the empirical tuples sampled from the empirical replay pool, construct the critic network loss function based on the target value. On this basis, update the critic network parameters using gradient descent. S05. Update the policy network: Calculate the policy network loss function, and update the policy network parameters based on the policy network loss function; S06. Introduce a state confidence-based entropy adjustment mechanism: Calculate the mean of the outputs of the two commentator networks, then calculate the value variance corresponding to the mean, construct the state confidence based on the value variance, adaptively adjust the target entropy after obtaining the state confidence, then construct the entropy coefficient loss function, and update the entropy coefficient using gradient descent based on the entropy coefficient loss function. S07. Update the target critic network: Perform a soft update on the parameters of the target critic network; S08. Repeat steps S02 to S07 until the training process converges or the preset number of training iterations is reached.

[0015] Another aspect of the present invention provides a fleet control system based on enhanced state-aware soft actor commentator, the system comprising: Units used to establish vehicle platoon model: The vehicle platoon consists of three cars, with vehicle 1 as the lead car and vehicles 2 and 3 as the following cars. The vehicle platoon model includes a vehicle platoon following model, a vehicle platoon communication topology model, and a vehicle longitudinal dynamics model. The unit used for lead vehicle control based on linear quadratic optimal control: After establishing the corresponding linear error state equation by taking the target acceleration or target driving force of vehicle 1 as the control input, the optimal feedback gain matrix is ​​obtained by solving the Riccati equation, thereby obtaining the optimal control law of vehicle 1. The unit used for reinforcement learning-based vehicle following control defines the state, actions, rewards, and constraints of vehicle 2; defines the state, actions, rewards, and constraints of vehicle 3; trains the weight parameters of the reinforcement learning network, takes the states of vehicle 2 and vehicle 3 as input, and outputs the corresponding actions of vehicle 2 and vehicle 3 for controlling vehicle 2 and vehicle 3.

[0016] The beneficial effects of the method described in this invention are as follows: By employing a reinforcement learning-based vehicle platoon cooperative control method, the vehicle platoon following control process is transformed into a strategy self-learning and dynamic optimization process. This enables real-time adjustment of control output based on vehicle operating states in complex dynamic traffic scenarios, thereby improving the cooperative control accuracy, overall stability, and environmental adaptability of the vehicle platoon. Compared to traditional rule-based and optimization-based control methods, this invention does not rely on fixed control parameters or precise mathematical models, and can more flexibly address issues such as speed fluctuations between vehicles, changes in vehicle distance, and state uncertainties, thus enhancing the consistency, robustness, and cooperative control effect of the vehicle platoon under dynamic conditions.

[0017] By introducing a stability-enhancing exploration mechanism, the system adaptively adjusts the policy exploration noise based on temporal difference errors, dynamically adjusting the exploration intensity according to the estimation deviation at different learning stages and under different states. When the estimation deviation is large, the system automatically enhances the exploration capability to improve the policy's ability to escape local optima and adapt to complex scenarios; when the estimation deviation decreases, the system automatically weakens the exploration disturbance to improve the stability of the policy convergence process. Therefore, this invention can ensure exploration efficiency while maintaining learning stability, thereby improving the convergence performance, generalization ability, and control reliability of reinforcement learning policies in vehicle platoon cooperative control under dynamic scenarios.

[0018] By introducing a state confidence-aware entropy adjustment mechanism, the uncertainty of the current state is assessed using the variance of the value function output by a dual-critic network, and the target entropy is dynamically adjusted based on the state confidence level, thereby achieving adaptive control of the policy exploration level. Compared to fixed entropy adjustment methods, this invention enables the policy to maintain necessary exploration capabilities under high uncertainty and enhances the stability of policy updates under low uncertainty, thereby improving the temperature parameter update process, increasing sample utilization efficiency and training stability, and further enhancing the robustness of vehicle platoon cooperative control. Attached Figure Description

[0019] Figure 1 This is a framework diagram of the vehicle cooperative control method based on enhanced state-aware soft actor commentator described in this embodiment of the invention; Figure 2 This is a schematic diagram of the vehicle queue as described in an embodiment of the present invention; Figure 3 This is a diagram of the enhanced state-aware soft actor critic network structure described in this embodiment of the invention. Detailed Implementation

[0020] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0021] This embodiment proposes a platoon cooperative control method based on an enhanced state-aware soft-actor commentator, aiming to address the problems of insufficient exploration efficiency, inability to adjust fixed entropy based on state differences, and oscillation in policy updates inherent in the standard soft-actor commentator algorithm for platoon cooperative control. By introducing a stability-enhancing exploration mechanism and a state confidence-aware entropy adjustment mechanism into the basic soft-actor commentator algorithm framework, this method can adaptively adjust exploration noise based on temporal differential errors and dynamically correct the target entropy by calculating state confidence based on the variance of the dual-commentator network output. This improves the following accuracy, comfort, and platoon consistency of vehicle platoons in dynamic traffic scenarios. Figure 1 The specific technical solutions will be introduced.

[0022] S1: Establish a vehicle queuing model The vehicle convoy consists of three pure electric vehicles, such as Figure 2 As shown, the leading vehicle is vehicle 1, and the following vehicles are vehicles 2 and 3. The vehicle queuing model includes a vehicle queuing car-following model, a vehicle queuing communication topology model, and a vehicle longitudinal dynamics model. The vehicle queuing car-following model, the vehicle queuing communication topology model, and the vehicle longitudinal dynamics model are invoked in subsequent processes such as leading vehicle control, following vehicle state construction, reward calculation, and vehicle motion state update.

[0023] S1-1: Establish a vehicle platoon following model For the For each following vehicle, the distance error and speed error are defined as follows: ; in, and They represent Time of the first The distance and speed errors of the following vehicles; and They represent Always in the lead car and the first The expected and actual distance between following vehicles; and They represent The speed of the lead car at that moment and the first The speed of the following vehicle.

[0024] Actual vehicle spacing is expressed as: ; in, and They represent The displacement and length of the lead car at all times. express Time of the first The displacement of the vehicle following the car.

[0025] The desired distance between vehicles is determined by a vehicle spacing strategy, which defines the safe following distance between vehicles. The desired distance is expressed as: ; in, and They represent the first The constant headway and safe stationary distance of the following vehicle.

[0026] S1-2: Establish a vehicle queue communication topology model Inter-vehicle platooning information exchange adopts a lead vehicle-to-leader communication topology model based on vehicle-to-vehicle communication, and its information flow relationship is represented as follows: ; in, Indicates the number of following vehicles; Represents a directed graph; Indicates a group of vehicles following; Represents the set of edges; Represents the adjacency matrix; This represents an element in the adjacency matrix.

[0027] Among the following vehicles, a degree matrix is ​​introduced to represent the set of following vehicles receiving information, defined as follows: ; in, Degree matrix, Indicates vehicle The degree.

[0028] To describe the information interaction between the leading vehicle and the following vehicles, a connection matrix is ​​introduced, which is defined as: ; in, Represents the connection matrix; The elements in the connection matrix represent the following vehicles. Whether it can directly receive information from the leading vehicle. The elements in the adjacency matrix, degree matrix, and connectivity matrix can be represented as: ; in, Indicates from vehicle To the vehicle Directed communication. Therefore, the adjacency matrix, degree matrix, and connectivity matrix are expressed as follows: ; To maintain the coordination and stability of the vehicle platoon, an augmented Laplace matrix is ​​introduced, which is expressed as: ; in, The augmented Laplace matrix represents the PLF topology model, which integrates the communication relationships between following vehicles and the global information input from the leading vehicle.

[0029] S1-3: Establish the vehicle longitudinal dynamics model For the The longitudinal dynamic equation of the following vehicle is: ; in, , and They represent Time vehicle Displacement, velocity, and acceleration; Indicates vehicle The quality. and They represent Time vehicle The driving force and resistance are expressed as follows: ; in, express Time vehicle The actual torque of the motor; and These represent the transmission ratio and the final drive ratio, respectively. Indicates the efficiency of the power transmission system; Indicates vehicle The tire radius; Indicates the rolling resistance coefficient; Represents gravitational acceleration; Indicates air density; Indicates the air drag coefficient; Indicates the windward area; express Road gradient angle at all times.

[0030] S2: Leading vehicle control based on linear quadratic optimal control The leading vehicle is identified as Vehicle 1. The longitudinal dynamics model of the vehicle established in step S1 is used as the controlled object. The vehicle speed curve output under standard driving conditions is used as the target vehicle speed input for the leading vehicle. The speed tracking error of the leading vehicle is defined as: ; in, express The vehicle speed tracking error at time 1. express The reference speed of vehicle 1 at that moment. express The actual speed of vehicle 1 at that moment.

[0031] To improve steady-state tracking accuracy, an error integral state is further introduced: ; in, express The integral state of the velocity tracking error at any given time is used to accumulate the velocity deviation in order to reduce the steady-state error of the system.

[0032] Based on this, construct the augmented state vector of the leading vehicle: ; in, express The augmented state vector of the leading car at time step.

[0033] Using the target acceleration or target driving force of the guided vehicle as the control input, a corresponding linear error state equation is established.

[0034] in, This represents the error state matrix of the leading vehicle. This represents the control input matrix for the lead vehicle. This represents the control input for the lead vehicle. The first line of the state equation describes the change in the speed error integral state; the second line describes the effect of the control input on the speed tracking error.

[0035] Based on this, construct the performance index function: ; in, The state weighting matrix, To control the weighting matrix, for The time-series control input of the leading vehicle is defined. The performance index function is used to determine the optimization objective in the speed tracking control process of the leading vehicle, wherein the state weighting term is used to constrain the speed tracking error and its integral error of the leading vehicle, and the control weighting term is used to constrain the amplitude of the control input. Under the condition of satisfying the aforementioned linear error state equation, the corresponding continuous-time algebraic Riccati equation can be obtained by minimizing the performance index function.

[0036] Based on the aforementioned linear error state equation and performance index function, a continuous-time algebraic Riccati equation is established: ; in, This represents the error state matrix of the leading vehicle. This represents the control input matrix for the lead vehicle. Denotes the symmetric positive definite solution matrix of the Riccati equation. Represents the state weighting matrix. This represents the control weighting matrix.

[0037] Solving the above continuous-time algebraic Riccati equation yields the optimal feedback gain matrix: ; in, This represents the optimal feedback gain matrix. Represents the control weighting matrix. This represents the control input matrix for the lead vehicle. Let represent the symmetric positive definite solution matrix of the Riccati equation.

[0038] This leads to the optimal control law for the lead vehicle: ; S3: Reinforcement Learning-Based Vehicle Following Control The following vehicles include a first following vehicle and a second following vehicle, with the first following vehicle corresponding to vehicle 2 and the second following vehicle corresponding to vehicle 3. The reinforcement learning control of the following vehicles invokes the vehicle queuing car-following model, the vehicle queuing communication topology model, and the vehicle longitudinal dynamics model established in step S1. The vehicle queuing car-following model is used to calculate the following vehicle's distance error, speed error, and safety constraints; the vehicle queuing communication topology model is used to determine the state information of the leading and preceding vehicles received by the following vehicle; and the vehicle longitudinal dynamics model is used to update the following vehicle's speed, acceleration, and displacement based on the reinforcement learning actions. The reinforcement learning-based following vehicle control includes the following steps: S3-1: Define the state, actions, rewards, and constraints of vehicle 2. Define the state, actions, and rewards of vehicle 2 as follows:

[0039] in, , and They are respectively The status, actions, and rewards of vehicle 2 at any given moment. and They represent The spacing error and speed error of vehicle 2 at any given time; , and They represent The speed, acceleration, and jerk of vehicle 2 at any given moment; and They represent The speed and acceleration of vehicle 1 at time 1. express The acceleration correction amount generated by vehicle 2 based on the acceleration of vehicle 1 at any given time. , , and These represent the weighting coefficients corresponding to the spacing error term, velocity error term, acceleration term, and jerk term, respectively. This represents the global scaling factor.

[0040] Define the constraints for vehicle 2 as follows: ; in, Indicates that vehicle 2 is in Longitudinal displacement at time t, Indicates that vehicle 1 is in Longitudinal displacement at time t, Indicates the minimum permissible distance between vehicles; Indicates that vehicle 2 is in The speed of time and These represent the lower and upper limits of the vehicle's permitted speed, respectively. Indicates that vehicle 2 is in acceleration at any moment and These represent the lower and upper limits of the vehicle's permissible acceleration, respectively. Indicates that vehicle 2 is in The acceleration of time, and These represent the lower and upper limits of the vehicle's permissible acceleration, respectively. Indicates that vehicle 2 is in Momentary action output and These represent the lower and upper limits of the action output, respectively.

[0041] Proceed to step S3-3.

[0042] S3-2: Define the state, actions, rewards, and constraints of vehicle 3. Define the state, actions, and rewards of vehicle 3 as follows:

[0043] in, , and They are respectively The status, actions, and rewards of vehicle 3 at any given moment. and They represent The spacing error and speed error of vehicle 3 at any given time; , and They represent The speed, acceleration, and jerk of vehicle 3 at any given moment; and They represent The speed and acceleration of vehicle 1 at time 1. express The acceleration correction amount generated by vehicle 3 based on the acceleration of vehicle 1 at any given time. , , and These represent the weighting coefficients corresponding to the spacing error term, velocity error term, acceleration term, and jerk term, respectively. This represents the global scaling factor.

[0044] Define the constraints for vehicle 3 as follows: ; in, Indicates vehicle 3 is in The longitudinal displacement at any given moment; Indicates vehicle 3 is in The speed of time; Indicates vehicle 3 is in Acceleration at any moment; Indicates vehicle 3 is in The acceleration of time; Indicates vehicle 3 is in Momentary action output.

[0045] Proceed to step S3-3.

[0046] S3-3: Training Reinforcement Learning Network Weight Parameters The reinforcement learning network adopts an enhanced state-aware soft actor-critic algorithm network architecture, including a policy network, two critic networks, and target critic networks corresponding to the two critic networks respectively, such as... Figure 3As shown. The policy network takes the current state as input and outputs the action; the two critic networks take the current state and action as input and output an estimate; the target critic network is only used for subsequent parameter updates.

[0047] The training process for the reinforcement learning network parameters includes the following steps: S3-3-1: Initialize network parameters Initialize the policy network parameters as follows The initialization parameters for the value network and the target value network are as follows: Initialize the experience replay pool as follows: The experience replay pool capacity is .

[0048] S3-3-2: Introducing a stability enhancement exploration mechanism based on TD error First, at the current moment For vehicles 2 and 3, the status, actions, and rewards are set as follows: ; in for Time vehicle state, for Time vehicle action, for Time vehicle award, For the next moment of the vehicle state.

[0049] Construct the target value based on the sample transition from the previous time step: The timing difference error of the previous time step is expressed as: ; in, Indicates targeting vehicles At any moment The constructed target state action value; Indicates vehicle At any moment The timing difference error; Indicates vehicle At any moment The instant reward received; Indicates the discount factor; and Representing vehicles At any moment and The state; Indicates vehicle At any moment The actual actions performed; Indicates time vehicle In the state The following actions are sampled from the policy network. These actions are only used to construct the Bellman objective and are not the final actions to be executed. This indicates the current network of critics' opinions on vehicles. At any moment Value estimation of state-action pairs; and These represent two target commentator networks regarding the vehicle. At any moment Value estimation of state-action pairs; Indicates the network parameters of the policy A defined strategy function; This represents the entropy regularization coefficient.

[0050] The dynamic perturbation coefficients for the current time step are constructed based on the absolute value of the time-series difference error from the previous time step. ; in, Indicates vehicle At any moment The dynamic disturbance coefficient; This represents the lower bound of the minimum perturbation; This represents the sensitivity adjustment coefficient.

[0051] Furthermore, the dynamic perturbation coefficient is used to perturb the output action of the policy network at the current moment to obtain the final execution action: ; in, Indicates vehicle At any moment The final action to be performed; Indicates the policy network in state The mean of the output; This represents zero-mean, unit-covariance Gaussian noise.

[0052] S3-3-3: Sampling experience from the experience playback pool A batch of empirical tuples is sampled sequentially from the empirical replay pool: ; S3-3-4: Update the critics' network The commentator network loss function is constructed based on the target value and is expressed as follows: ; in, Represents the commentator network loss function; Indicates the first A network of commentators estimates the value of the current state-action pair; This represents the experience replay pool; , This indicates sampling empirical tuples from the empirical replay pool and calculating their expected values. This indicates the object of the expectation operation, that is, to calculate the expectation of the squared error term within the square brackets.

[0053] Based on this, the commentator network parameters are updated using gradient descent, with the following update formula: ; in, Indicates the first A commentator's network parameters; Indicates the first Calculate the gradient of the parameters of the commentator network; This indicates the online learning rate of the commentators.

[0054] S3-3-5: Update Policy Network The network loss function for calculating the policy is expressed as: ; in, Represents the policy network loss function; and Let each represent the state-action-value function of the two commentator networks, i.e. and ; Indicates the network parameters of the policy The final policy function is determined.

[0055] Based on the aforementioned policy network loss function, the policy network parameters are updated using the following formula: ; in, Indicates the policy network parameters; This represents the learning rate of the policy network. This indicates that the gradient of the policy network parameters is calculated.

[0056] S3-3-6: Introducing a state confidence-based entropy adjustment mechanism To characterize the confidence level of the current state-action pair, the state confidence is calculated. First, the mean of the two critic network outputs is calculated, expressed as: ; in, This indicates that two online critics are discussing the vehicles. At any moment State Action The average value estimate is then calculated; subsequently, the corresponding value variance is expressed as: ; in, Represents the variance of value. for .

[0057] The state confidence score is constructed based on the value variance and is expressed as follows: ; in, Indicates the state confidence level. This represents the confidence level adjustment coefficient.

[0058] After obtaining the state confidence, the target entropy is adaptively adjusted, as follows: ; in, This represents the adaptively adjusted target entropy; Represents the initial target entropy; This represents the target entropy adjustment coefficient.

[0059] Subsequently, the entropy coefficient loss function is constructed, expressed as: ; in, This represents the entropy coefficient loss function.

[0060] Based on the aforementioned entropy coefficient loss function, the entropy coefficient is updated using gradient descent. : ; in, Indicates temperature parameter, This represents the learning rate for the temperature parameter. Represents the temperature parameter loss function. Represents the temperature parameter loss function. The gradient.

[0061] S3-3-7: Update the target commentator network The target commentator network parameters are soft-updated using the following formula: ; in, Indicates the first Parameters of a target commentator network; This represents the soft update coefficient.

[0062] S3-3-8: Repeated Updates Repeat steps S3-3-2 to S3-3-7 until the training process converges or the preset number of training iterations is reached.

Claims

1. A vehicle control method based on enhanced state-aware soft actor critics, characterized in that, The method includes the following steps: S1. Establish a vehicle queue model: The vehicle queue consists of three cars, with vehicle 1 as the lead car and vehicles 2 and 3 as the following cars. The vehicle queue model includes a vehicle queue following model, a vehicle queue communication topology model, and a vehicle longitudinal dynamics model. S2. Lead vehicle control based on linear quadratic optimal control: Using the target acceleration or target driving force of vehicle 1 as the control input, after establishing the corresponding linear error state equation, the optimal feedback gain matrix is ​​obtained by solving the Riccati equation, thereby obtaining the optimal control law of vehicle 1. S3. Reinforcement learning-based vehicle following control: Define the state, actions, rewards, and constraints of vehicle 2; Define the state, actions, rewards, and constraints of vehicle 3; The weight parameters of the reinforcement learning network are trained, taking the states of vehicle 2 and vehicle 3 as inputs, and outputting the corresponding actions of vehicle 2 and vehicle 3 for the control of vehicle 2 and vehicle 3.

2. The vehicle control method based on enhanced state-aware soft actor commentator according to claim 1, characterized in that, The vehicle platoon following model is used to calculate the following vehicle's distance error, speed error, and safety constraints. The vehicle platoon communication topology model is used to determine the state information of the leading vehicle and the preceding vehicle received by the following vehicle. The vehicle longitudinal dynamics model is used to update the following vehicle's speed, acceleration, and displacement based on reinforcement learning actions.

3. The vehicle control method based on enhanced state-aware soft actor commentator according to claim 2, characterized in that, The linear error state equation is as follows: ； in, This represents the error state matrix of vehicle 1. This represents the control input matrix for vehicle 1. express Control input for vehicle 1 at any given time. express The augmented state vector of vehicle 1 at time 1.

4. The vehicle control method based on enhanced state-aware soft actor commentator according to claim 3, characterized in that, The optimal control law for vehicle 1 is as follows: ,in, This is the optimal feedback gain matrix.

5. The vehicle control method based on enhanced state-aware soft actor commentator according to claim 4, characterized in that, The specific status, actions, and rewards for vehicle 2 are as follows: in, , and They are respectively The status, actions, and rewards of vehicle 2 at any given moment; and They represent The spacing error and speed error of vehicle 2 at any given time; , and They represent The speed, acceleration, and jerk of vehicle 2 at any given moment; and They represent The speed and acceleration of vehicle 1 at time 1; express The acceleration correction amount generated by vehicle 2 based on the acceleration of vehicle 1 at any given time; , , and These represent the weighting coefficients corresponding to the spacing error term, velocity error term, acceleration term, and jerk term, respectively. This represents the global scaling factor.

6. The vehicle control method based on enhanced state-aware soft actor commentator according to claim 5, characterized in that, The specific status, actions, and rewards for vehicle 3 are as follows: in, , and They are respectively The status, actions, and rewards of vehicle 3 at any given moment; and They represent The spacing error and speed error of vehicle 3 at any given time; , and They represent The speed, acceleration, and jerk of vehicle 3 at any given moment; and They represent The speed and acceleration of vehicle 1 at time 1; express The acceleration correction amount generated by vehicle 3 based on the acceleration of vehicle 1 at any given time; , , and These represent the weighting coefficients corresponding to the spacing error term, velocity error term, acceleration term, and jerk term, respectively. This represents the global scaling factor.

7. The vehicle control method based on enhanced state-aware soft actor critics according to claim 6, characterized in that, The reinforcement learning network selects a soft actor-critic algorithm network architecture and trains the reinforcement learning network weight parameters.

8. The vehicle control method based on enhanced state-aware soft actor critics according to claim 7, characterized in that, The specific weight parameters for training a reinforcement learning network are as follows: S01. Initialize network parameters: Initialize the policy network parameters as follows: The initialization parameters for the value network and the target value network are as follows: Initialize the experience replay pool as follows: The experience replay pool capacity is ; S02. Introduce a stability enhancement exploration mechanism based on TD error: at the current time... The states, actions, and rewards of vehicles 2 and 3 are set respectively. The target value is constructed based on the sample transfer of the previous moment. The dynamic perturbation coefficient of the current moment is constructed based on the absolute value of the temporal difference error of the previous moment. The dynamic perturbation coefficient is used to perturb the output action of the policy network at the current moment to obtain the final execution action. S03. Sample experience from the experience replay pool: Sample a batch of experience tuples from the experience replay pool in chronological order; S04. Update the critic network: Using the empirical tuples sampled from the empirical replay pool, construct the critic network loss function based on the target value. On this basis, update the critic network parameters using gradient descent. S05. Update the policy network: Calculate the policy network loss function, and update the policy network parameters based on the policy network loss function; S06. Introduce a state confidence-based entropy adjustment mechanism: Calculate the mean of the outputs of the two commentator networks, then calculate the value variance corresponding to the mean, construct the state confidence based on the value variance, adaptively adjust the target entropy after obtaining the state confidence, then construct the entropy coefficient loss function, and update the entropy coefficient using gradient descent based on the entropy coefficient loss function. S07. Update the target critic network: Perform a soft update on the parameters of the target critic network; S08. Repeat steps S02 to S07 until the training process converges or the preset number of training iterations is reached.

9. A fleet control system based on enhanced state-aware soft actor critics, characterized in that, The system includes: Units used to establish vehicle platoon model: The vehicle platoon consists of three cars, with vehicle 1 as the lead car and vehicles 2 and 3 as the following cars. The vehicle platoon model includes a vehicle platoon following model, a vehicle platoon communication topology model, and a vehicle longitudinal dynamics model. The unit used for lead vehicle control based on linear quadratic optimal control: After establishing the corresponding linear error state equation by taking the target acceleration or target driving force of vehicle 1 as the control input, the optimal feedback gain matrix is ​​obtained by solving the Riccati equation, thereby obtaining the optimal control law of vehicle 1. The unit used for reinforcement learning-based vehicle following control defines the state, actions, rewards, and constraints of vehicle 2; defines the state, actions, rewards, and constraints of vehicle 3; trains the weight parameters of the reinforcement learning network, takes the states of vehicle 2 and vehicle 3 as input, and outputs the corresponding actions of vehicle 2 and vehicle 3 for controlling vehicle 2 and vehicle 3.

Images