Transform and deep reinforcement learning-based four-wheel steering control method and system

Abstract

The invention belongs to the technical field of vehicle four-wheel steering control, and discloses a four-wheel steering control method and system based on Transform and deep reinforcement learning. The method comprises the following steps: S1, obtaining an output state quantity based on a vehicle steering model comprising crosswind disturbance; and S2, according to the output state quantity and an improved PPO-Transformer algorithm, solving a feasible optimal solution of front and rear wheel turning angles in the four-wheel steering system, and determining a control quantity acting on the vehicle steering model based on the feasible optimal solution. Specifically, on one hand, the stability factor and disturbance factor of vehicle steering are fully considered, and external disturbances such as crosswind are introduced into a vehicle steering model, so that the robustness of a reinforcement learning training result is effectively improved; and on the other hand, the improved PPO-Transform algorithm is obtained by embedding a Transform architecture with a simplified structure into an Actor network, and the recognition capability of the algorithm on abnormal data is improved by utilizing a self-attention mechanism of a Transform core while the real-time control and generalization advantages of the PPO algorithm are not influenced.

Description

Technical Field

[0001] This invention belongs to the field of vehicle four-wheel steering control technology, specifically relating to a four-wheel steering control method and system based on Transformer and deep reinforcement learning. Background Technology

[0002] Vehicle steering technology is a crucial part of the vehicle control system. As the core actuator, it translates the directional commands from the driver or intelligent system into steering actions of the wheels, thereby changing the vehicle's trajectory. It is the fundamental technical support for ensuring driving safety, handling, and maneuverability.

[0003] Currently, vehicle steering technology is transitioning from traditional front-wheel steering to more advanced four-wheel steering. This technology, through rear-wheel coordinated steering, significantly improves vehicle agility at low speeds and stability at high speeds. Because four-wheel steering models are complex and have high safety requirements, model predictive control, which can handle constraints, is generally used. However, this optimization-based approach often suffers from poor real-time performance under complex and changing real-world conditions, thus compromising actual control effectiveness.

[0004] Reinforcement learning, as an intelligent control method, can optimize control strategies through continuous interaction with the environment and provide control values ​​through simple "lookup table" operations. It can efficiently solve the real-time control problem of four-wheel steering systems, demonstrating great application potential. However, reinforcement learning technology lacks an effective response mechanism to anomalies in sensor data, hindering its widespread adoption in the safety-sensitive field of vehicle four-wheel steering control.

[0005] On the other hand, neural network models based on the Transformer architecture have achieved great success in many fields due to their powerful sequential data processing capabilities. Their ability to capture long-short-term dependencies through self-attention mechanisms is beneficial for identifying erroneous data in sensor time-series data.

[0006] In summary, combining Transformer with reinforcement learning is a possible approach to handling anomalous data. However, the Transformer architecture suffers from structural complexity and long computation time. Therefore, it is necessary to simplify and innovate the structure to facilitate embedding the Transformer architecture into reinforcement learning and improve real-time computation efficiency. Summary of the Invention

[0007] To address the problems mentioned in the background section, this invention aims to provide a four-wheel steering control method and system based on Transformer and deep reinforcement learning. It effectively embeds the Transformer model into a traditional reinforcement learning framework, utilizing the core self-attention mechanism of Transformer to improve the algorithm's ability to identify abnormal data, thereby enhancing the robustness and adaptability of the four-wheel steering system. Simultaneously, considering the real-time requirements of the steering system, this invention simplifies the complexity of the Transformer model, effectively improving the real-time performance of the control while ensuring overall performance.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A four-wheel steering control method based on Transformer and deep reinforcement learning includes:

[0010] S1. Obtain the output state variables based on the vehicle steering model that includes crosswind disturbances;

[0011] S2. Based on the output state variables and the improved PPO-Transformer algorithm, solve for feasible optimization solutions of the front and rear wheel steering angles in the four-wheel steering system, and determine the control variables acting on the vehicle steering model based on the feasible optimization solutions;

[0012] The solution obtained using the improved PPO-Transformer algorithm includes:

[0013] The Actor network is implemented based on the Transformer architecture, by fitting parameters. To output the optimal action And maximize the expectation of the advantage function. , where the shearing function probability ratio function Advantage function Timing difference error State value function Therefore The parameters to be learned are derived from the state space. To the real number space The mapping, For the state dimension, For instant rewards, As a discount factor, and For hyperparameters;

[0014] The Critic network part is used to estimate the advantage function. And by minimizing the loss of the value function Implement the state value function The estimate, in which the advantage function Estimation calculation of the target value function .

[0015] Preferably, establishing the vehicle steering model including crosswind disturbance includes:

[0016] Establish a nonlinear two-degree-of-freedom model under the influence of crosswinds;

[0017] Based on the nonlinear two-degree-of-freedom model, a vehicle steering model including crosswind disturbances is established.

[0018] Preferably, the vehicle steering model including crosswind disturbance is expressed as follows:

[0019] ;

[0020] in, For vehicle quality, , The corresponding distances from the vehicle's center of gravity to the front and rear axles. and These are the steering stiffness of the front and rear wheels, The yaw rate is angular velocity. The sideslip angle is the angle of the center of mass. For the longitudinal speed of the vehicle, For the moment of inertia of yaw rotation, The horizontal distance from the center of wind pressure to the vehicle's center of mass. The lateral force caused by crosswinds and For the front and rear wheel steering angles;

[0021] The vehicle steering model takes the system state. System input .

[0022] Preferably, the Actor network part outputs the optimal action based on a Gaussian policy. Among them, expectations and variance It is determined by strategy.

[0023] Preferably, when the Actor network part and the Critic network part update the network parameters, the network update rule is as follows: .

[0024] Preferably, the Actor network is implemented based on the Transformer architecture, and its process includes:

[0025] Original state High-dimensional features are generated by mapping to a high-dimensional space through a linear transformation layer. ;

[0026] The high-dimensional features The input encoder is processed to obtain a context feature vector; based on this feature vector, the output layer of the Actor network generates the optimal action. .

[0027] Preferably, the encoder includes an encoding layer; the encoding layer employs a Fourier function containing learnable parameters for the high-dimensional features. Perform positional encoding to obtain the position-encoded high-dimensional features. The Fourier function is expressed as: In the formula, This represents the learnable parameters.

[0028] Preferably, the encoder further includes a multi-head attention layer; the multi-head attention layer employs a multi-head attention mechanism to process the high-dimensional features after position encoding. Express from multiple perspectives:

[0029] High-dimensional features after position encoding respectively with the weight matrix , , Multiply to obtain a vector ;

[0030] Calculate the attention weight matrix Divide the attention weight matrix by After scaling, Softmax normalization is applied.

[0031] The processed attention weight matrix and vector Multiply to obtain the self-attention output. ;

[0032] A multi-head attention mechanism is employed to compute multiple self-attention outputs in parallel, and the concatenation of these outputs is then combined with the weight matrix. Multiplication yields multi-dimensional feature representations. .

[0033] Preferably, the encoder further includes:

[0034] The first residual connection layer connects the multi-angle features with the high-dimensional features. The intermediate features are obtained by inputting them together into the first residual connection layer and normalizing them. ;in, This indicates a normalization operation. ,in, It is the mean of input 'a'. It is the standard deviation. It is to prevent the minimum value of division by zero. , These are learnable scaling and offset parameters;

[0035] The intermediate features are input into the feedforward network layer to obtain the feedforward features. ; where, linear transformation function W is the weight matrix, b is the bias vector, both of which are learnable parameters, and b is the non-linear activation function. ;

[0036] The second residual connection layer inputs the feedforward features and intermediate features together to obtain the encoder output. .

[0037] A four-wheel steering control system based on Transformer and deep reinforcement learning, comprising:

[0038] The module obtains the ideal state variables of a four-wheel steering vehicle system based on a vehicle steering model that includes crosswind disturbances.

[0039] The optimized control module, based on the output state variables and the improved PPO-Transformer algorithm, solves the feasible optimization solutions for the front and rear wheel steering angles in the four-wheel steering system, and determines the control variables based on the feasible optimization solutions;

[0040] The execution module applies the control quantity to the vehicle steering model.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] (1) In this invention, a nonlinear two-degree-of-freedom model under the influence of crosswind is established, and on this basis, a vehicle steering model including crosswind disturbance is further established. This fully considers the stability factors and disturbance factors of vehicle steering. Introducing external disturbances such as crosswind can ensure the accuracy of the training environment and effectively improve the robustness of reinforcement learning training results.

[0043] (2) In this invention, the PPO-Transformer algorithm is improved by embedding the simplified Transformer architecture into the Actor network. While not affecting the real-time control and generalization advantages of the PPO algorithm itself, the self-attention mechanism of the Transformer core is used to improve the algorithm's ability to identify abnormal data, giving the agent a powerful ability to model time-series dependencies and perceive abnormal states. Attached Figure Description

[0044] Figure 1 This is a block diagram illustrating the control principle of the four-wheel steering control method of the present invention.

[0045] Figure 2 The flowchart below shows the optimization solution process of the improved PPO-Transformer algorithm of this invention.

[0046] Figure 3 This is a schematic diagram of the vehicle steering model including crosswind disturbance in this invention. Detailed Implementation

[0047] To further understand the content of this invention, a detailed description of the invention is provided in conjunction with the accompanying drawings and embodiments. The structures, proportions, sizes, etc., depicted in the accompanying drawings are merely for illustrative purposes and to aid those skilled in the art, and are not intended to limit the implementation conditions of the invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to size, without affecting the effects and objectives of the invention, should still fall within the scope of the technical content disclosed in this invention. Furthermore, terms such as "upper," "lower," "left," "right," and "middle" used in this specification are merely for clarity and not intended to limit the scope of implementation. Changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention. It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein.

[0048] Figure 1 This is a block diagram illustrating the control principle of the four-wheel steering control method based on Transformer and deep reinforcement learning according to the present invention. As shown in the diagram, the four-wheel steering control method includes:

[0049] S1. Obtain the output state variables based on the vehicle steering model that includes crosswind disturbances.

[0050] Specifically, assuming a constant longitudinal speed for the vehicle, and considering stability and disturbance factors during vehicle steering, a nonlinear two-degree-of-freedom model under the influence of crosswinds is first established. Then, based on this nonlinear two-degree-of-freedom model, a model is established as follows: Figure 3 The vehicle steering model including crosswind disturbances is shown to characterize the relationship between vehicle handling stability and vehicle yaw and roll motions, and to construct the interactive environment required for reinforcement learning training. The vehicle steering model including crosswind disturbances is expressed as follows:

[0051] ;

[0052] in, For vehicle quality, , The corresponding distances from the vehicle's center of gravity to the front and rear axles. and These are the steering stiffness of the front and rear wheels, The yaw rate is angular velocity. The sideslip angle is the angle of the center of mass. For the longitudinal speed of the vehicle, For the moment of inertia of yaw rotation, The horizontal distance from the center of wind pressure to the vehicle's center of mass. The lateral force caused by crosswinds and For the front and rear wheel steering angles;

[0053] The vehicle steering model takes the system state. System input .

[0054] Furthermore, this invention performs domain randomization processing on the simulation training environment, that is, it introduces complex and variable real-world environmental disturbances into an ideal nonlinear two-degree-of-freedom vehicle steering model, mainly including crosswind disturbance dynamics. ,Pick During each training of the vehicle steering model, the crosswind disturbance dynamics are considered. Random values ​​are selected from the interval to be introduced into the steering model as environmental disturbances, thereby improving the generalization and robustness of the reinforcement learning training results of the vehicle steering model.

[0055] S2. Based on the output state variables and the improved PPO-Transformer algorithm, solve for feasible optimization solutions of the front and rear wheel steering angles in the four-wheel steering system, and determine the control variables acting on the vehicle steering model based on the feasible optimization solutions.

[0056] Specifically, this invention designs a targeted reinforcement learning reward function for vehicle four-wheel steering control systems. Among them, system status System input Expected state Two additional penalty functions are also introduced. The steering angles of the front and rear wheels of the vehicle are constrained respectively to ensure that the vehicle does not overturn due to excessive steering angles during control. , These correspond to the steering angles of the front and rear wheels of the vehicle, respectively.

[0057] In a typical embodiment, the improved PPO-Transformer algorithm used in this invention mainly includes the following: Figure 2 The diagram shows two parts: the Actor network and the Critic network.

[0058] Firstly, regarding Actor networks:

[0059] In the Actor network section, a Transformer-based encoder structure is innovatively introduced. Leveraging the Transformer encoder's ability to model long-range dependencies in sequential data, the representation learning and anomaly detection of time-series state data are enhanced. This effectively identifies and suppresses abnormal data caused by sensor noise, intermittent faults, or external interference, significantly improving the robustness and reliability of the control system. Specifically, the Transformer encoder includes an input layer, an encoding layer, a multi-head attention layer, a first residual connection layer, a feedforward network layer, and a second residual connection layer.

[0060] (1) Input layer: the original state at each time step A linear transformation layer is used to map to a high-dimensional space, generating an 8-dimensional high-dimensional feature representation. The core purpose of this step is to transform the low-dimensional raw data into a higher-dimensional feature vector, thereby capturing richer state details and laying the foundation for subsequent temporal relationship preservation and anomaly identification.

[0061] (2) Encoding layer: To further preserve the temporal relationship of the sequence data and ensure that the model can accurately perceive the order of the data in the time dimension, the encoding layer uses a Fourier function with learnable parameters to encode the high-dimensional features. Perform positional encoding to obtain the position-encoded high-dimensional features. The Fourier function is expressed as: In the formula, This represents learnable parameters that enable the model to adaptively learn temporal structures.

[0062] (3) Multi-head attention layer: The multi-head attention layer uses a multi-head attention mechanism to encode the high-dimensional features after position. Express from multiple perspectives;

[0063] First, the high-dimensional features after position encoding respectively with the weight matrix , , Multiply to obtain a vector And through calculation The attention weight matrix is ​​obtained; the value of each element in this matrix directly represents the correlation strength between data points at different positions in the sequence. The higher the value, the closer the correlation between the two data points; the lower the value, the more distant the correlation. Since abnormal data deviates significantly from normal data in high-dimensional space and has a weaker correlation with other data, the weight values ​​corresponding to abnormal data in the initial attention weight matrix will be significantly lower, providing a basis for subsequent suppression of abnormal interference.

[0064] Then, divide the attention weight matrix by After scaling, Softmax normalization is applied; then the processed attention weight matrix and vector are compared. Multiply to obtain the self-attention output. This output incorporates information from all positions in the sequence.

[0065] Furthermore, a multi-head attention mechanism is employed to compute multiple self-attention outputs in parallel, and then the concatenation of these outputs is combined with the weight matrix. Multiplication yields multi-dimensional feature representations. Each attention head focuses on features in different representation subspaces, thereby capturing data patterns more comprehensively and identifying and reducing the impact of outlier data through comprehensive comparison.

[0066] (4) The first residual connection layer connects the multi-angle features with the high-dimensional features. The intermediate features are obtained by inputting them together into the first residual connection layer and normalizing them. ;in, The high-dimensional input features after position encoding are the input to the encoder layer. It is a multi-head self-attention output, which is obtained by concatenating and linearly transforming the results of multiple self-attention heads. This indicates a normalization operation. ,in, It is the mean of input 'a'. It is the standard deviation. It is to prevent the minimum value of division by zero. , These are learnable scaling and offset parameters. This step preserves underlying information, mitigates gradient vanishing, and enhances information transfer efficiency.

[0067] (5) Feedforward network layer, consisting of two linear transformations , and the nonlinear activation functions between them Composition. The intermediate features are input into the feedforward network layer to obtain feedforward features. ; where, linear transformation function W is the weight matrix, b is the bias vector, both of which are learnable parameters, and b is the non-linear activation function. .

[0068] (6) Second residual connection layer: The feedforward features and intermediate features are input together into the second residual connection layer to obtain the encoder output. .

[0069] Through the above structure, the Transformer's encoder can progressively enhance effective features and suppress anomalous information, thereby achieving more robust control strategy generation in complex temporal environments. Specifically, the Actor network is implemented based on the Transformer architecture, and by fitting parameters... To output the optimal action The purpose is to more efficiently approximate the optimal strategy, thereby maximizing the expected value of the advantage function.

[0070] In the formula, the shearing function probability ratio function , For the dominant function, This is a hyperparameter (preferably 0.2 in this invention). Furthermore, the Actor network part outputs the optimal action based on a Gaussian policy. ,expect and variance It is determined by strategy.

[0071] It should be noted that the improved PPO-Transformer algorithm of this invention employs a clipping mechanism to limit the step size of policy updates, thereby improving the stability of the optimization process. During optimization, a clipping function is used to compare the probabilities of the old and new policies. Limited to the range Within the trust domain, the smaller of the original target and the pruned target is selected as the final loss through a minimum value operation. When the advantage function is positive, this mechanism prevents the policy from overconfidently updating too much on good actions; when the advantage function is negative, it prevents over-penalizing bad actions and updating too aggressively, thus enabling robust policy optimization within the trust domain and effectively avoiding training crashes.

[0072] Secondly, regarding the Critic network:

[0073] The Critic network part is used to estimate the advantage function. And by minimizing the loss of the value function Implement the state value function The estimate, in which the advantage function Estimation calculation of the target value function .

[0074] Furthermore, this invention improves the PPO-Transformer algorithm by incrementally estimating the dominance function using a generalized dominance estimation algorithm. Among them, timing difference error , For instant rewards, As a discount factor, This is a hyperparameter (preferably 0.95 in this invention). Specifically, this is achieved by introducing a parameter... To mix multi-step timing difference errors ,when When it degenerates into a high-bias, low-variance single-step time-series difference estimate; when When this happens, it becomes a low-bias, high-variance Monte Carlo estimate. By choosing an appropriate... The generalized advantage estimation mechanism can smoothly combine information from different time steps, thus providing a more effective and stable learning signal for policy gradients.

[0075] In this invention, the optimization training of the improved PPO-Transformer algorithm mainly consists of two cyclical steps: a data acquisition process and an optimization process.

[0076] Data acquisition process; using the current strategy Collection Track The Critic network part calculates the value of each state. Calculate the advantage estimate Calculate the target value ;

[0077] Optimization process: Randomly sample small batches of data from the collected trajectory data, and calculate for each sample. , , Optimize and update the network parameters of the Actor network and the Critic network. Each round repeats for K epochs.

[0078] Repeat the above two steps until convergence.

[0079] In summary, the four-wheel steering control method based on Transformer and deep reinforcement learning proposed in this invention effectively embeds the Transformer model into the traditional reinforcement learning framework. It leverages the core self-attention mechanism of Transformer to improve the algorithm's ability to identify abnormal data, thereby enhancing the robustness and adaptability of the four-wheel steering system. Furthermore, considering the real-time requirements of the steering system, this invention simplifies the model complexity of Transformer, effectively improving the real-time performance of the control while ensuring overall performance.

[0080] This invention also discloses a four-wheel steering control system based on Transformer and deep reinforcement learning, the system comprising:

[0081] The module obtains the ideal state variables of a four-wheel steering vehicle system based on a vehicle steering model that includes crosswind disturbances.

[0082] The optimized control module, based on the output state variables and the improved PPO-Transformer algorithm, solves the feasible optimization solutions for the front and rear wheel steering angles in the four-wheel steering system, and determines the control variables based on the feasible optimization solutions;

[0083] The execution module applies the control quantity to the vehicle steering model.

[0084] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the method, and will not be elaborated upon here.

[0085] In summary, the four-wheel steering control method and system based on Transformer and deep reinforcement learning provided in this invention, on the one hand, establishes a nonlinear two-degree-of-freedom model under the influence of crosswinds, and further establishes a vehicle steering model including crosswind disturbances on this basis. This fully considers the stability and disturbance factors of vehicle steering. By introducing external disturbances such as crosswinds, it can ensure the accuracy of the training environment and effectively improve the robustness of reinforcement learning training results. On the other hand, by embedding a simplified Transformer architecture (encoder) into the Actor network, an improved PPO-Transformer algorithm is obtained. While not affecting the real-time control and generalization advantages of the PPO algorithm itself, the self-attention mechanism of the Transformer core is used to improve the algorithm's ability to identify abnormal data, giving the agent a strong ability to model temporal dependencies and perceive abnormal states.

[0086] In another exemplary embodiment, an electronic device is also provided, the electronic device including a memory and a processor, and a program stored in the memory, the processor executing the program to implement one or more steps of the aforementioned method.

[0087] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the four-wheel steering control method based on Transformer and deep reinforcement learning described above. For example, the computer-readable storage medium may be a first memory including program instructions that can be executed by a first processor of an electronic device to complete the four-wheel steering control method based on Transformer and deep reinforcement learning described above.

[0088] In another exemplary embodiment, a computer program product is also provided, comprising a computer program executable by a programmable device, the computer program having a code portion for performing the aforementioned Transformer and deep reinforcement learning-based four-wheel steering control method when executed by the programmable device. In some embodiments, part or all of the computer program may be loaded into and / or installed onto a device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the aforementioned method may be performed. Alternatively, in other embodiments, the CPU may be configured to perform one or more steps of the aforementioned method by any other suitable means (e.g., by means of firmware).

[0089] In the description of this invention, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0090] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A four-wheel steering control method based on Transformer and deep reinforcement learning, characterized in that, Includes the following steps: S1. Obtain the output state variables based on the vehicle steering model that includes crosswind disturbances; S2. Based on the output state variables and the improved PPO-Transformer algorithm, solve for feasible optimization solutions of the front and rear wheel steering angles in the four-wheel steering system, and determine the control variables acting on the vehicle steering model based on the feasible optimization solutions; The solution obtained using the improved PPO-Transformer algorithm includes: The Actor network is implemented based on the Transformer architecture, by fitting parameters. To output the optimal action And maximize the expectation of the advantage function. , where the shearing function probability ratio function Advantage function Timing difference error State value function Therefore The parameters to be learned are derived from the state space. To the real number space The mapping, For the state dimension, For instant rewards, As a discount factor, and For hyperparameters; The Critic network part is used to estimate the advantage function. And by minimizing the loss of the value function Implement the state value function The estimate, in which the advantage function Estimation calculation of the target value function .

2. The four-wheel steering control method based on Transformer and deep reinforcement learning according to claim 1, characterized in that, Establishing the vehicle steering model that includes crosswind disturbances includes: Establish a nonlinear two-degree-of-freedom model under the influence of crosswinds; Based on the nonlinear two-degree-of-freedom model, a vehicle steering model including crosswind disturbances is established.

3. The four-wheel steering control method based on Transformer and deep reinforcement learning according to claim 2, characterized in that, The vehicle steering model including crosswind disturbance is expressed as follows: ； in, For vehicle quality, , The corresponding distances from the vehicle's center of gravity to the front and rear axles. and These are the steering stiffness of the front and rear wheels, The yaw rate is angular velocity. The sideslip angle is the angle of the center of mass. For the longitudinal speed of the vehicle, For the moment of inertia of yaw rotation, The horizontal distance from the center of wind pressure to the vehicle's center of mass. The lateral force caused by crosswinds and For the front and rear wheel steering angles; The vehicle steering model takes the system state. System input .

4. The four-wheel steering control method based on Transformer and deep reinforcement learning according to claim 1, characterized in that: The Actor network part outputs the optimal action based on a Gaussian policy. Among them, expectations and variance It is determined by strategy.

5. The four-wheel steering control method based on Transformer and deep reinforcement learning according to claim 1, characterized in that: When the Actor network part and the Critic network part update the network parameters, the network update rule is as follows: .

6. The four-wheel steering control method based on Transformer and deep reinforcement learning according to claim 1, characterized in that, The Actor network is implemented based on the Transformer architecture, and its process includes: Original state High-dimensional features are generated by mapping to a high-dimensional space through a linear transformation layer. ; The high-dimensional features The input encoder is processed to obtain a context feature vector; based on this feature vector, the output layer of the Actor network generates the optimal action. .

7. The four-wheel steering control method based on Transformer and deep reinforcement learning according to claim 6, characterized in that: The encoder includes an encoding layer; The encoding layer employs a Fourier function with learnable parameters for the high-dimensional features. Perform positional encoding to obtain the position-encoded high-dimensional features. The Fourier function is expressed as: In the formula, This represents the learnable parameters.

8. A four-wheel steering control method based on Transformer and deep reinforcement learning according to claim 7, characterized in that: The encoder also includes a multi-head attention layer; the multi-head attention layer employs a multi-head attention mechanism to process the high-dimensional features after position encoding. Express from multiple perspectives: High-dimensional features after position encoding respectively with the weight matrix , , Multiply to obtain a vector ; Calculate the attention weight matrix Divide the attention weight matrix by After scaling, Softmax normalization is applied. The processed attention weight matrix and vector Multiply to obtain the self-attention output. ; A multi-head attention mechanism is employed to compute multiple self-attention outputs in parallel, and the concatenation of these outputs is then combined with the weight matrix. Multiplication yields multi-dimensional feature representations. .

9. A four-wheel steering control method based on Transformer and deep reinforcement learning according to claim 8, characterized in that, The encoder further includes a first residual connection layer, a feedforward network layer, and a second residual connection layer; Combine the multi-angle features with high-dimensional features The intermediate features are obtained by inputting them together into the first residual connection layer and normalizing them. ;in, This indicates a normalization operation. ,in, It is the mean of input 'a'. It is the standard deviation. It is to prevent the minimum value of division by zero. , These are learnable scaling and offset parameters; The intermediate features are input into the feedforward network layer to obtain the feedforward features. ; where, linear transformation function W is the weight matrix, b is the bias vector, both of which are learnable parameters, and b is the non-linear activation function. ; The feedforward features and intermediate features are input together into the second residual connection layer to obtain the encoder output. .

10. A four-wheel steering control system based on Transformer and deep reinforcement learning, characterized in that, Includes the following structure: The module obtains the ideal state variables of a four-wheel steering vehicle system based on a vehicle steering model that includes crosswind disturbances. The optimized control module, based on the output state variables and the improved PPO-Transformer algorithm, solves the feasible optimization solutions for the front and rear wheel steering angles in the four-wheel steering system, and determines the control variables based on the feasible optimization solutions; The execution module applies the control quantity to the vehicle steering model.

Images