Distributed driving unmanned wheel type vehicle system control method, device and equipment

Abstract

The invention discloses a distributed driving unmanned wheel type vehicle system control method, device and equipment, and relates to the field of automatic driving, and the method comprises the steps: constructing a distributed driving unmanned wheel type vehicle system with Ackerman steering and differential steering; based on the distributed driving unmanned wheeled vehicle system and the two-degree-of-freedom analysis model of the vehicle, constructing a kinematics model based on the auxiliary yawing moment, and determining operation environment data according to the auxiliary yawing moment; constructing a reinforcement learning control system based on a double-layer deep Q learning network-priority experience return visit algorithm according to the operation environment data and the working condition of the distributed driving unmanned wheeled vehicle system; training the reinforcement learning control system; according to the current working condition and the current operation environment data, the trained reinforcement learning control system is utilized to determine the auxiliary yaw moment; and controlling the distributed driving unmanned wheel type vehicle system according to the auxiliary yaw moment. According to the invention, the vehicle roll stability can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of autonomous driving, and in particular to a distributed drive unmanned wheeled vehicle system control method, device and equipment. BACKGROUND

[0002] In recent years, small and medium-sized unmanned vehicles cancel the traditional differential, transmission and transmission shaft and other complex transmission structure, and adopt distributed motor drive, that is, each driving wheel is provided with a driving motor that can act independently, forming a full-line control technology upgrade, and bringing a cross-domain improvement in the maneuverability of unmanned vehicles.

[0003] However, the control technology of the full-line control vehicle still adopts a dynamic / kinematic analysis control method, which cannot realize end-to-end control between instructions and components, and the existing control method cannot realize distributed control through independent control of the line control components in the face of high-scale steering and other high-maneuvering methods, resulting in excessive roll angle of the vehicle during high-speed steering, and even rollover accidents.

[0004] Therefore, in order to solve the above problems, a new distributed drive unmanned wheeled vehicle system control method is needed, which can reduce the roll angle of the vehicle during high-speed steering by fusing Ackerman steering and differential steering, and improve the roll stability of the vehicle. SUMMARY

[0005] The purpose of the present application is to provide a distributed drive unmanned wheeled vehicle control method, device and equipment, which can improve the roll stability of the vehicle.

[0006] To achieve the above purpose, the present application provides the following solutions:

[0007] In a first aspect, the present application provides a distributed drive unmanned wheeled vehicle system control method, comprising:

[0008] A distributed drive unmanned wheeled vehicle system with Ackerman steering and differential steering is constructed; the steering configuration of the front wheels of the vehicle of the distributed drive unmanned wheeled vehicle system adopts Ackerman structure; and the front and rear axle transmission configuration of the distributed drive unmanned wheeled vehicle system adopts a power structure driven by four hub motors;

[0009] Based on the two-degree-of-freedom analysis model of the distributed drive unmanned wheeled vehicle system and the vehicle, a kinematic model based on auxiliary yaw moment is constructed; the kinematic model based on auxiliary yaw moment is used to determine running environment data according to auxiliary yaw moment; the running environment data includes yaw angular velocity, yaw angular velocity change, roll angle and roll angle change;

[0010] Based on the operating environment data and the working conditions of the distributed drive unmanned wheeled vehicle system, a reinforcement learning control system is constructed based on a two-layer deep Q learning network-priority experience revisit algorithm; and the reinforcement learning control system is trained;

[0011] Based on the current working conditions and the current operating environment data, the auxiliary yaw moment is determined using the trained reinforcement learning control system;

[0012] Control of a distributed drive unmanned wheeled vehicle system based on auxiliary yaw torque.

[0013] Optionally, the kinematic model is:

[0014]

[0015] Where, ΔM represents the auxiliary yaw moment, Represents the change in roll angle, ω r represents the yaw angular velocity, Indicates the change in yaw rate, C f and C r Both represent the front and rear wheel suspension stiffness, β represents the roll angle, m represents the mass of the distributed drive unmanned wheeled vehicle system, and v x represents the lateral speed of the distributed drive unmanned wheeled vehicle system, δ represents the front wheel turning angle of the distributed drive unmanned wheeled vehicle system, and I z Represents the moment of inertia of the distributed drive unmanned wheeled vehicle system.

[0016] Optionally, the training of the reinforcement learning control system specifically includes:

[0017] Based on the roll angle and roll angle change in the operating environment data, the stability of the distributed drive unmanned wheeled vehicle system is scored and a stability reward is obtained;

[0018] Taking operating environment data, operating conditions of the distributed drive unmanned wheeled vehicle system, auxiliary yaw moment, and stability reward as input data;

[0019] Utilizing the experience pool in the reinforcement learning control system to store the input data, and calculating the time series difference error corresponding to each piece of input data;

[0020] Calculate the corresponding priority based on the timing difference error corresponding to each input data;

[0021] Calculate sample probability based on priority;

[0022] Select input data from the experience pool based on sample probability;

[0023] Based on the selected input data, a two-layer deep Q-learning network in the reinforcement learning control system is used to generate the control law;

[0024] The control law is optimized based on the Munchausen optimization algorithm to obtain the optimal control law.

[0025] Optionally, the utilizing an experience pool in a reinforcement learning control system to store the input data and calculating a time series difference error corresponding to each piece of input data specifically includes:

[0026] Using the formula δ t =r t +γ*Q2(s t ,argmaxQ1(s t ,a t ))-Q1(s t+1 ,a t+1 ) Calculate the time series difference error δ corresponding to each input data at time t t ;

[0027] Among them, r t represents the stability reward at time t, γ represents the discount factor, Q1 represents the evaluation network, Q2 represents the target network, s t represents the operating environment data at time t and the working condition of the distributed drive unmanned wheeled vehicle system, argmax represents the aggregate function, a t represents the auxiliary yaw moment at time t, s t+1 represents the operating environment data at time t+1 and the working condition of the distributed drive unmanned wheeled vehicle system, a t+1 represents the auxiliary yaw moment at time t+1.

[0028] Optionally, the calculating of the corresponding priority based on the timing differential error corresponding to each piece of output data specifically includes:

[0029] Using the formula P priority (t)=|δ t |+ε,ε>0Calculate the priority P of each input data priority (t);

[0030] Here, ε represents the dielectric constant.

[0031] Optionally, calculating the sample probability based on the priority specifically includes:

[0032] Using the formula Calculate the sample probability P data(t) ;

[0033] Among them, the adjustment factor a≥0, when a=0, the sampling algorithm of the experience pool adopts random sampling, when a>0, by adjusting the specific value of a, the probability of the experience pool extracting training from the input data is controlled, P pariority(t) Indicates the priority of each input data. represents the total priority of the input data, and k represents the number of input data.

[0034] Optionally, optimizing the control law based on the Munchausen optimization algorithm to obtain the optimal control law specifically includes:

[0035] Utilize the formula TargetQ t =r t +α*ln(π(at|s t ))+γ*Q2(s t+1 , argmaxQ1(s t+1 , a t+1 ))Get the optimal control law;

[0036] Wherein, α represents the custom proportional coefficient. When α=0, the Munchausen optimization algorithm does not optimize the control law. π(a t |s t ) represents the input data selection control strategy, TargetQ t represents the optimal control law.

[0037] Optionally, controlling the distributed drive unmanned wheeled vehicle system according to the auxiliary yaw torque specifically includes:

[0038] Based on the auxiliary yaw moment, the equal division principle algorithm is used to obtain the auxiliary torque corresponding to each hub motor;

[0039] The wheel hub motors are driven accordingly based on the assist torque corresponding to each wheel hub motor.

[0040] In a second aspect, the present application provides a distributed drive unmanned wheeled vehicle system control device, which is applied to the distributed drive unmanned wheeled vehicle system control method described above, and the distributed drive unmanned wheeled vehicle system control device includes:

[0041] A distributed drive unmanned wheeled vehicle system building module is used to build a distributed drive unmanned wheeled vehicle system with Ackermann steering and differential steering; the steering configuration of the vehicle front wheels of the distributed drive unmanned wheeled vehicle system adopts an Ackermann structure; the front and rear axle transmission configuration of the distributed drive unmanned wheeled vehicle system adopts a power structure driven by four wheel hub motors;

[0042] a kinematics model construction module, configured to construct a kinematics model based on auxiliary yaw moment based on the distributed driving unmanned wheeled vehicle system and a two-degree-of-freedom analysis model of the vehicle; the kinematics model based on auxiliary yaw moment is configured to determine operating environment data according to the auxiliary yaw moment; the operating environment data includes yaw angular velocity, yaw angular velocity change, roll angle and roll angle change;

[0043] a reinforcement learning control system construction and training module, configured to construct a reinforcement learning control system based on a double-layer deep Q learning network-priority experience revisiting algorithm according to the operating environment data and the working condition of the distributed driving unmanned wheeled vehicle system, and train the reinforcement learning control system;

[0044] an auxiliary yaw moment determination module, configured to determine the auxiliary yaw moment by using the trained reinforcement learning control system according to the current working condition and the current operating environment data;

[0045] a driving module, configured to control the distributed driving unmanned wheeled vehicle system according to the auxiliary yaw moment.

[0046] In a third aspect, the present application provides a computer device, comprising a memory, a processor, a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of the distributed driving unmanned wheeled vehicle system control method.

[0047] According to the specific embodiments provided by the present application, the present application has the following technical effects:

[0048] The present application provides a control method, device and equipment for a distributed drive unmanned wheeled vehicle system. By constructing a distributed drive unmanned wheeled vehicle system with Ackermann steering and differential steering, that is, adopting an Ackermann structure to form a steering configuration for front wheel steering, and simultaneously adopting four hub motors to replace the traditional front and rear axle transmission configuration, the distributed drive unmanned wheeled vehicle system can adopt a steering method that integrates Ackermann steering and differential steering during the steering process, thereby enhancing the steering maneuverability of the distributed drive unmanned wheeled vehicle system. According to the two-degree-of-freedom analysis model of the distributed drive unmanned wheeled vehicle system and the vehicle, a kinematic model based on the auxiliary yaw torque is constructed. By using the kinematic model to determine the operating environment, the roll angle of the distributed drive unmanned wheeled vehicle system can be changed when the vehicle speed and steering remain unchanged; by constructing and training a reinforcement learning control system based on the Double-Deep Q_learning Network-Prioritized Experience Replay (DDQN-PER) algorithm according to the operating environment data and the working conditions of the distributed drive unmanned wheeled vehicle system, the optimal control law can be iteratively generated; according to the current working conditions and the current operating environment data, the auxiliary yaw torque is determined by using the optimal control law generated by the trained reinforcement learning control system; the distributed drive unmanned wheeled vehicle system is controlled according to the auxiliary yaw torque, which can improve the vehicle's roll stability. This application reduces the roll angle of the vehicle during high-speed steering by integrating Ackerman steering and differential steering, which can improve the vehicle's roll stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 This is a flow chart of a distributed drive unmanned wheeled vehicle system control method in one embodiment of the present application;

[0051] Figure 2 Schematic diagram of the distributed drive unmanned wheeled vehicle system with Ackerman steering and differential steering;

[0052] Figure 3 Schematic diagram of the reinforcement learning control system;

[0053] Figure 4 Schematic diagram of a distributed drive unmanned wheeled vehicle system controlled by auxiliary yaw torque;

[0054] Figure 5 A schematic diagram of the speed of a distributed drive unmanned wheeled vehicle system provided in one embodiment of the present application;

[0055] Figure 6 A schematic diagram of vehicle speed steering for a distributed drive unmanned wheeled vehicle system provided in one embodiment of the present application;

[0056] Figure 7 A schematic diagram of the auxiliary yaw torque output by a reinforcement learning control system is provided for one embodiment of the present application;

[0057] Figure 8 Schematic diagram of the roll angle of a distributed drive unmanned wheeled vehicle system without auxiliary yaw torque and the roll angle of a distributed drive unmanned wheeled vehicle system with auxiliary yaw torque applied in one embodiment of the present application;

[0058] Figure 9 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0060] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0061] In an exemplary embodiment, Figure 1 As shown, a distributed drive unmanned wheeled vehicle system control method is provided. The method is executed by a computer device. Specifically, it can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In an embodiment of the present application, a distributed drive unmanned wheeled vehicle system control method provided by the present application includes S101-S105:

[0062] S101: Construct a distributed drive unmanned wheeled vehicle system with Ackermann steering and differential steering. The front-wheel steering configuration of the distributed drive unmanned wheeled vehicle system adopts an Ackermann structure. The front and rear axle transmission configuration of the distributed drive unmanned wheeled vehicle system adopts a power structure driven by four wheel hub motors. Each of the four wheel hub motors can operate independently. A coordinated control model is established for the left front and rear motors and the right front and rear motors to generate an auxiliary yaw torque with the vehicle's center of mass as the origin. This enhances or suppresses the vehicle's yaw angular velocity during steering, thereby changing the vehicle's steering roll stability while maintaining the front wheel steering angle.

[0063] In an exemplary embodiment, Figure 2 As shown in the figure, the four wheel hub motors of the distributed drive unmanned wheeled vehicle system use GEEKBRO brand K210 model wheel-side motors. The motor has a supply voltage of 48-60VDC, a rated power of 1.2kW, a peak power of 3.5kW, and can output a maximum motor torque of 50Nm at rated speed. The peak torque difference of the vehicle composed of four wheel hub motors is 200Nm.

[0064] S102: Based on the distributed drive unmanned wheeled vehicle system and the two-degree-of-freedom analysis model of the vehicle, a kinematic model based on the auxiliary yaw moment is constructed; the kinematic model based on the auxiliary yaw moment is used to determine operating environment data according to the auxiliary yaw moment; the operating environment data includes: yaw angular velocity, yaw angular velocity change, roll angle and roll angle change.

[0065] The effect of the auxiliary yaw moment on the vehicle differential steering process is analyzed by using a two-degree-of-freedom analysis model of the vehicle. The two-degree-of-freedom analysis model of the vehicle is a vehicle analysis model; the two-degree-of-freedom analysis model of the vehicle is:

[0066]

[0067] Taking into account the combined effects of differential steering and Ackerman steering, the influence of the auxiliary yaw moment on the vehicle kinematic differential equation is analyzed through the vehicle's two-degree-of-freedom analysis model, and then a kinematic model based on the auxiliary yaw moment is obtained; the kinematic model based on the auxiliary yaw moment is:

[0068]

[0069] Where m represents the mass of the distributed drive unmanned wheeled vehicle system, v x and v y They represent the lateral speed and longitudinal speed of the distributed drive unmanned wheeled vehicle system, and They represent the changes in the lateral velocity and longitudinal velocity of the distributed drive unmanned wheeled vehicle system, ω r represents the yaw angular velocity, ω r Indicates the change in yaw rate, ΣF x represents the sum of the lateral forces of the distributed drive unmanned wheeled vehicle system, ΣF y represents the sum of the longitudinal forces of the distributed drive unmanned wheeled vehicle system, ΔM represents the auxiliary yaw moment, and I z represents the moment of inertia of the distributed drive unmanned wheeled vehicle system, Represents the change in roll angle, ω r represents the yaw angular velocity, Indicates the change in yaw angular velocity, C f and C r Both represent the front and rear wheel suspension stiffness, β represents the roll angle, and δ represents the front wheel turning angle of the distributed drive unmanned wheeled vehicle system.

[0070] Through the kinematic model based on the auxiliary yaw torque, it can be seen that when the vehicle speed and steering remain unchanged, by controlling the auxiliary yaw torque, the change in the vehicle's yaw angular velocity can be affected, and then the yaw angular velocity can be increased or decreased. This change will affect the numerical value of the change in the vehicle's roll angle, and ultimately change the value of the vehicle's roll angle when the vehicle speed and steering remain unchanged.

[0071] S103: Based on the operating environment data and the operating conditions of the distributed drive unmanned wheeled vehicle system, a reinforcement learning control system is constructed based on a double-layer deep Q learning network-priority experience revisit algorithm; and the reinforcement learning control system is trained.

[0072] The reinforcement learning control system uses an offline training method and a kinematic model as its operating environment. Based on the operating conditions and operating environment data of the distributed-drive unmanned wheeled vehicle at time t, as well as the auxiliary yaw torque, it calculates and transmits the operating environment data of the distributed-drive unmanned wheeled vehicle at time t+1 to the reinforcement learning control system. Simultaneously, the vehicle's roll stability is scored based on the values ​​of the roll angle and the amount of roll angle change, resulting in a stability reward, which is then returned to the reinforcement learning control system.

[0073] like Figure 3As shown, the reinforcement learning control system is constructed based on a deep Q-learning network-prioritized experience revisiting algorithm. The reinforcement learning control system comprises the right half, which includes an experience pool, a two-layer deep Q-learning network (DDQN), and a Munchausen optimization algorithm. The experience pool is responsible for storing input data; the Munchausen optimization algorithm is responsible for iterating the optimal control law of the reinforcement learning control system; and the DDQN, as the main body of the reinforcement learning control system, is responsible for generating and storing the optimal control law. By interacting with the "operating environment" and outputting an auxiliary yaw force, the operating conditions of the distributed drive unmanned wheeled vehicle system, along with driving environment data, auxiliary yaw torque, and stability rewards calculated by the operating environment are used as input data. The internal neural network transmits this input data to the operating conditions, causing the operating environment to enter the operating conditions of the distributed drive unmanned wheeled vehicle system at time t+1, and the above process repeats. The operating environment also evaluates the input data at that time, which determines the iterative upgrade status of the neural network.

[0074] S104: Determine an auxiliary yaw moment based on the current working condition and the current operating environment data using the trained reinforcement learning control system.

[0075] The trained reinforcement learning control system includes a trained optimal control law. In order to eliminate the dimensional differences between the operating environment data and the operating conditions of the distributed drive unmanned wheeled vehicle system, a normalization process is used to speed up the calculation of the control strategy.

[0076] As a specific embodiment, the maximum value of the auxiliary yaw moment is 200 and the minimum value is -200, where a positive value represents an auxiliary yaw moment clockwise around the vertical direction of the vehicle, and a negative value represents an auxiliary yaw moment counterclockwise around the vertical direction of the vehicle.

[0077] S105: Controlling the distributed drive unmanned wheeled vehicle system according to the auxiliary yaw torque.

[0078] In an exemplary embodiment of the present application, the training process of the reinforcement learning control system in the above S103 can be replaced by the following S301-S308:

[0079] S301: Based on the roll angle and roll angle change in the operating environment data, the stability of the distributed drive unmanned wheeled vehicle system is scored to obtain a stability reward.

[0080] S302: Taking operating environment data, operating conditions of the distributed drive unmanned wheeled vehicle system, auxiliary yaw moment, and stability reward as input data.

[0081] S303: Using the experience pool in the reinforcement learning control system to store the input data, and using the formula δ t =r t+γ*Q2(s t , argmaxQ1(s t , a t ))-Q1(s t+1 , a t+1 ) Calculate the time series difference error δ corresponding to each input data at time t t .

[0082] Among them, r t represents the stability reward at time t, γ represents the discount factor, which is 0.9, Q1 represents the evaluation network, Q2 represents the target network, and s t represents the operating environment data at time t and the working condition of the distributed drive unmanned wheeled vehicle system, argmax represents the aggregate function, a t represents the auxiliary yaw moment at time t, s t+1 represents the operating environment data at time t+1 and the working condition of the distributed drive unmanned wheeled vehicle system, a t+1 represents the auxiliary yaw moment at time t+1.

[0083] During the experience pool input data extraction process, random sampling is used to ensure that each piece of input data has an equal probability of being extracted for training and updating. Since each piece of input data corresponds to a different temporal difference error (TD-error) value, and this value will be used in the gradient calculation to update the network parameters, reflecting that different data have different degrees of influence on the network parameter update, to speed up network training, data with larger TD-error values ​​are given higher priority, making them more likely to be used in network training. This paper adopts a ratio-based priority algorithm, which is currently used in most applications. Its priority calculation method can significantly identify higher-priority data and ensure that such data is frequently selected for training.

[0084] S304: Based on the timing difference error corresponding to each input data, use the formula P priority (t)=|δ t |+ε,ε>0Calculate the priority P of each input data priority (t); where ε represents the dielectric constant.

[0085] S305: Utilize formula based on priority Calculate the sample probability P data(t) ; Among them, the adjustment factor a≥0, when a=0, the sampling algorithm of the experience pool adopts random sampling, when a>0, by adjusting the specific value of a, the probability of the experience pool extracting training from the input data is controlled, P pariority(t) Indicates the priority of each input data. represents the total priority of the input data, and k represents the number of input data.

[0086] S306: Select input data from the experience pool based on the sample probability.

[0087] With the help of the PER algorithm, the input data entering the experience pool includes the vehicle's operating environment data and the operating conditions of the distributed drive unmanned wheeled vehicle system at time t, the operating environment data and the operating conditions of the distributed drive unmanned wheeled vehicle system at time t+1, the auxiliary yaw torque at time t, and the vehicle stability reward. Each input data is processed by calculating the TD-error. After priority experience sampling, the experience pool will output sampling data with a higher TD-error for neural network training.

[0088] S307: Based on the selected input data, a control law is generated using a two-layer deep Q learning network in a reinforcement learning control system.

[0089] The reinforcement learning control system utilizes a two-layer deep Q-learning network. The second-layer network updates parameters at a slower rate than the first-layer network. This delayed synchronization between the two layers addresses the overestimation problem inherent in deep reinforcement learning and leverages the time-efficient offline training of discrete neural networks. The two-layer deep Q-learning network iteratively generates control laws by analyzing the relationship between input data. Specifically, based on the operating conditions and operating environment data of the distributed drive unmanned wheeled vehicle system at time t, it generates an auxiliary yaw torque that maximizes vehicle roll stability.

[0090] DDQN sets up a two-layer neural network Q1 target network and Q2 evaluation network. Q1 and Q2 are independent of each other, and the Q1 network parameters are updated in real time as training progresses. The Q2 network parameters are synchronized with the Q1 network parameters after a certain number of training rounds. The two-layer network together constitutes the objective function and estimates the selected input data. The objective function is:

[0091] DQN: TargetQ t =r t +γ*max(Q(s t+1 , a t ));

[0092] DDQN:Targe t Q t =r t +γ*Q2(s t+1 ,argmaxQ1(s t+1 ,a t+1 ));

[0093] The DDQN improvement involves first selecting the input data with the highest evaluation using the Q1 target network based on the state. The Q2 evaluation network then reevaluates the selected input data, which is then used to update the neural network. Because the Q2 network parameters lag behind the updated Q1 network parameters, the secondary evaluation value is slightly lower than the initial evaluation of the input data. This offsets overestimation and improves the optimization of the energy management control strategy.

[0094] S308: Based on the Munchausen optimization algorithm, use the formula Tar getQ t =r t +α*ln(π(a t |s t ))+γ*Q2(s t+1 , argmaxQ1(s t+1 , a t+1 )) optimizes the control law to obtain the optimal control law; wherein α represents a custom proportional coefficient. When α=0, the Munchausen optimization algorithm does not optimize the control law. π(a t |s t ) represents the input data selection control strategy, Tar getQ t represents the optimal control law.

[0095] The Munchausen optimization algorithm iteratively optimizes network parameters by using a back propagation method based on mean square error. The iterative optimization process is expressed as:

[0096]

[0097] Among them, W t represents the neural network parameters, W t+1 Represents the optimized neural network parameters, learning_rate represents the neural network learning rate, which is set to 0.0001, Loss represents the loss of one round of neural network training, TargetQ (i) represents the control law of the energy management control strategy trained once, and n is a variable used to store the number of calculations.

[0098] Combining the objective function in DDQN with analysis, the starting point of the network parameter optimization process is the process of the neural network selecting input data based on the operating environment. Assuming that the network has been optimized to the optimal strategy, the probability of selecting high-priority input data in the experience pool is 1, and the probability of selecting the remaining data is 0. Based on the above analysis, the Munchausen algorithm takes the logarithm of the probability of selecting high-priority input data, and the corresponding value of high-priority input data is 0, and the corresponding values ​​of the remaining data are -∞. By combining the selection probability with the stability reward, low-priority input data is suppressed, achieving the effect of focusing training on selecting high-priority input data. The calculation process is:

[0099]

[0100] in, Indicates the effect of high-priority input data, α indicates a custom scale factor, π(a t |s t ) represents the high-priority input data selection control strategy in the energy management system. In order to make the selection probability of input data computable, it is necessary to use the softmax strategy to perform flexible maximum calculation on the selection probability. The softmax strategy is:

[0101]

[0102] Among them, Q(s t ,a t ) represents the network evaluation of the control strategy when the auxiliary yaw moment at time t is used under the working conditions and operating environment data of the distributed drive unmanned wheeled vehicle system at time t. The maximum flexibility calculation formula is substituted into the objective function in DDQN to obtain the expression of the optimal control law.

[0103] The optimal control law can output an auxiliary yaw torque based on the operating environment data and the working conditions of the distributed drive unmanned wheeled vehicle system. By executing this auxiliary yaw torque, the vehicle can adjust its roll state without changing the vehicle speed and steering.

[0104] In an exemplary embodiment, the above step 105 can be replaced by S501-S502:

[0105] S501: Based on the auxiliary yaw moment, an equal division principle algorithm is used to obtain the auxiliary torque corresponding to each hub motor.

[0106] S502: Driving the wheel hub motors based on the assist torque corresponding to each wheel hub motor.

[0107] like Figure 4As shown in the figure, the reinforcement learning control system will be deployed on the vehicle controller with an operating system, and a CAN communication network needs to be built between the vehicle controller, drive motor, pitch angle sensor, and roll angle sensor. Based on the above deployment, the process of using auxiliary yaw torque for control is as follows: (1) The four drive motors, yaw angle sensor, and roll angle sensor transmit the vehicle speed, steering, yaw angular velocity, yaw angular velocity change, roll angle, and roll angle change data to the vehicle controller through the bus; (2) The auxiliary yaw torque is then generated through the deployed reinforcement learning control system technology; (3) The vehicle controller calculates the generated auxiliary yaw torque according to the principle of equal division to obtain the auxiliary torque of the four motors; (4) The torque is sent to the corresponding drive motor through the CAN bus for execution; (5) Repeat the above steps (1)-(4). The vehicle speed and steering used are as follows: Figure 5 and Figure 6 As shown, the vehicle speed is maintained at 15 m / s (54 km / h) and the steering is 0.6 rad (the front wheels are turning clockwise at an angle of 34 degrees).

[0108] like Figure 8 As shown in the figure, without the auxiliary yaw torque, the roll angle of the vehicle during the steering process is different from the roll angle of the vehicle during the steering process when the reinforcement learning control system is applied as shown in the figure. Figure 7 As shown in the figure, the roll angle of the vehicle during the steering process under the action of the auxiliary yaw torque is compared. It can be seen that by applying the auxiliary yaw torque, the auxiliary roll angle of the vehicle does not undergo a large mutation during the initial steering process, and during the vehicle steering process, the body roll angle is reduced by an average of 0.75 degrees, effectively improving the vehicle's roll stability during the steering process.

[0109] Distributed drive unmanned wheeled vehicle systems use remote control commands to control the vehicle's forward movement, steering, and other maneuvers. Without driver tolerance restrictions, they can perform extreme driving maneuvers such as high-speed, large-scale steering. To avoid roll accidents caused by excessive roll angles during high-speed steering, a distributed drive unmanned wheeled vehicle control method based on reinforcement learning has been developed. By leveraging the independent operation of the four in-wheel motors in a distributed drive configuration, a reinforcement learning neural network is used to control the auxiliary yaw torque generated by the four motors, reducing the roll angle during high-speed steering and improving the vehicle's roll stability.

[0110] In another exemplary embodiment, a distributed drive unmanned wheeled vehicle system control device is provided, including: a distributed drive unmanned wheeled vehicle system construction module, a kinematic model construction module, a reinforcement learning control system construction and training module, an auxiliary yaw moment determination module and a drive module.

[0111] The distributed drive unmanned wheeled vehicle system construction module is used to construct a distributed drive unmanned wheeled vehicle system with Ackermann steering and differential steering; the steering configuration of the front wheel steering of the distributed drive unmanned wheeled vehicle system adopts an Ackermann structure; the front and rear axle transmission configuration of the distributed drive unmanned wheeled vehicle system adopts a power structure driven by four wheel hub motors.

[0112] The kinematic model construction module is used to construct a kinematic model based on the auxiliary yaw moment based on the distributed drive unmanned wheeled vehicle system and the two-degree-of-freedom analysis model of the vehicle; the kinematic model based on the auxiliary yaw moment is used to determine operating environment data according to the auxiliary yaw moment; the operating environment data includes: yaw angular velocity, yaw angular velocity change, roll angle and roll angle change.

[0113] The reinforcement learning control system construction and training module is used to construct a reinforcement learning control system based on the operating environment data and the operating conditions of the distributed drive unmanned wheeled vehicle system, based on the double-layer deep Q learning network-priority experience revisit algorithm; and train the reinforcement learning control system.

[0114] The auxiliary yaw moment determination module is used to determine the auxiliary yaw moment according to the current working condition and the current operating environment data using the trained reinforcement learning control system.

[0115] The driving module is used to control a distributed drive unmanned wheeled vehicle system according to the auxiliary yaw torque.

[0116] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a distributed drive unmanned wheeled vehicle system control method is implemented.

[0117] Those skilled in the art will understand that Figure 9The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above method embodiments.

[0118] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0119] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0120] The database involved in each of the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, and the like, without being limited thereto. The processor involved in each of the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, and the like, without being limited thereto.

[0121] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but it should be considered that any combination of the technical features is within the scope of the present disclosure, as long as there is no contradiction.

[0122] The principles and implementation manners of the present application are described by using specific examples herein, and the above embodiments are only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, the specific implementation manners and application ranges can be changed according to the idea of the present application. In summary, the content of the present description should not be understood as a limitation of the present application.

Claims

1. A distributed drive unmanned wheeled vehicle system control method, characterized in that: The distributed drive unmanned wheeled vehicle system control method includes: Constructing a distributed drive unmanned wheeled vehicle system with Ackermann steering and differential steering; the steering configuration of the front wheels of the distributed drive unmanned wheeled vehicle system adopts an Ackermann structure; the front and rear axle transmission configuration of the distributed drive unmanned wheeled vehicle system adopts a power structure driven by four wheel hub motors; Based on the distributed drive unmanned wheeled vehicle system and the two-degree-of-freedom analysis model of the vehicle, a kinematic model based on the auxiliary yaw moment is constructed; the kinematic model based on the auxiliary yaw moment is used to determine operating environment data according to the auxiliary yaw moment; the operating environment data includes: yaw angular velocity, yaw angular velocity change, roll angle, and roll angle change; Based on the operating environment data and the working conditions of the distributed drive unmanned wheeled vehicle system, a reinforcement learning control system is constructed based on a two-layer deep Q learning network-priority experience revisit algorithm; and the reinforcement learning control system is trained; Based on the current working conditions and the current operating environment data, the auxiliary yaw moment is determined using the trained reinforcement learning control system; Control of a distributed drive unmanned wheeled vehicle system based on auxiliary yaw torque.

2. The distributed drive unmanned wheeled vehicle system control method according to claim 1, characterized in that: The kinematic model is: Where, ΔM represents the auxiliary yaw moment, Represents the change in roll angle, ω r represents the yaw angular velocity, Indicates the change in yaw rate, C f and C r Both represent the front and rear wheel suspension stiffness, β represents the roll angle, m represents the mass of the distributed drive unmanned wheeled vehicle system, and v x represents the lateral speed of the distributed drive unmanned wheeled vehicle system, δ represents the front wheel turning angle of the distributed drive unmanned wheeled vehicle system, and I z Represents the moment of inertia of the distributed drive unmanned wheeled vehicle system.

3. The distributed drive unmanned wheeled vehicle system control method according to claim 1, characterized in that: The reinforcement learning control system is trained, specifically including: Based on the roll angle and roll angle change in the operating environment data, the stability of the distributed drive unmanned wheeled vehicle system is scored and a stability reward is obtained; Taking operating environment data, operating conditions of the distributed drive unmanned wheeled vehicle system, auxiliary yaw moment, and stability reward as input data; Utilizing the experience pool in the reinforcement learning control system to store the input data, and calculating the time series difference error corresponding to each piece of input data; Calculate the corresponding priority based on the timing difference error corresponding to each input data; Calculate sample probability based on priority; Select input data from the experience pool based on sample probability; Based on the selected input data, a two-layer deep Q-learning network in the reinforcement learning control system is used to generate the control law; The control law is optimized based on the Munchausen optimization algorithm to obtain the optimal control law.

4. The distributed drive unmanned wheeled vehicle system control method according to claim 3, characterized in that: The process of using the experience pool in the reinforcement learning control system to store the input data and calculating the time series difference error corresponding to each piece of input data specifically includes: Using the formula δ t =r t +γ*Q2(s t , argmaxQ1(s t , a t ))-Q1(s t+1 , a t+1 ) Calculate the time series difference error δ corresponding to each input data at time t t ; Among them, r t represents the stability reward at time t, γ represents the discount factor, Q1 represents the evaluation network, Q2 represents the target network, s t represents the operating environment data at time t and the working condition of the distributed drive unmanned wheeled vehicle system, argmax represents the aggregate function, a t represents the auxiliary yaw moment at time t, s t+1 represents the operating environment data at time t+1 and the working condition of the distributed drive unmanned wheeled vehicle system, a t+1 represents the auxiliary yaw moment at time t+1.

5. The distributed drive unmanned wheeled vehicle system control method according to claim 4, characterized in that: The calculating of the corresponding priority based on the timing differential error corresponding to each piece of output data specifically includes: Using the formula P priority (t)=|δ t |+ε,ε>0Calculate the priority P of each input data priority (t); Here, ε represents the dielectric constant.

6. The distributed drive unmanned wheeled vehicle system control method according to claim 5, characterized in that: The priority-based sample probability calculation specifically includes: Using the formula Calculate the sample probability P data(t) ; Among them, the adjustment factor a≥0, when a=0, the sampling algorithm of the experience pool adopts random sampling, when a>0, by adjusting the specific value of a, the probability of the experience pool extracting training from the input data is controlled, P pariority(t) Indicates the priority of each input data. represents the total priority of the input data, and k represents the number of input data.

7. The distributed drive unmanned wheeled vehicle system control method according to claim 4, characterized in that: The control law is optimized based on the Munchausen optimization algorithm to obtain the optimal control law, specifically including: Utilize the formula TargetQ t =r t +α*ln(π(a t |s t ))+γ*Q2(s t+1 ,argmaxQ1(s t+1 ,a t+1 ))Get the optimal control law; Wherein, α represents the custom proportional coefficient. When α=0, the Munchausen optimization algorithm does not optimize the control law. π(a t |s t ) represents the input data selection control strategy, TargetQ t represents the optimal control law.

8. The distributed drive unmanned wheeled vehicle system control method according to claim 1, characterized in that: The distributed drive unmanned wheeled vehicle system controlled according to the auxiliary yaw torque specifically includes: Based on the auxiliary yaw moment, the equal division principle algorithm is used to obtain the auxiliary torque corresponding to each hub motor; The wheel hub motors are driven accordingly based on the assist torque corresponding to each wheel hub motor.

9. A distributed drive unmanned wheeled vehicle system control device, applied to the distributed drive unmanned wheeled vehicle system control method according to any one of claims 1 to 8, characterized in that: The distributed drive unmanned wheeled vehicle system control device includes: A distributed drive unmanned wheeled vehicle system building module is used to build a distributed drive unmanned wheeled vehicle system with Ackermann steering and differential steering; the steering configuration of the vehicle front wheels of the distributed drive unmanned wheeled vehicle system adopts an Ackermann structure; the front and rear axle transmission configuration of the distributed drive unmanned wheeled vehicle system adopts a power structure driven by four wheel hub motors; a kinematic model construction module, configured to construct a kinematic model based on the auxiliary yaw moment based on the distributed drive unmanned wheeled vehicle system and the two-degree-of-freedom analysis model of the vehicle; the kinematic model based on the auxiliary yaw moment is configured to determine operating environment data based on the auxiliary yaw moment; the operating environment data comprising: yaw angular velocity, yaw angular velocity variation, roll angle, and roll angle variation; A reinforcement learning control system construction and training module is used to build and train a reinforcement learning control system based on operating environment data and the operating conditions of the distributed drive unmanned wheeled vehicle system, using a two-layer deep Q learning network-priority experience revisit algorithm; An auxiliary yaw moment determination module is used to determine the auxiliary yaw moment based on the current working conditions and the current operating environment data using the trained reinforcement learning control system; A drive module is used to control a distributed drive unmanned wheeled vehicle system based on an auxiliary yaw moment.

10. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the distributed drive unmanned wheeled vehicle system control method according to any one of claims 1 to 8.

Images