Low adhesion road four-wheel drive torque distribution method based on reinforcement learning

Abstract

The application relates to the technical field of vehicle control systems, in particular to a low-attached-pavement four-wheel-drive torque distribution method based on reinforcement learning; the method comprises the following steps: acquiring initial state information of a target vehicle, and determining whether the target vehicle is in an out-of-control state based on the state information; when it is determined that the target vehicle is not in the out-of-control state, processing the initial state information by using a pre-trained main strategy network to obtain a control strategy output by the main strategy network with the maximum reward value of a reward function as a target; wherein the reward function comprises a speed tracking reward constructed based on a speed item and a control item, a wheel slip penalty constructed based on the speed item, and a stability reward constructed based on an angle item; and then adjusting the output torque of a driving motor corresponding to each wheel based on the control strategy. The scheme of the application can improve the adaptability of vehicle stability control when facing complex road conditions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle control systems, and particularly relates to a low adhesion road surface four-wheel drive torque distribution method based on reinforcement learning. BACKGROUND

[0002] In complex working conditions (such as icy, snowy and muddy road surfaces), the adhesion coefficient between the vehicle tire and the road surface is low and changes frequently, thus higher requirements are put forward for the stability control strategy of the vehicle.

[0003] In the related art, a linear control strategy based on PID control or sliding mode control is often used to adjust the four-wheel torque distribution of the vehicle; the strategy based on linear control in the related art has a certain implementability in a single working condition (such as an icy road surface); however, in a complex working condition, the adhesion coefficient between the vehicle tire and the road surface may change suddenly and fluctuate at a high frequency, thus having a high nonlinear characteristic; therefore, the adjustment based on the linear strategy in the related art is prone to a state of control overshoot or response lag, thus causing the vehicle to be unstable; that is, the adaptability of the stability control of the vehicle in the related art when facing a complex road condition is poor.

[0004] Therefore, how to improve the adaptability of the stability control of the vehicle when facing a complex road condition is a problem to be solved. SUMMARY

[0005] Therefore, it is necessary to provide a low adhesion road surface four-wheel drive torque distribution method, device and vehicle based on reinforcement learning, which can improve the adaptability of the stability control of the vehicle when facing a complex road condition.

[0006] In a first aspect, the present application provides a low adhesion road surface four-wheel drive torque distribution method based on reinforcement learning, comprising:

[0007] obtaining initial state information of a target vehicle, the state information comprising parameters of a control item, parameters of a speed item and parameters of an angle item, and determining whether the target vehicle is in an out-of-control state based on the state information;

[0008] when it is determined that the target vehicle is not in the out-of-control state, processing the initial state information by using a pre-trained main policy network to obtain a control policy output by the main policy network with a reward value of a reward function being maximized as a target; wherein the reward function includes a speed tracking reward constructed based on the speed term and the control term, a wheel slip penalty constructed based on the speed term, and a stability reward constructed based on the angle term; the control policy includes a torque ratio corresponding to each wheel; the main policy control network is obtained by reinforcement learning training of experience samples of multiple driving scenes in an experience pool with the reward value of the reward function being maximized as a target value;

[0009] adjusting output torques of driving motors corresponding to each wheel based on the control policy.

[0010] In one of the embodiments, the method further includes:

[0011] obtaining response state information of the target vehicle after the control policy is executed, and determining a reward value of the reward function based on the response state information and the initial state information;

[0012] constructing an experience sample from the initial state information, the control policy, the reward value corresponding to the control policy, and the response state information, and writing the experience sample into the experience pool;

[0013] training a main value network by using the experience samples in the experience pool to obtain an updated main value network;

[0014] updating the main policy network based on the updated main value network.

[0015] In one of the embodiments, the training of the main value network by using the experience samples in the experience pool to obtain the updated main value network includes:

[0016] extracting a set number of the experience samples from the experience pool as training samples of the value network;

[0017] for each of the training samples, calling the main value network to process the initial state information and the control policy in the training sample to obtain an expected cumulative reward output by the main value network;

[0018] The target policy network is used to process the response state information in the training sample, to obtain a stable policy output by the target policy network with the maximum reward value of the reward function as the target, and the target value network is used to process the initial response state information and the stable policy, to obtain a stable cumulative reward; wherein the target policy network has the same network structure as the main policy network; the target value network has the same network structure as the main value network.

[0019] The main value network is trained based on a value loss function with the minimum value loss as the target, to obtain an updated main value network; the value loss function is constructed based on the reward value in the training sample and the stable cumulative reward and the expected cumulative reward corresponding to the training sample.

[0020] In one of the embodiments, the updating of the main policy network based on the updated main value network comprises:

[0021] Based on the gradient update strategy, the parameters of the main policy network are updated using the updated parameters of the main value network, to obtain an updated main policy network.

[0022] In one of the embodiments, the method further comprises:

[0023] Based on the updated parameters of the main value network and a first penetration ratio, the parameters of the target value network are updated; wherein the first penetration ratio is the ratio of the respective use ratios of the updated parameters of the main value network and the current parameters of the target value network.

[0024] Based on the updated parameters of the main policy network and a second penetration ratio of the main policy network to the target policy network, the parameters of the target policy network are updated; wherein the second penetration ratio is the ratio of the respective use ratios of the updated parameters of the main policy network and the current parameters of the target policy network.

[0025] In one of the embodiments, the adjusting of the output torque of the driving motor corresponding to each wheel based on the control strategy comprises:

[0026] The road adhesion coefficient is determined, and the change rate is determined according to the update frequency of the road adhesion coefficient;

[0027] When the change rate is greater than a fluctuation threshold, noise is added in the control strategy, and the output torque of the driving motor corresponding to each wheel is adjusted based on the control strategy with the added noise;

[0028] When the variation rate is less than or equal to the fluctuation threshold, the output torque of the drive motor corresponding to each wheel is adjusted based on the control strategy.

[0029] In one of the embodiments, the speed term includes the vehicle speed of the target vehicle and the wheel speed of the tire, and the determining the road adhesion coefficient comprises:

[0030] An image of a road on which the target vehicle travels is acquired, and a first adhesion coefficient is determined based on the image of the road;

[0031] A reflection coefficient of the road on which the target vehicle travels is acquired, and a second adhesion coefficient is determined according to the reflection coefficient;

[0032] An expected wheel speed is determined based on the vehicle speed and the radius of the tire, and a correction coefficient is determined based on the deviation between the expected wheel speed and the wheel speed;

[0033] An initial adhesion coefficient is determined based on the first adhesion coefficient and the second adhesion coefficient, and the initial adhesion coefficient is corrected based on the correction coefficient to obtain the road adhesion coefficient.

[0034] In one of the embodiments, the out-of-control state includes a slipping state and a limit deflection state, and the method further comprises:

[0035] When the target vehicle is in the slipping state, an out-of-control wheel in the slipping state is determined based on the parameters of the speed term, and a safe torque range corresponding to the out-of-control wheel is determined;

[0036] A first control instruction for each wheel is generated based on the safe torque range corresponding to the out-of-control wheel, and the first control instruction is executed; the first control instruction is used to adjust the output torque of the drive motor corresponding to each wheel to the safe torque range corresponding to the out-of-control wheel;

[0037] When the target vehicle is in the limit deflection state, a safe torque range corresponding to each wheel is determined;

[0038] A second control instruction for each wheel is generated based on the safe torque range corresponding to each wheel, and the second control instruction is executed; the second control instruction is used to adjust the output torque of the drive motor corresponding to each wheel to the safe torque range corresponding to the wheel.

[0039] In a second aspect, the application further provides a torque distribution device, which comprises a state information acquisition module, a strategy generation module and an adjustment module, wherein:

[0040] obtain state information of the target vehicle, the state information comprising parameters of a control term, a speed term and an angle term, and determine whether the target vehicle is in an out-of-control state based on the state information;

[0041] generate a control strategy for the target vehicle based on the state information when it is determined that the target vehicle is not in the out-of-control state, the control strategy being output by a pre-trained master policy network based on a reward function that maximizes a reward value, the reward function comprising a speed tracking reward based on the speed term and the control term, a wheel slip penalty based on the speed term, and a stability reward based on the angle term, the control strategy comprising a torque ratio for each wheel, the master policy network being trained based on experience samples of multiple driving scenarios in an experience pool and the reward value of the reward function;

[0042] adjust an output torque of a drive motor corresponding to each wheel based on the control strategy.

[0043] In a third aspect, the present application also provides a vehicle, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the low adhesion road four-wheel drive torque distribution method based on reinforcement learning according to any one of the above first aspect when executing the computer program.

[0044] In a fourth aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the low adhesion road four-wheel drive torque distribution method based on reinforcement learning according to any one of the above first aspect when executed by a processor.

[0045] In a fifth aspect, the present application also provides a computer program product, comprising a computer program, and the computer program implements the steps of the low adhesion road four-wheel drive torque distribution method based on reinforcement learning according to any one of the above first aspect when executed by a processor.

[0046] The low adhesion road four-wheel drive torque distribution method based on reinforcement learning, the device and the vehicle described above, by means of reinforcement learning, utilize experience samples of multiple driving scenarios in an experience pool to train a policy model, so that the policy model can adapt to more driving scenarios; that is, the policy model has good adaptability to complex road conditions.

[0047] And, the reinforcement learning belongs to unsupervised training, and is training aiming at maximizing a reward function; and the reward function is a speed tracking reward based on a speed constraint, a stability reward of an angle constraint, and a slip penalty based on the speed constraint. Therefore, the control strategy generated by the main policy network for the state information of the target vehicle can make the target vehicle maintain the related speed state expected by the driver based on the accelerator opening degree, the brake opening degree, and the steering angle as much as possible, can make the target vehicle maintain the side angle and the yaw angle in a range not exceeding the limit value as much as possible, and can make the target vehicle avoid the slip state of the wheel as much as possible. Therefore, the control strategy generated based on the main policy network adjusts the output torque of the driving motor corresponding to each wheel, thereby improving the stability of the target vehicle and making the target vehicle maintain the stable state as much as possible.

[0048] And, the advantage of the reinforcement learning is that the trained network can continuously learn, that is, the main policy model can continuously learn the control strategy made each time, thereby better optimizing the decision-making ability of the main policy model, and making the main policy model generate a better control strategy for execution when facing complex road conditions, thereby improving the stability of the target vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other related drawings can be obtained by those skilled in the art without creative labor.

[0050] Figure 1 A flowchart of a low-attached road four-wheel drive torque distribution method based on reinforcement learning in an embodiment;

[0051] Figure 2 A flowchart of adjusting the output torque in an embodiment;

[0052] Figure 3 A flowchart of updating the main value network in an embodiment;

[0053] Figure 4 A schematic diagram of the relationship between the networks in an embodiment;

[0054] Figure 5 A flowchart of a low-attached road four-wheel drive torque distribution method based on reinforcement learning in another embodiment;

[0055] Figure 6A diagram for verifying the contrast effect in one embodiment;

[0056] Figure 7 A structural block diagram of the torque distribution device in one embodiment;

[0057] Figure 8 An internal structure diagram of the vehicle in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0059] It should be noted that the terms "first", "second", and the like used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "multiple" used in the present application refers to two and more than two. The term "and / or" used in the present application refers to one of the options, or any combination of multiple options.

[0060] The low-attached road four-wheel drive torque distribution method based on reinforcement learning provided by the embodiments of the present application can be applied to the control system of the vehicle, wherein the torque control system includes a main policy network, a target main policy network, a main value network, and a target value network; wherein the main policy network and the target main policy network use the same model architecture, but the parameters of the target main policy network are always lagging behind the main policy network updates; the main value network and the target value network use the same model architecture, but the parameters of the target value network are always lagging behind the main value network updates. Among them, the main policy network is configured in the vehicle, and the target main policy network, the main value network, and the target value network can be configured in the vehicle or in a cloud server capable of communicating with the vehicle. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0061] It should be noted that the low-attached road four-wheel drive torque distribution method based on reinforcement learning of the present application is applicable to four-wheel vehicles, and each wheel is driven by a corresponding electric motor; thus the torque corresponding to each wheel is adjustable.

[0062] In one exemplary embodiment, as Figure 1As shown, a low-μ road four-wheel drive torque distribution method based on reinforcement learning is provided, specifically comprising the following steps 10-30, wherein:

[0063] Step 10, obtaining the initial state information of the target vehicle, the state information including the parameters of the control item, the parameters of the speed item and the parameters of the angle item, and determining whether the target vehicle is in an out-of-control state based on the state information.

[0064] For the embodiment of the present application, the parameters of the control item include the active control items of the driver on the vehicle, including the throttle opening, the brake opening and the steering angle of the steering wheel. The parameters of the speed item include the vehicle speed, the speed of each wheel, the lateral acceleration and the longitudinal acceleration. The parameters of the angle item include the yaw rate and the side slip angle. Among them, some parameters in the state information can be directly measured by the sensors arranged on the target vehicle, and the rest of the parameters can be calculated by the physical law based on the measured parameters of the sensors. Further, the state information further includes the road adhesion coefficient, which is obtained based on the road image of the road traveled and the measurement data of the road traveled by the radar.

[0065] Specifically, the out-of-control state of the vehicle includes two states, one is the slip state of at least one wheel slipping, and the other is the limit side slip state of the side slip angle exceeding the corresponding side slip limit value. The action determined by the main strategy network of step 20 is used to control the target vehicle to try not to enter the out-of-control state; and once the target vehicle enters the out-of-control state, the core pursuit is to control the target vehicle to recover to a stable state. Therefore, only when it is determined that the target vehicle is not in the out-of-control state, the control strategy corresponding to the main strategy network in step 20 is generated.

[0066] Step 20, when it is determined that the target vehicle is not in the out-of-control state, the pre-trained main strategy network is used to process the initial state information to obtain the control strategy output by the main strategy network with the maximum reward value of the reward function as the target.

[0067] For the embodiment of the present application, the reward function includes a speed tracking reward constructed based on the speed item and the control item, a wheel slip penalty constructed based on the speed item, and a stability reward constructed based on the angle item; the control strategy includes the torque ratio of each wheel; the main strategy control network is obtained by reinforcement learning training of the experience samples of multiple driving scenes in the experience pool with the reward value of the reward function as the maximum target value.

[0068] Specifically, before processing the state information by using the main policy network, each parameter needs to be normalized according to a preset rule, so that each parameter is mapped to the interval [-1, 1] after normalization; then the state vector S of the target vehicle is constructed based on each parameter after normalization; wherein S includes 9 dimensions, as shown in formula (1):

[0069] , formula (1);

[0070] wherein, is a longitudinal velocity, is a lateral velocity; is a yaw angle, is a side slip angle; are wheel speeds of four wheels respectively, wherein 1, 2, 3 and 4 are numbers corresponding to four tires respectively; is a road adhesion coefficient, is an acceleration, is a steering angle of a steering wheel.

[0071] The control strategy includes a torque ratio of each wheel, which is represented by an action vector A, as shown in formula (2):

[0072] , formula (2); wherein, ∈(0, 1), and ; is a torque ratio, and the labels 1, 2, 3 and 4 are labels of corresponding wheels; that is, is a torque ratio corresponding to the wheel with label 1.

[0073] Specifically, the training core of the main policy network pursues to maximize the reward value of the reward function, and the reward function is a speed tracking reward constructed based on a speed term and a control term, a wheel slip penalty constructed based on a speed term, and a stability reward constructed based on an angle term. Therefore, the control strategy generated by the main policy network for the state information of the target vehicle can make the target vehicle maintain the related speed state expected by the driver based on the accelerator opening degree, brake opening degree and steering angle as much as possible, on the other hand, it can also make the target vehicle maintain the side slip angle and yaw angle within the range not exceeding the limit value as much as possible, and it can also make the target vehicle avoid the wheel slip state as much as possible.

[0074] The specific training and updating process of the main policy network model will be described in more detail in subsequent embodiments, and will not be described in more detail here.

[0075] Step 30, adjusting the output torque of the driving motor corresponding to each wheel based on the control strategy.

[0076] For the embodiments of the present application, firstly, the total demand torque pursued by the driver is determined based on the opening degree of the accelerator of the target vehicle , and then the target torque required by each wheel is determined in sequence according to the torque proportion in the control strategy.

[0077] Specifically, ; wherein, is the target torque of the wheel numbered , that is, ; that is, After determining the target torque corresponding to each wheel, , the target torque is checked for feasibility based on the maximum allowable output torque of the drive motor of the wheel, and only in the case that , it is determined that the feasibility check is passed, and then the adjustment instruction is generated and sent to the drive motor corresponding to each wheel, so that the drive motor outputs the target torque corresponding to the wheel; wherein, is the maximum allowable output torque of the wheel corresponding to the wheel.

[0078] Further, if the target torque corresponding to at least one wheel is greater than the maximum allowable output torque of the drive motor, the torque proportions corresponding to all wheels are scaled in equal proportion according to a preset proportion, but the proportional relationship between the torque proportions corresponding to each wheel remains unchanged. Then, the new target torque is determined again based on the scaled torque proportions corresponding to each wheel, and the feasibility check is performed again, and then the adjustment instruction (PWM signal) is generated and sent to the drive motor corresponding to each wheel, so that the drive motor outputs the target torque corresponding to the wheel.

[0079] In the above low-attached road four-wheel drive torque distribution method based on reinforcement learning, the policy model is obtained by training a plurality of experience samples of driving scenes in the experience pool by means of reinforcement learning, so that the policy model can adapt to more driving scenes; that is, the policy model has good adaptability to complex road conditions.

[0080] Furthermore, reinforcement learning is an unsupervised training method aimed at maximizing the reward function. This reward function consists of a speed-tracking reward based on speed constraints, a stability reward based on angle constraints, and a slippage penalty based on speed constraints. Therefore, the control strategy generated by the master policy network based on the target vehicle's state information can, on the one hand, enable the target vehicle to maintain the desired speed state based on the driver's throttle opening, brake opening, and steering angle; on the other hand, it can keep the lateral angle and yaw angle within their limits; and on the other hand, it can minimize wheel slippage. Thus, the control strategy generated by the master policy network adjusts the output torque of the drive motors corresponding to each wheel, thereby improving the stability of the target vehicle and keeping it in a stable state as much as possible.

[0081] Furthermore, the advantage of reinforcement learning is that it enables the trained network to learn continuously. In other words, the master policy model can continuously learn from each control policy it makes, thereby better optimizing the decision-making ability of the master policy model. This allows the master policy model to generate better control policy execution when facing complex road conditions, thereby improving the stability of the target vehicle.

[0082] In one embodiment, the runaway control adjustment strategy specifically includes a slippage adjustment strategy and a limit deflection adjustment strategy. In step 10, determining whether the target vehicle is in a runaway state based on the state information includes determining whether the target vehicle is in a slippage state and determining whether the target vehicle is in a limit deflection state.

[0083] The steps to determine whether a target vehicle is in a slippery state may specifically include: for each wheel, determining whether the wheel is in a slippery state based on the wheel speed and the expected wheel speed; and determining that the target vehicle is in a slippery state when any wheel is in a slippery state.

[0084] Specifically, in At that time, determine the tires Slippery; among them, The desired wheel speed is expressed by formula (3):

[0085] Formula (3); where, For vehicle speed, For tires The radius.

[0086] Furthermore, the step of determining whether the target vehicle is in a state of extreme deflection may specifically include: determining... At that time, it is determined that the target vehicle is in a state of extreme deflection; among which, Side slip angle, a safety threshold corresponding to the side slip angle.

[0087] In one of the embodiments, the reward function corresponding to the main policy network includes not only the speed tracking reward, the wheel slip penalty and the stability reward, but also the uniform distribution reward; the composition of the reward function of the main policy network is described in detail as follows.

[0088] Specifically, the reward function is represented by formula (4)-formula (8):

[0089] Formula (4);

[0090] Formula (5);

[0091] Formula (6);

[0092] Formula (7);

[0093] Formula (8);

[0094] wherein, is the reward value of the reward function; is the weight, wherein, is determined according to the road adhesion coefficient ; when the road adhesion coefficient is in a lower first interval, the priorities of (the stability coefficient) and (the safety coefficient) are promoted to prevent slip out of control; when the road adhesion coefficient is in a higher second interval, the (the dynamic coefficient) and (the efficiency coefficient) are enhanced to optimize the performance of the target vehicle.

[0095] The stability reward is represented by formula (9) and formula (10), and the stability reward is limited based on the yaw rate tracking error and the side slip angle.

[0096] Formula (9);

[0097] Formula (10);

[0098] wherein, is the yaw angle, is the expected yaw angle; is the front and rear wheelbase of the target vehicle, is the stability coefficient of the target vehicle, is the steering angle of the steering wheel.

[0099] Speed tracking reward is expressed by equation (11) and equation (12), which is used to encourage tracking of the desired acceleration.

[0100] , equation (11);

[0101] , equation (12);

[0102] wherein, is the maximum acceleration, is the desired acceleration, is the longitudinal acceleration.

[0103] Slip penalty is expressed by equation (13) and equation (14), which is negative when the wheel slips.

[0104] , equation (13);

[0105] , equation (14);

[0106] wherein, in which, is the slip ratio of the wheel, is the number of the wheel; is the slip tolerance threshold value; is the wheel speed of the wheel; is the wheel radius, which is the same for each wheel.

[0107] Uniform distribution reward is expressed by equation (15), which encourages uniform distribution of the torque of each wheel.

[0108] , equation (15); wherein, is the variance function.

[0109] The reward function is the core basis for the master policy network to generate a control policy based on state information (state vector S), that is, to maximize the reward value of the reward function to generate a control policy.

[0110] In one embodiment, with reference to Figure 2 , step 30 adjusts the output torque of the driving motor corresponding to each wheel based on the control policy, which can specifically include steps 31-33, wherein:

[0111] Step 31, determine the road adhesion coefficient, and determine the change rate according to the update frequency of the road adhesion coefficient.

[0112] Specifically, the determination process of the road adhesion coefficient includes two stages, the first stage is a stage of determining an initial adhesion coefficient based on image and radar data; the second stage is a stage of determining a compensation coefficient based on wheel speed deviation, and then the initial adhesion coefficient is compensated according to the compensation coefficient to obtain the final road adhesion coefficient.

[0113] In the first stage, a road image of a road on which the target vehicle travels is acquired, and a first adhesion coefficient is determined based on the road image; a reflection coefficient of the road on which the target vehicle travels is acquired, and a second adhesion coefficient is determined according to the reflection coefficient.

[0114] The road image of the road on which the target vehicle travels is input into the pre-trained adhesion coefficient recognition model, and the adhesion coefficient output by the coefficient recognition model is obtained as the first adhesion coefficient. The data generated by the radar includes the reflection coefficient of the road, and the second adhesion coefficient is determined by using the relationship between the reflection coefficient and the road adhesion coefficient. The first adhesion coefficient and the second adhesion coefficient can be weighted and summed to obtain the initial adhesion coefficient, and the weights corresponding to the first adhesion coefficient and the second adhesion coefficient can be preset.

[0115] In the second stage, an expected wheel speed is determined based on the vehicle speed and the radius of the tire, and a correction coefficient is determined based on the deviation of the wheel speed from the expected wheel speed; an initial adhesion coefficient is determined based on the first adhesion coefficient and the second adhesion coefficient, and the initial adhesion coefficient is corrected based on the correction coefficient to obtain the road adhesion coefficient.

[0116] Specifically, the expected wheel speed has been determined in the aforementioned process of judging the vehicle state, and the wheel speeds of the wheels have also been acquired in the state information, so the wheel speed deviation between the wheel speeds of the wheels and the expected wheel speed is underestimated; the average value of the wheel speed deviations of the four wheels is further determined as an average wheel speed deviation, the correction coefficient corresponding to the average wheel speed deviation is determined based on the corresponding relationship between the wheel speed deviation and the correction coefficient; and then the product of the correction coefficient and the initial adhesion coefficient is taken as the road adhesion coefficient.

[0117] Further, the sampling frequency of the state information can be 100Hz, so the update frequency of the road adhesion coefficient can be greater than or equal to 100Hz. Then the change rate of the road adhesion coefficient is determined according to the update frequency of the road adhesion coefficient.

[0118] Step 32, when the change rate is greater than the fluctuation threshold, adding noise in the control strategy, and adjusting the output torque of the driving motor corresponding to each wheel based on the control strategy after adding noise;

[0119] Step 33, when the change rate is less than or equal to the fluctuation threshold, adjusting the output torque of the driving motor corresponding to each wheel based on the control strategy.

[0120] Specifically, when the rate of change of the road adhesion coefficient is greater than the fluctuation threshold, it is considered that the scene switching is realized, that is, a new driving environment appears; at this time, Gaussian noise is added to the control strategy (action vector A), so that the main policy network can adapt to the new driving environment faster. Each execution of steps 10-30 is a decision-making process, and in the new driving environment, multiple decision-making processes often need to be executed continuously, therefore, the variance of the added Gaussian noise is gradient descent until 0; wherein the proportion of the gradient is not specifically limited in the embodiments of the present application. In addition, Gaussian noise with fixed variance can also be added to the control strategy generated in each decision-making process.

[0121] In one possible implementation, the fluctuation threshold can be 0.02 / S; the variance of the added Gaussian noise is fixed at 0.05.

[0122] In one embodiment, the main policy network M, the target policy network M* corresponding to the main policy network, and the main value network Q and the target value network Q* corresponding to the main value network are further trained.

[0123] Specifically, the main policy network and the target policy network have the same architecture, which is the FCNN (Fully Connected Neural Network) architecture, and the main policy network includes three fully connected neural network layers, and the number of neurons in the three neural network layers is 64, 32 and 4 respectively. Among them, the first and second neural network layers are parameter layers, and the corresponding activation functions are both ReLU loss functions; and the third neural network layer is an output layer, and the loss function is a Softmax loss function.

[0124] In one embodiment, the main policy network, the target policy network, the main value network and the target value network are trained and updated using the DDPG (Deep Deterministic Policy Gradient) algorithm. The following content further describes the relationship between the main policy network, the target policy network, the main value network and the target value network, and the training and updating process between the networks.

[0125] The core pursuit of the main policy network is to maximize the reward value of the reward function, and it generates a corresponding control policy according to the pursuit; after the main policy network generates a control policy, the main value network processes the state information and the corresponding control policy to obtain an evaluation value representing the pros and cons of the control policy, that is, the expected cumulative reward (Q value). The core pursuit of the main value network is how to make the evaluation of the expected cumulative reward more accurate. The parameters of the main value network are also used to update the parameters of the main policy network; in simple terms, the main value network tells the main policy network how to adjust the parameters to change the output control policy, and how to adjust the parameters to maximize the Q value corresponding to the control policy generated for a state information S.

[0126] Further, the update of the target policy network is synchronized with the update of the main policy network, and the update of the target value network is also synchronized with the update of the main value network. After the main policy network completes the update each time, a small part of the updated parameters of the main policy network is used to update the target policy network, which can make the target policy network follow the update of the main policy network, but because it updates fewer parameters; therefore, compared with the main policy network, the target policy network has higher stability.

[0127] Similarly, after the main value network completes the update each time, a small part of the updated parameters of the main value network is used to update the target value network, which can make the target value network follow the update of the main value network, but because it updates fewer parameters; therefore, compared with the main value network, the target value network has higher stability.

[0128] The update process of the main value network and the main policy network is further described in detail below. In the training phase, the main value network and the main policy network are offline trained and do not participate in the interaction with the real environment; and in the update phase, the main value network is configured in the vehicle and interacts with the real driving environment to obtain real data as experience samples, and at this time, the update of the main policy network and the main value network is online update. Further, in the training process of the main policy network, Gaussian noise can also be added to the control policy to enhance the exploratory nature in the early training, and the variance of the Gaussian noise is gradually reduced as the number of training increases, so as to change from exploration to more accurate policy generation.

[0129] It needs to be explained that the training phase of the main value network and the continuous update phase are the same way; therefore, in the embodiments of the present application, only the update steps of the main policy network and the main value network in the application phase are described.

[0130] As steps 10-30 above, each execution of steps 10-30 is a decision-making process, and there will be an initial state information for each decision-making process And a decision-making action and a reward value , where t is the time step corresponding to the decision-making process.

[0131] Furthermore, after each decision-making process, the process specifically includes: acquiring the target vehicle's response state information after executing the control strategy, and determining the reward value of the reward function based on the response state information and the initial state information; constructing an experience sample using the initial state information, the control strategy, the reward value corresponding to the control strategy, and the response state information, and writing the experience sample into the experience pool. The experience pool already stores the experience samples generated by the main policy network during the training phase.

[0132] Specifically, after completing decision process t, the next decision process t+1 begins; at this time, the status of the target vehicle after completing its decision action is acquired in real time. The subsequent response status information is also the initial state information of the target vehicle in the decision-making process t+1. Then, based on the response status information... The initial state information and the reward function determine the reward value of the decision-making process. Furthermore, the status information Decision-making actions Reward Value and response status information Construct as an empirical sample ; and further, this empirical sample Place it into the experience pool.

[0133] Furthermore, the experience samples in the experience pool are used to update the main value network. The main value network is trained using the experience samples in the experience pool to obtain the updated main value network, and the main policy network is updated based on the updated main value network.

[0134] In one embodiment, reference Figure 3 The main value network is trained using experience samples from the experience pool to obtain an updated main value network. This process can specifically include steps A-D, where:

[0135] Step A: Extract a set number of experience samples from the experience pool as training samples for the value network.

[0136] Step B: For each training sample, the main value network is invoked to process the initial state information and control strategy in the training sample to obtain the expected cumulative reward output by the main value network.

[0137] Specifically, refer to Figure 4Each update of the main value network needs to be spaced N decision processes, where N > 1; each update of the main value network only needs to extract part of the sample experience from the experience pool; where the set number can be set in advance, for example, it can be 32. And the main value network is updated by randomly extracting a set number of experience samples from the experience pool, which can avoid the correlation between experience samples and reduce the probability of overfitting of the main value network.

[0138] Further, after each decision process is executed, the main value network evaluates the control strategy generated by the main decision network in the decision process to obtain an expected cumulative reward, which is also called a Q value, which represents the pros and cons of the control strategy generated by the main decision network in the decision process. The main value network takes the state information in the decision process corresponding to the time step t and the action information as input, and outputs the evaluation of the decision process t, i.e. the expected cumulative reward . .

[0139] Step C, using the target policy network to process the response state information in the training sample to obtain a stable policy output by the target policy network to maximize the reward value of the reward function, and using the target value network to process the initial response state information and the stable policy to obtain a stable cumulative reward;

[0140] Step D, based on the value loss function, the main value network is trained to minimize the value loss to obtain an updated main value network; the value loss function is constructed based on the reward value in the training sample and the stable cumulative reward and the expected cumulative reward corresponding to the training sample.

[0141] Specifically, referring to Figure 4 , since the control strategy generated by the target policy network is more stable than the control strategy generated by the main policy network, in the process of updating the main value network, a relatively stable decision process is performed using the target policy network, i.e. using the target policy model to process the response state information , so as to obtain a stable policy ; and then using the target value network to evaluate this stable decision process, i.e. taking the response state information and the stable policy as input of the network, obtaining the stable cumulative reward output by the target policy network.

[0142] Specifically, the value loss function can be represented by formula (16):

[0143] , formula (16);

[0144] ​Wherein, L is a loss value, N is the number of experience samples participating in training; is the expected cumulative reward of the decision-making process corresponding to time step t, generated by the main decision-making network ; is the control policy corresponding to the reward value obtained from the experience sample; is the discount factor; is the target value network generated expected cumulative reward (stable cumulative reward) of the decision-making process corresponding to time step t+1.

[0145] In another embodiment, the process of training the main value network using the experience samples in the experience pool to obtain the updated main value network, the above step C and step D have another implementation, and another possible implementation is further described, that is, after step A and step B, steps C1 and D1 can also be used, wherein:

[0146] Step C1, using the target policy network to process the initial state information in the training sample to obtain the control policy output by the target policy network to maximize the reward value of the reward function, and using the target value network to process the initial state information and the control policy to obtain the stable cumulative reward;

[0147] Step D1, training the main value network based on the value loss function to minimize the value loss to obtain the updated main value network; the value loss function is constructed based on the reward value in the training sample and the stable cumulative reward and the expected cumulative reward corresponding to the training sample.

[0148] Specifically, since the control policy generated by the target policy network is more stable than the control policy generated by the main policy network, in the process of updating the main value network, the target policy network is used to perform a relatively stable decision-making process, that is, for each experience sample, the target policy model is used to process the initial state information in the experience sample, so as to obtain a stable control policy ; and then the target value network is used to evaluate this stable decision-making process, that is, the initial state information and the stable control policy are used as the input of the network to obtain the stable cumulative reward output by the target policy network.

[0149] Specifically, the value loss function can be represented by formula (17):

[0150] , formula (17);

[0151] wherein, is the loss value, N is the number of experience samples participating in training; is the expected cumulative reward of the decision-making process corresponding to time step t, generated by the main decision-making network; is the control policy is the corresponding reward value, obtained from the experience sample; is the discount factor; is the stable control policy generated by the target value network for the target policy network for the decision-making process of time step t generated by the target value network is the expected cumulative reward (stable cumulative reward) of the decision-making process.

[0152] By using the value loss function and the extracted N experience samples to train the main value network, and according to the value of the loss function, the parameters of the main value network are updated using the gradient descent algorithm (such as Adam). After the main value network is trained, the parameters of the main value network are updated, thereby improving the accuracy of the main value network in evaluating the pros and cons of the decision-making process.

[0153] In one embodiment, after the parameters of the main value network are updated, the parameters of the main policy network are updated using the updated parameters of the main value network based on the gradient update strategy, to obtain the updated main policy network.

[0154] Specifically, the strategy for updating the main policy network is the gradient ascent strategy, and the parameters of the main policy network are updated using the updated parameters of the main value network . The purpose of updating the main policy network is to adjust the parameters of the main policy network θ^M to maximize the Q value corresponding to the control policy output by the main policy network.

[0155] Specifically, the process of deterministic policy gradient can be represented by formula (18):

[0156] , formula (18);

[0157] wherein, is the update gradient; characterizes how the main policy network fine-tunes the parameters to change the actuator output action; characterizes how to fine-tune the generated control policy A under the current state information S to make the Q value larger. The dot product is that the main policy network adjusts itself according to the "guiding direction" provided by the main value network, the purpose being to make the control policy output by the main policy network be able to obtain a higher Q value in the main value network.

[0158] Further, the calculated policy gradient The parameters applied to the main policy network are updated, which can be represented by formula (19):

[0159] , formula (19); wherein, is the learning rate.

[0160] Further, the target policy network is updated when the parameters of the main policy network are updated, and the target value network is updated when the parameters of the main value network are updated.

[0161] The updating process of the target value network includes: updating the parameters of the target value network based on the updated parameters of the main value network and the first penetration ratio; wherein the first penetration ratio is the ratio of the respective use ratios of the updated parameters of the main value network and the current parameters of the target value network.

[0162] Specifically, the parameters of the target value network are ; wherein, is the parameter of the target value network, is the parameter (updated) of the main value network; A1 is the use ratio of the parameter of the main value network, is the use ratio of the target policy network; , and .

[0163] The updating process of the target policy network includes: updating the parameters of the target policy network based on the updated parameters of the main policy network and the second penetration ratio; wherein the second penetration ratio is the ratio of the respective use ratios of the updated parameters of the main policy network and the current parameters of the target policy network.

[0164] Specifically, the parameters of the target policy network are ; wherein, is the parameter of the target policy network, is the parameter (updated) of the main policy network; B1 is the use ratio of the parameter of the main policy network, is the use ratio of the target policy network; , and . Wherein, Generally between 0.001-0.005.

[0165] As the above content, the parameter updating process of the main strategy network, the main value network, the target strategy network and the target value network is completed. The control strategy generated by the main strategy network in the actual driving scene according to the real-time state information of the target vehicle can be constructed as an experience sample and put into the experience pool; thus, the experience sample in the experience pool is used to continuously update the parameters of the main value network and the main strategy network. In this way, the judgment ability of the main value network and the reliability and accuracy of the control strategy generated by the main strategy network are further improved.

[0166] In one of the embodiments, the method provided by the embodiments of the present application further includes a loss-of-control adjustment strategy when the target vehicle is in a loss-of-control state, and the loss-of-control adjustment strategy specifically includes a slip adjustment strategy and a limit deflection adjustment strategy; therefore, before determining which adjustment strategy to execute, the loss-of-control state of the target vehicle is determined.

[0167] The slip adjustment strategy specifically includes: when the target vehicle is in a slip state, determining a loss-of-control wheel in the slip state based on the parameters of the speed term, and determining a safe torque interval corresponding to the loss-of-control wheel; taking the safe torque interval corresponding to the loss-of-control wheel as a reference, generating a first control instruction for each wheel, and executing the first control instruction; the first control instruction is used to adjust the output torque of the driving motor corresponding to each wheel to the safe torque interval corresponding to the loss-of-control wheel.

[0168] The limit deflection adjustment strategy specifically includes: when the target vehicle is in a limit deflection state, determining a safe torque interval corresponding to each wheel; taking the safe torque interval corresponding to each wheel as a reference, generating a second control instruction for each wheel, and executing the second control instruction; the second control instruction is used to adjust the output torque of the driving motor corresponding to each wheel to the safe torque interval corresponding to the wheel.

[0169] Specifically, the safe torque interval corresponding to each wheel can be pre-set, and each wheel can independently correspond to a safe torque interval, or each wheel can correspond to a safe torque interval; for this, no specific limitation is made in the embodiments of the present application. By limiting each wheel in the safe torque interval corresponding to the wheel in space and time, it is possible to both suppress slip and meet the basic power demand to maintain normal driving of the target vehicle.

[0170] Since the safe torque interval of each wheel is definitely smaller than the maximum allowable output torque of the driving motor corresponding to the wheel with respect to the target torque, before generating the first control instruction and the second control instruction, no feasibility check is needed, and the instructions can be directly generated and executed.

[0171] In summary, the torque strategy control method based on reinforcement learning provided by the present application has a specific process in the application stage as shown in Figure 5 the figure, which includes:

[0172] The system starts up, loads the parameters of the offline-trained main policy network, and initializes the experience pool (capacity). strip);

[0173] State acquisition: The sensor acquires and processes the state information to obtain a 9-dimensional state vector;

[0174] Action decision: The master strategy network outputs a control strategy that includes torque distribution ratio based on the state information;

[0175] Torque execution: Calculate the control strategy and verify its feasibility, then convert it into a control signal (PWM signal) to drive the motor.

[0176] Feedback learning: Collect response status information after each decision-making process, calculate the reward value, and then store the relevant parameters of the decision-making process into the experience pool;

[0177] Parameter Update (Value Network): 32 experience samples are randomly sampled from the experience pool, and the main value network is trained using the value loss function to update its parameters. Then, using the updated parameters of the main value network, the parameters of the main policy network are updated using a gradient ascent optimization strategy. The main policy network updates the target policy network using a soft update strategy, and the main value network also updates the target value network using a soft update strategy.

[0178] Dynamic adjustment: Check whether the target vehicle has triggered a scene switch (based on the rate of change of the road surface adhesion coefficient) or is in an out-of-control state. If it is in an out-of-control state, execute the out-of-control adjustment strategy.

[0179] In one embodiment, the reinforcement learning-based torque allocation strategy provided in this embodiment is verified on the CarSim-Simulink platform and compared with the uncontrolled state and the ideal control state. The verification scenario is set as an icy or snowy road surface with a certain road adhesion coefficient. ,result Figure 6 As shown.

[0180] It should be understood that although each step in the flowchart involved in the above embodiments is shown in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.

[0181] Based on the same inventive concept, the embodiments of the present application also provide a torque distribution device for implementing the above-mentioned low- adhesion road four-wheel drive torque distribution method based on reinforcement learning. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more torque distribution device embodiments provided below can refer to the limitations of the low-adhesion road four-wheel drive torque distribution method based on reinforcement learning in the above text, which will not be repeated here.

[0182] In one exemplary embodiment, as shown in Figure 7 A torque distribution device 700 is provided, the device comprising a state information acquisition module 701, a policy generation module 702, and an adjustment module 703, wherein:

[0183] The state information acquisition module 701 is configured to acquire initial state information of the target vehicle, the state information comprising parameters of control items, parameters of speed items, and parameters of angle items, and determine whether the target vehicle is in an out-of-control state based on the state information.

[0184] The policy generation module 702 is configured to, when it is determined that the target vehicle is not in an out-of-control state, process the initial state information using a pre-trained master policy network to obtain a control policy output by the master policy network with the goal of maximizing a reward value of a reward function; wherein the reward function comprises a speed tracking reward constructed based on the speed items and the control items, a wheel slip penalty constructed based on the speed items, and a stability reward constructed based on the angle items; the control policy comprises torque ratios corresponding to the wheels; and the master policy control network is obtained by reinforcement learning training of experience samples in an experience pool of multiple driving scenarios with the goal of maximizing the reward value of the reward function.

[0185] The adjusting module 703 is configured to adjust the output torque of the driving motor corresponding to each wheel based on the control strategy.

[0186] In one of the embodiments, the torque distribution device further comprises an updating module, which is specifically configured to:

[0187] obtain response state information of the target vehicle after the control strategy is executed, and determine a reward value of the reward function based on the response state information and the initial state information;

[0188] construct an experience sample based on the initial state information, the control strategy, the reward value corresponding to the control strategy, and the response state information, and write the experience sample into an experience pool;

[0189] train the main value network by using the experience sample in the experience pool to obtain an updated main value network;

[0190] update the main policy network based on the updated main value network.

[0191] In one of the embodiments, the updating module is specifically configured to:

[0192] extract a set number of experience samples from the experience pool as training samples of the value network;

[0193] for each training sample, call the main value network to process the initial state information and the control strategy in the training sample to obtain an expected cumulative reward output by the main value network;

[0194] process the response state information in the training sample by using the target policy network to obtain a stable strategy output by the target policy network with the objective of maximizing the reward value of the reward function, and process the initial response state information and the stable strategy by using the target value network to obtain a stable cumulative reward; the target policy network has the same network structure as the main policy network; and the target value network has the same network structure as the main value network;

[0195] train the main value network based on a value loss function with the objective of minimizing the value loss to obtain an updated main value network; the value loss function is constructed based on the reward value in the training sample and the stable cumulative reward and the expected cumulative reward corresponding to the training sample.

[0196] In one of the embodiments, the updating module is specifically configured to:

[0197] update the parameters of the main policy network by using the updated parameters of the main value network based on a gradient updating strategy to obtain an updated main policy network.

[0198] In one of the embodiments, the updating module is specifically configured to:

[0199] update the parameters of the target value network based on the updated parameters of the main value network and a first penetration ratio, wherein the first penetration ratio is a ratio of respective proportions of the updated parameters of the main value network and the current parameters of the target value network;

[0200] update the parameters of the target policy network based on the updated parameters of the main policy network and a second penetration ratio of the target policy network by the main policy network, wherein the second penetration ratio is a ratio of respective proportions of the updated parameters of the main policy network and the current parameters of the target policy network.

[0201] In one of the embodiments, the torque distribution device further comprises an adhesion coefficient determination module, wherein:

[0202] The adhesion coefficient determination module is specifically configured to determine the road adhesion coefficient, and determine the change rate according to the update frequency of the road adhesion coefficient.

[0203] The adjustment module 703 is specifically configured to add noise in the control strategy when the change rate is greater than the fluctuation threshold, and adjust the output torque of the driving motor corresponding to each wheel based on the control strategy after adding the noise.

[0204] The adjustment module 703 is further configured to adjust the output torque of the driving motor corresponding to each wheel based on the control strategy when the change rate is less than or equal to the fluctuation threshold.

[0205] In one of the embodiments, the adhesion coefficient determination module is specifically configured to:

[0206] Obtain a road image of a road on which the target vehicle travels, and determine a first adhesion coefficient based on the road image.

[0207] Obtain a reflection coefficient of the road on which the target vehicle travels, and determine a second adhesion coefficient based on the reflection coefficient.

[0208] Determine an expected wheel speed based on the vehicle speed and the radius of the tire, and determine a correction coefficient based on the deviation between the expected wheel speed and the wheel speed.

[0209] Determine an initial adhesion coefficient based on the first adhesion coefficient and the second adhesion coefficient, and correct the initial adhesion coefficient based on the correction coefficient to obtain the road adhesion coefficient.

[0210] In one of the embodiments, the out-of-control state includes a slipping state and a limit deflection state, and the torque distribution device further comprises an out-of-control adjustment module, which is specifically configured to:

[0211] When the target vehicle is in the slipping state, determine an out-of-control wheel in the slipping state based on the parameters of the speed term, and determine a safe torque interval corresponding to the out-of-control wheel.

[0212] generating a first control instruction for each wheel based on the safe torque range corresponding to the out-of-control wheel, and executing the first control instruction; the first control instruction is used to adjust the output torque of the drive motor corresponding to each wheel to the safe torque range corresponding to the out-of-control wheel;

[0213] determining the safe torque range corresponding to each wheel when the target vehicle is in the limit deflection state;

[0214] generating a second control instruction for each wheel based on the safe torque range corresponding to each wheel, and executing the second control instruction; the second control instruction is used to adjust the output torque of the drive motor corresponding to each wheel to the safe torque range corresponding to the wheel.

[0215] Each module in the torque distribution device described above can be realized by software, hardware and their combination in whole or in part. Each module described above can be embedded in or independent of the processor in the vehicle in hardware form, or can be stored in the memory in the vehicle in software form, so as to be called and executed by the processor to perform the operation corresponding to each module.

[0216] In an exemplary embodiment, a vehicle is provided, and a block diagram of the internal structure of the vehicle can be as shown in Figure 8 The vehicle includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the vehicle is used to provide computing and control capability. The memory of the vehicle includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the vehicle is used to exchange information between the processor and external devices. The communication interface of the vehicle is used for wired or wireless communication with external terminals, and wireless communication can be realized through WIFI, mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. The computer program is executed by the processor to realize a low adhesion road four-wheel drive torque distribution method based on reinforcement learning. The display unit of the vehicle is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the vehicle can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the vehicle shell, or an external keyboard, touchpad or mouse, etc.

[0217] Further, the vehicle further comprises a PWM controller and a sensor module, the PWM controller being configured to generate adjustment instructions for the driving motor corresponding to each wheel in response to the control strategy. The PWM controller and the sensor module are both connected to the system bus through the input / output interface.

[0218] The sensor module specifically comprises:

[0219] A wheel speed sensor is installed at the hub of each wheel to collect the real-time rotational speed of each wheel with a sampling frequency of 100 Hz and a measurement error of less than or equal to ±1 rpm.

[0220] An IMU (Inertial Measurement Unit) is installed at the center of mass of the vehicle to collect the longitudinal acceleration , lateral acceleration , and yaw rate of the vehicle, with a measurement range of ±500° / s and an accuracy of ±0.1° / s.

[0221] A millimeter wave radar and a camera are installed at the front bumper to jointly identify the road surface type (ice / snow / water / mud) and estimate the road surface adhesion coefficient with an error of less than or equal to ±0.05.

[0222] A pedal position sensor is configured to collect the opening degree of the accelerator pedal (0~100%) for calculating the total demand torque.

[0223] Those skilled in the art can understand that Figure 8 the structure shown in the above description is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the vehicle to which the scheme of the present application is applied. The specific vehicle can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0224] In one exemplary embodiment, a vehicle is provided, comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of any one of the above embodiments of the low-adhesion road surface four-wheel drive torque distribution method based on reinforcement learning.

[0225] In one embodiment, a computer-readable storage medium is provided, storing a computer program, and the computer program is executed by a processor to implement the steps of any one of the above embodiments of the low-adhesion road surface four-wheel drive torque distribution method based on reinforcement learning.

[0226] In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of any of the above embodiments of the method for low-mu road four-wheel drive torque distribution based on reinforcement learning.

[0227] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0228] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the processes of the above-mentioned embodiment methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., and is not limited thereto. The processor involved in the embodiments provided in the present application can be a general processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., and is not limited thereto.

[0229] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, it should be considered as the scope of the present application.

[0230] The above embodiments only express several implementation ways of the present application, and the description is specific and detailed, but it should not be understood as a limitation to the patent scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A low adhesion road surface four-wheel drive torque distribution method based on reinforcement learning, characterized by, The method comprises: acquiring initial state information of a target vehicle, the state information comprising parameters of a control item, parameters of a speed item, and parameters of an angle item, and determining whether the target vehicle is in an out-of-control state based on the state information; the speed item comprising a vehicle speed and a wheel speed of a tire of the target vehicle; when it is determined that the target vehicle is not in the out-of-control state, processing the initial state information by using a pre-trained master policy network to obtain a control policy output by the master policy network with a maximum reward value of a reward function as a target; wherein the reward function comprises a speed tracking reward constructed based on the speed item and the control item, a wheel slip penalty constructed based on the speed item, and a stability reward constructed based on the angle item; the control policy comprises a torque ratio corresponding to each wheel; the master policy network is obtained by reinforcement learning training of experience samples of multiple driving scenes in an experience pool with the maximum reward value of the reward function as a target value; acquiring a road image of a road on which the target vehicle travels, and determining a first adhesion coefficient based on the road image; acquiring a reflection coefficient of the road on which the target vehicle travels, and determining a second adhesion coefficient according to the reflection coefficient; determining an expected wheel speed based on the vehicle speed and a radius of the tire, and determining a correction coefficient based on a deviation between the expected wheel speed and the wheel speed; determining an initial adhesion coefficient based on the first adhesion coefficient and the second adhesion coefficient, and correcting the initial adhesion coefficient based on the correction coefficient to obtain a road adhesion coefficient; determining a change rate according to an update frequency of the road adhesion coefficient; when the change rate is greater than a fluctuation threshold, adding noise in the control policy, and adjusting output torques of driving motors corresponding to each wheel based on the control policy with the noise added; when the change rate is less than or equal to the fluctuation threshold, adjusting the output torques of the driving motors corresponding to each wheel based on the control policy.

2. The method of claim 1, wherein, The method further comprises: acquiring response state information of the target vehicle after the control policy is executed, and determining a reward value of the reward function based on the response state information and the initial state information; constructing an experience sample from the initial state information, the control policy, the reward value corresponding to the control policy, and the response state information, and writing the experience sample into the experience pool; training a master value network by using the experience samples in the experience pool to obtain an updated master value network; updating the master policy network based on the updated master value network.

3. The method of claim 2, wherein, The training of the master value network by using the experience samples in the experience pool to obtain the updated master value network comprises: extracting a set number of the experience samples from the experience pool as training samples of the master value network; for each training sample, calling the master value network to process the initial state information and the control policy in the training sample to obtain an expected cumulative reward output by the master value network; The target policy network is used to process the response state information in the training sample, to obtain a stable policy output by the target policy network with the maximum reward value of the reward function as the target, and the target value network is used to process the initial response state information and the stable policy, to obtain a stable cumulative reward; the target policy network has the same network structure as the main policy network; the target value network has the same network structure as the main value network; The main value network is trained based on a value loss function with the minimum value loss as the target, to obtain an updated main value network; the value loss function is constructed based on the reward value in the training sample and the stable cumulative reward and the expected cumulative reward corresponding to the training sample.

4. The method of claim 3, wherein, The main policy network is updated based on the updated main value network, including: The parameters of the main policy network are updated based on the updated parameters of the main value network, to obtain an updated main policy network.

5. The method according to claim 3 or 4, characterized in that, The method further includes: The parameters of the target value network are updated based on the updated parameters of the main value network and a first penetration ratio; the first penetration ratio is the ratio of the updated parameters of the main value network to the current parameters of the target value network; The parameters of the target policy network are updated based on the updated parameters of the main policy network and a second penetration ratio of the target policy network to the main policy network; the second penetration ratio is the ratio of the updated parameters of the main policy network to the current parameters of the target policy network.

6. The method of claim 1, wherein, The out-of-control state includes a slipping state and a limit deflection state, and the method further includes: When the target vehicle is in the slipping state, the out-of-control wheel in the slipping state is determined based on the parameters of the speed term, and a safe torque interval corresponding to the out-of-control wheel is determined; A first control instruction for each wheel is generated based on the safe torque interval corresponding to the out-of-control wheel, and the first control instruction is executed; the first control instruction is used to adjust the output torque of the driving motor corresponding to each wheel to the safe torque interval corresponding to the out-of-control wheel; When the target vehicle is in the limit deflection state, a safe torque interval corresponding to each wheel is determined; A second control instruction for each wheel is generated based on the safe torque interval corresponding to each wheel, and the second control instruction is executed; the second control instruction is used to adjust the output torque of the driving motor corresponding to each wheel to the safe torque interval corresponding to the wheel.

7. A torque distribution device characterized by, The device includes a state information acquisition module, a policy generation module, an adhesion coefficient determination module, and an adjustment module, wherein: The state information acquisition module is configured to acquire initial state information of the target vehicle, the state information including parameters of a control term, parameters of a speed term, and parameters of an angle term, and determine whether the target vehicle is in an out-of-control state based on the state information; the speed term including a vehicle speed of the target vehicle and a wheel speed of a tire; The policy generation module is configured to, when it is determined that the target vehicle is not in the out-of-control state, process the initial state information by using a pre-trained master policy network, to obtain a control policy output by the master policy network with a maximum reward value of a reward function as a target; the reward function including a speed tracking reward constructed based on the speed term and the control term, a tire slip penalty constructed based on the speed term, and a stability reward constructed based on the angle term; the control policy including a torque ratio corresponding to each wheel; and the master policy network being obtained by reinforcement learning training of experience samples of multiple driving scenes in an experience pool with the maximum reward value of the reward function as a target. The adhesion coefficient determination module is configured to acquire a road image of a road on which the target vehicle travels, determine a first adhesion coefficient based on the road image; acquire a reflection coefficient of the road on which the target vehicle travels, and determine a second adhesion coefficient according to the reflection coefficient; determine an expected wheel speed based on the vehicle speed and a radius of a tire, and determine a correction coefficient based on a deviation between the expected wheel speed and the wheel speed; determine an initial adhesion coefficient based on the first adhesion coefficient and the second adhesion coefficient, and correct the initial adhesion coefficient based on the correction coefficient to obtain a road adhesion coefficient; and determine a change rate according to an update frequency of the road adhesion coefficient. The adjustment module is configured to, when the change rate is greater than a fluctuation threshold, add noise to the control policy, and adjust output torques of driving motors corresponding to each wheel based on the control policy with the noise added; and when the change rate is less than or equal to the fluctuation threshold, adjust the output torques of the driving motors corresponding to each wheel based on the control policy.

8. The apparatus of claim 7, wherein, The device further includes an update module, which is specifically configured to: acquire response state information of the target vehicle after the control policy is executed, and determine a reward value of the reward function based on the response state information and the initial state information; construct an experience sample from the initial state information, the control policy, the reward value corresponding to the control policy, and the response state information, and write the experience sample into the experience pool; train a master value network by using the experience samples in the experience pool, to obtain an updated master value network; update the master policy network based on the updated master value network.

9. The apparatus of claim 8, wherein, The update module is specifically configured to: extract a set number of the experience samples from the experience pool as training samples of the master value network; for each training sample, call the master value network to process the initial state information and the control policy in the training sample, to obtain an expected cumulative reward output by the master value network. The target policy network is used to process the response state information in the training sample, to obtain a stable policy output by the target policy network with the maximum reward value of the reward function as a target, and the target value network is used to process the initial response state information and the stable policy, to obtain a stable cumulative reward; the target policy network has the same network structure as the main policy network; and the target value network has the same network structure as the main value network. The main value network is trained based on a value loss function with the minimum value loss as a target, to obtain an updated main value network; the value loss function is constructed based on the reward value in the training sample, the stable cumulative reward corresponding to the training sample, and the expected cumulative reward.

10. A vehicle comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor implements the steps of the method in any one of claims 1 to 6 when executing the computer program.

Images