Unmanned tracked vehicle safety decision self-learning and control method based on data mechanism hybrid enhancement

Abstract

The invention relates to the technical field of unmanned tracked vehicle decision and control, in particular to a data mechanism hybrid enhanced unmanned tracked vehicle safety decision self-learning and control method. According to the method, a mechanism-driven crawler motion control multi-constraint optimization mechanism is constructed by utilizing model predictive control, and the safe trajectory tracking capability is realized; according to the unmanned tracked vehicle decision-making self-learning and control method based on data mechanism hybrid enhancement, the powerful self-learning ability of reinforcement learning and the constraint processing advantage of model prediction control are considered in the method, decision making and trajectory tracking are conducted through the reinforcement learning and the model prediction control, and the safe self-learning ability is achieved; the basic safety effect of trajectory tracking on vehicle movement is considered, a self-learning decision-making mechanism of trajectory tracking capability boundary constraint is constructed, and optimization of the whole system performance is achieved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of unmanned tracked vehicle decision and control, in particular to a data-mechanism hybrid enhanced unmanned tracked vehicle safe decision self-learning and control method. BACKGROUND

[0002] The unmanned tracked vehicle is a typical safety-critical system, which not only needs to have high adaptability and self-learning ability to cope with dynamic and open traffic environment, but also needs to ensure the safety of the learning process. On the one hand, from the perspective of system development, the strategy trained in the virtual environment needs to meet the safety requirements in the real deployment application stage. On the other hand, the unmanned tracked vehicle needs to have the ability of autonomous safe exploration and learning in the real world, so as to optimize the driving strategy autonomously. As the most typical self-learning technology, deep reinforcement learning has the advantages of no model, no supervision and autonomous learning, and can guide the interaction between the agent and the environment, and autonomously learn to continuously optimize the decision through the pre-defined reward and optimization target. It is considered to be the most effective method to improve the intelligence of unmanned tracked vehicles in dynamic and open environment, and is very suitable for learning the decision behavior in complex scenarios which is difficult to model by mechanism.

[0003] However, the "black box" characteristics of deep learning and the random trial-and-error mechanism of reinforcement learning seriously affect the safety and rationality of strategy exploration and utilization in the learning process, making it difficult to meet the self-learning needs of unmanned tracked vehicles. In addition, most existing methods ignore the basic safety role of trajectory tracking on vehicle motion, and the ability boundary of trajectory tracking is usually affected by constraints such as vehicle kinematics and dynamics. Ignoring this factor will lead to the decision trajectory being too aggressive to be well tracked, or too conservative to waste the ability of trajectory tracking, resulting in the inability to achieve the optimal performance of the entire decision control system.

[0004] Therefore, how to realize safe and reasonable decision self-learning and control of unmanned tracked vehicles considering the ability boundary of trajectory tracking is also an important challenge. SUMMARY

[0005] The present application aims to solve the problem of realizing safe and reasonable decision self-learning and control of unmanned tracked vehicles considering the ability boundary of trajectory tracking, and proposes a data-mechanism hybrid enhanced unmanned tracked vehicle safe decision self-learning and control method.

[0006] The present application provides a data-mechanism hybrid enhanced unmanned tracked vehicle safe decision self-learning and control method, which comprises:

[0007] S1: Constructing a vehicle decision self-learning and control environment, providing a self-vehicle sensor model, a self-vehicle motion physical model, and an environment participant motion physical model for self-learning algorithm training;

[0008] S2: acquiring motion states of the ego vehicle and the environment participants at the current time based on an ego vehicle sensor model, an ego vehicle motion physical model, and an environment participant motion physical model;

[0009] S3: constructing a decision instruction set, a self-learning decision strategy action value network, and a decision action random exploration mechanism, and calculating an expected decision instruction of the ego vehicle;

[0010] S4: mapping the expected decision instruction of the ego vehicle to an expected trajectory of the ego vehicle;

[0011] S5: constructing a mechanism-driven motion control multi-constraint optimization problem using model predictive control to track the expected trajectory of the ego vehicle;

[0012] S6: constructing a decision strategy trial and error and deviation correction mechanism based on the mechanism-driven motion control multi-constraint optimization problem, and calculating a safe expected decision instruction and a safe expected motion control instruction of the ego vehicle;

[0013] S7: constructing an ego vehicle motion bottom layer controller to track the safe expected motion control instruction, and calculating a safe bottom layer control instruction of the ego vehicle;

[0014] S8: controlling the motion of the ego vehicle based on the safe bottom layer control instruction of the ego vehicle, and realizing single-step state transition to acquire motion states of the ego vehicle and the environment participants at the next time;

[0015] S9: constructing a self-learning state space, an action space, and a reward function;

[0016] S10: collecting training data based on the self-learning state space, the action space, and the reward function, and constructing an experience pool;

[0017] S11: constructing a data-driven decision self-learning algorithm using reinforcement learning based on the experience pool, and realizing the update of the decision strategy network.

[0018] Further, the acquiring of the motion states of the ego vehicle and the environment participants at the current time based on the ego vehicle sensor model, the ego vehicle motion physical model, and the environment participant motion physical model comprises:

[0019] acquiring the motion states of the ego vehicle and the environment participants at the current time, including: a longitudinal velocity and a lateral velocity of a vehicle center of mass in a vehicle body coordinate system , a yaw angular velocity of the vehicle at the center of mass, a vehicle heading angle , a longitudinal acceleration of the vehicle center of mass in the vehicle body coordinate system, and a yaw angular acceleration ; relative longitudinal distance between the vehicle and the obstacle in the vehicle body coordinate system ; relative lateral distance ; relative longitudinal velocity ; relative lateral velocity ; relative longitudinal distance between the vehicle and the end point in the vehicle body coordinate system ; relative lateral distance ; relative longitudinal velocity ; relative lateral velocity .

[0020] Further, the decision instruction set, the self-learning decision strategy action value network, and the decision action random exploration mechanism are constructed, and the expected decision instruction of the ego vehicle is calculated, including:

[0021] S3-1: Constructing a decision instruction space, including equal-interval velocity decision instructions and equal-interval angle decision instructions, wherein the velocity decision instruction ranges from 0 to a velocity upper limit value , and the angle decision instruction ranges from an angle lower limit value to an angle upper limit value ;

[0022] Each type of instruction is numbered from 1 in ascending order, and all numbers of the two types of instructions are combined to form a column vector set, which is used as the decision instruction set;

[0023] S3-2: Based on the decision instruction set, constructing a decision strategy action value network as a fully connected network, wherein , , , are the number of neurons of the 1st, 2nd, 3rd, and 4th neural networks respectively, and is the sum of the number of motion states of the ego vehicle and the environmental participants at the current time in step S2-1 and the dimension of any column vector in the decision instruction set, the activation function is the hyperbolic tangent function, and the network parameters are randomly initialized;

[0024] S3-3: Based on the decision strategy action value network, constructing a decision action random exploration mechanism: first, take a random real number between 0 and 1, if the random real number is greater than a set threshold , then randomly select a vector from the decision instruction set as the expected decision instruction, otherwise, the motion states of the ego vehicle and the environmental participants at the current time in step S2-1 and each vector in the decision instruction set are integrated to form a single column vector, and are respectively input into the decision strategy action value network to calculate the output value, i.e., the action value;

[0025] The decision instruction corresponding to the maximum action value is selected as the expected decision instruction of the vehicle.

[0026] Furthermore, the desired decision instruction of the ego vehicle is mapped to the desired trajectory of the ego vehicle, including:

[0027] S4-1: Based on the desired decision instruction of the vehicle, first the corresponding angle decision instruction Take it out and take the center of mass of the vehicle as the origin and the body coordinate system The axis is used as the reference, and the desired path of the vehicle is constructed according to the angle decision instruction, and then the corresponding speed decision instruction is used. Take it out and take the center of mass of the vehicle as the origin along the body coordinate system Axis forward direction The size of the equal spacing is constructed normals, where , is the total number of prediction time steps, To predict the time step, the coordinates of the intersection of each normal line and the desired path in the vehicle body coordinate system and the heading angle of the desired path are calculated and constructed as trajectory points ,in For the The longitudinal position of the desired trajectory point, For the The lateral position of the desired trajectory point, For the The heading angle of the desired trajectory point, and ,Finally, all trajectory points are constructed in sequence as the desired trajectory of the vehicle.

[0028] Furthermore, model predictive control is used to construct a mechanism-driven motion control multi-constraint optimization problem to track the desired trajectory of the ego vehicle, including:

[0029] S5-1: Construct the longitudinal and lateral dynamics and kinematics differential model of the vehicle:

[0030] ;

[0031] in, 、 are the longitudinal and lateral positions of the vehicle's center of mass in the body coordinate system, respectively; 、 、 、 、 、 They are 、 、 、 、 、 The first derivative of ; is the vehicle mass; is the vehicle lateral force, expressed as:

[0032] ;

[0033] in, is the lateral sliding friction coefficient between the track and the ground; is the acceleration due to gravity; tanh is the hyperbolic tangent function; is a set real number; It is the longitudinal ground contact length of the track.

[0034] S5-2: Define model state variables, output variables and control variables:

[0035] ;

[0036] in, 、 、 are state variables, output variables, and control variables respectively; T is the transposition symbol;

[0037] S5-3: Constructing a discrete time prediction model:

[0038] ;

[0039] in, is the index of the prediction time step; for State variables at the moment; 、 They are State variables and output variables at the moment; for Control variables at time; is the state-output transition matrix, defined as follows:

[0040] ;

[0041] also, 、 、 for The three matrices corresponding to the prediction model system at the moment are obtained by linearizing the longitudinal and lateral dynamics and kinematic differential models of the ego-vehicle constructed in step S5-1 at the reference system working point:

[0042] ;

[0043] in, is the identity matrix; Indicates the construction of the differential model in step S5-1; is the partial differential symbol; 、 for The state vector and control vector of the working point of the reference system are constructed using the control variables optimized at the previous moment and the corresponding state prediction trajectory;

[0044] S5-4: Constructing a multi-constraint model predictive control optimization problem:

[0045] ;

[0046] Among them, min is the minimum symbol; for Optimization goals at all times; for Output variables at time; is the expected trajectory point of the vehicle; To control the total number of time steps; is the weight matrix for tracking the desired reference signal; 、 are the weight matrices for suppressing the amplitude and rate of change of the control variable respectively; is the constraint relaxation factor; To suppress the weight coefficient of excessive relaxation factor; for The first control variable increment, i.e. control variables Hedi control variables The difference is defined as:

[0047] ;

[0048] for arrive Corresponding The control variable increment sequence is defined as follows:

[0049] ;

[0050] also, 、 They are The upper and lower limits of the state variables at each moment; 、 They are Output variable upper and lower limits at all times; 、 They are Control the upper and lower limits of variables at all times; 、 They are Control the upper and lower limits of variable increments at all times; is an upper bound of the constraint factor;

[0051] S5-5: For the output variable in step S5-4 Construct a prediction safety constraint:

[0052]

[0053] wherein, , are the predicted positions of the obstacle obtained by uniform speed prediction at the time t in the prediction time domain; , are the envelope circle radii of the ego vehicle and the obstacle, respectively; is the minimum safety distance.

[0054] Further, based on the mechanism-driven motion control multi-constraint optimization problem, a decision strategy trial and error and correction mechanism is constructed, and the safety expected decision instruction and safety expected motion control instruction of the ego vehicle are calculated, including:

[0055] S6-1: Solve the multi-constraint optimization problem in step S5 to obtain a control vector increment sequence;

[0056] S6-2: Use the control vector increment sequence to recursively propagate the time-discrete prediction model in step S5-3 in the prediction time domain to obtain a time sequence of output variables;

[0057] S6-3: Verify whether the time sequence of output variables satisfies the prediction safety constraint in step S5-5. If it does, set the expected decision instruction of the ego vehicle in step S3 as the safety expected decision instruction, and add the first group of vector elements in the control vector increment sequence in step S6-1 to the ego vehicle control variable at the last time to obtain the safety expected motion control instruction at the current time. If it does not, return to step S3-3 and execute the subsequent steps in turn from this step.

[0058] Further, a motion bottom controller of the ego vehicle is constructed to track the safety expected motion control instruction, and the safety bottom control instruction of the ego vehicle is calculated, including:

[0059] S7-1: Define the safety expected motion control instruction as the expected longitudinal acceleration and the expected yaw angular acceleration ;

[0060] S7-2: Convert the expected longitudinal acceleration and the expected yaw angular acceleration into the expected rotational speeds of the left and right side tracks:

[0061]

[0062] wherein,​​​ , are the desired rotation speeds of the left and right side tracks respectively; is the center distance between the left and right side tracks; is the equivalent rotation radius of the left and right side track motor drive wheels;

[0063] S7-3: Construct the left and right side track motor controller to track the desired rotation speed, and get the left and right side track motor driving torque, that is, the vehicle safety bottom layer control instruction:

[0064] ;

[0065] wherein, , are the left and right side track motor driving torques respectively; , are the left and right side track rotation speeds respectively; , , are the proportional parameters, integral parameters, and differential parameters of the left side track motor controller; , , are the proportional parameters, integral parameters, and differential parameters of the right side track motor controller.

[0066] Further, construct the self-learning state space, action space, and reward function, including:

[0067] S9-1: First, construct the self-learning state space:

[0068] ;

[0069] Then, the decision instruction set of step S3-1 is constructed as the action space ;

[0070] S9-2: Construct the self-learning reward function :

[0071] ;

[0072] wherein, is the collision reward function, constructed as:

[0073] ;

[0074] wherein, is the collision reward function value; is the time reward function, constructed as:

[0075] ;

[0076] wherein, is the maximum speed of the vehicle, is the weight coefficient; To reach the destination reward function, it is constructed as:

[0077] ;

[0078] in, is the weight coefficient.

[0079] Furthermore, based on the self-learning state space, action space, and reward function, training data is collected and an experience pool is constructed, including:

[0080] The motion state of the ego vehicle and the environmental participants at the current moment in step S2, the ego vehicle's safety expectation decision instruction in step S6, the motion state of the ego vehicle and the environmental participants at the next moment in step S8, and the state transition reward calculated based on the reward function in step S9 are constructed into a single state transition sequence and stored in the experience pool as a single self-learning training data.

[0081] Furthermore, based on the experience pool, reinforcement learning is used to build a data-driven decision-making self-learning algorithm to update the decision-making strategy network, including:

[0082] S11-1: Set the decision strategy action value network in step S3-2 to Network and Target network;

[0083] S11-2: Randomly extract batches of data from the experience pool and Network and Target Network construction loss function:

[0084] ;

[0085] in, is the loss function; for Constantly reinforce the state variables of the learning agent; for Constantly reinforce the decision-making actions of the learning agent; for State variables at the moment; Instant rewards for state transfers; For the experience pool; is the reward discount factor; 、 They are time Network and Target Network parameters; Represents the calculation of the neural network output value; To make the action that maximizes the network output value at the moment, defined as:

[0086] ;

[0087] wherein, is the decision action of the reinforcement learning agent at the moment;

[0088] S11-3: Based on the loss function, the gradient descent method is used to update the network parameters:

[0089] ;

[0090] wherein, is the network parameters at the moment ,is the learning rate, is the gradient symbol;

[0091] S11-4: Every step training step length, periodically copy network parameters to the target network;

[0092] S11-5: Repeat steps S11-2 to S11-4, and constantly update the decision policy network parameters until convergence.

[0093] Compared with the prior art, the significant progress of the present application is:

[0094] 1) A mechanism-driven tracked vehicle motion control multi-constraint optimization mechanism is constructed by using model predictive control, realizing safe trajectory tracking capability;

[0095] 2) A data mechanism hybrid enhanced unmanned tracked vehicle decision self-learning and control method is proposed, which considers the strong self-learning ability of reinforcement learning and the constraint processing advantage of model predictive control, respectively uses both for decision and trajectory tracking, and realizes safe self-learning ability;

[0096] 3) The basic safety effect of trajectory tracking on vehicle motion is considered, a self-learning decision mechanism with trajectory tracking ability boundary constraint is constructed, and the optimization of the whole system performance is realized.

[0097] In order to more clearly illustrate the functional characteristics and structural parameters of the present application, the following will be further described in combination with the drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0098] Figure 1 ​Schematic diagram of the self-learning and control process for unmanned tracked vehicle safety decision-making

[0099] Figure 2 Constructing a schematic diagram for the vehicle's expected decision and expected trajectory;

[0100] Figure 3 A schematic diagram is constructed for predicting safety constraints on the vehicle of the present invention;

[0101] Figure 4 This is a schematic diagram of the vehicle reinforcement learning self-learning decision-making process of the present invention. DETAILED DESCRIPTION

[0102] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0103] Figure 2 and Figure 3 middle 、 The center of mass of the vehicle in the body coordinate system Longitudinal and lateral velocities under the 、 are the envelope radii of the ego vehicle and the obstacle respectively; The division interval of the angle decision instruction; The distance of the speed decision instruction within a prediction time step; is the coordinate of the expected trajectory point; Predict trajectory point coordinates for obstacles.

[0104] Figure 4 middle for Constantly reinforce the state variables of the learning agent; 、 They are Moment and Constantly reinforce the decision-making actions of the learning agent; for State variables at the moment; For make The action instruction with the largest network value; Instant rewards for state transfers; 、 for Network and Target Network parameters.

[0105] See alsoFigures 1 to 3 The application provides a technical solution: a self-learning and control method for safety decision of unmanned tracked vehicle based on data mechanism mixed enhancement, comprising the following steps:

[0106] S1: constructing a vehicle decision self-learning and control environment to provide a self-vehicle sensor model, a self-vehicle motion physical model and an environment participant motion physical model for self-learning algorithm training;

[0107] S2: based on the self-vehicle sensor model, the self-vehicle motion physical model and the environment participant motion physical model, collecting motion states of the self-vehicle and the environment participant at the current time;

[0108] S3: constructing a decision instruction set, a self-learning decision strategy action value network and a decision action random exploration mechanism to calculate a self-vehicle expected decision instruction;

[0109] S4: mapping the self-vehicle expected decision instruction to a self-vehicle expected trajectory;

[0110] S5: constructing a mechanism-driven motion control multi-constraint optimization problem by using model predictive control to track the self-vehicle expected trajectory;

[0111] S6: based on the mechanism-driven motion control multi-constraint optimization problem, constructing a decision strategy trial and error and deviation correction mechanism to calculate a self-vehicle safety expected decision instruction and a safety expected motion control instruction;

[0112] S7: constructing a self-vehicle motion bottom controller to track the safety expected motion control instruction to calculate a self-vehicle safety bottom control instruction;

[0113] S8: based on the self-vehicle safety bottom control instruction, controlling the self-vehicle motion to realize single-step state transition to collect motion states of the self-vehicle and the environment participant at the next time;

[0114] S9: constructing a self-learning state space, an action space and a reward function;

[0115] S10: based on the self-learning state space, the action space and the reward function, collecting training data to construct an experience pool: the motion states of the self-vehicle and the environment participant at the current time obtained in step S2, the self-vehicle safety expected decision instruction obtained in step S6, the motion states of the self-vehicle and the environment participant at the next time obtained in step S8 and the state transition reward calculated based on the reward function in step S9 are constructed as a single state transition sequence to be stored in the experience pool as single self-learning training data;

[0116] S11: based on the experience pool, constructing a data-driven decision self-learning algorithm by using reinforcement learning to realize updating of the decision strategy network.

[0117] The step S2 specifically comprises:

[0118] S2-1: Collect the motion state of the ego vehicle and the environment participants at the current time, including: the longitudinal velocity and lateral velocity of the vehicle mass center in the vehicle body coordinate system; the yaw angular velocity of the vehicle at the mass center; the vehicle heading angle; the longitudinal acceleration of the vehicle mass center in the vehicle body coordinate system; the yaw angular acceleration of the vehicle at the mass center; the relative longitudinal distance, relative lateral distance, relative longitudinal velocity and relative lateral velocity of the vehicle and the obstacle in the vehicle body coordinate system; the relative longitudinal distance, relative lateral distance, relative longitudinal velocity and relative lateral velocity of the vehicle and the terminal in the vehicle body coordinate system.

[0119] The step S3 specifically comprises:

[0120] S3-1: Construct a decision instruction space, including an equal-interval velocity decision instruction and an equal-interval angle decision instruction, wherein the velocity decision instruction ranges from 0 to a velocity upper limit value , the angle decision instruction ranges from an angle lower limit value to an angle upper limit value , on the basis of which, each type of instruction is numbered from 1 in ascending order, and all possible combinations of the two types of instructions are constructed as a column vector set, which is used as the decision instruction set;

[0121] S3-2: Based on the decision instruction set, construct a decision strategy action value network as a fully connected network, wherein , , , are the neuron numbers of the 1st, 2nd, 3rd and 4th neural networks respectively, the activation function is the hyperbolic tangent function, and the network parameters are randomly initialized;

[0122] S3-3: Based on the decision strategy action value network, construct a decision action random exploration mechanism: first take a random real number between 0 and 1, if the number is greater than a set threshold ​​​​​​​​​​​​​​​, then a vector is randomly selected from the decision instruction set as the expected decision instruction. Otherwise, the motion state of the ego vehicle and the environmental participants at the current moment in step S2-1 and each vector in the decision instruction set are integrated into a single column vector, and respectively input into the decision strategy action value network to calculate the output value, that is, the action value. On this basis, the decision instruction corresponding to the maximum action value is selected as the expected decision instruction of the ego vehicle.

[0123] The step S4 specifically includes:

[0124] S4-1: Based on the desired decision instruction of the vehicle, first the corresponding angle decision instruction Take it out and take the center of mass of the vehicle as the origin and the body coordinate system The axis is used as the reference, and the desired path of the vehicle is constructed according to the angle decision instruction, and then the corresponding speed decision instruction is used. Take it out and take the center of mass of the vehicle as the origin along the body coordinate system Axis forward direction The size of the equal spacing is constructed normals, where , is the total number of prediction time steps, To predict the time step, the coordinates of the intersection of each normal line and the desired path in the vehicle body coordinate system and the heading angle of the desired path are calculated and constructed as trajectory points ,in For the The longitudinal position of the desired trajectory point, For the The lateral position of the desired trajectory point, For the The heading angle of the desired trajectory point, and ,Finally, all trajectory points are constructed in sequence as the desired trajectory of the vehicle.

[0125] The step S5 specifically includes:

[0126] S5-1: Construct the longitudinal and lateral dynamics and kinematics differential model of the vehicle:

[0127] ;

[0128] in, 、 are the longitudinal and lateral positions of the vehicle’s center of mass in the vehicle body coordinate system, 、 、 、 、 、 They are 、 、 , , , first derivative of the vehicle mass,

[0129] ;

[0130] wherein,

[0131] S5-2: Define model state variables, output variables and control variables:

[0132] ;

[0133] wherein, , ,

[0134] S5-3: Construct a time-discrete prediction model:

[0135] ;

[0136] wherein,

[0137] ;

[0138] Furthermore, , ,

[0139] ; ​​​​​​​​​​​​​​​​​​

[0140] in, is the identity matrix; Indicates the construction of the differential model in step S5-1; 、 for The operating point of the reference system at a certain moment can be constructed using the control variables optimized at the previous moment and the corresponding state prediction trajectory.

[0141] S5-4: Constructing a multi-constraint model predictive control optimization problem:

[0142] ;

[0143] in, for Optimization goals at all times; for Output variables at time; is the expected trajectory point of the vehicle; To control the total number of time steps; is the weight matrix for tracking the desired reference signal; 、 are the weight matrices for suppressing the amplitude and rate of change of the control variable respectively; is the constraint relaxation factor; To suppress the weight coefficient of excessive relaxation factor; for The first control variable increment, i.e. control variables Hedi control variables The difference is defined as:

[0144] ;

[0145] for arrive Corresponding The control variable increment sequence is defined as follows:

[0146] ;

[0147] also, 、 They are The upper and lower limits of the state variables at each moment; 、 They are Output variable upper and lower limits at all times; 、 They are Control the upper and lower limits of variables at all times; 、 respectively the upper and lower limit values of the control variable increment at the moment; is the upper limit value of the constraint factor.

[0148] S5-5: for the output variable in step S5-4 construct the prediction safety constraint:

[0149] ;

[0150] wherein, , respectively the predicted position of the obstacle obtained by uniform speed prediction of the obstacle at the moment; , , respectively the envelope circle radii of the ego vehicle and the obstacle; is the minimum safety distance.

[0151] The step S6 specifically comprises:

[0152] S6-1: solving the multi-constraint optimization problem in step S5 to obtain a control vector increment sequence;

[0153] S6-2: using the control vector increment sequence to recursively propagate the time-discrete prediction model in step S5-3 in the prediction time domain to obtain a time sequence of the output variable;

[0154] S6-3: verifying whether the time sequence of the output variable satisfies the prediction safety constraint in step S5-5, if yes, setting the expected decision instruction of the ego vehicle in step S3 as the safety expected decision instruction, and adding the first group of vector elements in the control vector increment sequence in step S6-1 to the control variable of the ego vehicle at the last moment to obtain the safety expected motion control instruction at the current moment; if not, returning to step S3-3 and executing the subsequent steps in turn from this step.

[0155] The step S7 specifically comprises:

[0156] S7-1: defining the safety expected motion control instruction as the expected longitudinal acceleration and the expected yaw angular acceleration ;

[0157] S7-2: converting the expected longitudinal acceleration and the expected yaw angular acceleration into the expected rotational speeds of the left and right side tracks:

[0158] ;

[0159] wherein, , respectively the expected rotational speeds of the left and right side tracks; is the center distance between the left and right side tracks; is the equivalent rotation radius of the left and right side track motor driving wheels.

[0160] S7-3: Constructing the left and right side track motor controllers to track the desired rotation speed, obtaining the left and right side track motor driving torque, that is, the vehicle safety bottom layer control instruction:

[0161] ;

[0162] wherein, , are the left and right side track motor driving torques, respectively; , are the left and right side track rotation speeds, respectively; , , is the left side track motor controller parameter; , , is the right side track motor controller parameter.

[0163] The step S9 specifically comprises:

[0164] S9-1: First, construct a self-learning state space:

[0165] ;

[0166] Then, the decision instruction set of step S3-1 is constructed as an action space .

[0167] S9-2: Constructing a self-learning reward function :

[0168] ;

[0169] wherein, is the collision reward function, constructed as:

[0170] ;

[0171] wherein, is the collision reward function value; is the time reward function, constructed as:

[0172] ;

[0173] wherein, is the highest speed of the vehicle, is the weight coefficient; is the destination reward function, constructed as:

[0174] ;

[0175] wherein, is a weight coefficient.

[0176] The step S11 specifically comprises:

[0177] S11-1: setting the decision-making policy action value network in step S3-2 as network and target network;

[0178] S11-2: randomly extracting batch data from the experience pool, and constructing a loss function based on network and target network;

[0179] ;

[0180] wherein, is a loss function; is a state variable of the reinforcement learning agent at the moment; is a decision-making action of the reinforcement learning agent at the moment; is a state variable at the moment; is a state transition immediate reward; is an experience pool; is a reward discount factor; , respectively are network and target network parameters at the moment; denotes neural network output value calculation; is an action that maximizes the neural network output value at the moment, defined as:

[0181] ;

[0182] wherein, is a decision-making action of the reinforcement learning agent at the moment;

[0183] S11-3: based on the loss function, updating network parameters using gradient descent method:

[0184] ;

[0185] wherein, is network parameters at the moment, ​​​learning rate, gradient sign;

[0186] S11-4: every training step, periodically copy network parameters to target network;

[0187] S11-5: repeat steps S11-2 to S11-4, constantly update decision policy network parameters until convergence.

[0188] While embodiments of the present application have been shown and described with reference to particular embodiments thereof, it will be understood by those skilled in the art that various changes in form and details can be made therein without departing from the spirit and scope of the application as defined by the appended claims and their equivalents.

Claims

1. A data-mechanism hybrid enhanced self-learning and control method for safety decision-making of unmanned tracked vehicles, characterized by: The method comprises: S1: Build a vehicle decision-making self-learning and control environment to provide the vehicle sensor model, vehicle motion physics model, and environment participant motion physics model for self-learning algorithm training; S2: Based on the ego vehicle sensor model, ego vehicle motion physics model, and environment participant motion physics model, the motion state of the ego vehicle and environment participants at the current moment is collected; S3: Construct a decision instruction set, a self-learning decision strategy action value network, and a decision action random exploration mechanism to calculate the expected decision instruction of the vehicle; S4: Mapping the desired decision instructions of the ego vehicle into the desired trajectory of the ego vehicle; S5: Use model predictive control to formulate a mechanism-driven multi-constraint optimization problem for motion control and track the desired trajectory of the ego vehicle. S6: Based on the mechanism-driven multi-constraint optimization problem of motion control, a trial-and-error decision-making strategy and a correction mechanism are constructed to calculate the desired safety decision instructions and desired safety motion control instructions for the ego vehicle. S7: Build the ego vehicle motion bottom-level controller to track the safety desired motion control instructions and calculate the ego vehicle safety bottom-level control instructions; S8: Based on the vehicle's underlying safety control instructions, control the vehicle's motion, implement single-step state transfer, and collect the motion states of the vehicle and environmental participants at the next moment; S9: Construct self-learning state space, action space, and reward function; S10: Based on the self-learning state space, action space, and reward function, collect training data and build an experience pool; S11: Based on the experience pool, reinforcement learning is used to build a data-driven decision-making self-learning algorithm to update the decision-making strategy network.

2. The method according to claim 1, characterized in that The motion states of the ego vehicle and the environmental participants at the current moment are collected based on the ego vehicle sensor model, the ego vehicle motion physics model, and the environmental participant motion physics model, including: Collect the motion status of the vehicle and the environment participants at the current moment, including: the center of mass of the vehicle in the body coordinate system Longitudinal speed under and lateral velocity ; The yaw rate of the vehicle at the center of mass ; Vehicle heading angle ; Longitudinal acceleration of the vehicle's center of mass in the body coordinate system ; The vehicle's yaw angular acceleration at the center of mass ; The relative longitudinal distance between the vehicle and the obstacle in the vehicle body coordinate system , relative lateral distance , relative longitudinal velocity , relative lateral velocity ; The relative longitudinal distance between the vehicle and the end point in the vehicle body coordinate system , relative lateral distance , relative longitudinal velocity , relative lateral velocity .

3. The method according to claim 2, characterized in that Construct a decision instruction set, a self-learning decision strategy action value network, and a decision action random exploration mechanism to calculate the expected decision instructions of the vehicle, including: S3-1: Constructing a decision instruction space, including equally spaced speed decision instructions and The speed decision instruction ranges from 0 to the upper speed limit. , the angle decision instruction is from the angle lower limit To the upper limit of the angle ; Number each type of instruction from 1 in ascending order, and construct a column vector set of all number combinations of the two types of instructions as the decision instruction set; S3-2: Based on the decision instruction set, the decision strategy action value network is constructed as A fully connected network, where 、 、 、 are the number of neurons in the 1st, 2nd, 3rd and 4th layers of the neural network respectively, and is the sum of the number of motion states of the ego vehicle and the environment participants at the current moment in step S2-1 and the dimension of any column vector in the decision instruction set. The activation function is the hyperbolic tangent function, and the network parameters are randomly initialized. S3-3: Based on the decision strategy action value network, a random exploration mechanism for decision actions is constructed: First, a random real number between 0 and 1 is taken. If the random real number is greater than the set threshold , then a vector is randomly selected from the decision instruction set as the expected decision instruction. Otherwise, the motion state of the ego vehicle and the environment participants at the current moment in step S2-1 and each vector in the decision instruction set are integrated into a single column vector, and are input into the decision strategy action value network respectively to calculate the output value, i.e., the action value. The decision instruction corresponding to the maximum action value is selected as the expected decision instruction of the vehicle.

4. The method according to claim 3, characterized in that Mapping the desired decision instructions of the ego vehicle to the desired trajectory of the ego vehicle includes: S4-1: Based on the desired decision instruction of the vehicle, first the corresponding angle decision instruction Take it out and take the center of mass of the vehicle as the origin and the body coordinate system The axis is used as the reference, and the desired path of the vehicle is constructed according to the angle decision instruction, and then the corresponding speed decision instruction is used. Take it out and take the center of mass of the vehicle as the origin along the body coordinate system Axis forward direction The size of the equal spacing is constructed normals, where , is the total number of prediction time steps, To predict the time step, the coordinates of the intersection of each normal line and the desired path in the vehicle body coordinate system and the heading angle of the desired path are calculated and constructed as trajectory points ,in For the The longitudinal position of the desired trajectory point, For the The lateral position of the desired trajectory point, For the The heading angle of the desired trajectory point, and ,Finally, all trajectory points are constructed in sequence as the desired trajectory of the vehicle.

5. The method according to claim 4, characterized in that Model predictive control is used to formulate a mechanism-driven multi-constraint optimization problem for motion control to track the desired trajectory of the ego vehicle, including: S5-1: Construct the longitudinal and lateral dynamics and kinematics differential model of the vehicle: ； in, 、 are the longitudinal and lateral positions of the vehicle's center of mass in the body coordinate system, respectively; 、 、 、 、 、 They are 、 、 、 、 、 The first derivative of ; is the vehicle mass; is the vehicle lateral force, expressed as: ； in, is the lateral sliding friction coefficient between the track and the ground; is the acceleration due to gravity; tanh is the hyperbolic tangent function; is a set real number; is the longitudinal ground contact length of the track; S5-2: Define model state variables, output variables and control variables: ； in, 、 、 are state variables, output variables, and control variables respectively; T is the transposition symbol; S5-3: Constructing a discrete time prediction model: ； in, is the index of the prediction time step; for State variables at the moment; 、 They are State variables and output variables at the moment; for Control variables at time; is the state-output transition matrix, defined as follows: ； also, 、 、 for The three matrices corresponding to the prediction model system at the moment are obtained by linearizing the longitudinal and lateral dynamics and kinematic differential models of the ego-vehicle constructed in step S5-1 at the reference system working point: ； in, is the identity matrix; Indicates the construction of the differential model in step S5-1; is the partial differential symbol; 、 for The state vector and control vector of the working point of the reference system are constructed using the control variables optimized at the previous moment and the corresponding state prediction trajectory; S5-4: Constructing a multi-constraint model predictive control optimization problem: ； Among them, min is the minimum symbol; for Optimization goals at all times; for Output variables at time; is the expected trajectory point of the vehicle; To control the total number of time steps; is the weight matrix for tracking the desired reference signal; 、 are the weight matrices for suppressing the amplitude and rate of change of the control variable respectively; is the constraint relaxation factor; To suppress the weight coefficient of excessive relaxation factor; for The first control variable increment, i.e. control variables Hedi control variables The difference is defined as: ； for arrive Corresponding The control variable increment sequence is defined as follows: ； also, 、 They are The upper and lower limits of the state variables at each moment; 、 They are Output variable upper and lower limits at all times; 、 They are Control the upper and lower limits of variables at all times; 、 They are Control the upper and lower limits of variable increments at all times; is the upper limit of the constraint factor; S5-5: Output variables in step S5-4 Constructing predictive safety constraints: ； in, 、 In the prediction time domain The predicted obstacle position obtained by predicting the obstacle at a constant speed at all times; 、 are the envelope radii of the ego vehicle and the obstacle respectively; The minimum safe distance.

6. The method according to claim 5, characterized in that Based on the mechanism-driven multi-constraint optimization problem of motion control, a trial-and-error and error correction mechanism for decision-making strategies is constructed to calculate the desired safety decision instructions and desired safety motion control instructions for the ego-vehicle, including: S6-1: Solve the multi-constraint optimization problem in step S5 to obtain a control vector increment sequence; S6-2: Using the control vector increment sequence, the time discrete prediction model in step S5-3 is recursively applied in the prediction time domain to obtain an output variable time series; S6-3: Verify whether the output variable time series satisfies the predicted safety constraints in step S5-5. If so, set the vehicle expected decision instruction in step S3 as the safety expected decision instruction, and at the same time add the first group of vector elements in the control vector increment sequence in step S6-1 to the vehicle control variable at the previous moment to obtain the safety expected motion control instruction at the current moment; if not, return to step S3-3 and execute subsequent steps in sequence starting from this step.

7. The method according to claim 6, characterized in that Construct the ego vehicle motion bottom-level controller to track the safety desired motion control instructions and calculate the ego vehicle safety bottom-level control instructions, including: S7-1: Define safe desired motion control instructions as desired longitudinal acceleration and the desired yaw acceleration ; S7-2: Convert the desired longitudinal acceleration and desired yaw angular acceleration into the desired rotational speeds of the left and right tracks: ； in, 、 are the expected speeds of the left and right tracks respectively; The distance between the center of the left and right tracks; is the equivalent rotation radius of the left and right crawler motor drive wheels; S7-3: Build left and right track motor controllers to track the desired speed and obtain the left and right track motor driving torques, which are the underlying control instructions for vehicle safety: ； in, 、 are the driving torques of the left and right track motors respectively; 、 The track speeds for the left and right sides respectively; 、 、 are the proportional parameters, integral parameters, and differential parameters of the left track motor controller; 、 、 are the proportional parameters, integral parameters, and differential parameters of the right track motor controller.

8. The method according to claim 7, characterized in that Construct self-learning state space, action space, and reward function, including: S9-1: First, construct the self-learning state space: ； Then, the decision instruction set of step S3-1 is constructed into an action space ; S9-2: Building a self-learning reward function : ； in, is the collision reward function, constructed as: ； in, is the collision reward function value; is the time reward function, which is constructed as: ； in, is the maximum speed of the vehicle, is the weight coefficient; To reach the destination reward function, it is constructed as: ； in, is the weight coefficient.

9. The method according to claim 8, characterized in that Based on the self-learning state space, action space, and reward function, we collect training data and build an experience pool, including: The motion state of the ego vehicle and the environmental participants at the current moment in step S2, the ego vehicle's safety expectation decision instruction in step S6, the motion state of the ego vehicle and the environmental participants at the next moment in step S8, and the state transition reward calculated based on the reward function in step S9 are constructed into a single state transition sequence and stored in the experience pool as a single self-learning training data.

10. The method according to claim 9, characterized in that Based on the experience pool, reinforcement learning is used to build a data-driven decision-making self-learning algorithm to update the decision-making strategy network, including: S11-1: Set the decision strategy action value network in step S3-2 to Network and Target network; S11-2: Randomly extract batches of data from the experience pool and Network and Target Network construction loss function: ； in, is the loss function; for Constantly strengthen the state variables of the learning agent; for Constantly reinforce the decision-making actions of the learning agent; for State variables at the moment; Instant rewards for state transfers; For the experience pool; is the reward discount factor; 、 They are time Network and Target Network parameters; Represents the calculation of the neural network output value; for Always The action with the largest network output value is defined as: ； in, for Constantly reinforce the decision-making actions of the learning agent; S11-3: Update based on loss function using gradient descent method Network parameters: ； in, for time Network parameters, is the learning rate, is the gradient symbol; S11-4: Every Step training step, periodically Network parameters are copied to the target network; S11-5: Repeat steps S11-2 to S11-4, and continuously update the decision strategy network parameters until convergence.

Images