A data mechanism-based hybrid enhancement method for self-learning and control of safety decisions in unmanned tracked vehicles

Abstract

This invention relates to the field of decision-making and control technology for unmanned tracked vehicles, specifically a data-mechanism hybrid enhancement method for self-learning and control of unmanned tracked vehicle safety decisions. This invention utilizes model predictive control to construct a mechanism-driven multi-constraint optimization mechanism for tracked vehicle motion control, achieving safe trajectory tracking capabilities. It proposes a data-mechanism hybrid enhancement method for self-learning and control of unmanned tracked vehicles, considering the powerful self-learning capabilities of reinforcement learning and the constraint handling advantages of model predictive control, utilizing both for decision-making and trajectory tracking respectively, thus achieving safe self-learning capabilities. Considering the fundamental safety role of trajectory tracking in vehicle motion, a self-learning decision-making mechanism with boundary constraints for trajectory tracking capabilities is constructed, achieving optimization of the overall system performance.

Description

Technical Field

[0001] This invention belongs to the field of decision-making and control technology for unmanned tracked vehicles, and in particular relates to a data mechanism-enhanced self-learning and control method for safety decision-making of unmanned tracked vehicles. Background Technology

[0002] Unmanned tracked vehicles are typical safety-critical systems, requiring not only high adaptability and self-learning capabilities to cope with dynamic and open traffic environments, but also ensuring the safety of the learning process. On one hand, from a system development perspective, strategies pre-trained in a virtual environment must meet safety requirements during real-world deployment. On the other hand, unmanned tracked vehicles need the ability to autonomously and safely explore and learn in the real world to optimize their driving strategies. Deep reinforcement learning, as the most typical self-learning technology, offers advantages such as model-free, unsupervised, and autonomous learning. It can guide the agent's interaction with the environment through predefined rewards and optimization goals, enabling autonomous learning and continuous decision-making optimization. It is considered the most effective method to improve the intelligence of unmanned tracked vehicles in dynamic and open environments and is particularly suitable for learning decision-making behaviors in complex scenarios that are difficult to model using mechanistic methods.

[0003] However, the "black box" nature of deep learning and the random trial-and-error mechanism of reinforcement learning severely affect the safety and rationality of policy exploration and utilization during the learning process, making it difficult to meet the self-learning requirements of unmanned tracked vehicles. Furthermore, most existing methods neglect the fundamental safety role of trajectory tracking in vehicle motion. The capability boundary of trajectory tracking is typically constrained by vehicle kinematics and dynamics; ignoring this factor leads to either overly aggressive decision trajectories that cannot be well tracked, or overly conservative trajectories that waste trajectory tracking capabilities, ultimately preventing the achievement of optimal performance for the entire decision-making and control system.

[0004] Therefore, how to achieve safe and reasonable decision-making self-learning and control of unmanned tracked vehicles while considering the boundary of trajectory tracking capability is also an important challenge. Summary of the Invention

[0005] The purpose of this invention is to solve the problem of achieving safe and reasonable decision-making self-learning and control of unmanned tracked vehicles under the premise of considering the boundary of trajectory tracking capability, as proposed in the background technology. This invention proposes a data mechanism hybrid enhancement method for safe decision-making self-learning and control of unmanned tracked vehicles.

[0006] This application provides a data mechanism-enhanced self-learning and control method for safety decision-making of unmanned tracked vehicles, the method comprising:

[0007] S1: Construct a vehicle decision-making self-learning and control environment to provide a vehicle sensor model, a vehicle motion physics model, and an environmental participant motion physics model for training the self-learning algorithm;

[0008] S2: Based on the vehicle sensor model, the vehicle motion physics model, and the environmental participant motion physics model, the motion state of the vehicle and the environmental participants at the current moment is collected;

[0009] S3: Construct a set of decision instructions, a self-learning decision strategy action value network, and a random exploration mechanism for decision actions to calculate the vehicle's expected decision instructions;

[0010] S4: Map the vehicle's desired decision command to the vehicle's desired trajectory;

[0011] S5: Utilize model predictive control to construct a mechanism-driven multi-constraint optimization problem for motion control, tracking the desired trajectory of the vehicle;

[0012] S6: Based on the mechanism-driven multi-constraint optimization problem of motion control, a decision strategy trial and error and correction mechanism is constructed to calculate the vehicle's safety expectation decision command and safety expectation motion control command.

[0013] S7: Construct a vehicle motion underlying controller to track the desired motion control commands for safety and calculate the vehicle safety underlying control commands;

[0014] S8: Based on the vehicle's safety underlying control commands, control the vehicle's movement, realize single-step state transition, and collect the movement state of the vehicle and environmental participants in the next moment;

[0015] S9: Construct the self-learning state space, action space, and reward function;

[0016] S10: Based on the self-learning state space, action space, and reward function, collect training data and build an experience pool;

[0017] S11: Based on the experience pool, a data-driven decision self-learning algorithm is constructed using reinforcement learning to update the decision policy network.

[0018] Furthermore, the motion states of the vehicle and environmental participants at the current moment are collected based on the vehicle sensor model, the vehicle motion physics model, and the environmental participant motion physics model, including:

[0019] The motion states of the vehicle and environmental participants at the current moment are collected, including: the vehicle's center of mass in the vehicle coordinate system. longitudinal velocity and lateral speed ; Yaw rate of the vehicle at its center of mass Vehicle heading angle Longitudinal acceleration of the vehicle's center of mass in the vehicle body coordinate system Yaw acceleration of the vehicle at its center of gravity The relative longitudinal distance between the vehicle and the obstacle in the vehicle coordinate system Relative lateral distance Relative longitudinal velocity Relative lateral velocity The relative longitudinal distance between the vehicle and the destination in the vehicle coordinate system. Relative lateral distance Relative longitudinal velocity Relative lateral velocity .

[0020] Furthermore, a decision instruction set, a self-learning decision strategy action value network, and a decision action stochastic exploration mechanism are constructed to calculate the vehicle's expected decision instructions, including:

[0021] S3-1: Construct the decision instruction space, including Equal-interval speed decision commands and There are several equally spaced angle decision commands, among which the speed decision command ranges from 0 to the upper speed limit. The angle decision command starts from the lower limit of the angle. Up to the upper limit of the angle ;

[0022] Each type of instruction is numbered from 1 in ascending order, and all the numbers of the two types of instructions are combined to construct a column vector set, which is used as the decision instruction set.

[0023] S3-2: Construct a decision-making strategy action value network based on the decision instruction set. A fully connected network, in which , , , These represent the number of neurons in layers 1, 2, 3, and 4 of the neural network, respectively. The sum of the number of motion states of the vehicle and environmental participants at the current moment in step S2-1 and the dimension of any column vector in the decision instruction set is used as the activation function, which is the hyperbolic tangent function, and the network parameters are randomly initialized.

[0024] S3-3: Based on the decision-making strategy action value network, construct a random exploration mechanism for decision-making actions: First, take a random real number between 0 and 1. If the number of random real numbers is greater than a set threshold... If the desired decision instruction is selected, a vector is randomly selected from the decision instruction set. Otherwise, the motion states of the vehicle and environmental participants at the current moment in step S2-1 are integrated with each vector in the decision instruction set to construct a single column vector, which is then input into the decision strategy action value network to calculate the output value, i.e., the action value.

[0025] The decision instruction corresponding to the maximum action value is selected as the vehicle's desired decision instruction.

[0026] Furthermore, mapping the vehicle's desired decision instructions to the vehicle's desired trajectory includes:

[0027] S4-1: Based on the vehicle's desired decision-making instructions, first send the corresponding angle decision-making instructions. Take it out, and with the vehicle's center of gravity as the origin, and the vehicle coordinate system Using the axis as a reference, the desired path of the vehicle is constructed according to the angle decision command, and then the corresponding speed decision command is executed. Take it out, and with the vehicle's center of gravity as the origin, along the vehicle's coordinate system The direction of the axis forward is Constructing equal-spaced sizes 3 normal lines, of which , To predict the total number of steps in time, To predict the time step, the coordinates of the intersection points of each normal and the desired path in the vehicle coordinate system, as well as the heading angle of the desired path, are calculated and constructed as trajectory points. ,in For the first The vertical position of the desired trajectory point For the first The lateral position of the desired trajectory point For the first The heading angle of the desired trajectory point, and Finally, all trajectory points are constructed in sequence to form the desired trajectory of the vehicle.

[0028] Furthermore, model predictive control is used to construct a mechanism-driven multi-constraint optimization problem for motion control, tracking the desired trajectory of the vehicle, including:

[0029] S5-1: Constructing the longitudinal and lateral dynamic and kinematic differential models of the vehicle:

[0030] ;

[0031] in, , These represent the longitudinal and lateral positions of the vehicle's center of gravity in the vehicle body coordinate system, respectively. , , , , , They are respectively , , , , , The first derivative; For vehicle quality; The lateral force on the vehicle is expressed as:

[0032] ;

[0033] in, The coefficient of lateral sliding friction between the track and the ground; ρ is the acceleration due to gravity; tanh is the hyperbolic tangent function; Let it be a given real number; This refers to the longitudinal ground contact length of the track.

[0034] S5-2: Define the model's state variables, output variables, and control variables:

[0035] ;

[0036] in, , , These represent the state variable, output variable, and control variable, respectively; T is the transpose sign.

[0037] S5-3: Constructing a time-discrete prediction model:

[0038] ;

[0039] in, For the prediction time step index; for The state variable at any given time; , They are respectively The state variables and output variables at each moment; for Control variables at any given time; The state-output transition matrix is ​​defined as follows:

[0040] ;

[0041] also, , , for The three matrices corresponding to the prediction model system at time points are obtained by linearizing the longitudinal and lateral dynamics and kinematic differential models of the vehicle constructed in step S5-1 at the reference system operating point:

[0042] ;

[0043] in, It is the identity matrix; This indicates that the differential model is constructed in step S5-1; The symbol is for partial differentials; , for The system references the state vector and control vector at each operating point and constructs the trajectory using the control variables and corresponding state prediction obtained from the previous time step.

[0044] S5-4: Constructing a multi-constraint model for predictive control optimization:

[0045] ;

[0046] Where min is the minimum sign; for The optimization goal at any given moment; for Output variables at any given time; The desired trajectory points of the vehicle; To control the total number of steps in time; The weight matrix is ​​used to track the desired reference signal; , These are the weight matrices for suppressing the magnitude and rate of change of the control variables, respectively; For constraint relaxation factor; To suppress excessively large weighting coefficients for relaxation factors; for The increment of the control variable, i.e. the th control variable. One control variable and the One control variable The difference is defined as:

[0047] ;

[0048] for arrive corresponding The resulting sequence of control variable increments is defined as follows:

[0049] ;

[0050] also, , They are respectively Upper and lower limits of the state variable at any given time; , They are respectively Output the upper and lower limits of the variable at all times; , They are respectively Always control the upper and lower limits of variables; , They are respectively Constantly control the upper and lower limits of variable increments; This is the upper limit of the constraint factor;

[0051] S5-5: Regarding the output variables in step S5-4 Constructing predictive security constraints:

[0052] ;

[0053] in, , In the prediction time domain The predicted position of the obstacle is obtained by making a constant velocity prediction of the obstacle at any time; , These are the radii of the envelope circles for the vehicle and the obstacle, respectively. This is the minimum safe distance.

[0054] Furthermore, based on the mechanism-driven multi-constraint optimization problem of motion control, a decision strategy trial-and-error and correction mechanism is constructed to calculate the vehicle's desired safety decision command and desired safety motion control command, including:

[0055] S6-1: Solve the multi-constraint optimization problem in step S5 to obtain the control vector increment sequence;

[0056] S6-2: Use the control vector increment sequence to recursively extrapolate the time discrete prediction model in step S5-3 in the prediction time domain to obtain the output variable time series;

[0057] S6-3: Verify whether the output variable time series meets the prediction safety constraints in step S5-5. If it does, set the vehicle's expected decision command in step S3 as the safety expected decision command. At the same time, add the first set of vector elements in the control vector increment sequence in step S6-1 to the vehicle's control variables at the previous time to obtain the safety expected motion control command at the current time. If it does not meet the constraints, return to step S3-3 and execute the subsequent steps sequentially from this step.

[0058] Furthermore, a vehicle motion underlying controller is constructed to track the desired motion control commands for safety and calculate the vehicle's safety underlying control commands, including:

[0059] S7-1: Define the desired motion control command for safety as the desired longitudinal acceleration. and expected yaw acceleration ;

[0060] S7-2: Convert the desired longitudinal acceleration and desired yaw acceleration into the desired rotational speeds of the left and right tracks:

[0061] ;

[0062] in, , These are the expected rotational speeds of the left and right tracks, respectively. The distance between the centers of the left and right tracks; The equivalent rotation radius of the left and right track motor drive wheels;

[0063] S7-3: Construct left and right track motor controllers to track the desired speed and obtain the driving torque of the left and right track motors, i.e., the vehicle's underlying safety control command:

[0064] ;

[0065] in, , These represent the driving torque of the left and right track motors, respectively. , These represent the rotational speeds of the left and right tracks, respectively. , , These are the proportional, integral, and derivative parameters of the left track motor controller. , , These are the proportional, integral, and derivative parameters of the right-side track motor controller.

[0066] Furthermore, the self-learning state space, action space, and reward function are constructed, including:

[0067] S9-1: First, construct the self-learning state space:

[0068] ;

[0069] Then, the set of decision instructions from step S3-1 is constructed into an action space. ;

[0070] S9-2: Constructing a self-learning reward function :

[0071] ;

[0072] in, The collision reward function is constructed as follows:

[0073] ;

[0074] in, The collision reward function value; The time-based reward function is constructed as follows:

[0075] ;

[0076] in, The maximum speed of the vehicle. These are the weighting coefficients; The reward function for reaching the destination is constructed as follows:

[0077] ;

[0078] in, These are the weighting coefficients.

[0079] Furthermore, based on the self-learning state space, action space, and reward function, training data is collected to construct an experience pool, including:

[0080] The motion states of the vehicle and environmental participants at the current moment in step S2, the vehicle safety expectation decision instruction in step S6, the motion states of the vehicle and environmental participants at the next moment in step S8, and the state transition reward calculated based on the reward function in step S9 are constructed into a single state transition sequence and stored in the experience pool as a single self-learning training data.

[0081] Furthermore, based on the experience pool, a data-driven decision self-learning algorithm is constructed using reinforcement learning to update the decision policy network, including:

[0082] S11-1: Set the decision-making strategy action value network in step S3-2 as follows: Network and target network;

[0083] S11-2: Randomly draw a batch of data from the experience pool, and based on... Network and Target Network construction loss function:

[0084] ;

[0085] in, The loss function; for Continuously reinforce the learning agent's state variables; for Continuously reinforce the decision-making actions of the learning agent; for The state variable at any given time; Instant reward for state transition; For experience pool; As a reward discount factor; , They are respectively time Network and Target Network parameters; This indicates the calculation of the neural network output value; for Time makes The action with the largest network output value is defined as:

[0086] ;

[0087] in, for Continuously reinforce the decision-making actions of the learning agent;

[0088] S11-3: Update based on the loss function using gradient descent. Network parameters:

[0089] ;

[0090] in, for time Network parameters, For learning rate, Gradient symbol;

[0091] S11-4: Every [time] Step training stride length, periodically... Copy network parameters to target network;

[0092] S11-5: Repeat steps S11-2 to S11-4, continuously updating the network parameters of the decision policy until convergence.

[0093] The significant advancement of this invention compared to existing technologies lies in:

[0094] 1) A mechanism-driven multi-constraint optimization mechanism for tracked vehicle motion control was constructed using model predictive control, achieving safe trajectory tracking capability;

[0095] 2) A data mechanism hybrid enhancement method for decision-making self-learning and control of unmanned tracked vehicles is proposed. This method takes into account the powerful self-learning ability of reinforcement learning and the constraint handling advantages of model predictive control. The two are used for decision-making and trajectory tracking respectively, and a safe self-learning capability is achieved.

[0096] 3) Considering the fundamental safety role of trajectory tracking in vehicle motion, a self-learning decision-making mechanism for trajectory tracking capability boundary constraints was constructed, achieving the optimization of the entire system performance.

[0097] To more clearly illustrate the functional characteristics and structural parameters of the present invention, further explanation is provided below in conjunction with the accompanying drawings and specific embodiments. Attached Figure Description

[0098] Figure 1Schematic diagram of self-learning and control process for safety decision-making of unmanned tracked vehicles

[0099] Figure 2 This is a schematic diagram illustrating the vehicle expectation decision and expectation trajectory construction of the present invention;

[0100] Figure 3 This is a schematic diagram illustrating the construction of vehicle predictive safety constraints according to the present invention;

[0101] Figure 4 This is a schematic diagram illustrating the self-learning decision-making process of vehicle reinforcement learning according to the present invention. Detailed Implementation

[0102] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0103] Figure 2 and Figure 3 middle , The vehicle's center of mass in the vehicle body coordinate system The longitudinal and lateral velocities are below. , These are the radii of the envelope circles for the vehicle and the obstacle, respectively. The interval for dividing the angle decision command; The distance for the speed decision instruction within a prediction time step; The coordinates of the points on the desired trajectory; The coordinates of the predicted trajectory points for the obstacle.

[0104] Figure 4 middle for Continuously reinforce the learning agent's state variables; , They are respectively Time and Continuously reinforce the decision-making actions of the learning agent; for The state variable at any given time; In order to be in make The action command with the highest network value; Instant reward for state transition; , for Network and Target Network parameters.

[0105] Please see Figures 1 to 3 This invention provides a technical solution: a data mechanism-enhanced self-learning and control method for safety decision-making of unmanned tracked vehicles, comprising the following steps:

[0106] S1: Construct a vehicle decision-making self-learning and control environment to provide a vehicle sensor model, a vehicle motion physics model, and an environmental participant motion physics model for training the self-learning algorithm;

[0107] S2: Based on the vehicle sensor model, the vehicle motion physics model, and the environmental participant motion physics model, the motion state of the vehicle and the environmental participants at the current moment is collected;

[0108] S3: Construct a set of decision instructions, a self-learning decision strategy action value network, and a random exploration mechanism for decision actions to calculate the vehicle's expected decision instructions;

[0109] S4: Map the vehicle's desired decision command to the vehicle's desired trajectory;

[0110] S5: Utilize model predictive control to construct a mechanism-driven multi-constraint optimization problem for motion control, thereby tracking the desired trajectory of the vehicle;

[0111] S6: Based on the mechanism-driven multi-constraint optimization problem of motion control, a decision strategy trial and error and correction mechanism is constructed to calculate the vehicle's safety expectation decision command and safety expectation motion control command.

[0112] S7: Construct a vehicle motion underlying controller to track the desired motion control commands for safety, and calculate the vehicle safety underlying control commands accordingly;

[0113] S8: Based on the vehicle's safety underlying control commands, control the vehicle's movement to achieve single-step state transition, thereby collecting the movement state of the vehicle and environmental participants in the next moment;

[0114] S9: Construct the self-learning state space, action space, and reward function;

[0115] S10: Based on the self-learning state space, action space, and reward function, collect training data to construct an experience pool: The motion states of the vehicle and environmental participants obtained in step S2 at the current moment, the vehicle safety expectation decision instructions obtained in step S6, the motion states of the vehicle and environmental participants obtained in step S8 at the next moment, and the state transition rewards calculated based on the reward function in step S9 are constructed into a single state transition sequence, which is then stored in the experience pool as a single self-learning training data.

[0116] S11: Based on the experience pool, a data-driven decision self-learning algorithm is constructed using reinforcement learning to update the decision policy network.

[0117] Step S2 specifically includes:

[0118] S2-1: The motion state of the vehicle and environmental participants at the current moment, including: the vehicle's center of mass in the vehicle coordinate system. longitudinal velocity and lateral speed ; Yaw rate of the vehicle at its center of mass Vehicle heading angle Longitudinal acceleration of the vehicle's center of mass in the vehicle body coordinate system Yaw acceleration of the vehicle at its center of gravity The relative longitudinal distance between the vehicle and the obstacle in the vehicle coordinate system Relative lateral distance Relative longitudinal velocity Relative lateral velocity The relative longitudinal distance between the vehicle and the destination in the vehicle coordinate system. Relative lateral distance Relative longitudinal velocity Relative lateral velocity .

[0119] Step S3 specifically includes:

[0120] S3-1: Construct the decision instruction space, including Equal-interval speed decision commands and There are several equally spaced angle decision commands, among which the speed decision command ranges from 0 to the upper speed limit. The angle decision command starts from the lower limit of the angle. Up to the upper limit of the angle Based on this, each type of instruction is numbered from 1 in ascending order, and all possible combinations of numbers for the two types of instructions are constructed into a set of column vectors, which are used as the decision instruction set.

[0121] S3-2: Construct a decision-making strategy action value network based on the decision instruction set. A fully connected network, in which , , , These represent the number of neurons in layers 1, 2, 3, and 4 of the neural network, respectively. The activation function is the hyperbolic tangent function, and the network parameters are randomly initialized.

[0122] S3-3: Based on the decision-making strategy action value network, construct a random exploration mechanism for decision-making actions: First, take a random real number between 0 and 1. If the number is greater than a set threshold... If the desired decision is achieved, a vector is randomly selected from the decision instruction set as the expected decision instruction. Otherwise, the motion states of the vehicle and environmental participants at the current moment in step S2-1 are integrated with each vector in the decision instruction set to construct a single column vector, which is then input into the decision strategy action value network to calculate the output value, i.e., the action value. Based on this, the decision instruction corresponding to the maximum action value is selected as the desired decision instruction for the vehicle.

[0123] Step S4 specifically includes:

[0124] S4-1: Based on the vehicle's desired decision-making instructions, first send the corresponding angle decision-making instructions. Take it out, and with the vehicle's center of gravity as the origin, and the vehicle coordinate system Using the axis as a reference, the desired path of the vehicle is constructed according to the angle decision command, and then the corresponding speed decision command is executed. Take it out, and with the vehicle's center of gravity as the origin, along the vehicle's coordinate system The direction of the axis forward is Constructing equal-spaced sizes 3 normal lines, of which , To predict the total number of steps in time, To predict the time step, the coordinates of the intersection points of each normal and the desired path in the vehicle coordinate system, as well as the heading angle of the desired path, are calculated and constructed as trajectory points. ,in For the first The vertical position of the desired trajectory point For the first The lateral position of the desired trajectory point For the first The heading angle of the desired trajectory point, and Finally, all trajectory points are constructed in sequence to form the desired trajectory of the vehicle.

[0125] Step S5 specifically includes:

[0126] S5-1: Constructing the longitudinal and lateral dynamic and kinematic differential models of the vehicle:

[0127] ;

[0128] in, , These represent the longitudinal and lateral positions of the vehicle's center of gravity in the vehicle body coordinate system, respectively. , , , , , They are respectively , , , , , The first derivative, For vehicle quality, The lateral force of the vehicle can be approximated by the following formula:

[0129] ;

[0130] in, The coefficient of lateral sliding friction between the track and the ground; ρ is the acceleration due to gravity; tanh is the hyperbolic tangent function; Let it be a small real number; This refers to the longitudinal ground contact length of the track.

[0131] S5-2: Define the model's state variables, output variables, and control variables:

[0132] ;

[0133] in, , , These represent the state variable, output variable, and control variable, respectively; T is the transpose symbol.

[0134] S5-3: Constructing a time-discrete prediction model:

[0135] ;

[0136] in, For the prediction time step index; for The state variable at any given time; , They are respectively The state variables and output variables at each moment; for Control variables at any given time; The state-output transition matrix is ​​defined as follows:

[0137] ;

[0138] also, , , for The prediction model system matrix at time step S5-1 can be obtained by linearizing the vehicle's longitudinal and lateral dynamics and kinematic differential models constructed in step S5-1 at the reference system operating point:

[0139] ;

[0140] in, It is the identity matrix; This indicates that the differential model is constructed in step S5-1; , for The operating point of the reference system can be constructed using the control variables and corresponding state prediction trajectories obtained from the previous time step optimization.

[0141] S5-4: Constructing a multi-constraint model for predictive control optimization:

[0142] ;

[0143] in, for The optimization goal at any given moment; for Output variables at any given time; The desired trajectory points of the vehicle; To control the total number of steps in time; The weight matrix is ​​used to track the desired reference signal; , These are the weight matrices for suppressing the magnitude and rate of change of the control variables, respectively; For constraint relaxation factor; To suppress excessively large weighting coefficients for relaxation factors; for The increment of the control variable, i.e. the th control variable. One control variable and the One control variable The difference is defined as:

[0144] ;

[0145] for arrive corresponding The resulting sequence of control variable increments is defined as follows:

[0146] ;

[0147] also, , They are respectively Upper and lower limits of the state variable at any given time; , They are respectively Output the upper and lower limits of the variable at all times; , They are respectively Always control the upper and lower limits of variables; , They are respectively Constantly control the upper and lower limits of variable increments; This represents the upper limit of the constraint factor.

[0148] S5-5: Regarding the output variables in step S5-4 Constructing predictive security constraints:

[0149] ;

[0150] in, , In the prediction time domain The predicted position of the obstacle is obtained by making a constant velocity prediction of the obstacle at any time; , These are the radii of the envelope circles for the vehicle and the obstacle, respectively. This is the minimum safe distance.

[0151] Step S6 specifically includes:

[0152] S6-1: Solve the multi-constraint optimization problem in step S5 to obtain the control vector increment sequence;

[0153] S6-2: Use the control vector increment sequence to recursively extrapolate the time discrete prediction model in step S5-3 in the prediction time domain to obtain the output variable time series;

[0154] S6-3: Verify whether the output variable time series meets the prediction safety constraints in step S5-5. If it does, set the vehicle's expected decision command in step S3 as the safety expected decision command. At the same time, add the first set of vector elements in the control vector increment sequence in step S6-1 to the vehicle's control variables at the previous time to obtain the safety expected motion control command at the current time. If it does not meet the constraints, return to step S3-3 and execute the subsequent steps sequentially from this step.

[0155] Step S7 specifically includes:

[0156] S7-1: Define the desired motion control command for safety as the desired longitudinal acceleration. and expected yaw acceleration ;

[0157] S7-2: Convert the desired longitudinal acceleration and desired yaw acceleration into the desired rotational speeds of the left and right tracks:

[0158] ;

[0159] in, , These are the expected rotational speeds of the left and right tracks, respectively. The distance between the centers of the left and right tracks; The equivalent rotation radius of the left and right track motor drive wheels is given.

[0160] S7-3: Construct left and right track motor controllers to track the desired speed and obtain the driving torque of the left and right track motors, i.e., the vehicle's underlying safety control command:

[0161] ;

[0162] in, , These represent the driving torque of the left and right track motors, respectively. , These represent the rotational speeds of the left and right tracks, respectively. , , The parameters for the left track motor controller; , , These are the parameters for the right track motor controller.

[0163] Step S9 specifically includes:

[0164] S9-1: First, construct the self-learning state space:

[0165] ;

[0166] Then, the set of decision instructions from step S3-1 is constructed into an action space. .

[0167] S9-2: Constructing a self-learning reward function :

[0168] ;

[0169] in, The collision reward function is constructed as follows:

[0170] ;

[0171] in, The collision reward function value; The time-based reward function is constructed as follows:

[0172] ;

[0173] in, The maximum speed of the vehicle. These are the weighting coefficients; The reward function for reaching the destination is constructed as follows:

[0174] ;

[0175] in, These are the weighting coefficients.

[0176] Step S11 specifically includes:

[0177] S11-1: Set the decision-making strategy action value network in step S3-2 as follows: Network and target network;

[0178] S11-2: Randomly draw a batch of data from the experience pool, and based on... Network and Target Network construction loss function:

[0179] ;

[0180] in, The loss function; for Continuously reinforce the learning agent's state variables; for Continuously reinforce the decision-making actions of the learning agent; for The state variable at any given time; Instant reward for state transition; For experience pool; As a reward discount factor; , They are respectively time Network and Target Network parameters; This indicates the calculation of the neural network output value; for Time makes The action with the largest network output value is defined as:

[0181] ;

[0182] in, for Continuously reinforce the decision-making actions of the learning agent;

[0183] S11-3: Update based on the loss function using gradient descent. Network parameters:

[0184] ;

[0185] in, for time Network parameters, For learning rate, Gradient symbol;

[0186] S11-4: Every [time] Step training stride length, periodically... Copy network parameters to target network;

[0187] S11-5: Repeat steps S11-2 to S11-4, continuously updating the network parameters of the decision policy until convergence.

[0188] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A data mechanism-based hybrid enhancement method for self-learning and control of safety decision-making in unmanned tracked vehicles, characterized in that, The method includes: S1: Construct a vehicle decision-making self-learning and control environment to provide a vehicle sensor model, a vehicle motion physics model, and an environmental participant motion physics model for training the self-learning algorithm; S2: Based on the vehicle sensor model, the vehicle motion physics model, and the environmental participant motion physics model, the motion state of the vehicle and the environmental participants at the current moment is collected; S3: Construct a set of decision instructions, a self-learning decision strategy action value network, and a random exploration mechanism for decision actions to calculate the vehicle's expected decision instructions; S4: Map the vehicle's desired decision command to the vehicle's desired trajectory; S5: Utilize model predictive control to construct a mechanism-driven multi-constraint optimization problem for motion control, tracking the desired trajectory of the vehicle; S6: Based on the mechanism-driven multi-constraint optimization problem of motion control, a decision strategy trial and error and correction mechanism is constructed to calculate the vehicle's safety expectation decision command and safety expectation motion control command. S7: Construct a vehicle motion underlying controller to track the desired motion control commands for safety and calculate the vehicle safety underlying control commands; S8: Based on the vehicle's safety underlying control commands, control the vehicle's movement, realize single-step state transition, and collect the movement state of the vehicle and environmental participants in the next moment; S9: Construct the self-learning state space, action space, and reward function; S10: Based on the self-learning state space, action space, and reward function, collect training data and build an experience pool; S11: Based on the experience pool, a data-driven decision self-learning algorithm is constructed using reinforcement learning to update the decision policy network.

2. The method according to claim 1, characterized in that, The motion states of the vehicle and environmental participants at the current moment are collected based on the vehicle sensor model, the vehicle motion physics model, and the environmental participant motion physics model, including: The motion states of the vehicle and environmental participants at the current moment are collected, including: the vehicle's center of mass in the vehicle coordinate system. longitudinal velocity and lateral speed ; Yaw rate of the vehicle at its center of mass Vehicle heading angle Longitudinal acceleration of the vehicle's center of mass in the vehicle body coordinate system Yaw acceleration of the vehicle at its center of gravity The relative longitudinal distance between the vehicle and the obstacle in the vehicle coordinate system Relative lateral distance Relative longitudinal velocity Relative lateral velocity The relative longitudinal distance between the vehicle and the destination in the vehicle coordinate system. Relative lateral distance Relative longitudinal velocity Relative lateral velocity .

3. The method according to claim 2, characterized in that, A set of decision instructions, a self-learning decision strategy action value network, and a random exploration mechanism for decision actions are constructed to calculate the vehicle's expected decision instructions, including: S3-1: Construct the decision instruction space, including Equal-interval speed decision commands and There are several equally spaced angle decision commands, among which the speed decision command ranges from 0 to the upper speed limit. The angle decision command starts from the lower limit of the angle. Up to the upper limit of the angle ; Each type of instruction is numbered from 1 in ascending order, and all the numbers of the two types of instructions are combined to construct a column vector set, which is used as the decision instruction set. S3-2: Construct a decision-making strategy action value network based on the decision instruction set. A fully connected network, in which , , , These represent the number of neurons in layers 1, 2, 3, and 4 of the neural network, respectively. The sum of the number of motion states of the vehicle and environmental participants at the current moment in step S2-1 and the dimension of any column vector in the decision instruction set is used as the activation function, which is the hyperbolic tangent function, and the network parameters are randomly initialized. S3-3: Based on the decision-making strategy action value network, construct a random exploration mechanism for decision-making actions: First, take a random real number between 0 and 1. If the random real number is greater than a set threshold... If the desired decision instruction is selected, a vector is randomly selected from the decision instruction set. Otherwise, the motion states of the vehicle and environmental participants at the current moment in step S2-1 are integrated with each vector in the decision instruction set to construct a single column vector, which is then input into the decision strategy action value network to calculate the output value, i.e., the action value. The decision instruction corresponding to the maximum action value is selected as the vehicle's desired decision instruction.

4. The method according to claim 3, characterized in that, Mapping the vehicle's desired decision instructions to the vehicle's desired trajectory includes: S4-1: Based on the vehicle's desired decision-making instructions, first send the corresponding angle decision-making instructions. Take it out, and with the vehicle's center of gravity as the origin, and the vehicle coordinate system Using the axis as a reference, the desired path of the vehicle is constructed according to the angle decision command, and then the corresponding speed decision command is executed. Take it out, and with the vehicle's center of gravity as the origin, along the vehicle's coordinate system The direction of the axis forward is Constructing equal-spaced sizes 3 normal lines, of which , To predict the total number of steps in time, To predict the time step, the coordinates of the intersection points of each normal and the desired path in the vehicle coordinate system, as well as the heading angle of the desired path, are calculated and constructed as trajectory points. ,in For the first The vertical position of the desired trajectory point For the first The lateral position of the desired trajectory point For the first The heading angle of the desired trajectory point, and Finally, all trajectory points are constructed in sequence to form the desired trajectory of the vehicle.

5. The method according to claim 4, characterized in that, Model predictive control is used to construct a mechanism-driven multi-constraint optimization problem for motion control, tracking the desired trajectory of a vehicle, including: S5-1: Constructing the longitudinal and lateral dynamic and kinematic differential models of the vehicle: ； in, , These represent the longitudinal and lateral positions of the vehicle's center of gravity in the vehicle body coordinate system, respectively. , , , , , They are respectively , , , , , The first derivative; For vehicle quality; The lateral force on the vehicle is expressed as: ； in, The coefficient of lateral sliding friction between the track and the ground; ρ is the acceleration due to gravity; tanh is the hyperbolic tangent function; Let it be a given real number; This refers to the longitudinal ground contact length of the track. S5-2: Define the model's state variables, output variables, and control variables: ； in, , , These represent the state variable, output variable, and control variable, respectively; T is the transpose sign. S5-3: Constructing a time-discrete prediction model: ； in, For the prediction time step index; for The state variable at any given time; , They are respectively The state variables and output variables at each moment; for Control variables at any given time; The state-output transition matrix is ​​defined as follows: ； also, , , for The three matrices corresponding to the prediction model system at time points are obtained by linearizing the longitudinal and lateral dynamics and kinematic differential models of the vehicle constructed in step S5-1 at the reference system operating point: ； in, It is the identity matrix; This indicates that the differential model is constructed in step S5-1; The symbol is for partial differentials; , for The system references the state vector and control vector at each operating point and constructs the trajectory using the control variables and corresponding state prediction obtained from the previous time step. S5-4: Constructing a multi-constraint model for predictive control optimization: ； Where min is the minimum sign; for The optimization goal at any given moment; for Output variables at any given time; The desired trajectory points of the vehicle; To control the total number of steps in time; The weight matrix is ​​used to track the desired reference signal; , These are the weight matrices for suppressing the magnitude and rate of change of the control variables, respectively; For constraint relaxation factor; To suppress excessively large weighting coefficients for relaxation factors; for The increment of the control variable, i.e. the th control variable. One control variable and the One control variable The difference is defined as: ； for arrive corresponding The resulting sequence of control variable increments is defined as follows: ； also, , They are respectively Upper and lower limits of the state variable at any given time; , They are respectively Output the upper and lower limits of the variable at all times; , They are respectively Control the upper and lower limits of variables at all times; , They are respectively Constantly control the upper and lower limits of variable increments; This is the upper limit of the constraint factor; S5-5: Regarding the output variables in step S5-4 Constructing predictive security constraints: ； in, , In the prediction time domain The predicted position of the obstacle is obtained by making a constant velocity prediction of the obstacle at any time; , These are the radii of the envelope circles for the vehicle and the obstacle, respectively. This is the minimum safe distance.

6. The method according to claim 5, characterized in that, Based on the mechanism-driven multi-constraint optimization problem of motion control, a decision strategy trial-and-error and correction mechanism is constructed to calculate the vehicle's expected safety decision command and expected safety motion control command, including: S6-1: Solve the multi-constraint optimization problem in step S5 to obtain the control vector increment sequence; S6-2: Use the control vector increment sequence to recursively extrapolate the time discrete prediction model in step S5-3 in the prediction time domain to obtain the output variable time series; S6-3: Verify whether the output variable time series meets the prediction safety constraints in step S5-5. If it does, set the vehicle's expected decision command in step S3 as the safety expected decision command. At the same time, add the first set of vector elements in the control vector increment sequence in step S6-1 to the vehicle's control variables at the previous time to obtain the safety expected motion control command at the current time. If it does not meet the constraints, return to step S3-3 and execute the subsequent steps sequentially from this step.

7. The method according to claim 6, characterized in that, A vehicle motion underlying controller is constructed to track the desired motion control commands for safety and calculate the vehicle's safety underlying control commands, including: S7-1: Define the desired motion control command for safety as the desired longitudinal acceleration. and expected yaw acceleration ; S7-2: Convert the desired longitudinal acceleration and desired yaw acceleration into the desired rotational speeds of the left and right tracks: ； in, , These are the expected rotational speeds of the left and right tracks, respectively. The distance between the centers of the left and right tracks; The equivalent rotation radius of the left and right track motor drive wheels; S7-3: Construct left and right track motor controllers to track the desired speed and obtain the driving torque of the left and right track motors, i.e., the vehicle's underlying safety control command: ； in, , These represent the driving torque of the left and right track motors, respectively. , These represent the rotational speeds of the left and right tracks, respectively. , , These are the proportional, integral, and derivative parameters of the left track motor controller. , , These are the proportional, integral, and derivative parameters of the right-side track motor controller.

8. The method according to claim 7, characterized in that, Construct the self-learning state space, action space, and reward function, including: S9-1: First, construct the self-learning state space: ； Then, the set of decision instructions from step S3-1 is constructed into an action space. ; S9-2: Constructing a self-learning reward function : ； in, The collision reward function is constructed as follows: ； in, The collision reward function value; The time-based reward function is constructed as follows: ； in, The maximum speed of the vehicle. These are the weighting coefficients; The reward function for reaching the destination is constructed as follows: ； in, These are the weighting coefficients.

9. The method according to claim 8, characterized in that, Based on the self-learning state space, action space, and reward function, training data is collected to construct an experience pool, including: The motion states of the vehicle and environmental participants at the current moment in step S2, the vehicle safety expectation decision instruction in step S6, the motion states of the vehicle and environmental participants at the next moment in step S8, and the state transition reward calculated based on the reward function in step S9 are constructed into a single state transition sequence and stored in the experience pool as a single self-learning training data.

10. The method according to claim 9, characterized in that, Based on an experience pool, a data-driven decision self-learning algorithm is constructed using reinforcement learning to update the decision policy network, including: S11-1: Set the decision-making strategy action value network in step S3-2 as follows: Network and target network; S11-2: Randomly draw a batch of data from the experience pool, and based on... Network and Target Network construction loss function: ； in, The loss function; for Continuously reinforce the learning agent's state variables; for Continuously reinforce the decision-making actions of the learning agent; for The state variable at any given time; Instant reward for state transition; For experience pool; As a reward discount factor; , They are respectively time Network and Target Network parameters; This indicates the calculation of the neural network output value; for Time makes The action with the largest network output value is defined as: ； in, for Continuously reinforce the decision-making actions of the learning agent; S11-3: Update based on the loss function using gradient descent. Network parameters: ； in, for time Network parameters, For learning rate, Gradient symbol; S11-4: Every [time] Step training stride length, periodically... Copy network parameters to target network; S11-5: Repeat steps S11-2 to S11-4, continuously updating the network parameters of the decision policy until convergence.

Images