Learning training acceleration method for mountain logistics unmanned vehicle driving control
By conducting fitness assessment and diversity elite selection on the intelligent agent of the unmanned logistics vehicle in the mountainous terrain, the training process was optimized, solving the problems of slow convergence speed and local optima in traditional methods. This enabled the vehicle to quickly learn the optimal driving control strategy in mountainous terrain, thereby improving the traffic efficiency and stability of the unmanned vehicle.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-02-14
- Publication Date
- 2026-05-12
AI Technical Summary
Traditional reinforcement learning methods suffer from sparse reward characteristics in training autonomous vehicles for mountain logistics, resulting in slow agent convergence. Furthermore, the elite selection mechanism of the cross-entropy algorithm is prone to homogenization, causing the training process to easily fall into local optima and affecting the effectiveness of the final driving control strategy.
By evaluating the fitness of individuals with weights in the population, a diverse elite set is selected. Based on this diverse elite set, the Gaussian distribution of the policy parameters for the next iteration is updated. By combining the linear action value function and the greedy policy, the training process of the agent is optimized.
It enables intelligent agents to quickly learn the optimal driving control strategy in mountainous terrain with fewer environmental interactions, improving the traffic efficiency and control stability of unmanned vehicles in complex terrain.
Smart Images
Figure CN121706834B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent logistics technology, specifically to a method for accelerating the learning and training of unmanned vehicle driving control in mountainous logistics areas. Background Technology
[0002] In mountainous logistics transportation scenarios, unmanned vehicles need to cope with complex challenges such as undulating slopes and power constraints, and their automated driving control relies on reinforcement learning algorithms to obtain the optimal strategy.
[0003] Traditional reinforcement learning methods have significant drawbacks in training unmanned logistics vehicles in mountainous terrain: on the one hand, the sparse reward characteristics of the mountainous environment lead to slow convergence of the agent, requiring a large amount of computing power and time; on the other hand, the elite selection mechanism of the traditional cross-entropy algorithm is prone to homogenization, with similar parameters of the selected elite individuals, making the training process prone to local optima, which in turn affects the effectiveness of the final driving control strategy. Summary of the Invention
[0004] The proposed method for accelerating the learning and training of unmanned vehicles for mountain logistics addresses the technical problems of slow convergence speed of agents due to the sparse reward characteristics of existing mountain environments, which requires a large amount of computing power and time; and the fact that the elite selection mechanism of the cross-entropy algorithm is prone to homogenization, with similar parameters of the selected elite individuals, making the training process prone to local optima, which in turn affects the effectiveness of the final driving control strategy.
[0005] To achieve the above objectives, the present invention adopts the following technical solution: The learning and training acceleration method for driving control of unmanned mountain logistics vehicles of the present invention includes the following steps:
[0006] S1. Construct intelligent agents based on the needs of mountain logistics transportation;
[0007] S2. The agent generates a population based on the Gaussian distribution of the policy parameters during iterative training, and assigns a set of weight parameters to each individual in the population to obtain a weighted individual.
[0008] S3. Evaluate the fitness of each individual with weight in the population to obtain the fitness score of each individual with weight.
[0009] S4. Elite selection is performed on the current population using the fitness score to select a diverse elite set;
[0010] S5. Update the Gaussian distribution of the strategy parameters for the next iteration based on the diverse elite set;
[0011] S6. Repeat the iterative training until the average fitness score of the individuals in the population reaches the preset threshold.
[0012] Preferably, step S1 includes the following steps:
[0013] S11. Construct a corresponding reinforcement learning environment based on the needs of mountain logistics transportation to simulate the operating environment of unmanned vehicles in complex mountainous terrain;
[0014] S12. Construct an intelligent agent based on the reinforcement learning environment;
[0015] S13. The intelligent agent interacts with the working environment, calculates the action value of each executable action in the current state through a linear action value function, and selects the action to be executed using a greedy strategy.
[0016] Preferably, step S3 includes the following steps:
[0017] S31. After loading the weight parameters into the agent, it starts to interact with the working environment from the initial state.
[0018] S32. The agent continuously outputs control commands based on its strategy parameters until it reaches the top of the slope.
[0019] S33. Record the maximum position reached by the unmanned vehicle within the round, and also record the actual number of steps consumed, and calculate the fitness score.
[0020] Preferably, step S4 includes the following steps:
[0021] S41. Sort all individuals in the current population in descending order according to the fitness score, and then sequentially traverse and select elites.
[0022] S42. Introduce parameter space diversity constraints and calculate the minimum Euclidean distance between the currently traversed individual and the selected elite.
[0023] S43. If the minimum Euclidean distance is greater than the selection threshold, the currently traversed individual is added to the diversity elite set until the diversity elite set reaches the preset number of elites.
[0024] S44. If the number of elites in the diversity elite set does not reach the preset number, then the remaining highest-scoring individuals will be added to the diversity elite set in turn.
[0025] Preferably, the Gaussian distribution for updating the strategy parameters of the next iteration based on the diverse elite set includes:
[0026] The mean is updated as follows:
[0027]
[0028] The standard deviation is updated as follows:
[0029]
[0030] And set a minimum constraint on the standard deviation:
[0031] in, The mean of the population's Gaussian distribution. For a diverse group of elites, As individuals within a diverse elite group, Let be the standard deviation of the Gaussian distribution of the population. This is the minimum standard deviation.
[0032] Preferably, step S6 includes the following steps:
[0033] S61. The agent continuously monitors the average fitness score of the weighted individuals in the population during the current iteration, and determines whether the average fitness score has reached the preset fitness threshold and whether the total number of iterations has reached the maximum limit.
[0034] S62. If neither is true, continue iterating; if one of them is true, stop iterating and output the final optimal weight parameters.
[0035] Preferably, the linear action value function is as follows:
[0036]
[0037] in, It is a linear action value function. For action The corresponding weight vector, Tile encoding features;
[0038] The greedy strategy is as follows:
[0039] .
[0040] Preferably, the fitness score is calculated using the following formula:
[0041]
[0042] in, For fitness score, This represents the maximum value of the autonomous vehicle's location in a single evaluation. To limit the maximum number of steps, The actual number of steps completed. This is the threshold for whether unmanned vehicles can reach their targets in mountainous logistics areas.
[0043] Preferably, the formula for calculating the minimum Euclidean distance is as follows:
[0044]
[0045] in, This represents the minimum Euclidean distance between the currently traversed individuals and the already selected elites. This refers to the currently traversed individual.
[0046] Preferably, the formula for calculating the average fitness score is as follows:
[0047]
[0048] in, The preset size of the population, The first in the population Fitness scores of individuals with weights.
[0049] As can be seen from the above technical solution, the present invention provides a method for accelerating the learning and training of driving control for unmanned vehicles in mountainous terrain. Compared with the prior art, the present invention has the following advantages: Fitness evaluation is performed on each weighted individual in the population to obtain a fitness score for each weighted individual; elite selection is performed on the current population using the fitness scores to select a diverse elite set; and the Gaussian distribution of the policy parameters for the next iteration is updated based on the diverse elite set. This allows the agent to quickly learn the optimal driving control strategy in mountainous terrain with fewer interactions with the environment, effectively improving the travel efficiency and control stability of unmanned vehicles in complex terrain. Attached Figure Description
[0050] Figure 1 This is a flowchart illustrating the learning and training acceleration method for the driving control of unmanned vehicles in mountain logistics according to the present invention.
[0051] Figure 2 This is a schematic diagram of the algorithm flow for the learning and training acceleration method of the unmanned vehicle control for mountain logistics in this invention.
[0052] Figure 3 This is a schematic diagram of the working environment for the learning and training acceleration method of the unmanned vehicle control for mountain logistics of the present invention.
[0053] Figure 4 This is a statistical chart comparing the learning and training acceleration method for unmanned mountain logistics vehicle control of the present invention with traditional methods. Detailed Implementation
[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.
[0055] like Figure 1 As shown, the learning and training acceleration method for the driving control of unmanned vehicles in mountain logistics in this embodiment includes the following steps:
[0056] S1. Construct intelligent agents based on the needs of mountain logistics transportation;
[0057] S2. During iterative training, the agent generates a population based on the Gaussian distribution of the policy parameters, and assigns a set of weight parameters to each individual in the population to obtain the weighted individual.
[0058] Specifically, in each training iteration, the agent is based on the current Gaussian distribution. The size of the generated scale is In the population, each sampled individual represents a set of potential policy weight parameters. and reshaped into The matrix format, where For the size of the action set, This represents the number of dimensions of the state features after tile encoding, so that they can be directly loaded into a linear policy network for action value estimation.
[0059] S3. Evaluate the fitness of each individual with a weight in the population to obtain the fitness score of each individual with a weight.
[0060] S4. Select elites from the current population by using fitness scores to identify a diverse set of elites.
[0061] S5. Update the Gaussian distribution of the strategy parameters for the next iteration based on the diverse elite set;
[0062] S6. Repeat the iterative training until the average fitness score of the individuals in the population reaches the preset threshold.
[0063] Furthermore, S1 includes the following steps:
[0064] S11. Construct a corresponding reinforcement learning environment based on the needs of mountain logistics transportation to simulate the operating environment of unmanned vehicles in complex mountainous terrain;
[0065] S12. Construct an intelligent agent based on the reinforcement learning environment;
[0066] S13. The agent interacts with the working environment, calculates the action value of each executable action in the current state through the linear action value function, and selects the action to be executed using a greedy strategy.
[0067] like Figure 2 As shown, the operational requirements of this environment are as follows: When performing material transportation and route passage tasks, mountain logistics unmanned vehicles are prone to problems due to the dramatic changes in slope and frequent terrain undulations. If the driving force output of the unmanned vehicle is insufficient, it may be unable to directly cross the top of the slope and stop in the middle. Therefore, the training objective is to enable the unmanned vehicle to learn a control strategy of "swaying to build up momentum - sprinting to climb over".
[0068] Strengthen the construction of the learning environment, such as Figure 3 As shown, the autonomous vehicle's state observation includes its current position and current speed; the agent selects from a pre-set set of actions at each decision-making moment. The system selects control actions, which include at least three basic actions: applying driving force forward, applying driving force backward, and not applying driving force, thereby enabling adaptive adjustment of the autonomous vehicle's driving behavior in complex sloping environments.
[0069] Furthermore, the linear action value function is as follows:
[0070]
[0071] in, It is a linear action value function. For action The corresponding weight vector, Tile encoding features;
[0072] Furthermore, the greedy strategy is as follows:
[0073] .
[0074] Furthermore, S3 includes the following steps:
[0075] S31. After loading the weight parameters into the agent, it starts to interact with the working environment from the initial state.
[0076] S32. The agent continuously outputs control commands based on its strategy parameters until it reaches the top of the slope.
[0077] S33. Record the maximum position reached by the unmanned vehicle within the round, and also record the actual number of steps consumed, and calculate the fitness score.
[0078] Specifically,
[0079] The formula for calculating the fitness score is as follows:
[0080]
[0081] in, For fitness score, This represents the maximum value of the autonomous vehicle's location in a single evaluation. To limit the maximum number of steps, The actual number of steps completed. This is the threshold for whether unmanned vehicles can reach their targets in mountainous logistics areas.
[0082] Specifically, the fitness score formula is divided into two parts: the maximum position of the autonomous vehicle in a single evaluation. When, explain the corresponding weight. The driving strategy enables the autonomous vehicle to successfully complete the task, at which point the score is determined by... (Basic score for ensuring task completion) and (This consists of efficiency items that reward completing tasks with fewer steps, encouraging strategies to pursue high efficiency while ensuring task completion; when...) When this occurs, it indicates that the strategy cannot support the autonomous vehicle to reach the target, and at this point, it should be directly... As a scoring system, it focuses on quantifying hill-climbing ability and avoids invalid interference from the step count indicator when the core task is not completed. The overall formula realizes dual quantification of hill-climbing ability and strategy efficiency. The algorithm convergence is improved through clear hierarchical evaluation logic, which is in line with the engineering application needs of unmanned vehicles for mountain logistics.
[0083] Furthermore, S4 includes the following steps:
[0084] S41. Sort all individuals in the current population in descending order according to their fitness scores, and then sequentially traverse and select elites.
[0085] S42. Introduce parameter space diversity constraints and calculate the minimum Euclidean distance between the currently traversed individual and the selected elite.
[0086] S43. If the minimum Euclidean distance is greater than the selection threshold, the currently traversed individual is added to the diversity elite set until the diversity elite set reaches the preset number of elites.
[0087] S44. If the number of elites in the diversity elite set does not reach the preset number, then the remaining highest-scoring individuals will be added to the diversity elite set in turn.
[0088] Furthermore, the formula for calculating the minimum Euclidean distance is as follows:
[0089]
[0090] in, This represents the minimum Euclidean distance between the currently traversed individuals and the already selected elites. This refers to the currently traversed individual.
[0091] Specifically, if (where the distance constraint coefficient) , (where the mean of the current population standard deviation is), then... Join a diverse elite group ;when Reach the preset number of elites Stop when the score is reached; if the score is insufficient, supplement the remaining highest-scoring individuals in turn to reach the required level. This distance constraint based on dynamic thresholds forces elite samples to maintain a different distribution in the weight space, which can prevent the algorithm from getting trapped in local optima too early due to excessive convergence.
[0092] The Gaussian distribution for updating the policy parameters for the next iteration based on a diverse elite set includes:
[0093] The mean is updated as follows:
[0094]
[0095] The standard deviation is updated as follows:
[0096]
[0097] And set a minimum constraint on the standard deviation:
[0098] in, The mean of the population's Gaussian distribution. For a diverse group of elites, As individuals within a diverse elite group, The standard deviation of the Gaussian distribution of the population is given; finally, the standard deviation used to update the Gaussian distribution in the next round is obtained. , is the minimum standard deviation, where This is to prevent the strategy search space from collapsing prematurely.
[0099] Furthermore, S6 includes the following steps:
[0100] S61. The agent continuously monitors the average fitness score of the individuals with weights in the population during the current iteration, and determines whether the average fitness score has reached the preset fitness threshold and whether the total number of iterations has reached the maximum limit.
[0101] S62. If neither is true, continue iterating; if one of them is true, stop iterating and output the final optimal weight parameters.
[0102] Specifically, the autonomous vehicle's intelligent system will continuously monitor the current... The system will stop iterating and output the final optimal weight parameters once either of the following termination conditions is met: whether the average fitness score of the strategy population reaches a preset threshold during the iteration, or whether the total number of iterations has reached the maximum limit.
[0103] The formula for calculating the average fitness score is as follows:
[0104]
[0105] in, The preset size of the population, The first in the population Fitness scores of individuals with weights.
[0106] like Figure 4 As shown, the proposed diversity elite cross-entropy learning training acceleration method (represented by DE-CEM in the figure) compared with the Q-learning algorithm in this embodiment achieves an average elite step count of only 4 rounds (i.e., interaction with the environment). The maximum number of steps decreased from 200 to around 110, indicating that the DE-CEM method requires fewer interactions with the environment and can quickly converge to a better solution with only a relatively limited number of interactions. Furthermore, relying on the core mechanisms of population iteration and distance-based elite selection, DE-CEM can quickly focus on better strategies while avoiding elite convergence.
[0107] In summary, the method of this invention can accelerate the training speed of automated driving control for unmanned vehicles in mountainous logistics by using a distance-constrained diversity elite cross-entropy method, thereby improving training efficiency, saving computing resources, and enabling unmanned vehicles to learn the optimal driving control strategy for mountainous terrain more quickly.
[0108] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).
[0109] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.
[0110] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0111] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for accelerating learning and training of mountain logistics unmanned vehicle driving control, characterized in that, Includes the following steps: S1. Construct intelligent agents based on the needs of mountain logistics transportation; S1 includes the following steps: S11. Construct a corresponding reinforcement learning environment based on the needs of mountain logistics transportation to simulate the operating environment of unmanned vehicles in complex mountainous terrain; S12. Construct an intelligent agent based on the reinforcement learning environment; S13. The intelligent agent interacts with the working environment, calculates the action value of each executable action in the current state through a linear action value function, and selects the action to be executed using a greedy strategy. S2. The agent generates a population based on the Gaussian distribution of the policy parameters during iterative training, and assigns a set of weight parameters to each individual in the population to obtain a weighted individual. S3. Evaluate the fitness of each individual with weight in the population to obtain the fitness score of each individual with weight. S3 includes the following steps: S31. After loading the weight parameters into the agent, it starts to interact with the working environment from the initial state. S32. The agent continuously outputs control commands based on its strategy parameters until it reaches the top of the slope. S33. Record the maximum position reached by the unmanned vehicle within the round, and also record the actual number of steps consumed, and calculate the fitness score. S4. Elite selection is performed on the current population using the fitness score to select a diverse elite set; S5. Update the Gaussian distribution of the strategy parameters for the next iteration based on the diverse elite set; S6. Repeat the iterative training until the average fitness score of the individuals in the population reaches the preset threshold. 2.The method of claim 1, wherein: S4 includes the following steps: S41. Sort all individuals in the current population in descending order according to the fitness score, and then sequentially traverse and select elites. S42. Introduce parameter space diversity constraints and calculate the minimum Euclidean distance between the currently traversed individual and the selected elite. S43. If the minimum Euclidean distance is greater than the selection threshold, the currently traversed individual is added to the diversity elite set until the diversity elite set reaches the preset number of elites. S44. If the number of elites in the diversity elite set does not reach the preset number, then the remaining highest-scoring individuals will be added to the diversity elite set in turn. 3.The method of claim 2, wherein: The Gaussian distribution of the strategy parameters for updating the next iteration based on a diverse elite set includes: The mean is updated as follows: The standard deviation is updated as follows: and setting a minimum constraint on the standard deviation: in, The mean of the population's Gaussian distribution. For a diverse group of elites, As individuals within a diverse elite group, Let be the standard deviation of the Gaussian distribution of the population. This is the minimum standard deviation.
4. The learning and training acceleration method for driving control of unmanned vehicles in mountain logistics according to claim 3, characterized in that: S6 includes the following steps: S61. The agent continuously monitors the average fitness score of the weighted individuals in the population during the current iteration, and determines whether the average fitness score has reached the preset fitness threshold and whether the total number of iterations has reached the maximum limit. S62. If neither is true, continue iterating; if one of them is true, stop iterating and output the final optimal weight parameters.
5. The learning and training acceleration method for driving control of unmanned vehicles in mountain logistics according to claim 4, characterized in that: The linear action value function is as follows: in, It is a linear action value function. For action The corresponding weight vector, Tile encoding features; The greedy strategy is as follows: 。 6. The learning and training acceleration method for driving control of unmanned vehicles in mountain logistics according to claim 5, characterized in that: The fitness score is calculated using the following formula: in, For fitness score, This represents the maximum value of the autonomous vehicle's location in a single evaluation. To limit the maximum number of steps, The actual number of steps completed. This is the threshold at which unmanned vehicles can reach their targets in mountainous logistics areas.
7. The learning and training acceleration method for driving control of unmanned vehicles in mountain logistics according to claim 6, characterized in that: The formula for calculating the minimum Euclidean distance is as follows: in, This represents the minimum Euclidean distance between the currently traversed individuals and the already selected elites. This refers to the currently traversed individual.
8. The learning and training acceleration method for driving control of unmanned vehicles in mountain logistics according to claim 7, characterized in that: The formula for calculating the average fitness score is as follows: in, The preset size of the population, The first in the population Fitness scores of individuals with weights.