Proximal policy optimization training method, system, storage medium, and vehicle
By introducing a phase load-aware scheduling module and a latency consistency alignment mechanism, the problem of low resource utilization in the PPO training system is solved, decoupling and acceleration of each phase are achieved, and training efficiency and stability are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 安徽蔚来智驾科技有限公司
- Filing Date
- 2026-05-07
- Publication Date
- 2026-08-04
AI Technical Summary
Existing PPO training systems suffer from low resource utilization and limited training efficiency, especially in asynchronous computing environments, making it difficult to decouple and accelerate each stage without changing the PPO optimization objective and convergence characteristics.
A phased load-aware scheduling module is introduced. Through a latency-consistency alignment mechanism, it asynchronously executes joint sampling of policies and values, calculation of rewards and advantages, and optimization of policy and value networks. It adopts version-aware filtering strategies and dynamic resource scheduling to achieve decoupling and acceleration of each phase.
It improves the parallel processing capability of the training system, ensures the stability and convergence of the PPO training process, avoids inconsistencies between training samples and model parameters, and improves resource utilization and training efficiency.
Smart Images
Figure CN122173934B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent driving technology, specifically providing a near-end strategy optimization training method, system, storage medium, and vehicle. Background Technology
[0002] In the field of autonomous driving, reinforcement learning algorithms are commonly used to train various intelligent agents. Among them, Proximal Policy Optimization (PPO) is a policy gradient algorithm widely used in complex reinforcement learning tasks. Its training process typically includes stages such as trajectory sampling, reward calculation, and network parameter updates. Traditional PPO training systems often execute these stages synchronously or in a weakly asynchronous manner. However, as task complexity and the computational cost of reward functions increase, the differences in computational and execution efficiency between different training stages become increasingly apparent. This can easily lead to resource idleness, pipeline congestion, and other adverse situations, thus limiting overall training efficiency.
[0003] Existing reinforcement learning methods based on asynchronous computation attempt to improve parallelism through asynchronous Actor-Learner architectures, but they often rely on off-policy training assumptions or require significant adjustments to the algorithm structure. Therefore, they are difficult to directly apply to PPO algorithms that strictly depend on on-policy constraints. Thus, without changing the PPO optimization objective and convergence characteristics, and while ensuring the consistency and stability of training data, how to decouple and accelerate the training stages, and improve the problems of tight coupling between stages, low resource utilization, and algorithmic modifications required for training acceleration in existing PPO training architectures, has become an urgent problem to be solved.
[0004] Accordingly, there is a need in this field for a new proximal policy optimization training scheme to address the above problems. Summary of the Invention
[0005] To overcome the above-mentioned shortcomings, this application is proposed to solve, or at least partially solve, the technical problem of how to construct a PPO training system based on asynchronous computing and achieve decoupling and acceleration of training at each stage without changing the PPO optimization objective and convergence characteristics.
[0006] In a first aspect, a proximal policy optimization training method is provided, applied to the stage load-aware scheduling module of a proximal policy optimization training system. The system further includes a policy and value joint sampling module, a reward and advantage calculation module, and a policy and value network optimization module. The method includes: Acquire and cache the sampling trajectory data generated by the strategy and value joint sampling module, wherein each sampling trajectory data corresponds to a parameter version, and the version numbers of all or part of the parameter versions are the same. The sampled trajectory data is sent to the reward and advantage calculation module to obtain the reward value and advantage value corresponding to each sampled trajectory data, thereby generating training trajectory data; The training trajectory data is filtered based on a preset first filtering strategy to generate training samples. The training samples are sent to the policy and value network optimization module to upgrade the parameter version from the first version to the second version, wherein the parameter version corresponding to each of the sampling trajectory data is no higher than the first version; The parameters corresponding to the second version are sent to the policy and value joint sampling module to update the parameters of the policy and value joint sampling module.
[0007] In one technical solution of the above-mentioned proximal policy optimization training method, the step of sending the sampled trajectory data to the reward and advantage calculation module includes: Send all the sampled trajectory data to the reward and advantage calculation module; or... Based on a preset second filtering strategy, the sampled trajectory data is filtered, and the filtered sampled trajectory data is sent to the reward and advantage calculation module.
[0008] In one technical solution of the above-mentioned proximal policy optimization training method, the second selection strategy includes: The version difference between the parameter version of each of the sampled trajectory data and the first version is obtained, and the sampled trajectory data whose version difference is less than or equal to a preset version filtering threshold is sent to the reward and advantage calculation module.
[0009] In one technical solution of the above-mentioned proximal policy optimization training method, the method further includes: Based on the first version, the parameter version corresponding to each training trajectory data, and the preset version allowable delay threshold, the minimum effective parameter version and effective training trajectory data are determined, wherein the effective training trajectory data are the training trajectory data with a parameter version not lower than the minimum effective parameter version; When the number of valid training trajectory data is greater than or equal to the number of samples required for training, the training samples are generated based on the first screening strategy.
[0010] In one technical solution of the above-mentioned proximal policy optimization training method, the first screening strategy includes at least one of the following strategies: Prioritize selecting valid training trajectory data corresponding to parameter versions with higher version numbers; Based on the parameter version, the effective training trajectory data is deweighted, wherein the higher the version number of the parameter version, the greater the weight of the data to be selected and retained.
[0011] In one technical solution of the above-mentioned near-end policy optimization training method, the policy and value joint sampling module includes one or more policy and value joint sampling nodes, and the policy and value network optimization module includes one or more policy and value network optimization nodes. When the number of effective training trajectory data is greater than or equal to a preset high threshold for the number of training samples, at least one of the following operations is performed: Increase computing resources for the strategy and value network optimization module; Reduce computing resources used for the joint policy and value sampling module; Add one or more strategy and value network optimization nodes; Disable some strategy and value joint sampling nodes; When the number of valid training trajectory data is less than a preset low threshold for the number of training samples, at least one of the following operations is performed: Increase computing resources for the joint policy and value sampling module; Reduce computing resources used for the strategy and value network optimization module; Add one or more policy and value joint sampling nodes; Disable some strategy and value network optimization nodes.
[0012] In a second aspect, a proximal policy optimization training system is provided, comprising: The strategy and value joint sampling module includes one or more strategy and value joint sampling nodes for generating sampling trajectory data; The reward and advantage calculation module is used to calculate the reward value and advantage value corresponding to each of the sampled trajectory data; The strategy and value network optimization module includes one or more strategy and value network optimization nodes, which are used to perform parameter optimization updates. The phased load-aware scheduling module is configured to execute the near-end strategy optimization training method described in any of the above technical solutions.
[0013] In one technical solution of the aforementioned proximal policy optimization training system, the system further includes: a first GPU, a second GPU, a first CPU, and a second CPU, wherein, The strategy and value joint sampling node is deployed on the first GPU, and the number of the first GPU can be one or more. The reward and advantage calculation module is deployed on the first CPU; The strategy and value network optimization node is deployed on the second GPU, and the number of the second GPU can be one or more. The phase load awareness scheduling module is deployed on the second CPU.
[0014] In a third aspect, a storage medium is provided, the storage medium being adapted to store multiple lines of program code, the program code being adapted to be loaded and run by a processor to implement the near-end policy optimization training method described in any of the above technical solutions.
[0015] In a fourth aspect, a vehicle is provided, the vehicle comprising one or more intelligent agents, the intelligent agents being trained based on the proximal policy optimization training method described in any of the above technical solutions, or being trained based on the proximal policy optimization training system described in any of the above technical solutions.
[0016] The above-mentioned technical solutions of this application have at least one or more of the following beneficial effects: By introducing a delayed consistency alignment mechanism, this application achieves effective control over asynchronous sampling trajectory data, reward value, advantage value, and parameter updates, avoiding serious version inconsistencies between training samples and model parameters, ensuring that the parameter update process meets the on-policy constraint requirements of the PPO algorithm, and the asynchronous computation-based PPO training system of this application can also improve the parallel processing capability of the training system and ensure the stability and convergence of the PPO training process. Attached Figure Description
[0017] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application.
[0018] Figure 1 This is a schematic diagram of a near-end policy optimization training system according to an embodiment of this application.
[0019] Figure 2 This is a schematic flowchart of the main steps of a near-end policy optimization training method according to an embodiment of this application.
[0020] Figure 3 This is an embodiment according to this application. Figure 2 A detailed flowchart of step S203. Detailed Implementation
[0021] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.
[0022] In the description of this application, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.
[0023] First, please refer to the appendix. Figure 1 , Figure 1 This is a schematic diagram of a proximal policy optimization training system according to an embodiment of this application. Figure 1 As shown, the near-end policy optimization training system of this application includes: a training data generation module, a policy and value joint sampling module, a reward and advantage calculation module, a policy and value network optimization module, and a stage load-aware scheduling module.
[0024] The following section uses an intelligent driving agent as an example to illustrate the functions of each module in the proximal policy optimization training system of this application, as well as the proximal policy optimization training method. It should be noted that this application does not limit the architecture of the intelligent driving agent. As an example, an autonomous driving agent can be developed independently based on a Vision-Language-Action (VLA) model, or it can be built on top of an open-source autonomous driving model, such as building an intelligent driving agent based on the OpenDriveVLA model.
[0025] The training data generation module is deployed on a CPU or GPU to generate initial training data, such as initial trajectory data and environmental data. As an example, the training data generation module is deployed on a third CPU.
[0026] The policy and value joint sampling module comprises M (M≥1) independent policy and value joint sampling nodes (sampling nodes). Each sampling node has the same model structure for its policy and value networks, and is used to generate sampling trajectory data based on the initial training data. As an example, the M sampling nodes are deployed on a first GPU, which can be one or more GPUs.
[0027] The reward and advantage calculation module is deployed on a CPU or GPU to calculate the reward and advantage values corresponding to each sampled trajectory data. As an example, the reward and advantage calculation module is deployed on the first CPU.
[0028] The policy and value network optimization module includes N (N≥1) policy and value network optimization nodes (optimization nodes). Each optimization node has the same model structure for its policy and value networks, and is used to optimize and update network parameters based on training samples. The N optimization nodes are deployed on a second GPU, and the number of second GPUs can be one or more.
[0029] The stage load-aware scheduling module (scheduling module) can be deployed on CPU or GPU. The scheduling module is the main control module of the near-end policy optimization training system, used to control the data interaction between various modules, perform parameter version management, etc. As an example, the scheduling module is deployed on the second CPU.
[0030] It should be noted that the first GPU and the second GPU can be the same or different. When they are the same GPU, different nodes can run different program processes. Similarly, the first CPU, the second CPU, and the third CPU can be the same or different. When they are the same CPU, different modules can run different program processes.
[0031] Next reading Figure 2 and combined Figure 1 This application describes the training method for optimizing the near-end strategy. Figure 2 This is a flowchart illustrating the main steps of a near-end policy optimization training method according to an embodiment of this application, applied to a stage load-aware scheduling module. The near-end policy optimization training method in this embodiment includes: Step S201: Obtain and cache the sampling trajectory data generated by the strategy and value joint sampling module, wherein each sampling trajectory data corresponds to a parameter version, and the version numbers of each parameter version can be all the same or partially the same; Step S202: Send the sampled trajectory data to the reward and advantage calculation module to obtain the reward value and advantage value corresponding to each sampled trajectory data, and then generate training trajectory data, wherein the training trajectory data includes the sampled trajectory data and the reward value and advantage value corresponding to each sampled trajectory data; Step S203: Based on the preset first filtering strategy, the training trajectory data is filtered to generate training samples; Step S204: Send the training samples to the policy and value network optimization module to upgrade the parameter version from the first version to the second version, wherein the parameter version corresponding to each sampling trajectory data is no higher than the first version; Step S205: Send the model parameters corresponding to the second version to the policy and value joint sampling module to update the model parameters of the policy and value joint sampling module.
[0032] In step S201, the sampled trajectory data typically includes a state sequence, an action sequence, an environmental feedback information sequence, and a corresponding value estimation sequence. Each sampled trajectory data corresponds to a parameter version, and the version numbers of all parameter versions are the same or partially the same. In other words, the sampled trajectory data temporarily stored in the scheduling module can correspond to multiple parameter versions.
[0033] The reason for multiple parameter versions is that at the start of training, the initial model parameters of each sampling node in the joint policy and value sampling module are the same. After training begins, differences in computational resources and trajectory prediction results among the sampling nodes lead to variations in the time required to generate each sample trajectory data. Furthermore, to ensure version consistency of the sample trajectory data, each sampling node typically updates its model parameters only after generating a complete batch of sample trajectory data. This results in different parameter versions of the sample trajectory data generated by different sampling nodes.
[0034] Meanwhile, since the joint sampling module of strategy and value, the reward and advantage calculation module and the strategy and value network optimization module are asynchronous computing structures, the reward and advantage calculation module and the strategy and value network optimization module may sometimes fail to consume all the sampled trajectory data in a timely manner. This will also cause the scheduling module to cache multiple versions of sampled trajectory data with different parameters.
[0035] In step S202, the scheduling module sends all the cached sampling trajectory data to the reward and advantage calculation module, which calculates the reward value and advantage value corresponding to each sampling trajectory data and returns the calculation results to the scheduling module.
[0036] The scheduling module concatenates the sampled trajectory data, along with the corresponding reward and advantage values, to generate training trajectory data. This training trajectory data includes a state sequence, an action sequence, an environmental feedback information sequence, a value estimation sequence, a reward value, and an advantage value.
[0037] In another embodiment, the scheduling module may also filter the sampled trajectory data based on a preset second filtering strategy and send the filtered sampled trajectory data to the reward and advantage calculation module.
[0038] Specifically, the version difference between the parameter version of each sampled trajectory data and the first version (the current parameter version of the strategy and value network optimization module) is obtained, and the sampled trajectory data whose version difference is less than or equal to the preset version filtering threshold is sent to the reward and advantage calculation module.
[0039] As an example, the version numbers of the sampled trajectory data parameters include 3, 4, 5, and 6. The first version has a version number of 6. In this case, the version difference corresponding to version number 3 is 3, the version difference corresponding to version number 4 is 2, the version difference corresponding to version number 5 is 1, and the version difference corresponding to version number 6 is 0. When the version filtering threshold is 2, the sampled trajectory data corresponding to versions 4, 5, and 6 will be sent to the reward and advantage calculation module for calculating reward and advantage values, while the sampled trajectory data corresponding to version number 2 will be discarded.
[0040] By setting a version filtering threshold, trajectories that do not meet the parameter version delay consistency constraint can be directly discarded, avoiding meaningless calculation of reward and advantage values and saving computing power of the reward and advantage calculation module.
[0041] Next reading Figure 3 , combined Figure 3 Explain the specific implementation method of step S203. Figure 3 This is a detailed flowchart illustrating step S203 according to an embodiment of this application.
[0042] In step S2031, the minimum effective parameter version and effective training trajectory data are determined based on the first version, the parameter version corresponding to each training trajectory data, and the preset version allowable delay threshold.
[0043] As an example, the parameter version numbers corresponding to the training trajectory data include 3, 4, 5 and 6. The first version has a version number of 6. When the allowed latency threshold of the version is 2, the lowest effective parameter version number is: the first version number - the allowed latency threshold of the version, that is, the lowest effective parameter version number is . At this time, the training trajectory data with parameter version not lower than the lowest effective parameter version is the effective training trajectory data, that is, the training trajectory data corresponding to version numbers 4, 5 and 6 is the effective training trajectory data.
[0044] In step S2032, it is determined whether the number of valid training trajectory data is greater than or equal to the number of samples required for training, wherein the number of samples required for training is the preset number of samples for each batch, which can be set by those skilled in the art according to the actual situation.
[0045] In step S2033, when the number of valid training trajectory data is greater than or equal to the number of samples required for training, training samples are generated based on the first screening strategy.
[0046] In step S2034, the first screening strategy is to prioritize selecting valid training trajectory data corresponding to parameter versions with higher version numbers. For example, if the required number of training samples is 100, and there are 30 valid training trajectory data corresponding to version number 6, 80 corresponding to version number 5, and 20 corresponding to version number 4, then 30 valid training trajectory data corresponding to version number 6 and 70 corresponding to version number 5 are selected as training samples. If there are 130 valid training trajectory data corresponding to version number 6, then the training samples will only include 100 valid training trajectory data corresponding to version number 6. The selection method can be random or based on the order of trajectory generation time from most recent to oldest.
[0047] In step S2035, the first screening strategy is to reduce the weight of effective training trajectory data based on the parameter version. The higher the version number of the parameter version, the heavier the weight of the data to be screened and retained. In other words, the higher the version number of the parameter version, the higher the pre-assigned weight ratio of the effective training trajectory data in the training samples.
[0048] As an example, the effective training trajectory data includes three version numbers, with weights corresponding to the version numbers from highest to lowest as follows: 50%, 30%, and 20%. When the version numbers of the effective training trajectory data are 4, 5, and 6, it is desired that the effective training trajectory data of version number 6 accounts for 50% of the training samples, the effective training trajectory data of version number 5 accounts for 30%, and the effective training trajectory data of version number 4 accounts for 20%. It should be noted that if the effective training trajectory data of higher versions is insufficient to meet the corresponding weight requirements, the remaining weight will be allocated to the effective training trajectory data of the second highest version number.
[0049] This application introduces a delay-consistent alignment mechanism. Specifically, the scheduling module maintains corresponding parameter versions for sampled trajectory data, reward values, advantage values, and training samples. Within the allowed delay threshold of the version, the training samples and the current parameter version (first version) are aligned. This avoids serious inconsistencies between the training sample parameters and the model parameter versions, while allowing limited delays in parameter versions due to asynchronous execution. It ensures that the parameter update process complies with the on-policy constraints of the PPO algorithm, while also improving the parallel processing capability of the training system and guaranteeing the stability and convergence of the PPO training process.
[0050] In step S204, the scheduling module distributes the training samples to each optimization node of the policy and value network optimization module. At this time, the parameter versions of all optimization nodes are the same, namely the first version. Each optimization node works in parallel, updating the gradient of the policy network parameters based on the objective function of the PPO algorithm, and simultaneously performing regression optimization on the value network parameters. After summing the calculation results of all optimization nodes, the parameter versions of all optimization nodes are synchronously upgraded from the first version to the second version, and the second version model parameters are sent to the scheduling module.
[0051] It should be noted that this application does not limit the algorithm for multiple optimization nodes to perform gradient updates and regression optimization of the network model in parallel. As an example, the circular All Reduce algorithm can be used, and those skilled in the art can also use other parallel algorithms.
[0052] In step S205, after the scheduling module obtains the updated model parameters, it distributes the latest version of the model parameters to the policy and value joint sampling module so that the model parameters of each sampling node of the policy and value joint sampling module can be updated in a timely manner.
[0053] This application achieves decoupling and pipelined collaboration of each training stage through asynchronous communication and buffering mechanisms, thereby avoiding resource idleness or blocking caused by inconsistent computational efficiency in different training stages and improving training speed.
[0054] This application distributes the modules of the near-end policy optimization training system across multiple GPUs and CPUs on computer nodes, further decoupling the modules physically, achieving efficient utilization of distributed computing resources, and significantly improving the overall throughput and system stability of complex reinforcement learning training.
[0055] In this embodiment of the application, the scheduling module is also used to control the computing resources of the relevant GPU / CPU and / or the working status of the relevant nodes according to the number of valid training trajectory data cached.
[0056] When the number of valid training trajectory data is greater than or equal to a preset high threshold for the number of training samples, it indicates a backlog of valid training trajectory data, and / or the policy and value network optimization module is unable to consume valid training trajectory data in a timely manner. At least one of the following operations will be performed: Increase computing resources for the policy and value network optimization module, increase the consumption of effective training trajectory data, and thus improve training efficiency; Reduce computing resources used for the joint policy and value sampling module and reduce the production of effective training trajectory data to achieve a slowdown balance; Add one or more policy and value network optimization nodes to increase the consumption of effective training trajectory data, thereby improving training efficiency; Disabling some strategy and value joint sampling nodes reduces the production of effective training trajectory data, thus achieving a slowdown balance.
[0057] When the number of valid training trajectory data is less than the preset low threshold of the number of training samples, it indicates that the number of valid training trajectory data generated is insufficient, and / or the policy and value network optimization module consumes valid training trajectory data too quickly, and at least one of the following operations is performed: Increase computing resources for the joint policy and value sampling module to produce more effective training trajectory data, thereby improving training efficiency; Reduce computing resources used for the strategy and value network optimization module to achieve a speed-down balance; Add one or more policy and value joint sampling nodes to generate more effective training trajectory data, thereby improving training efficiency; Disabling some strategy and value network optimization nodes will achieve a slowdown balance.
[0058] The scheduling module adjusts software / hardware resources to ensure that the near-end policy optimization training system of this application maintains overall load balance under high parallelism, thereby improving resource utilization and training efficiency.
[0059] It should be noted that the reward calculation method, advantage calculation method, and objective function design used in the PPO algorithm are all well-known technologies and will not be elaborated here.
[0060] Furthermore, this application also provides a storage medium that can be configured to store a program for executing the proximal policy optimization training method of the above-described method embodiments. This program can be loaded and run by a processor to implement the proximal policy optimization training method of the above-described embodiments. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The storage medium can be a storage device device comprising various computer devices. Optionally, in the embodiments of this application, the storage medium is a non-transitory read-write storage medium.
[0061] Furthermore, a vehicle is also provided, comprising one or more intelligent agents trained based on the proximal policy optimization training method described in the above embodiments, or trained based on the proximal policy optimization training system described in the above embodiments. As an example, the vehicle is a new energy vehicle, and the intelligent agent is an intelligent driving agent.
[0062] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.
[0063] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above-described method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0064] The technical solution of this application has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.
Claims
1. A proximal policy optimization training method, characterized in that, A stage load-aware scheduling module is applied to a proximal policy optimization training system. The system further includes a policy and value joint sampling module, a reward and advantage calculation module, and a policy and value network optimization module. The proximal policy optimization training system is applied to an intelligent driving agent. The method includes: Acquire and cache the sampling trajectory data generated by the policy and value joint sampling module, wherein the sampling trajectory data includes state sequence, action sequence, environmental feedback information sequence and corresponding value estimation sequence, and each of the sampling trajectory data corresponds to a parameter version, and the version numbers of all or part of the parameter versions are the same. The sampled trajectory data is sent to the reward and advantage calculation module to obtain the reward value and advantage value corresponding to each sampled trajectory data, thereby generating training trajectory data; The training trajectory data is filtered based on a preset first filtering strategy to generate training samples. The training samples are sent to the policy and value network optimization module to upgrade the parameter version from the first version to the second version, wherein the parameter version corresponding to each of the sampling trajectory data is no higher than the first version; The parameters corresponding to the second version are sent to the policy and value joint sampling module to update the parameters of the policy and value joint sampling module.
2. The near-end policy optimization training method of claim 1, wherein, Sending the sampled trajectory data to the reward and advantage calculation module includes: Send all the sampled trajectory data to the reward and advantage calculation module; or... Based on a preset second filtering strategy, the sampled trajectory data is filtered, and the filtered sampled trajectory data is sent to the reward and advantage calculation module.
3. The near-end policy optimization training method of claim 2, wherein, The second screening strategy includes: The version difference between the parameter version of each of the sampled trajectory data and the first version is obtained, and the sampled trajectory data whose version difference is less than or equal to a preset version filtering threshold is sent to the reward and advantage calculation module.
4. The near-end policy optimization training method according to any one of claims 1 to 3, characterized in that, The method further includes: Based on the first version, the parameter version corresponding to each training trajectory data, and the preset version allowable delay threshold, the minimum effective parameter version and effective training trajectory data are determined, wherein the effective training trajectory data are the training trajectory data with a parameter version not lower than the minimum effective parameter version; When the number of valid training trajectory data is greater than or equal to the number of samples required for training, the training samples are generated based on the first screening strategy.
5. The near-end policy optimization training method of claim 4, wherein, The first screening strategy includes at least one of the following strategies: Prioritize selecting valid training trajectory data corresponding to parameter versions with higher version numbers; Based on the parameter version, the effective training trajectory data is deweighted, wherein the higher the version number of the parameter version, the greater the weight of the data to be selected and retained.
6. The near-end policy optimization training method of claim 4, wherein, The policy and value joint sampling module includes one or more policy and value joint sampling nodes, and the policy and value network optimization module includes one or more policy and value network optimization nodes. When the number of effective training trajectory data is greater than or equal to a preset high threshold for the number of training samples, at least one of the following operations is performed: Increase computing resources for the strategy and value network optimization module; Reduce computing resources used for the joint policy and value sampling module; Add one or more strategy and value network optimization nodes; Disable some strategy and value joint sampling nodes; When the number of valid training trajectory data is less than a preset low threshold for the number of training samples, at least one of the following operations is performed: Increase computing resources for the joint policy and value sampling module; Reduce computing resources used for the strategy and value network optimization module; Add one or more policy and value joint sampling nodes; Disable some strategy and value network optimization nodes.
7. A proximal policy optimization training system, comprising: include: The strategy and value joint sampling module includes one or more strategy and value joint sampling nodes for generating sampling trajectory data; The reward and advantage calculation module is used to calculate the reward value and advantage value corresponding to each of the sampled trajectory data; The strategy and value network optimization module includes one or more strategy and value network optimization nodes, which are used to perform parameter optimization updates. The stage load-aware scheduling module is configured to execute the near-end policy optimization training method as described in claims 1 to 6.
8. The near-end policy optimisation training system of claim 7, wherein, The system further includes: a first GPU, a second GPU, a first CPU, and a second CPU, wherein, The strategy and value joint sampling node is deployed on the first GPU, and the number of the first GPU is one or more. The reward and advantage calculation module is deployed on the first CPU; The strategy and value network optimization nodes are deployed on the second GPU, and the number of the second GPU is one or more. The phase load awareness scheduling module is deployed on the second CPU.
9. A storage medium adapted to store a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the near-end policy optimization training method as described in any one of claims 1 to 6.
10. A vehicle characterized by comprising: include: One or more intelligent agents, which are trained based on the proximal policy optimization training method according to claims 1 to 6, or trained based on the proximal policy optimization training system according to claim 7 or 8.