Vehicle-unmanned aerial vehicle cooperative distribution path optimization method based on deep reinforcement learning

Abstract

The invention discloses a vehicle-unmanned aerial vehicle cooperative distribution path optimization method based on deep reinforcement learning, relates to the technical field of path optimization, and aims to solve the problem of logistics distribution uncertainty caused by customer dynamic demands and traffic flow time-varying characteristics in urban end distribution. The method comprises the following steps: firstly, establishing a dynamic logistics distribution time function by considering changes of vehicle speeds in different traffic flow states; secondly, a vehicle-unmanned aerial vehicle cooperative distribution path optimization model is constructed by taking the distribution timeliness meeting the customer demand as a target; thirdly, designing a reward function with demand expiration punishment according to the dynamic delivery time and the dynamic demand of the customer; and finally, solving the dynamic path optimization model by adopting a deep reinforcement learning algorithm. According to the method, the distribution strategy can be dynamically adjusted, the path planning efficiency and the distribution timeliness are improved, and the overall service level of urban terminal distribution is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of path optimization, and in particular to a vehicle-UAV collaborative delivery path optimization method based on deep reinforcement learning. Background Art

[0002] With the rapid development of e-commerce, the demand for on-demand delivery services continues to grow. The traditional logistics distribution model is inefficient, and drones are seen as a potential solution to this problem due to their high mobility and lack of ground traffic. The use of vehicle-drone collaborative delivery can make up for the lack of drone endurance and load capacity, and achieve more efficient urban logistics distribution.

[0003] The optimization of vehicle-drone collaborative delivery paths belongs to the traveling salesman problem (TSP), which is an NP-Hard problem. Most current studies use exact algorithms or heuristic algorithms to solve it. Exact algorithms are usually only applicable to solving small-scale instances; heuristic algorithms dominate current research, but even the most effective heuristic algorithms may take hours to generate high-quality solutions for large-scale problems. In addition, in urban terminal delivery, there are dynamic and uncertainties in demand and environment, such as the randomness of order quantity and location and time-varying traffic flow, all of which will affect the efficiency and timeliness of terminal delivery. However, most current studies only consider basic scenarios, ignoring the time-varying characteristics of traffic flow and the impact of dynamic demand on path optimization results; and considering the dynamic changes in the environment will further increase the complexity of the problem and reduce the efficiency of the algorithm solution. Therefore, it is necessary to comprehensively consider the impact of dynamic demand and dynamic traffic flow on delivery time, design a more efficient deep reinforcement learning algorithm, dynamically adjust the allocation strategy based on real-time traffic flow data, and optimize the delivery path and time. Summary of the invention

[0004] The present invention discloses a vehicle-UAV collaborative delivery path optimization method based on deep reinforcement learning, which aims to optimize the vehicle-UAV delivery path and time in real time by considering the ever-changing traffic conditions and customer needs, and improve the solution efficiency.

[0005] To achieve the above purpose, the technical solution provided by the present invention is: A vehicle-UAV collaborative delivery path optimization method based on deep reinforcement learning, characterized by comprising the following steps: Step 1: Considering the different driving speeds of vehicles under different traffic flow conditions, a dynamic logistics delivery time function is established to quantify the changes in vehicle-UAV collaborative delivery time under dynamic traffic flow conditions; Step 2: Considering the dynamic delivery time, with the goal of meeting all customer needs with the highest delivery timeliness, establish a vehicle-drone team low-altitude logistics collaborative delivery path optimization model; Step 3: Establish a reward function with demand expiration penalty based on dynamic traffic flow status and customer dynamic demand; Step 4: Use deep reinforcement learning algorithm to solve the dynamic optimization model of vehicle-UAV low-altitude logistics collaborative delivery path.

[0006] In order to optimize the above technical solutions, the specific measures / limitations taken also include: In step 1, the delivery time between two points is defined as a variable, which changes based on the time of day and the traffic flow state. First, the time-varying characteristics of dynamic traffic flow are considered, and the vehicle speeds under three traffic flow states are defined; Smooth traffic: In the late night or early morning hours, the average speed of vehicles on the main roads is 32 km / h, and the average speed of vehicles on the secondary roads is 23 km / h; Congestion status: During peak hours, the average speed of vehicles on main roads is 24 km / h, and the average speed of vehicles on secondary roads is 16 km / h; Medium state: After the peak hours, the average vehicle speed on the main roads is 28km / h, and the average vehicle speed on the secondary roads is 20km / h.

[0007] The Euclidean distance between nodes is divided by the time-varying speed under different traffic flow conditions to establish a dynamic logistics distribution time function. The model captures the fluctuating nature of different traffic flows throughout the day.

[0008] In step 2, a vehicle-drone team low-altitude logistics collaborative delivery path optimization model is established. Considering the dynamic delivery time that changes with time, the objective function is to minimize the total delivery time. The formula is as follows: ; in, T is the set of time periods, K is the set of transportation modes, including vehicles and drones, and A is the set of arcs of the road sections between nodes; Indicated in t Time, from node i To Node j transportation time; is a binary variable, indicating t Whether to use the mode of transportation at the moment k Slave Node i To Node j .

[0009] To adapt to the dynamic delivery environment, constraints cover multiple aspects such as customer service allocation, capacity constraints, access sequence and time windows, ensuring that the model can cope with new demands that change over time.

[0010] In step 3, a reward function with demand expiration penalty is established according to the dynamic traffic flow status and dynamic customer demand.

[0011] In the reinforcement learning framework, the reward function guides the agent's behavior towards the expected result. At each time step, the agent receives a feedback signal that includes not only the main reward from the original function, but also a penalty for overdue requirements. The requirement expiration penalty is defined as follows: For each node in the network, if a demand is overdue (i.e. not met within the specified time window), a penalty of -1 is applied. Reward function with demand overdue penalty The calculation formula is as follows: ; in, C is the reward obtained in the initial function, P is the cumulative penalty per time step, P The calculation formula is as follows: ; in, N represents the set of all nodes, is the indicator function, if the node i If the requirement has expired, it returns 1, otherwise it returns 0.

[0012] In step 4, we design a deep reinforcement learning algorithm by first defining the state space and action space: State Space S At each time step t The multidimensional space that captures the relevant information needed for decision making is defined as follows: ; in, s t Represents the unit time step t Multi-dimensional information within L t Indicates the current position of the vehicle and drone, C t represents the remaining capacity of vehicles and drones, D t Indicates the delivery status of the customer. T t Indicates the traffic conditions of the current road section. E t Indicates the estimated time of arrival. B t Indicates whether the customer demand is overdue; Action Space A Defines the vehicles and drones at each momentt Optional actions include selecting a destination node or maintaining the current position: ; in, and Represents vehicles and drones in t Moment of action.

[0013] Based on the definition of state space and action space, a dynamic demand matrix that changes over time is established. This matrix not only records the demand status of each node, but also updates it according to the movement of vehicles or drones and the passage of time. Before deciding the next action, first check whether the demand on each node has expired. If the demand is expired, the node is removed from the demand matrix or marked as satisfied, ensuring the effective allocation of resources and optimizing path planning.

[0014] The mathematical framework of deep reinforcement learning model is constructed using Markov decision process. The specific steps are as follows: Step 1: The agent observes the current state of the environment, including dynamic traffic flow and dynamic demand with time-varying characteristics; Step 2: The agent takes specific actions based on the current state; Step 3: Use a reward function to measure the value of the agent taking a specific action in the current state; Step 4: Train the agent’s decision-making ability, with the goal of maximizing the cumulative reward, and optimize the agent’s action selection strategy in each state.

[0015] In the process of solving the model, the Graph Attention Network (GAT) is used as an encoder to capture the structural information of the road network and the relationship between nodes, and convert state information such as dynamic traffic flow and dynamic demand into potential representations that are easy to understand for the reinforcement learning model; Using a Transformer-based decoder, based on the latent representation output by the encoder, a self-attention mechanism is used to model the dependencies between nodes, generate the next actions of the vehicle and drone, and make decisions for the agent; The model is trained using techniques such as experience replay and target network update. Based on the collected experience, the agent's strategy and value function are iteratively updated to adapt to the dynamic environment and improve the stability and efficiency of the training process.

[0016] Compared with the prior art, the present invention has the following beneficial effects: In view of the deficiencies in existing research on vehicle-UAV collaborative delivery path optimization, the present invention proposes a deep reinforcement learning method that comprehensively considers dynamic demand and time-varying effects of traffic flow, which is used to solve large-scale dynamic TSP problems. Compared with traditional precise algorithms and heuristic algorithms, the present invention adopts a deep reinforcement learning framework, which can adaptively handle random changes in order quantity and location, and update path planning in real time. At the same time, by integrating real-time traffic flow data, the model can dynamically adjust the allocation strategy, improve path optimization efficiency in complex and changeable urban environments, and shorten delivery time. This method not only significantly improves the solution efficiency, but also enhances the adaptability in actual dynamic scenarios, overcomes the traditional method's reliance on static scenarios and idealized assumptions, and meets the requirements for efficiency and timeliness in urban terminal distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0018] The above contents of the present invention are further described in detail below in the form of embodiments, but this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the above contents of the present invention belong to the scope of the present invention.

[0019] This paper proposes a vehicle-UAV collaborative delivery path optimization method based on deep reinforcement learning. The flow chart is as follows: Figure 1 As shown, the following steps are included: (1) Considering the different vehicle speeds under different traffic flow conditions, a dynamic logistics delivery time function is established to quantify the changes in vehicle-UAV collaborative delivery time under dynamic traffic flow conditions.

[0020] The specific steps in step (1) include: 1.1, considering the time-varying characteristics of dynamic traffic flow, define the vehicle speed under different traffic flow conditions; The delivery time between two points is defined as a variable, which changes with the time of day and the traffic flow state. First, consider the impact of the time-varying characteristics of dynamic traffic flow on vehicle speed, while the speed of the drone is not affected, and define the vehicle speed under three traffic flow states: Smooth state: In the late night or early morning hours, the average speed of vehicles on the main roads is about 32km / h, and the average speed of vehicles on the secondary roads is about 23km / h, indicating that the traffic flow is light and in a smooth state; Congestion status: During peak hours, the average speed of vehicles on main roads is about 24km / h, and the average speed of vehicles on secondary roads is about 16km / h. This significant reduction in speed highlights the serious traffic congestion and affects the efficiency of delivery; Medium state: After the peak period, the average speed of vehicles on the main roads is about 28 km / h, and the average speed of vehicles on the secondary roads is about 20 km / h, indicating that the traffic flow is gradually stabilizing; 1.2, create a time-dependent travel time matrix, divide the Euclidean distance between nodes by the time-varying speed under different traffic flow states, and establish a dynamic logistics delivery time function. The model captures the fluctuating nature of different traffic flows throughout the day.

[0021] (2) Taking into account the dynamic delivery time, with the goal of meeting all customer needs with the highest delivery timeliness, a vehicle-UAV fleet low-altitude logistics collaborative delivery path optimization model is established.

[0022] In step (2), a vehicle-drone fleet low-altitude logistics collaborative delivery path optimization model is established. Considering the dynamic delivery time that changes with time, the objective function is to minimize the total delivery time. The formula is as follows: ; in, T is the set of time periods, K is the set of transportation modes, including vehicles and drones, and A is the set of arcs of the road sections between nodes; Indicated in t Time, from node i To Node j transportation time; is a binary variable, indicating t Whether to use the mode of transportation at the moment k Slave Node i To Node j .

[0023] To adapt to the dynamic delivery environment, the constraints ensure that each customer is served at most once in all time periods up to the current time period, reflecting the accumulation of new demand over time. In addition, the remaining constraints also cover customer service allocation, capacity constraints, access order, and time windows.

[0024] (3) Based on the dynamic traffic flow status and dynamic customer demand, a reward function with demand expiration penalty is established.

[0025] In step (3), in the reinforcement learning framework, the reward function guides the agent's behavior towards the expected result. At each time step, the agent receives a feedback signal, which includes not only the main reward from the original function, but also a penalty for expired requirements. This additional penalty function is intended to prevent the occurrence of expired requirements, which is not conducive to the overall performance of the system. The penalty for expired requirements is defined as follows: For each node in the network, if a demand is overdue (i.e. not met within the specified time window), a penalty of -1 is applied. Reward function with demand overdue penalty The calculation formula is as follows: ; in, C is the reward obtained in the initial function, P is the cumulative penalty per time step, P The calculation formula is as follows: ; in, N represents the set of all nodes, is the indicator function, if the node i If the requirement has expired, it returns 1, otherwise it returns 0.

[0026] The setting of the reward function ensures that the agent is incentivized to meet demands in a timely manner, so that its goals are consistent with the operational goals of minimizing demand overdue and improving service reliability.

[0027] (4) A deep reinforcement learning algorithm is used to solve the dynamic optimization model of vehicle-UAV low-altitude logistics collaborative delivery paths.

[0028] The specific steps in step (4) include: 4.1, define state space and action space: State Space S At each time step t The multidimensional space that captures the relevant information needed for decision making is defined as follows: ; in, s t Represents the unit time step t Multi-dimensional information within L t Indicates the current position of the vehicle and drone, C t represents the remaining capacity of vehicles and drones, D t Indicates the delivery status of the customer. T t Indicates the traffic conditions of the current road section. E t Indicates the estimated time of arrival. B t Indicates whether the customer demand is overdue; Action Space A Defines the vehicles and drones at each moment tOptional actions include selecting a destination node or maintaining the current position: ; in, and Represents vehicles and drones in t Moment of action.

[0029] 4.2, establish a dynamic demand matrix that changes over time to provide real-time feedback for path optimization.

[0030] Based on the definition of state space and action space, a dynamic demand matrix that changes over time is established. This matrix not only records the demand status of each node, but also updates it according to the movement of vehicles or drones and the passage of time. Before deciding the next action, first check whether the demand on each node has expired. If the demand is expired, the node is removed from the demand matrix or marked as satisfied, ensuring the effective allocation of resources and optimizing path planning.

[0031] 4.3, the mathematical framework of deep reinforcement learning model is constructed using Markov decision process. The specific steps are as follows: Step 1: The agent observes the current state of the environment, including dynamic traffic flow and dynamic demand with time-varying characteristics; Step 2: The agent takes specific actions based on the current state; Step 3: Use a reward function to measure the value of the agent taking a specific action in the current state; Step 4: Train the agent’s decision-making ability, with the goal of maximizing the cumulative reward, and optimize the agent’s action selection strategy in each state.

[0032] 4.4, an encoder-decoder architecture is used to process the agent’s decision-making, and experience replay and target network update techniques are used to improve the stability of the model training process.

[0033] In order to effectively handle the complexity of the TSP problem, an encoder-decoder architecture is adopted for the agent's policy and value functions. The encoder-decoder architecture allows the agent to process the input state, which includes the structure of the graph, the time interval-based traffic flow state, and dynamic demand information, and generate appropriate actions for vehicles and drones.

[0034] In the process of solving the model, the Graph Attention Network (GAT) is used as an encoder to capture the structural information of the road network and the relationship between nodes, and convert state information such as dynamic traffic flow and dynamic demand into potential representations that are easy to understand for the reinforcement learning model; Using a Transformer-based decoder, based on the latent representation output by the encoder, a self-attention mechanism is used to model the dependencies between nodes, generate the next actions of the vehicle and drone, and make decisions for the agent; The model is trained using techniques such as experience replay and target network update. Based on the collected experience, the agent's strategy and value function are iteratively updated to adapt to the dynamic environment and improve the stability and efficiency of the training process.

[0035] The technical solution of the present invention is further illustrated by a specific embodiment below.

[0036] The Shanghai e-commerce parcel delivery system is selected as the research object. Shanghai has 124 warehouse nodes responsible for handling all e-commerce parcels, and these nodes have non-overlapping areas. Delivery personnel are dispatched from these nodes starting at 8 am with the goal of delivering parcels within the specified time. Assuming that all parcels are delivered to 9,214 delivery points, delivery to customers is abstracted as transporting the parcel to the nearest delivery point and further processing at the delivery point.

[0037] (1) Impact of dynamic demand changes on delivery time Two different scenarios were set up: one was to train the model based on static demand characteristics, and the other was to train the model based on fluctuating demand characteristics. The two models solved the delivery time under three conditions: basic environment (no demand change), normal demand change, and specific demand change (3-5 demand points were cancelled or added). The number of demand points was 50. The results are shown in Table 1 below, in minutes: Table 1 The impact of demand changes on delivery performance

[0038] In the model trained with static demand, the introduction of normal demand changes led to a significant improvement in delivery time, which was 72.58% higher than the basic environment; when the impact of specific demand changes (cancellation or increase) was taken into account, the delivery time increased by 10.20%. In the model trained with fluctuating demand, the introduction of normal demand changes and specific demand changes led to an increase in delivery time of 30.90% and 9.84%, respectively, which were both lower than the model trained with static demand. The results show that the model considering dynamic demand proposed in this paper has a good ability to adapt to demand fluctuations and can effectively adapt to and optimize delivery time.

[0039] (2) Impact of traffic time variability on delivery time Four delivery models are studied, and each model is trained based on different traffic conditions: all-day (including peak and non-peak), peak, non-peak, and event-driven congestion. The number of demand points is 50. The delivery time comparison of different models under different traffic conditions is shown in Table 2 below, in minutes: Table 2 Comparison of delivery time of different models under different traffic conditions

[0040] The all-day traffic model showed relatively consistent performance across different traffic conditions, with percentage differences ranging from -3.81% to 4.10%, indicating that the model was able to handle a variety of traffic scenarios well. The peak-hour traffic model performed best in the peak environment, however, it encountered difficulties in the congested environment, with a 9.48% increase in delivery time, indicating that event-driven congestion posed a significant challenge to the model. The off-peak traffic model performed best in the off-peak hours, while in the congested environment, the off-peak traffic model showed a significant improvement of -5.48%, indicating that the off-peak hours may provide some resilience against event-driven congestion. The event-driven congestion model showed its advantage in the congested environment, with a significant reduction in delivery time of -11.43% compared to the all-day environment. Among all models, except for the event-driven congestion model, the event environment posed the greatest challenge. This highlights the significant impact of event-driven congestion on delivery times and the importance of developing dedicated strategies to mitigate its impact.

[0041] (3) Path planning results Path planning is performed for two cases with different demand scopes.

[0042] The optimal paths shown in Case 1 and Case 2 demonstrate the algorithm’s ability to generate efficient and tailored solutions based on the specific characteristics and constraints of each scenario. The results of this case study highlight the significant potential of reinforcement learning in enhancing the coordination and optimization of truck-drone delivery systems. By enabling dynamic route planning and adaptive decision making, this approach can effectively minimize delivery time, reduce operating costs, and improve overall logistics performance in practical applications. The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession, without departing from the scope of the technical solution of the present invention, according to the technical essence of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. A vehicle-UAV collaborative delivery path optimization method based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Considering the different driving speeds of vehicles under different traffic flow conditions, a dynamic logistics delivery time function is established to quantify the changes in vehicle-UAV collaborative delivery time under dynamic traffic flow conditions; Step 2: Considering the dynamic delivery time, with the goal of meeting all customer needs with the highest delivery timeliness, establish a vehicle-drone team low-altitude logistics collaborative delivery path optimization model; Step 3: Establish a reward function with demand expiration penalty based on dynamic traffic flow status and customer dynamic demand; Step 4: Based on the reward function, a deep reinforcement learning algorithm is used to solve the vehicle-UAV low-altitude logistics collaborative distribution path optimization model.

2. According to the method of optimizing the vehicle-UAV collaborative delivery path based on deep reinforcement learning in claim 1, it is characterized in that: The specific process of step 1 is as follows: Step 1.1: Consider the time-varying characteristics of dynamic traffic flow and define the vehicle speed under different traffic flow conditions; Step 1.2: Divide the Euclidean distance by the time-varying speed under different traffic flow conditions to establish a dynamic logistics delivery time function.

3. The vehicle-UAV collaborative delivery path optimization method based on deep reinforcement learning according to claim 2 is characterized in that: In step 1.1, three vehicle speeds under different traffic flow conditions are defined: Smooth condition: The average speed of vehicles on the main roads is 32km / h, and the average speed of vehicles on the secondary roads is 23km / h; Congestion status: The average speed of vehicles on the main roads is 24km / h, and the average speed of vehicles on the secondary roads is 16km / h; Medium state: The average vehicle speed on the main roads is 28km / h, and the average vehicle speed on the secondary roads is 20km / h.

4. The vehicle-UAV collaborative delivery path optimization method based on deep reinforcement learning according to claim 1 is characterized in that: In step 2, the specific model is as follows: Considering the dynamic delivery time that changes with time, the objective function is to minimize the total delivery time. The formula is as follows: ； in, T is the set of time periods, K is the set of transportation modes, including vehicles and drones, and A is the set of arcs of the road sections between nodes; Indicated in t Time, from node i To Node j transportation time; is a binary variable, indicating t Whether to use the mode of transportation at the moment k Slave Node i To Node j .

5. The vehicle-UAV collaborative delivery path optimization method based on deep reinforcement learning according to claim 1 is characterized in that: The reward function with demand expiration penalty in step 3 is set as follows: To ensure that the agent in the reinforcement learning framework meets the requirements in time, a requirement expiration penalty is defined: for each node in the network, if the requirement is expired, that is, it is not met within the specified time window, a penalty of -1 is used; Reward function with demand expiration penalty The calculation formula is as follows: ； in, C is the reward obtained in the initial function, P is the cumulative penalty per time step, P The calculation formula is as follows: ； in, N represents the set of all nodes, is the indicator function, if the node i If the requirement has expired, it returns 1, otherwise it returns 0.

6. The vehicle-UAV collaborative delivery path optimization method based on deep reinforcement learning according to claim 1 is characterized in that: The specific process of step 4 is as follows: Step 4.1: Define the state space and action space of the deep reinforcement learning model; Step 4.2: Establish a dynamic demand matrix that changes over time to provide real-time feedback for route optimization; Step 4.3: Use Markov decision process to build a mathematical framework for deep reinforcement learning model; Step 4.4: Use the encoder-decoder architecture to process the agent’s decision making, and use experience replay and target network update techniques to improve the stability of the model training process.

7. The vehicle-UAV collaborative delivery path optimization method based on deep reinforcement learning according to claim 6 is characterized in that: In step 4.1: State Space S At each time step t The multidimensional space that captures the relevant information needed for decision making is defined as follows: ； in, s t Represents the unit time step t Multi-dimensional information within L t Indicates the current position of the vehicle and drone, C t represents the remaining capacity of vehicles and drones, D t Indicates the delivery status of the customer. T t Indicates the traffic conditions of the current road section. E t Indicates the estimated time of arrival. B t Indicates whether the customer demand is overdue; Action Space A Defines the vehicles and drones at each moment t Optional actions include selecting a destination node or maintaining the current position: ； in, and Represents vehicles and drones in t Moment of action.

8. The vehicle-UAV collaborative delivery path optimization method based on deep reinforcement learning according to claim 6 is characterized in that: In step 4.2: Establish a dynamic demand matrix that changes over time. This matrix not only records the demand status of each node, but also updates it according to the movement of vehicles or drones and the passage of time. Before deciding on the next action, first check whether the demand on each node has expired. If a demand is overdue, the node is removed from the demand matrix or marked as satisfied, ensuring efficient allocation of resources and optimized path planning.

9. The method for optimizing vehicle-UAV collaborative delivery paths based on deep reinforcement learning according to claim 6, characterized in that: The specific process of the Markov decision process in step 4.3 is as follows: Step 1: The agent observes the current environment state, including dynamic traffic flow and dynamic demand information with time-varying characteristics; Step 2: The agent takes specific actions based on the current state; Step 3: Use a reward function to measure the value of the agent taking a specific action in the current state; Step 4: Train the agent’s decision-making ability, with the goal of maximizing the cumulative reward, and optimize the agent’s action selection strategy in each state.

10. The vehicle-UAV collaborative delivery path optimization method based on deep reinforcement learning according to claim 6 is characterized in that: In step 4.4: Use graph attention network as encoder to convert dynamic traffic flow and dynamic demand state information into latent representation that is easy to understand for reinforcement learning model; Use a Transformer-based decoder to generate the next actions of the vehicle and drone based on the latent representation output by the encoder to make decisions for the agent; The model is trained using experience replay and target network update technology to collect experience, update strategies and value functions for the agent, adapt to dynamic environments, and improve the stability and efficiency of the training process.

Images