Dynamic path planning optimization method based on reinforcement learning

Abstract

The invention belongs to the technical field of path planning, and particularly relates to a dynamic path planning optimization method based on reinforcement learning, which comprises the following specific steps: S1, environment perception: acquiring environment information in real time, including the position, speed and direction of an intelligent agent, the position of an obstacle in the surrounding environment, the motion state of a dynamic object and target position information; and S2, environment pre-sensing and dynamic prediction: after original environment information is collected in environment sensing, analyzing and predicting the future change trend of the environment, and then transmitting the processed information. According to the invention, the intelligent agent can sense the environment state change in real time, the intelligent agent can timely adapt to the dynamic conditions of traffic flow change, temporary road control and the like by continuously interacting with the environment and adjusting the strategy according to the reward signal, for example, in urban traffic, sudden congestion road sections can be avoided in real time, the optimal path can be re-planned in real time, and the traffic efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of path planning, and in particular to a dynamic path planning optimization method based on reinforcement learning. Background Art

[0002] In today's intelligent transportation, logistics distribution, robot navigation and many other fields, path planning is a critical and basic task. As application scenarios become increasingly complex and diverse, extremely high requirements are placed on the real-time, high efficiency and adaptability of path planning. Traditional path planning methods, such as the A* algorithm and the Dijkstra algorithm, can find the optimal path from the starting point to the end point relatively well when facing simple and static environments. However, when in a dynamically changing environment, such as real-time road conditions in urban traffic, temporary road control caused by emergencies, or new goods are constantly piled up when handling robots are working in logistics warehouses, these traditional methods expose obvious limitations. Specifically, there are the following problems:

[0003] 1. Dynamic environmental adaptation: In real-world applications, the environment is often in a state of dynamic change. For example, in the field of intelligent transportation, urban traffic flow can vary significantly depending on factors such as peak hours, weekdays versus weekends, and emergencies (such as traffic accidents and road construction). Traditional path planning algorithms struggle to perceive and adapt to these dynamic changes in real time. As a result, the planned path may no longer be optimal due to changing road conditions within a short period of time, and may even lead to congestion, significantly reducing traffic efficiency.

[0004] 2. High-dimensional state space processing issues: When complex environmental factors are taken into account, the state space becomes extremely large and complex. For example, in a large logistics warehouse, a handling robot must not only consider its own position and the location of the target cargo, but also the positions of numerous shelves within the warehouse, the dynamic positions of other handling robots, and the traffic conditions in different aisles. Such a high-dimensional state space causes the computational complexity of traditional algorithms to increase exponentially, making it difficult to find an effective path planning solution within a limited time, seriously affecting the real-time and practicality of the system.

[0005] 3. Real-time requirements: Many practical application scenarios, such as autonomous vehicles requiring real-time route planning to respond to emergencies, or emergency rescue vehicles needing to plan the optimal route to the rescue site in the shortest possible time, place strict demands on the real-time performance of path planning systems. Traditional path planning methods, due to their cumbersome and time-consuming calculations in complex environments, often fail to meet the real-time requirements of these scenarios, potentially leading to serious consequences.

[0006] 4. Local Optima Trap: Traditional path planning algorithms are prone to getting stuck in local optimal solutions in complex environments. For example, in a scenario with multiple similar path options, the algorithm may select a seemingly optimal path based solely on current local information, even though a more optimal path exists from a global perspective. Once trapped in a local optimal solution, the agent may continue to follow a path that is not the global optimal solution, failing to find the true optimal solution and reducing the performance and efficiency of the entire system.

[0007] 5. Prior knowledge dependency: Traditional path planning algorithms typically require a significant amount of prior knowledge, such as detailed map information, road traffic rules, and historical traffic flow data. However, in some emerging or complex scenarios, this prior knowledge may be incomplete or difficult to obtain. For example, in some temporary venues, a transport robot may not have readily available accurate maps and detailed site information; or in some sudden emergency rescue scenarios, due to the urgency of the situation, it is impossible to obtain comprehensive and accurate environmental information in advance. In these situations, the application of traditional algorithms is greatly limited. However, a path planning system based on reinforcement learning can reduce reliance on prior knowledge through real-time interactive learning between the agent and the environment, making it better adaptable to such scenarios.

[0008] Based on the above, a dynamic path planning optimization method based on reinforcement learning is invented. Summary of the Invention

[0009] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions:

[0010] The dynamic path planning optimization method based on reinforcement learning includes the following specific steps:

[0011] S1, environmental perception: real-time acquisition of environmental information, including the agent's own position, speed, direction, as well as the location of obstacles in the surrounding environment, the motion state of dynamic objects, and target location information;

[0012] S2, environmental pre-perception and dynamic prediction: After environmental perception collects raw environmental information, it first analyzes and predicts future environmental change trends, and then transmits the processed information;

[0013] S3, state encoding: Encodes the information transmitted by environmental pre-perception and dynamic prediction and converts it into a state vector suitable for input of reinforcement learning algorithm;

[0014] S4, reinforcement learning: selects appropriate actions based on the current state vector and learns the optimal path planning strategy through interaction with the environment;

[0015] S5, action decision: Based on the action probability distribution or action value output by reinforcement learning, combined with a certain exploration-exploitation strategy, the actual action performed by the intelligent agent is determined;

[0016] S6, path generation and optimization module: Generates the final path planning result based on a series of actions selected by the intelligent agent during its interaction with the environment; at the same time, optimizes the generated path to improve the feasibility and efficiency of the path; and during the optimization process, it can use the path optimization algorithm to perform local or global optimization adjustments on the path generated by reinforcement learning to ensure that the path is as close to the optimal solution as possible while meeting real-time requirements.

[0017] As a preferred solution of the dynamic path planning optimization method based on reinforcement learning described in the present invention, the specific steps of S2 are as follows:

[0018] S21, multi-source data collection: collecting environmental data from multiple channels;

[0019] S22, data preprocessing: cleaning, conversion and standardization of the collected raw data;

[0020] S23, Feature Extraction and Fusion: Use data mining and machine learning algorithms to extract key features from preprocessed data. Then, different types of features are fused to form a feature vector that can comprehensively characterize the environmental state and change trends.

[0021] S24, Prediction Model Construction and Training: Select an appropriate prediction model based on the specific prediction task and data characteristics. Then, train the model using historical and real-time data to optimize model parameters so that the model can accurately capture the patterns and trends of environmental changes. During the training process, use cross-validation to evaluate model performance, and adjust the model structure and parameters based on the evaluation results to improve prediction accuracy.

[0022] S25, dynamic environment prediction: input the real-time data after preprocessing and feature extraction into the trained prediction model to predict the state changes of the environment in the future;

[0023] S26, Verification and Correction of Prediction Results: Verify the prediction results and evaluate their accuracy by comparing them with actual observation data. If a significant deviation between the prediction results and the actual situation is found, analyze the cause of the deviation and take appropriate corrective measures.

[0024] S27, prediction information output: output the verified and corrected prediction result to the state code.

[0025] As a preferred solution of the dynamic path planning optimization method based on reinforcement learning described in the present invention, wherein: the environment-related data in S21 includes sensor data, historical data, and external data obtained in real time by environmental perception; the cleaning in S22 is to remove noise, outliers and duplicate data; the conversion is to convert data in different formats into a unified format for subsequent analysis; and the standardization is to standardize the numerical data so that they are in the same scale range.

[0026] As a preferred solution of the dynamic path planning optimization method based on reinforcement learning described in the present invention, the prediction content in S25 includes the movement trajectory of the obstacle, the changing trend of traffic flow, and the state change of the target position.

[0027] As a preferred solution of the dynamic path planning optimization method based on reinforcement learning described in the present invention, the specific steps of S4 are as follows:

[0028] S41, Initialization: Initializes the agent's policy network and value network, setting the network structure, parameters, and hyperparameters. It also creates an experience replay buffer to store experience data from the agent's interaction with the environment. It also defines the agent's action space and state space, clarifying the actions the agent can take and the environmental state information it can perceive in the path planning scenario.

[0029] S42, state perception: Receive the current environment state vector from the state encoder. The vector is the result of encoding the original environment information and contains the key information required by the agent during path planning;

[0030] S43, action selection: Based on the current state, the agent outputs the action probability distribution through the policy network, or calculates the value of each action through the value network, and then combines the exploration-exploitation strategy to determine the actual action to be performed;

[0031] S44, Environment Interaction: The agent performs selected actions and interacts with the environment, so that the environment changes state according to the agent's actions, resulting in a new environment state.

[0032] S45, reward feedback: The environment gives the agent a reward signal based on the new state after the agent's action according to a pre-set reward mechanism; the reward setting is closely centered around the path planning goal;

[0033] S46, Experience Storage: The state, action, reward, and next state information of this interaction process are stored in the experience replay buffer. The experience data provides rich samples for subsequent algorithm training. By randomly sampling samples for training, the temporal correlation between data can be broken, which improves the stability and convergence speed of the algorithm.

[0034] S47, Strategy Update: Randomly extract a batch of experience data from the experience replay buffer, or directly use the current state-action-reward-next-state sequence to calculate the loss function according to the selected reinforcement learning algorithm; the loss function is used to measure the gap between the current strategy and the optimal strategy. Through the backpropagation algorithm, the loss error is transferred to the policy network and value network, the network parameters are updated, and the strategy of the agent is gradually optimized, so that the agent can make better action choices in subsequent path planning to obtain greater long-term cumulative rewards;

[0035] S48, continuous iteration: Repeat the above steps to allow the agent to continue learning through continuous interaction with the environment, and gradually learn the strategy that can plan the optimal path in a dynamic environment; as the training progresses, the agent's path planning ability continues to improve, and it can better adapt to environmental changes and find efficient path solutions.

[0036] As a preferred solution of the dynamic path planning optimization method based on reinforcement learning described in the present invention, the specific steps of S5 are as follows:

[0037] S51, receiving input data: receiving action probability distribution or action value information from reinforcement learning;

[0038] S52, generate a random number: To implement the exploration-exploitation strategy, generate a random number between 0 and 1. The random number will be used to determine whether the agent is exploring or exploiting existing knowledge in the current state;

[0039] S53, exploration-exploitation judgment: Compare the generated random number with the preset exploration probability:

[0040] Exploration phase: If the random number is less than the exploration probability, the agent enters the exploration mode. At this time, an action is randomly selected from the action space. This action may not be the currently known optimal action, but it helps the agent discover new path planning strategies and potentially better paths.

[0041] Exploitation phase: If the random number is greater than or equal to the exploration probability, the agent enters the exploitation mode; based on the action probability distribution or action value output by the reinforcement learning module, the agent selects the action with the highest probability or the action with the greatest value;

[0042] S54, action screening and adjustment: if the selected action may cause a collision with other objects or hinder its normal operation, the action is adjusted according to certain rules and a suitable action is reselected;

[0043] S55, output execution action: output the finalized action to the agent's actuator, so that the agent can perform the action in the environment, thereby completing the interaction with the environment and providing a data basis for subsequent state updates, reward feedback and strategy optimization.

[0044] As a preferred solution of the dynamic path planning optimization method based on reinforcement learning described in the present invention, the specific steps of S6 are as follows:

[0045] S61, Action Sequence Arrangement: Receives the actions selected by the agent during its interaction with the environment, organizes and integrates the actions in chronological order, and clarifies the action trajectory taken by the agent from the starting point to the current state;

[0046] S62, Preliminary Path Generation: Based on the action sequence and the state transition of the agent in the environment, the action sequence is converted into a sequence of path points. Combined with the environment map or spatial information, the specific position of the agent in space after each action is determined, and the position points are connected in sequence to form a preliminary path planning result.

[0047] S63, remove redundant path points: Analyze the path point sequence through an algorithm to identify and remove redundant points that have little impact on the overall path direction and length;

[0048] S64, path smoothing: Use interpolation or filtering algorithms to optimize the path curve to make the path smoother and smoother, avoid sharp turns in the path, and reduce energy consumption and mechanical wear during the movement of the intelligent body;

[0049] S65, local optimization adjustment: use the classic path optimization algorithm to perform local optimization on the path;

[0050] S66, Global Optimization Evaluation: Evaluate and optimize the path from a holistic perspective to determine whether the current path is globally optimal. If a better solution exists, adjust and optimize the path globally using a global search algorithm to ensure that the path is as close to the global optimal solution as possible while meeting real-time requirements.

[0051] S67, path result output: The final path planning result after optimization is output for use by the intelligent agent actuator or feedback to the user. The output path result can be a sequence of coordinate points, route markings on a map, or an instruction set that conforms to a specific format, so that the intelligent agent can move according to the planned path or guide the user to complete the journey.

[0052] Compared with existing technologies:

[0053] Dynamic environment adaptation problem: The present invention enables the intelligent agent to perceive changes in environmental status in real time. By continuously interacting with the environment, the intelligent agent adjusts its strategy based on reward signals and promptly adapts to dynamic situations such as changes in traffic flow and temporary road controls. For example, in urban traffic, it can avoid sudden congested sections in real time and re-plan the optimal route, greatly improving traffic efficiency.

[0054] High-dimensional state space processing problem: This invention adopts a suitable encoding method to convert the original information into a low-dimensional state vector, combined with a deep reinforcement learning algorithm to effectively process high-dimensional information. In large logistics warehouses, it can comprehensively consider multiple factors and quickly plan a reasonable path.

[0055] Real-time requirement issue: The present invention improves the algorithm convergence speed and operating efficiency by optimizing the algorithm. It can quickly provide path planning solutions in scenarios such as autonomous driving responding to emergencies and emergency rescue vehicle path planning, meeting strict real-time requirements.

[0056] Local optimal trap problem: This invention uses the exploration-exploitation strategy of reinforcement learning. During the training process, the intelligent agent explores new actions with a certain probability to avoid being limited to the local optimal path, thereby finding the global optimal solution and improving the overall performance and efficiency of the system.

[0057] Prior knowledge dependency: This invention reduces reliance on prior knowledge through real-time interactive learning between the intelligent agent and the environment, and can effectively plan paths in scenarios where there is a lack of complete prior knowledge, such as temporary construction sites and emergency rescue. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below.

[0059] The present invention provides a dynamic path planning optimization method based on reinforcement learning, which includes the following specific steps:

[0060] S1, environmental perception: real-time acquisition of environmental information, including the agent's own position, speed, direction, as well as the location of obstacles in the surrounding environment, the motion state of dynamic objects, and target location information;

[0061] S2, environmental pre-perception and dynamic prediction: After environmental perception collects raw environmental information, it first analyzes and predicts future environmental change trends, and then transmits the processed information;

[0062] The specific steps of S2 are as follows:

[0063] S21, multi-source data collection: collect environmental data from multiple channels; environmental data includes sensor data acquired in real time by environmental perception, historical data, and external data; cleaning in S22 is to remove noise, outliers, and duplicate data; conversion is to convert data in different formats into a unified format to facilitate subsequent analysis; standardization is to standardize numerical data to bring them into the same scale range;

[0064] S22, data preprocessing: cleaning, conversion and standardization of the collected raw data;

[0065] S23, Feature Extraction and Fusion: Use data mining and machine learning algorithms to extract key features from preprocessed data. For image data, convolutional neural networks are used to extract spatial features. For time series data, long short-term memory networks (LSTMs) or Transformers are used to extract temporal features. These different types of features are then fused to form a feature vector that comprehensively characterizes the state of the environment and its changing trends.

[0066] S24, Prediction Model Construction and Training: Based on the specific prediction task and data characteristics, select an appropriate prediction model, such as a time series prediction model based on deep learning (such as LSTM, GRU, Transformer) or a probabilistic prediction model (such as Gaussian process regression and Markov chain model). Then, use historical data and real-time data to train the model to optimize model parameters so that the model can accurately capture the patterns and trends of environmental changes. During the training process, use cross-validation methods to evaluate the model's performance, and adjust the model structure and parameters based on the evaluation results to improve the accuracy of the prediction.

[0067] S25, Dynamic Environment Prediction: Inputting pre-processed and feature-extracted real-time data into a trained prediction model to predict future environmental changes. This includes the movement trajectory of obstacles, traffic flow trends, and target location status changes.

[0068] S26, Verification and Correction of Prediction Results: Verify the prediction results by comparing them with actual observation data to evaluate the accuracy of the prediction. If a large deviation between the prediction results and the actual situation is found, analyze the cause of the deviation and take appropriate corrective measures. The model can be retrained, model parameters can be adjusted, or other prediction methods can be combined to correct the results to ensure the reliability of the prediction results.

[0069] S27, output of prediction information: The verified and corrected prediction results are output to the state code. The output prediction information is the probability distribution of future environmental states, a set of possible state scenarios, or the predicted values ​​of key environmental parameters. This information is combined with the current environmental state information to provide a more comprehensive and forward-looking basis for the agent's decision-making, helping the agent to plan paths in advance and respond to environmental changes.

[0070] S3, state encoding: Encodes the information transmitted by environmental pre-perception and dynamic prediction and converts it into a state vector suitable for input of reinforcement learning algorithm;

[0071] S4, reinforcement learning: selects appropriate actions based on the current state vector and learns the optimal path planning strategy through interaction with the environment;

[0072] The specific steps of S4 are as follows:

[0073] S41, Initialization: Initializes the agent's policy network and value network, setting the network structure, parameters, and hyperparameters. It also creates an experience replay buffer to store experience data from the agent's interaction with the environment. It also defines the agent's action space and state space, clarifying the actions the agent can take and the environmental state information it can perceive in the path planning scenario.

[0074] S42, state perception: Receive the current environment state vector from the state encoder. The vector is the result of encoding the original environment information and contains the key information required by the agent during path planning;

[0075] S43, action selection: Based on the current state, the agent outputs the action probability distribution through the policy network, or calculates the value of each action through the value network, and then combines the exploration-exploitation strategy to determine the actual action to be performed;

[0076] S44, Environmental Interaction: The agent performs selected actions and interacts with the environment, causing the environment to transition state based on the agent's actions, resulting in a new state. In a path planning scenario, if the agent selects the "forward" action, its position will change, and the state of the surrounding environment (such as distance to obstacles and proximity to the target) will also change accordingly.

[0077] S45, Reward Feedback: Based on the agent's new state after an action, the environment provides the agent with a reward signal according to a pre-defined reward mechanism. Rewards are closely aligned with the path planning objectives, for example, positive rewards are given for moving toward the goal, negative rewards are given for approaching obstacles or moving away from the goal, and high positive rewards are given for reaching the target. This reward signal allows the agent to perceive the quality of its actions, providing guidance for learning the optimal strategy.

[0078] S46, Experience Storage: The state, action, reward, and next state information of this interaction process are stored in the experience replay buffer. The experience data provides rich samples for subsequent algorithm training. By randomly sampling samples for training, the temporal correlation between data can be broken, which improves the stability and convergence speed of the algorithm.

[0079] S47, Strategy Update: Randomly extract a batch of experience data from the experience replay buffer, or directly use the current state-action-reward-next-state sequence to calculate the loss function according to the selected reinforcement learning algorithm; the loss function is used to measure the gap between the current strategy and the optimal strategy. Through the backpropagation algorithm, the loss error is transferred to the policy network and value network, the network parameters are updated, and the strategy of the agent is gradually optimized, so that the agent can make better action choices in subsequent path planning to obtain greater long-term cumulative rewards;

[0080] S48, Continuous Iteration: Repeat the above steps to allow the agent to continuously learn through continuous interaction with the environment, gradually learning the strategy that can plan the optimal path in a dynamic environment. As training progresses, the agent's path planning ability continues to improve, and it can better adapt to environmental changes and find efficient path solutions.

[0081] S5, action decision: Based on the action probability distribution or action value output by reinforcement learning, combined with a certain exploration-exploitation strategy, the agent determines the actual action to be performed. In the exploration phase, the agent randomly selects actions with a certain probability to discover new and potentially better paths. In the exploitation phase, the agent selects the action currently considered optimal to obtain the maximum known reward. As learning progresses, the ratio of exploration and exploitation is gradually adjusted, allowing the agent to fully explore the environment while efficiently utilizing the learned knowledge.

[0082] The specific steps of S5 are as follows:

[0083] S51, receiving input data: receiving action probability distribution or action value information from reinforcement learning; if a value-based reinforcement learning algorithm is used, the received value score corresponding to each possible action is received, and a higher score indicates a higher expected long-term cumulative reward for the action in the current state; if a policy-based reinforcement learning algorithm is used, the received probability distribution of each action is received, and a higher probability indicates a higher probability that the agent will choose that action;

[0084] S52, generate a random number: To implement the exploration-exploitation strategy (taking the ε-greedy strategy as an example), generate a random number between 0 and 1. The random number will be used to determine whether the agent is exploring or exploiting existing knowledge in the current state;

[0085] S53, exploration-exploitation judgment: Compare the generated random number with the preset exploration probability:

[0086] Exploration phase: If the random number is less than the exploration probability, the agent enters the exploration mode. At this time, an action is randomly selected from the action space. This action may not be the currently known optimal action, but it helps the agent discover new path planning strategies and potentially better paths.

[0087] Exploitation phase: If the random number is greater than or equal to the exploration probability, the agent enters the exploitation mode; based on the action probability distribution or action value output by the reinforcement learning module, the agent selects the action with the highest probability or the action with the greatest value;

[0088] S54, action screening and adjustment: If the selected action may cause a collision with other objects or hinder its normal operation, the action is adjusted according to certain rules (such as priority avoidance principle, waiting mechanism, etc.) and a new appropriate action is selected;

[0089] S55, output execution action: Output the finalized action to the agent's actuator, allowing the agent to execute the action in the environment, thereby completing the interaction with the environment and providing a data basis for subsequent state updates, reward feedback, and strategy optimization;

[0090] S6, Path Generation and Optimization Module: This module generates the final path planning result based on a series of actions selected by the agent during its interaction with the environment. It also optimizes the generated path, such as removing unnecessary path points and smoothing path curves, to improve the feasibility and efficiency of the path. During the optimization process, it can use path optimization algorithms such as the Dijkstra algorithm and the A* algorithm to perform local or global optimization adjustments on the path generated by reinforcement learning, ensuring that the path is as close to the optimal solution as possible while meeting real-time requirements.

[0091] The specific steps of S6 are as follows:

[0092] S61, Action Sequence Arrangement: Receives the actions selected by the agent during its interaction with the environment, organizes and integrates the actions in chronological order, and clarifies the action trajectory taken by the agent from the starting point to the current state;

[0093] S62, Preliminary Path Generation: Based on the action sequence and the state transition of the agent in the environment, the action sequence is converted into a sequence of path points. Combined with the environment map or spatial information, the specific position of the agent in space after each action is determined, and the position points are connected in sequence to form a preliminary path planning result.

[0094] S63, remove redundant path points: Analyze the path point sequence through an algorithm to identify and remove redundant points that have little impact on the overall path direction and length;

[0095] S64, path smoothing: Use interpolation or filtering algorithms to optimize the path curve to make the path smoother and smoother, avoid sharp turns in the path, and reduce energy consumption and mechanical wear during the movement of the intelligent body;

[0096] S65, Local Optimization Adjustment: Utilizes classic path optimization algorithms to perform local optimization of the route. Focusing on a specific section or several points along the route, combined with surrounding environmental information, the algorithm recalculates whether a more optimal local route solution exists. For example, in urban traffic route planning, if a section of road experiences temporary congestion, the algorithm A can be used to replan the local area of ​​that section, finding a more optimal detour solution and replacing the corresponding portion of the original route.

[0097] S66, Global Optimization Evaluation: Evaluate and optimize the path from a holistic perspective to determine whether the current path is globally optimal. If a better solution exists, adjust and optimize the path globally using a global search algorithm to ensure that the path is as close to the global optimal solution as possible while meeting real-time requirements.

[0098] S67, path result output: The final path planning result after optimization is output for use by the intelligent agent actuator or feedback to the user. The output path result can be a sequence of coordinate points, route markings on a map, or an instruction set that conforms to a specific format, so that the intelligent agent can move according to the planned path or guide the user to complete the journey.

[0099] Although the present invention has been described above with reference to embodiments, various modifications may be made thereto and equivalent components may be substituted without departing from the scope of the present invention. In particular, as long as there are no structural conflicts, the various features of the embodiments disclosed herein may be combined with each other in any manner, and the omission of an exhaustive description of such combinations in this specification is solely for the sake of space and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A dynamic path planning optimization method based on reinforcement learning, characterized in that: The specific steps are as follows: S1, environmental perception: real-time acquisition of environmental information, including the agent's own position, speed, direction, as well as the location of obstacles in the surrounding environment, the motion state of dynamic objects, and target location information; S2, environmental pre-perception and dynamic prediction: After environmental perception collects raw environmental information, it first analyzes and predicts future environmental change trends, and then transmits the processed information; S3, state encoding: Encodes the information transmitted by environmental pre-perception and dynamic prediction and converts it into a state vector suitable for input of reinforcement learning algorithm; S4, reinforcement learning: selects appropriate actions based on the current state vector and learns the optimal path planning strategy through interaction with the environment; S5, action decision: Based on the action probability distribution or action value output by reinforcement learning, combined with a certain exploration-exploitation strategy, the actual action performed by the intelligent agent is determined; S6, path generation and optimization module: Generates the final path planning result based on a series of actions selected by the intelligent agent during its interaction with the environment; at the same time, optimizes the generated path to improve the feasibility and efficiency of the path; and during the optimization process, it can use the path optimization algorithm to perform local or global optimization adjustments on the path generated by reinforcement learning to ensure that the path is as close to the optimal solution as possible while meeting real-time requirements.

2. The dynamic path planning optimization method based on reinforcement learning according to claim 1, characterized in that: The specific steps of S2 are as follows: S21, multi-source data collection: collecting environmental data from multiple channels; S22, data preprocessing: cleaning, conversion and standardization of the collected raw data; S23, Feature Extraction and Fusion: Use data mining and machine learning algorithms to extract key features from preprocessed data; Then different types of features are fused to form a feature vector that can fully characterize the environmental state and change trend; S24, Prediction Model Construction and Training: Select an appropriate prediction model based on the specific prediction task and data characteristics. Then, train the model using historical and real-time data to optimize model parameters so that the model can accurately capture the patterns and trends of environmental changes. During the training process, use cross-validation to evaluate model performance, and adjust the model structure and parameters based on the evaluation results to improve prediction accuracy. S25, dynamic environment prediction: input the real-time data after preprocessing and feature extraction into the trained prediction model to predict the state changes of the environment in the future; S26, Verification and Correction of Prediction Results: Verify the prediction results and evaluate their accuracy by comparing them with actual observation data. If a significant deviation between the prediction results and the actual situation is found, analyze the cause of the deviation and take appropriate corrective measures. S27, prediction information output: output the verified and corrected prediction result to the state code.

3. The dynamic path planning optimization method based on reinforcement learning according to claim 2, characterized in that: The environment-related data in S21 includes sensor data, historical data, and external data obtained in real time by environmental perception; the cleaning in S22 is to remove noise, outliers, and duplicate data; the conversion is to convert data in different formats into a unified format for subsequent analysis; and the standardization is to standardize numerical data so that they are in the same scale range.

4. The dynamic path planning optimization method based on reinforcement learning according to claim 2, characterized in that: The prediction content in S25 includes the movement trajectory of the obstacle, the change trend of the traffic flow, and the state change of the target position.

5. The dynamic path planning optimization method based on reinforcement learning according to claim 1, characterized in that: The specific steps of S4 are as follows: S41, Initialization: Initializes the agent's policy network and value network, setting the network structure, parameters, and hyperparameters. It also creates an experience replay buffer to store experience data from the agent's interaction with the environment. It also defines the agent's action space and state space, clarifying the actions the agent can take and the environmental state information it can perceive in the path planning scenario. S42, state perception: Receive the current environment state vector from the state encoder. The vector is the result of encoding the original environment information and contains the key information required by the agent during path planning; S43, action selection: Based on the current state, the agent outputs the action probability distribution through the policy network, or calculates the value of each action through the value network, and then combines the exploration-exploitation strategy to determine the actual action to be performed; S44, Environment Interaction: The agent performs selected actions and interacts with the environment; So that the environment can change state according to the action of the intelligent agent and generate a new environment state; S45, reward feedback: The environment gives the agent a reward signal based on the new state after the agent's action according to a pre-set reward mechanism; the reward setting is closely centered around the path planning goal; S46, Experience Storage: The state, action, reward, and next state information of this interaction process are stored in the experience replay buffer. The experience data provides rich samples for subsequent algorithm training. By randomly sampling samples for training, the temporal correlation between data can be broken, which improves the stability and convergence speed of the algorithm. S47, Strategy Update: Randomly extract a batch of experience data from the experience replay buffer, or directly use the current state-action-reward-next-state sequence to calculate the loss function according to the selected reinforcement learning algorithm; the loss function is used to measure the gap between the current strategy and the optimal strategy. Through the backpropagation algorithm, the loss error is transferred to the policy network and value network, the network parameters are updated, and the strategy of the agent is gradually optimized, so that the agent can make better action choices in subsequent path planning to obtain greater long-term cumulative rewards; S48, continuous iteration: Repeat the above steps to allow the agent to continuously learn through continuous interaction with the environment, and gradually learn the strategy that can plan the optimal path in the dynamic environment; As training progresses, the agent's path planning ability continues to improve, enabling it to better adapt to environmental changes and find efficient path solutions.

6. The dynamic path planning optimization method based on reinforcement learning according to claim 1, characterized in that: The specific steps of S5 are as follows: S51, receiving input data: receiving action probability distribution or action value information from reinforcement learning; S52, generate a random number: To implement the exploration-exploitation strategy, generate a random number between 0 and 1. The random number will be used to determine whether the agent is exploring or exploiting existing knowledge in the current state; S53, exploration-exploitation judgment: Compare the generated random number with the preset exploration probability: Exploration phase: If the random number is less than the exploration probability, the agent enters the exploration mode. At this time, an action is randomly selected from the action space. This action may not be the currently known optimal action, but it helps the agent discover new path planning strategies and potentially better paths. Exploitation phase: If the random number is greater than or equal to the exploration probability, the agent enters the exploitation mode; Based on the action probability distribution or action value output by the reinforcement learning module, select the action with the highest probability or the action with the greatest value; S54, action screening and adjustment: if the selected action may cause a collision with other objects or hinder its normal operation, the action is adjusted according to certain rules and a suitable action is reselected; S55, output execution action: output the finalized action to the agent's actuator, so that the agent can perform the action in the environment, thereby completing the interaction with the environment and providing a data basis for subsequent state updates, reward feedback and strategy optimization.

7. The dynamic path planning optimization method based on reinforcement learning according to claim 1, characterized in that: The specific steps of S6 are as follows: S61, Action Sequence Arrangement: Receives the actions selected by the agent during its interaction with the environment, organizes and integrates the actions in chronological order, and clarifies the action trajectory taken by the agent from the starting point to the current state; S62, Preliminary Path Generation: Based on the action sequence and the state transition of the agent in the environment, the action sequence is converted into a sequence of path points. Combined with the environment map or spatial information, the specific position of the agent in space after each action is determined, and the position points are connected in sequence to form a preliminary path planning result. S63, remove redundant path points: Analyze the path point sequence through an algorithm to identify and remove redundant points that have little impact on the overall path direction and length; S64, path smoothing: Use interpolation or filtering algorithms to optimize the path curve to make the path smoother and smoother, avoid sharp turns in the path, and reduce energy consumption and mechanical wear during the movement of the intelligent body; S65, local optimization adjustment: use the classic path optimization algorithm to perform local optimization on the path; S66, Global Optimization Evaluation: Evaluate and optimize the path from a holistic perspective to determine whether the current path is globally optimal. If a better solution exists, adjust and optimize the path globally using a global search algorithm to ensure that the path is as close to the global optimal solution as possible while meeting real-time requirements. S67, path result output: The final path planning result after optimization is output for use by the intelligent agent actuator or feedback to the user. The output path result can be a sequence of coordinate points, route markings on a map, or an instruction set that conforms to a specific format, so that the intelligent agent can move according to the planned path or guide the user to complete the journey.