AI path planning method and system based on deep reinforcement imitation learning

Abstract

The invention discloses an AI path planning method and system based on deep reinforcement imitation learning, and the method comprises the steps: carrying out the weighted fusion through introducing a state incidence matrix, and constructing an environment state space with time sequence correlation; on the basis of a generative adversarial imitation learning framework, adversarial training of a strategy generation network and a strategy discrimination network is carried out by utilizing a pre-trained expert strategy database; a master-slave distributed architecture is adopted to train a plurality of strategy network copies in parallel, and global parameters of the strategy generation network and the strategy discrimination network are optimized through an aggregation gradient and parameter soft update mechanism; and deploying the lightweight strategy generation network, outputting an action instruction according to a real-time environment state, and obtaining a real-time path planning result through a dynamic path correction mechanism and online performance fine adjustment. The problem that the robustness, the safety and the efficiency of path planning of the robot in a complex environment are difficult to meet actual requirements is solved.

Description

Technical Field

[0001] This invention belongs to the field of path planning technology, and in particular relates to an AI path planning method and system based on deep reinforcement imitation learning. Background Technology

[0002] With the rapid development of robotics technology, higher demands are being placed on robots' autonomous path planning capabilities in complex and dynamic environments. Traditional path planning algorithms, such as A* and Dijkstra's, perform relatively stably in static, known environments, but suffer from poor adaptability and low planning efficiency in scenarios with dynamic environmental changes and complex obstacle distributions. Reinforcement learning algorithms autonomously learn strategies through interaction with the environment, possessing a certain degree of environmental adaptability, but they suffer from slow convergence speed, susceptibility to getting trapped in local optima in complex environments, and require extensive interactive trial and error, posing safety risks in practical applications. Imitation learning quickly learns basic strategies by imitating expert demonstration data, but its over-reliance on expert data results in insufficient generalization ability and difficulty in coping with unseen environmental changes. Furthermore, existing path planning algorithms still have shortcomings in areas such as multi-source sensor information fusion, the stability and efficiency of policy network training, the rationality of reward function design, and real-time path correction in dynamic environments, making it difficult for robots to meet practical needs in terms of robustness, safety, and efficiency in path planning in complex environments. Summary of the Invention

[0003] This invention provides an AI path planning method and system based on deep reinforcement imitation learning, which addresses the technical problem that the robustness, safety, and efficiency of path planning for robots in complex environments are difficult to meet practical needs.

[0004] In a first aspect, the present invention provides an AI path planning method based on deep reinforcement imitation learning, comprising:

[0005] The system integrates multi-source sensor data from LiDAR, cameras, and odometers, and constructs an environmental state space with temporal correlation by introducing a state correlation matrix for weighted fusion.

[0006] Based on the generative adversarial imitation learning framework, a path planning policy is learned by using a pre-trained expert policy database and adversarial training between a policy generation network and a policy discrimination network. The policy generation network outputs actions based on the environmental state, and the policy discrimination network is used to evaluate the similarity between the generated actions and the expert actions.

[0007] A master-slave distributed architecture is used to train multiple policy network replicas in parallel. By aggregating gradients and using a soft parameter update mechanism, the global parameters of the policy generation network and the policy discrimination network are optimized.

[0008] A lightweight policy generation network is deployed to output action instructions based on the real-time environment status, and real-time path planning results are obtained through a dynamic path correction mechanism and online performance fine-tuning.

[0009] Secondly, the present invention provides an AI path planning system based on deep reinforcement imitation learning, comprising:

[0010] The module is configured to fuse multi-source sensor data from LiDAR, camera, and odometer, and constructs an environmental state space with temporal correlation by introducing a state correlation matrix for weighted fusion.

[0011] The learning module is configured to be based on a generative adversarial imitation learning framework. It utilizes a pre-trained expert policy database and learns path planning policies through adversarial training between a policy generation network and a policy discrimination network. The policy generation network outputs actions based on the environmental state, and the policy discrimination network is used to evaluate the similarity between the generated actions and the expert actions.

[0012] The optimization module is configured to train multiple policy network replicas in parallel using a master-slave distributed architecture, and optimize the global parameters of the policy generation network and the policy discrimination network through gradient aggregation and parameter soft update mechanism.

[0013] The output module is configured to deploy a lightweight policy generation network, output action commands based on the real-time environment status, and obtain real-time path planning results through a dynamic path correction mechanism and online performance fine-tuning.

[0014] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the AI ​​path planning method based on deep reinforcement imitation learning according to any embodiment of the present invention.

[0015] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor performs the steps of the AI ​​path planning method based on deep reinforcement imitation learning according to any embodiment of the present invention.

[0016] This application presents an AI path planning method and system based on deep reinforcement imitation learning. It constructs an environmental state space model that integrates multi-source sensor data, introduces a state association matrix to achieve weighted information fusion, and designs a dynamic update mechanism to ensure the continuity of state sequence, effectively reducing the limitations of single-sensor data and improving the robot's perception accuracy and robustness in complex environments. Employing a generative adversarial imitation learning architecture, combined with expert policy library pre-training and transfer learning modules, it integrates expert data and interactive data through a multi-priority experience replay pool, achieving a smooth transition from imitating experts to autonomous exploration, significantly accelerating model convergence and improving policy optimization efficiency. A multi-objective reward function integrating safety, efficiency, and economy is designed. By dynamically adjusting reward weights and a phased training strategy, it balances the requirements of collision avoidance, target orientation, and path smoothing in path planning, improving the overall quality of the planned path. The introduction of a dynamic path correction mechanism and an online performance evaluation closed loop, combined with lightweight model deployment and real-time inference strategies, enables rapid response to environmental changes and triggers path replanning. Simultaneously, the online fine-tuning mechanism dynamically balances path efficiency and safety, giving the robot stronger adaptive navigation capabilities in complex dynamic environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating an AI path planning method based on deep reinforcement imitation learning, provided as an embodiment of the present invention;

[0019] Figure 2 The following is a structural block diagram of an AI path planning system based on deep reinforcement imitation learning, provided in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Please see Figure 1 The diagram shows a flowchart of an AI path planning method based on deep reinforcement imitation learning, as presented in this application.

[0023] like Figure 1 As shown, the AI ​​path planning method based on deep reinforcement imitation learning specifically includes the following steps:

[0024] Step S101: Multi-source sensor data from LiDAR, camera and odometer are fused and processed, and weighted fusion is performed by introducing a state correlation matrix to construct an environmental state space with temporal correlation.

[0025] In this step, during the construction of the environmental state space model that integrates multi-source information, component analysis is used to perform coarse registration of point clouds in adjacent frames, and the source point cloud set O = {O} is calculated. i} and the target point set T = {T j The initial transformation matrix is ​​obtained, and then the registration accuracy is optimized by an improved point-to-line iterative nearest-point algorithm. The registration error function is defined as E = ∑ i ||R·O i +TT j || 2 , where R is the rotation matrix and T is the translation matrix.

[0026] Next, multimodal sensing data is integrated to construct the state space dimension, and the state space is defined as follows:

[0027] S(t)={P obs (t),F img (t),X robot (t),G(t),V robot (t)}

[0028] Among them, P obs (t) represents the set of obstacle coordinates at time t, obtained through a lidar point cloud clustering algorithm; X robot G(t) = {x(t), y(t), θ(t)} represents the robot's precise pose, obtained through optimization and fusion of odometry data and visual SLAM; G(t) = {x g ,y g} represents the coordinates of the target point; V robot (t) represents the robot's linear velocity and angular velocity.

[0029] Introducing the state correlation matrix Through M ij =I(s) i ,s j )=H(s i )+H(s j )+H(s i ,s j ) calculate, where element Mij It represents the mutual information entropy between the i-th state element and the j-th state element, which improves the robustness of environmental perception.

[0030] Finally, a dynamic state space update mechanism is designed, with experience priority based on p. t =|δ t |+ε is calculated, where δ t =r(t)+γmax a' Q(S(t+1), a')-Q(S(t), a(t)) represents the TD error, and ε is a minimal constant to avoid zero priority. Sampling is performed according to probability. Empirical tuples are selected, and a state decay factor λ is introduced, through S update (t)=λS(t)+(1-λ)S hist (t) Integrating historical state information S hist (t), where λ∈[0,1] dynamically adjusts the weights of the current state and the historical state.

[0031] Step S102: Based on the generative adversarial imitation learning framework, using a pre-trained expert policy database, a path planning policy is learned through adversarial training between a policy generation network and a policy discrimination network; the policy generation network outputs actions according to the environmental state, and the policy discrimination network is used to evaluate the similarity between the generated actions and the expert actions.

[0032] In this step, a policy network architecture based on generative adversarial imitation learning is designed, outputting a continuous action space A(t) = {v(t), w(t)}, with action probability distributions satisfying a ~ N(μ(s,θ), σ 2 (s,θ)), where μ(s,θ) and σ 2 (s,θ) are the mean and variance functions, respectively, and are determined by the network parameter θ.

[0033] The Critic network employs generalized advantage estimation (GAE) to optimize the advantage function estimation, as shown in the following formula: Where δ t =r t +γV(s t+1 )-V(s t ) represents the TD error, γ represents the discount factor, and λ represents the bias-variance balance factor, which effectively improves the stability of value estimation.

[0034] Determine if the network output satisfies Where τ D To determine network parameters, σ is the Sigmoid function.

[0035] Next, we design the adversarial training mechanism and loss function, as shown in the formula:

[0036]

[0037] Wherein, H(π) θ Let ζ be the policy entropy and ζ be the entropy coefficient. Policy optimization is achieved by maximizing this loss function. The loss function of the discriminator network adopts the form of binary cross-entropy, defined as:

[0038]

[0039] Where π E For expert policy distribution, π G To generate the policy distribution, the discriminative ability of the discriminative network is improved by minimizing this loss function.

[0040] An expert policy library and transfer learning module are introduced, and the generative network is pre-trained using the behavior cloning method. The pre-training loss function is: Simultaneously, a strategy smoothing factor α is designed, through a... smooth =α·a G +|1-a|·a E , merge to generate action a G With expert action a E .

[0041] In constructing an expert strategy database and experience playback mechanism, expert demonstration data is collected and filtered to build a high-quality strategy library. The robot is manually controlled to complete path planning tasks in various environmental states defined by S1, and the state-action-reward sequence is recorded. in For the environmental conditions observed by experts, Actions performed by experts Rewards for environmental feedback. Define the value of expert samples. The top 30% of samples by value were selected to form the core expert pool, with the remaining samples serving as the expansion pool. Data augmentation techniques were used to perturb the expert paths, generating new samples s'. E =s E +ε, where ε~N(0,σ) 2 ) represents Gaussian noise, and σ is dynamically adjusted according to the complexity of the environment.

[0042] Next, a multi-priority experience replay pool structure was designed, integrating reinforcement learning interactive data and expert demonstration data, with priority calculated based on TD error: in For agent TD error. Priority is set based on expert value: in Let α be the expert sample TD error, and α be the weighting coefficient.

[0043] Finally, a dynamic management and updating mechanism for empirical data is implemented, and the cosine similarity between new samples and existing samples in the replay pool is calculated. The replay pool is periodically optimized by retaining high-priority samples and randomly replacing low-priority samples, with the replacement probability inversely proportional to the sample priority. Simultaneously, an experience timeliness weight is designed, and the sample storage time t is considered. store The longer the length, the higher its priority decay coefficient. The smaller.

[0044] It should be noted that in designing the multi-objective reward function and adversarial training mechanism, the reward function is decomposed into the environmental reward R. env , and the reward for combat adv Expert-guided reward R exp The total reward is defined as: R(t) = w1R env (t)+w2R adv (t)+w3R exp (t), where w1, w2, w3 are the number of weight systems satisfying w1 + w2 + w3 = 1. Environmental rewards include collision penalties, goal-oriented rewards, and path-smoothing rewards: R env (t)=r goal I goal -r colide I colide , where I goal and I colide Let d(t) be the indicator function, d(t) be the distance from the robot to the target point, and Δw(t) be the change in angular velocity. The adversarial reward comes from the output of the discriminant network in S2. This reflects the similarity between the generated action and the expert strategy. The expert-guided reward is obtained by calculating the Euclidean distance between the generated action and the actions in the expert pool in S3: R exp (t)=-ε||a(t)-a E (t)||, where a E (t) represents the expert's optimal action in the corresponding state, enhancing the strategy's learning of expert knowledge.

[0045] Next, we design the adversarial loss functions for the generator network and the discriminator network. The loss function for the generator network is:

[0046]

[0047] Where A(s,a) is the dominance function calculated in S2, and H(π) θ )=-∑ a π θ (a|s)logπ θ (a|s) represents the policy entropy, and ζ represents the entropy coefficient. Policy optimization is achieved by maximizing this loss function. The loss function is defined as: Where π E For expert policy distribution, π G To generate a policy distribution.

[0048] Finally, a dynamic reward weight and training strategy adjustment mechanism is implemented, and a reward decay factor is introduced. Where γ0 is the initial discount factor, t is the current iteration number, and T is the total number of iterations, making later training focus more on long-term rewards.

[0049] Step S103: Multiple policy network replicas are trained in parallel using a master-slave distributed architecture. The global parameters of the policy generation network and the policy discrimination network are optimized by aggregating gradients and using a soft parameter update mechanism.

[0050] In this step, the global network parameter update formula for model optimization based on distributed training is: Where α is the learning rate and N is the number of local networks. Let be the loss function of the i-th local network.

[0051] Next, an adaptive gradient clipping and soft parameter update mechanism is designed to normalize the gradients uploaded from each local network. The gradient clipping threshold is dynamically adjusted to τ(t) = τ0·e -βt Where τ0 is the initial threshold, β is the decay coefficient, and t is the number of training steps, when the gradient norm At that time, perform the cropping operation. When synchronizing parameters from the global network to the local network, a soft update strategy is used, and the parameter update formula is θ. local ←τ·θ global +(1-τ)·θ local , where τ is the soft update coefficient, with a value of 0.001, to ensure a smooth transition of local network parameters and avoid training fluctuations caused by parameter mutations.

[0052] Finally, dynamic monitoring and adaptive adjustment of the training process are implemented, and the convergence evaluation index is defined as the average reward value. The rate of change of policy entropy is ΔH = H(t) - H(t-1), where K is the evaluation window size and H(t) is the policy entropy at time t.

[0053] Step S104: Deploy the lightweight policy generation network, output action instructions based on the real-time environment status, and obtain real-time path planning results through dynamic path correction mechanism and online performance fine-tuning.

[0054] In this step, during the inference phase of real-time path planning execution and dynamic optimization, the robot collects the environmental state s(t) in real time through sensors, inputs the feature vector into the lightweight model, and outputs action commands. The action prediction satisfies a(t) = μ(s(t), θ global )+ε(t), where ε(t) is the dynamic exploration noise, and its variance decreases with training maturity: σ(t)=σ0·e -λtThis ensures that a certain level of exploration capability is retained while guaranteeing the safety of the path.

[0055] To reduce inference latency, a sliding window mechanism is used to cache the state data of the most recent 5 frames, through s smooth (t)=α·s(t)+(1-α)s smooth (t-1) achieves state smoothing and reduces the interference of sensor noise on decision-making, where α = 0.7 is the smoothing coefficient.

[0056] Next, a dynamic path correction mechanism is designed to optimize the planned trajectory based on real-time environmental perception results. Using real-time point cloud data from LiDAR, an improved particle filter algorithm is employed to track obstacle positions, with the particle weight update formula being w. i (t)=w i (t-1)·p(z(t)|x i (t),m), where z(t) is the current observation data, x i (t) represents the state estimate of the i-th particle, and m represents the environment map. When a new obstacle is detected or the position of an existing obstacle changes, path replanning is triggered, and the robot pose is updated using the kinematic model.

[0057]

[0058] Where Δt is the control period. A path safety margin index is introduced. When d safe <δ th Emergency obstacle avoidance is triggered at this time.

[0059] Finally, an online performance evaluation and online optimization closed loop is constructed, defining performance evaluation metrics including path length. average speed and collision risk Where γ is the risk coefficient. Based on the evaluation results, an online fine-tuning mechanism using reinforcement learning is employed. Update policy network parameters, where R perf (t) represents the overall performance reward, and η represents the fine-tuning learning rate.

[0060] In summary, the method of this application can achieve the following technical effects:

[0061] By constructing an environmental state space model that integrates multi-source sensor data, introducing a state correlation matrix to achieve weighted information fusion, and designing a dynamic update mechanism to ensure the continuity of state time sequence, the limitations of single sensor data are effectively reduced, and the robot's perception accuracy and robustness in complex environments are improved.

[0062] By adopting a generative adversarial imitation learning architecture, combining expert policy library pre-training and transfer learning modules, and fusing expert data and interactive data through a multi-priority experience replay pool, a smooth transition from imitating experts to autonomous exploration is achieved, which significantly accelerates the model convergence speed and improves the efficiency of policy optimization.

[0063] The design incorporates a multi-objective reward function that integrates safety, efficiency, and economy. By dynamically adjusting reward weights and employing a phased training strategy, it balances the requirements for collision avoidance, target convergence, and path smoothing in path planning, thereby improving the overall quality of the planned path.

[0064] By introducing a dynamic path correction mechanism and an online performance evaluation closed loop, combined with lightweight model deployment and real-time inference strategies, the robot can quickly respond to environmental changes and trigger path replanning. At the same time, through an online fine-tuning mechanism, the robot can dynamically balance path efficiency and safety, enabling it to have stronger adaptive navigation capabilities in complex dynamic environments.

[0065] In a specific implementation, an AI path planning method based on deep reinforcement imitation learning in computer science includes the following steps:

[0066] S1. Construct an environmental state space model that integrates multi-source information;

[0067] S2. Design a policy network architecture based on generative adversarial imitation learning;

[0068] S3. Construct an expert strategy database and experience playback mechanism;

[0069] S4. Design a multi-objective reward function and adversarial training mechanism;

[0070] S5. Implement model optimization based on distributed training;

[0071] S6. Real-time execution and dynamic optimization of path planning;

[0072] In a preferred embodiment of the present invention, in step S1, constructing an environmental state space model that integrates multi-source information, multi-source sensor data acquisition and preprocessing are first performed. An environmental perception system is then constructed using lidar, a visual camera, and a wheeled odometer. The lidar employs an improved point cloud registration algorithm to process the raw scan data. First, component analysis is used to perform coarse registration of adjacent frame point clouds, and the source point cloud set O = {O} is calculated. i} and the target point set T = {T j The initial transformation matrix is ​​obtained, and then the registration accuracy is optimized by an improved point-to-line iterative nearest-point algorithm. The registration error function is defined as E = ∑ i ||R·O i +TT j || 2Where R is the rotation matrix and T is the translation matrix, this process effectively reduces the registration error of the traditional ICP algorithm in complex environments. Image data acquired by the visual camera is preprocessed by cropping, grayscale conversion, and Gaussian blurring, and then environmental features are extracted through a convolutional neural network, outputting a feature vector F. img The wheel odometry data is processed by Kalman filtering to obtain a preliminary estimate of the robot pose, X. odom ={x odom ,y odom ,θ odom} provides basic motion information for the state space.

[0073] Next, multimodal perception data is integrated to construct the state space dimension. By combining LiDAR obstacle detection results, visual features, robot pose, and target point information, the state space is defined as follows:

[0074] S(t)={P obs (t),F img (t),X robot (t),G(t),V robot (t)}

[0075] Among them, P obs (t) represents the set of obstacle coordinates at time t, obtained through a lidar point cloud clustering algorithm; X robot G(t) = {x(t), y(t), θ(t)} represents the robot's precise pose, obtained through optimization and fusion of odometry data and visual SLAM; G(t) = {x g ,y g} represents the coordinates of the target point; V robot (t) represents the robot's linear velocity and angular velocity. To quantify the correlation between elements in the state space, a state correlation matrix is ​​introduced. Through M ij =I(s) i ,s j )=H(s i )+H(s j )+H(s i ,s j ) calculate, where element M ij It represents the mutual information entropy between the i-th state element and the j-th state element, realizing the weighted fusion of state information and improving the robustness of environmental perception.

[0076] Finally, a dynamic state-space update mechanism is designed, combining the experience replay mechanism in deep reinforcement learning to perform temporal correlation processing on the state data. The state S(t), action a(t), reward r(t), and next state S(t+1) at each time step are combined into an experience tuple (S(t), a(t), r(t), S(t+1)) and stored in a priority experience replay pool. Experience priorities are assigned according to p...t =|δ t |+ε is calculated, where δ t =r(t)+γmax a' Q(S(t+1), a')-Q(S(t), a(t)) represents the TD error, and ε is a minimal constant to avoid zero priority. Sampling is performed according to probability. Empirical tuples are selected, and a state decay factor λ is introduced, through S update (t)=λS(t)+(1-λ)S hist (t) Integrating historical state information S hist (t), where λ∈[0,1] dynamically adjusts the weights of the current state and the historical state to achieve temporal continuity updates of the state space, providing high-quality environmental state data for subsequent policy learning.

[0077] In a preferred embodiment of the present invention, in step S2, designing a policy network architecture based on generative adversarial imitation learning, a dual-network architecture of a generator network and a discriminator network is first constructed. The generator network adopts an improved Actor-Critic structure, and its input is the environment state space S(t) constructed in S1. State features are extracted through multi-layer fully connected networks and convolutional layers. The Actor network outputs a continuous action space A(t) = {v(t), w(t)}, and the action probability distribution satisfies a ~ N(μ(s,θ), σ 2 (s,θ)), where μ(s,θ) and σ 2 (s, θ) represent the mean and variance functions, respectively, determined by the network parameter θ. The Critic network calculates the advantage function A(s, a) by fusing state and action features, and optimizes the advantage function estimate using generalized advantage estimation (GAE), as shown in the formula. Where δ t =r t +γV(s t+1 )-V(s t Let ) represent the TD error, γ be the discount factor, and λ be the bias-variance balance factor, effectively improving the stability of value estimation. The discriminator network takes the state-action pair (s, a) as input, and outputs the probability of the sample following the expert policy through a three-layer fully connected network and the LeakyReLU activation function. The network output satisfies , where τ D To determine network parameters, σ is the Sigmoid function, which enables effective differentiation between the generation strategy and the expert strategy.

[0078] Next, we designed an adversarial training mechanism and a loss function. The loss function of the generator network incorporates the editing target and policy regularization term of the PPO, and the formula is:

[0079]

[0080] Wherein, H(π) θ Let ζ be the policy entropy and ζ be the entropy coefficient. Policy optimization is achieved by maximizing this loss function. The loss function of the discriminator network adopts the form of binary cross-entropy, defined as follows:

[0081] Where π E For expert policy distribution, π G To generate the policy distribution, the discriminative network's ability to distinguish is improved by minimizing this loss function. During training, the generator network and the discriminative network alternately update the network iteratively. The generator network adjusts its policy based on feedback from the discriminative network to approximate expert behavior, while the discriminative network optimizes its distinguishing ability by comparing generated samples with expert samples, forming a closed-loop adversarial learning mechanism.

[0082] An expert policy library and a transfer learning module are introduced to construct an expert policy library containing high-quality path planning data. Each expert sample is the optimal action sequence corresponding to the state space in S1. The generative network is pre-trained using a behavior cloning method, with the pre-training loss function being: This allows the generative network to initially learn the basic characteristics of expert policies. During the adversarial training phase, a hybrid training dataset is constructed by sampling samples from the expert policy library and interaction samples from the generative network. Utilizing the parameter sharing mechanism in transfer learning, the network parameters from the pre-training phase are used as initial parameters for adversarial training, accelerating model convergence. Simultaneously, a policy smoothing factor α is designed to... smooth =α·a G +|1-a|·a E , merge to generate action a G With expert action a E α increases linearly from 0.1 to 0.9 as the training progresses, achieving a smooth transition from imitating expert policies to exploring optimal policies by the white-hat, thus improving the generalization ability of the policy network in complex environments.

[0083] In a preferred embodiment of the present invention, in step S3, which involves constructing an expert strategy database and an experience playback mechanism, a high-quality strategy library is built by collecting and filtering expert demonstration data. The robot is then manually controlled to complete path planning tasks in various environmental states defined in step S1, and the state-action-reward sequence is recorded. in For the environmental conditions observed by experts, Actions performed by experts Incentives are provided for environmental feedback. Drawing inspiration from the priority experience playback mechanism, expert data undergoes quality assessment, and the value of expert samples is defined. The top 30% of samples by value were selected to form the core expert pool, with the remaining samples serving as an expansion pool. To enhance data diversity, data augmentation techniques were used to perturb the expert paths, generating new samples s'. E =s E +ε, where ε~N(0,σ) 2 ) is Gaussian noise, and σ is dynamically adjusted with environmental complexity, so that the expert database covers more potential environmental changes and provides rich comparison samples for the discriminant network in S2.

[0084] Next, a multi-priority experience replay pool structure was designed, integrating reinforcement learning interactive data and expert demonstration data. The replay pool is divided into an expert experience area R. E and the agent's experience zone R G The agent's experience region stores the experience tuples (s,a,r,s') generated by the interaction between the generator network and the environment in S2, with their priority calculated according to the TD error. in This represents the agent's TD error. The expert experience region stores expert samples, with priority set based on expert value. in TD error for expert samples

[0085] The difference, α, is the weighting coefficient. A stratified sampling strategy is used during sampling, selecting samples from two empirical regions in a ratio β:(1-β), where β linearly decreases from 0.8 to 0.2 during training, achieving a smooth transition from relying on expert data to autonomously exploring data. The sampling probability satisfies... w is the priority adjustment factor.

[0086] Finally, a dynamic management and updating mechanism for empirical data is implemented. To address the sample redundancy problem, a sample similarity filtering strategy is introduced, calculating the cosine similarity between new samples and existing samples in the replay pool. New samples are rejected when the similarity exceeds a threshold τ, while samples with low similarity are retained to maintain data diversity. The replay pool is periodically optimized: when the number of samples in the pool reaches a threshold, high-priority samples are retained and low-priority samples are randomly replaced, with the replacement probability inversely proportional to the sample priority. Simultaneously, an experience timeliness weight is designed, and the sample storage time t is considered. store The longer the length, the higher its priority decay coefficient. The smaller, through p' i =p i ·η(t store Dynamically adjust the priority so that the replay pool always retains recent high-value samples, providing high-quality training data for parameter updates of the generator and discriminator networks in S2, and improving the convergence speed and stability of the model.

[0087] In a preferred embodiment of the present invention, in step S4, designing a multi-objective reward function and adversarial training mechanism, a multi-objective reward function integrating environmental feedback and expert knowledge is constructed to comprehensively consider the safety, efficiency, and economic indicators of path planning. The reward function is decomposed into environmental reward R. env , and the reward for combat adv Expert-guided reward R exp The total reward is defined as: R(t) = w1R env (t)+w2R adv (t)+w3R exp (t), where w1, w2, w3 are the number of weight systems satisfying w1 + w2 + w3 = 1. Environmental rewards include collision penalties, goal-oriented rewards, and path-smoothing rewards: R env (t)=r goal I goal -r colide I colide , where I goal and I colide Let d(t) be the indicator function, d(t) be the distance from the robot to the target point, and Δw(t) be the change in angular velocity. The adversarial reward comes from the output of the discriminant network in S2. This reflects the similarity between the generated action and the expert strategy. The expert-guided reward is obtained by calculating the Euclidean distance between the generated action and the actions in the expert database in S300: R exp (t)=-ε||a(t)-a E (t)||, where a E (t) represents the expert's optimal action in the corresponding state, enhancing the strategy's learning of expert knowledge.

[0088] Next, adversarial loss functions for the generator and discriminator networks are designed, and the generator network loss is fused with the PPO's clipping objective and policy entropy regularization. Referring to the near-end policy optimization algorithm, the generator network loss function is as follows:

[0089]

[0090] Where A(s,a) is the dominance function calculated in S2, and H(π) θ )=-∑ a π θ (a|s)logπ θ (a|s) represents the policy entropy, and ζ represents the entropy coefficient. Policy optimization is achieved by maximizing this loss function. The discriminant network loss adopts a binary cross-entropy form. Considering the need to distinguish between expert samples and generated samples in S3, the loss function is defined as: Where π E For expert policy distribution, π GTo generate the policy distribution, the ability of the discriminant network to distinguish between the two types of samples is improved by minimizing the loss function, thus forming a closed-loop adversarial mechanism of "generation-discrimination".

[0091] Finally, a dynamic reward weight and training strategy adjustment mechanism is implemented, adaptively adjusting the weights of each component in the reward function according to the training progress. In the initial stage, the expert-guided reward weight w3 is increased to 0.6, enabling the generative network to quickly learn basic path planning capabilities. With training iterations, w3 is linearly decreased to 0.1, while the adversarial reward weight w2 is increased to 0.5, enhancing the policy's autonomous exploration capabilities. A reward decay factor is introduced. Where γ0 is the initial discount factor, t is the current iteration number, and T is the total number of iterations, this design emphasizes long-term rewards in later training stages. Considering the sample distribution of the experience replay pool in S3, a phased training strategy is designed: expert samples dominate training in the early stage, a mixture of expert and generated samples is used in the middle stage, and generated samples dominate in the later stage, through N... E (t)=N0·e -λt Dynamically adjust the number of expert samples N E (t) enables a smooth transition from imitation learning to reinforcement learning, improving the model's robustness and optimal path planning capabilities in complex and unknown environments.

[0092] In a preferred embodiment of the present invention, in step S5, which implements model optimization based on distributed training, a distributed training architecture is first built, employing a master-slave network structure to coordinate the parallel training of multiple agents. Referring to the idea of ​​distributed near-end policy optimization algorithms, a global network and multiple local networks are set up. The global network is responsible for parameter updates and synchronization, while the local networks independently interact with the environment to collect data. The global network includes the generator network and discriminator network defined in S2, and its parameters are updated through gradient aggregation of each local network. Each local network loads the latest parameters from the global network, executes the policy in different environment instances constructed in S1, generates empirical data, and calculates gradients. The parameter interaction between the local and global networks is achieved through asynchronous gradient descent, and the global network parameter update formula is: Where α is the learning rate and N is the number of local networks. Let be the loss function of the i-th local network. Parallel computation across multiple nodes improves data acquisition efficiency and model training speed.

[0093] Next, an adaptive gradient clipping and soft parameter update mechanism is designed to address the gradient explosion and parameter inconsistency issues in distributed training. The gradients uploaded from each local network are normalized, and the gradient clipping threshold is dynamically adjusted to τ(t) = τ0·e -βt Where τ0 is the initial threshold, β is the decay coefficient, and t is the number of training steps, when the gradient norm At that time, perform the cropping operation. When synchronizing parameters from the global network to the local network, a soft update strategy is used, and the parameter update formula is θ. local ←τ·θ global +(1-τ)·θ local τ is the soft update coefficient, with a value of 0.001, ensuring a smooth transition of local network parameters and avoiding training fluctuations caused by parameter mutations. Combined with the reward function in S4, a reward weight factor is introduced into the gradient calculation, making the gradient weights corresponding to high-reward samples larger, thus increasing the contribution of key experiences to model optimization.

[0094] Finally, dynamic monitoring and adaptive adjustment of the training process are implemented, and the model convergence state is evaluated and the training strategy is optimized through quantitative metrics. The convergence evaluation metric is defined as the average reward value. The rate of change of policy entropy is ΔH = H(t) - H(t-1), where K is the evaluation window size and H(t) is the policy entropy at time t. When R... avg When the model stabilizes within the threshold range for 100 consecutive iterations and ΔH < ε, it is determined that the model has entered the convergence phase. At this point, the learning rate is reduced to α' = α·0.1, and the entropy coefficient ζ is increased to encourage the strategy to explore further within the convergence region. Referring to the priority experience replay mechanism, the distribution of training data is dynamically adjusted. When the model shows signs of overfitting, the sampling ratio of low-priority samples in S3 is increased to introduce more diverse data and improve generalization ability. By monitoring the training loss and environment interaction efficiency of each local network in real time, computational resources are dynamically allocated to allow poorly performing local networks to receive more training iterations, ensuring the consistency of overall efficiency and model performance in distributed training.

[0095] In a preferred embodiment of the present invention, in S6, real-time execution and dynamic optimization of path planning, lightweight deployment and real-time inference of the model are first achieved. The generative network model that has converged during training in S5 is pruned and quantized, retaining the core feature extraction layer and decision layer, and removing redundant parameters to reduce computational complexity. During the inference stage, the robot collects the environmental state s(t) in real time through sensors, converts it into a feature vector through the preprocessing process in S1, and then inputs it into the lightweight model to output action commands. The action prediction satisfies a(t) = μ(s(t), θ global )+ε(t), where ε(t) is the dynamic exploration noise, and its variance decreases with training maturity: σ(t)=σ0·e -λt This ensures that a certain level of exploration capability is retained while guaranteeing the safety of the path.

[0096] To reduce inference latency, a sliding window mechanism is used to cache the state data of the most recent 5 frames, through s smooth (t)=α·s(t)+(1-α)s smooth(t-1) achieves state smoothing and reduces the interference of sensor noise on decision-making, where α = 0.7 is the smoothing coefficient.

[0097] Next, a dynamic path correction mechanism is designed to optimize the planned trajectory based on real-time environmental perception results. Using real-time point cloud data from LiDAR, an improved particle filter algorithm is employed to track obstacle positions, with the particle weight update formula being w. i (t)=w i (t-1)·p(z(t)|x i (t),m), where z(t) is the current observation data, x i (t) represents the state estimate of the i-th particle, and m represents the environment map. When a new obstacle is detected or the position of an existing obstacle changes, path replanning is triggered, and the robot pose is updated using the kinematic model.

[0098]

[0099] Where Δt is the control period. A path safety margin index is introduced. When d safe <δ th Emergency obstacle avoidance is triggered when the robot is in a critical situation. The dynamic window method (DWA) is used to re-optimize the transition trajectory within a local area to ensure that the robot maintains a safe distance from obstacles.

[0100] Finally, an online performance evaluation and online optimization closed loop is constructed, and strategy parameters are dynamically adjusted through a multi-index quantification system. Performance evaluation metrics include path length. average speed and collision risk Where γ is the risk coefficient. Based on the evaluation results, an online fine-tuning mechanism using reinforcement learning is employed. Update policy network parameters, where R perf (t) represents the overall performance reward, and η represents the fine-tuning learning rate. Combining the multi-objective reward function in S4, path efficiency and safety are dynamically balanced during long-term operation. When environmental complexity increases (e.g., obstacle density exceeds a threshold), the collision penalty weight r is automatically increased. colide Conversely, the path progress reward weight r is increased. progress This enables robots to perform adaptive path planning in dynamic environments.

[0101] Please see Figure 2 The diagram shows a structural block diagram of an AI path planning system based on deep reinforcement imitation learning, as described in this application.

[0102] like Figure 2 As shown, the AI ​​path planning system 200 includes a construction module 210, a learning module 220, an optimization module 230, and an output module 240.

[0103] The system comprises the following modules: Construction module 210, configured to fuse multi-source sensor data from LiDAR, camera, and odometer, and construct an environmental state space with temporal correlation by introducing a state correlation matrix for weighted fusion; Learning module 220, configured to learn path planning strategies based on a generative adversarial learning framework, utilizing a pre-trained expert policy database, and through adversarial training between a policy generation network and a policy discriminator network; the policy generation network outputs actions based on the environmental state, and the policy discriminator network is used to evaluate the similarity between the generated actions and the expert actions; Optimization module 230, configured to train multiple policy network replicas in parallel using a master-slave distributed architecture, and optimize the global parameters of the policy generation network and the policy discriminator network through gradient aggregation and parameter soft update mechanisms; Output module 240, configured to deploy a lightweight policy generation network, output action instructions based on the real-time environmental state, and obtain real-time path planning results through a dynamic path correction mechanism and online performance fine-tuning.

[0104] It should be understood that Figure 2 The modules and references described in the document Figure 1 The steps described in the text correspond to those in the method described above. Therefore, the operations, features, and corresponding technical effects described above also apply to the method described in the text. Figure 2 The various modules in the document will not be described in detail here.

[0105] In other embodiments, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor performs the AI ​​path planning method based on deep reinforcement imitation learning in any of the above method embodiments.

[0106] In one embodiment, the computer-readable storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0107] The system integrates multi-source sensor data from LiDAR, cameras, and odometers, and constructs an environmental state space with temporal correlation by introducing a state correlation matrix for weighted fusion.

[0108] Based on the generative adversarial imitation learning framework, a path planning policy is learned by using a pre-trained expert policy database and adversarial training between a policy generation network and a policy discrimination network. The policy generation network outputs actions based on the environmental state, and the policy discrimination network is used to evaluate the similarity between the generated actions and the expert actions.

[0109] A master-slave distributed architecture is used to train multiple policy network replicas in parallel. By aggregating gradients and using a soft parameter update mechanism, the global parameters of the policy generation network and the policy discrimination network are optimized.

[0110] A lightweight policy generation network is deployed to output action instructions based on the real-time environment status, and real-time path planning results are obtained through a dynamic path correction mechanism and online performance fine-tuning.

[0111] Computer-readable storage media may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application program required for at least one function; the data storage area may store data created based on the use of the AI ​​path planning system based on deep reinforcement imitation learning, etc. Furthermore, the computer-readable storage medium may include high-speed random access memory, and may also include memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the computer-readable storage medium may optionally include memory remotely located relative to the processor, which can be connected to the AI ​​path planning system based on deep reinforcement imitation learning via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0112] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 3 As shown, the device includes a processor 310 and a memory 320. The electronic device may also include an input device 330 and an output device 340. The processor 310, memory 320, input device 330, and output device 340 can be connected via a bus or other means. Figure 3 Taking a bus connection as an example, the memory 320 is the computer-readable storage medium described above. The processor 310 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in the memory 320, thereby implementing the AI ​​path planning method based on deep reinforcement imitation learning as described in the above method embodiment. The input device 330 can receive input digital or character information and generate key signal inputs related to user settings and function control of the AI ​​path planning system based on deep reinforcement imitation learning. The output device 340 may include a display screen or other display device.

[0113] The aforementioned electronic device can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0114] In one implementation, the above-described electronic device is applied to an AI path planning system based on deep reinforcement imitation learning, for a client, and includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0115] The system integrates multi-source sensor data from LiDAR, cameras, and odometers, and constructs an environmental state space with temporal correlation by introducing a state correlation matrix for weighted fusion.

[0116] Based on the generative adversarial imitation learning framework, a path planning policy is learned by using a pre-trained expert policy database and adversarial training between a policy generation network and a policy discrimination network. The policy generation network outputs actions based on the environmental state, and the policy discrimination network is used to evaluate the similarity between the generated actions and the expert actions.

[0117] A master-slave distributed architecture is used to train multiple policy network replicas in parallel. By aggregating gradients and using a soft parameter update mechanism, the global parameters of the policy generation network and the policy discrimination network are optimized.

[0118] A lightweight policy generation network is deployed to output action instructions based on the real-time environment status, and real-time path planning results are obtained through a dynamic path correction mechanism and online performance fine-tuning.

[0119] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An AI path planning method based on deep reinforcement imitation learning, characterized in that, include: The system integrates multi-source sensor data from LiDAR, cameras, and odometers, and constructs an environmental state space with temporal correlation by introducing a state correlation matrix for weighted fusion. Based on the generative adversarial imitation learning framework, a path planning policy is learned by using a pre-trained expert policy database and adversarial training between a policy generation network and a policy discrimination network. The policy generation network outputs actions based on the environmental state, and the policy discrimination network is used to evaluate the similarity between the generated actions and the expert actions. A master-slave distributed architecture is used to train multiple policy network replicas in parallel. By aggregating gradients and using a soft parameter update mechanism, the global parameters of the policy generation network and the policy discrimination network are optimized. A lightweight policy generation network is deployed to output action instructions based on the real-time environment status, and real-time path planning results are obtained through a dynamic path correction mechanism and online performance fine-tuning.

2. The AI ​​path planning method based on deep reinforcement imitation learning according to claim 1, characterized in that, The construction of an environmental state space with temporal correlation by introducing a state correlation matrix for weighted fusion includes: The point cloud data of adjacent frames of lidar are registered. Coarse registration is performed by component analysis, and fine registration is performed by an improved point-to-line iterative nearest point algorithm to obtain the set of obstacle coordinates. The basic state space is constructed by fusing the obstacle coordinate set, the robot's precise pose obtained through visual SLAM optimization, the target point coordinates, and the robot's velocity information. Calculate the mutual information entropy of each element in the state space, construct the state correlation matrix, and perform weighted fusion of the basic state space; The state, action, reward, and next state are combined into an experience tuple, which is stored in the priority experience replay pool. The historical state information is then fused through a state decay factor to obtain the dynamic temporal sequence of the updated state space.

3. The AI ​​path planning method based on deep reinforcement imitation learning according to claim 1, characterized in that, The generative adversarial imitation learning framework utilizes a pre-trained expert policy database and learns path planning strategies through adversarial training between a policy generation network and a policy discrimination network. A generative adversarial imitation learning framework is constructed: the policy generation network adopts an Actor-Critic structure, where the Actor part outputs the action probability distribution in the continuous action space, and the Critic part uses the generalized advantage estimation algorithm to calculate the advantage function; the policy discrimination network is used to output the probability that the state-action pair originates from the expert policy. Building an expert strategy database: Collect state-action-reward sequences demonstrated by experts, evaluate the value of expert samples and screen high-quality samples, perform data augmentation by adding adaptive Gaussian noise, and build an expert strategy database; Pre-training and adversarial training: The behavior cloning method is used to pre-train the policy generation network using the expert policy library; in the adversarial training stage, the adversarial loss function and the discriminant loss function are alternately optimized to make the action distribution output by the policy generation network approximate the expert policy.

4. The AI ​​path planning method based on deep reinforcement imitation learning according to claim 1, characterized in that, The expression for the multi-objective reward function of the policy generation network is: R(t)=w1R env (t)+w2R adv (t)+w3R exp (t), R env (t)=r goal I goal -r colide I colide , R exp (t)=-ε||a(t)-a E (t)||, In the formula, R env (t) represents environmental rewards, R adv (t) represents the counter-reward, R exp (t) represents the expert guidance reward, where w1, w2, and w3 are weighting coefficients, and I goal and I colide Both are indicator functions, where d(t) is the distance from the robot to the target point, Δw(t) is the change in angular velocity, and a E (t) represents the expert's optimal action in the corresponding state.

5. The AI ​​path planning method based on deep reinforcement imitation learning according to claim 1, characterized in that, The method employs a master-slave distributed architecture to train multiple policy network replicas in parallel. By aggregating gradients and using a soft parameter update mechanism, the global parameters of the policy generation and policy discrimination networks are optimized, including: Set up a global network and multiple local networks, where the local networks interact and compute gradients in independent environment instances, and the global network is responsible for aggregating the gradients of all local networks and updating the parameters. An adaptive gradient clipping mechanism is adopted to dynamically adjust the gradient clipping threshold based on the number of training steps to prevent gradient explosion. A soft update strategy is employed to synchronize global network parameters to local network parameters, ensuring training stability. The soft update formula is as follows: In the formula, α is the learning rate, and N is the number of local networks. Let be the loss function of the i-th local network.

6. The AI ​​path planning method based on deep reinforcement imitation learning according to claim 1, characterized in that, The lightweight policy generation network is deployed to output action commands based on the real-time environment status, and obtains real-time path planning results through a dynamic path correction mechanism and online performance fine-tuning, including: Lightweight model deployment: The policy generation network that has converged during training is pruned and quantized to convert it into a lightweight model for real-time inference; Real-time motion generation: The robot's sensors collect real-time environmental conditions, which are preprocessed and then input into the lightweight model to output motion commands; the output introduces exploration noise that decays with training maturity. Dynamic path correction: Based on the particle filter algorithm, the position of dynamic obstacles is tracked. When the path safety margin is lower than the threshold, local replanning and emergency obstacle avoidance are triggered. Online performance fine-tuning: The performance is evaluated online based on path length, average speed and collision risk indicators, and the strategy network parameters are fine-tuned to obtain real-time path planning results.

7. An AI path planning system based on deep reinforcement imitation learning, characterized in that, include: The module is configured to fuse multi-source sensor data from LiDAR, camera, and odometer, and constructs an environmental state space with temporal correlation by introducing a state correlation matrix for weighted fusion. The learning module is configured to be based on a generative adversarial imitation learning framework. It utilizes a pre-trained expert policy database and learns path planning policies through adversarial training between a policy generation network and a policy discrimination network. The policy generation network outputs actions based on the environmental state, and the policy discrimination network is used to evaluate the similarity between the generated actions and the expert actions. The optimization module is configured to train multiple policy network replicas in parallel using a master-slave distributed architecture, and optimize the global parameters of the policy generation network and the policy discrimination network through gradient aggregation and parameter soft update mechanism. The output module is configured to deploy a lightweight policy generation network, output action commands based on the real-time environment status, and obtain real-time path planning results through a dynamic path correction mechanism and online performance fine-tuning.

8. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the method described in any one of claims 1 to 6.

Images