AGV self-learning navigation and obstacle avoidance method for large intelligent factory

Abstract

The application discloses an AGV self-learning navigation obstacle avoidance method for large-scale intelligent factories. First, the mobile robots interact with the environment respectively and update their local network parameters. After updating, the local network parameters and the performance of the mobile robots are transmitted to the central server. The central server compares the performance of the global network update of the current round and the global network update of the last round, gives different weights to the local network to update the global network. Then, the updated global network parameters are soft-updated with the local network of each mobile robot to obtain the updated local network. Finally, each mobile robot interacts with the environment again and updates its local network. The above steps are repeated until the obstacle avoidance success rate of all mobile robots reaches a specified threshold. Through the above process, the training of the mobile robots can be accelerated, the obstacle avoidance ability of the mobile robots can be increased, and the data privacy between the mobile robots can be ensured by transmitting only the parameters to the central server.

Description

Technical fields:

[0001] This invention relates to the field of AGV navigation and obstacle avoidance technology, and in particular to a self-learning navigation and obstacle avoidance method for AGVs in large-scale smart factories. Background technology:

[0002] AGVs are widely used in smart factories. The complex tasks in large smart factories require AGVs to reach their destinations as quickly and accurately as possible, while also ensuring the mobile robots can safely and effectively avoid obstacles in the environment. When the environmental map is unknown and relies on perception data from sensors such as LiDAR, deep reinforcement learning has solved many problems in the field of obstacle avoidance for mobile robots. However, the large amount of sensor data leads to slow convergence of deep reinforcement learning models, and the training efficiency for a single AGV is low, making it difficult to quickly train a neural network model with strong generalization capabilities. Therefore, an effective self-learning navigation and obstacle avoidance method for AGVs in large smart factories is needed.

[0003] Distributed machine learning techniques are commonly used to accelerate model training. However, traditional distributed training techniques require the transmission of raw data, which both exposes the privacy of local agents and increases communication overhead. Federated learning, as a novel distributed machine learning technique, can collaboratively train a global model using multiple distributed edge devices without requiring local agents to upload data. This effectively improves model training efficiency, reduces communication costs, and enhances data security.

[0004] In summary, existing AGV navigation and obstacle avoidance methods are inefficient and cannot quickly train a neural network model with navigation and obstacle avoidance capabilities to meet the complex work requirements of large-scale smart factories. Summary of the Invention:

[0005] To address the slow convergence of traditional deep reinforcement learning models and the privacy and communication challenges associated with traditional distributed learning, this invention proposes a self-learning navigation and obstacle avoidance method for AGVs in large-scale smart factories. This method accelerates model training by incorporating federated learning and increases model convergence speed by calculating weights based on growth values.

[0006] A self-learning navigation and obstacle avoidance method for AGVs in large-scale smart factories includes the following steps:

[0007] S1: Each AGV interacts with the environment and updates its local network. It determines whether to perform a global network update. If a global network update is performed, the updated local network parameters of each AGV and the performance of the AGV are transmitted to the central server. Otherwise, it continues to interact with the environment.

[0008] S2: The central server will compare the performance of the AGVs passed in this round of global update with that passed in the previous round, and calculate the growth value of each AGV. When the global update round is the first round, the same growth value will be assigned to the AGVs.

[0009] S3: Set a growth threshold M, and then determine whether the growth value of each AGV is greater than the threshold M. If it is greater than M, the local network of this AGV participates in the model aggregation; otherwise, it does not participate.

[0010] S4: The central server normalizes the growth value of each participating AGV and uses the normalized growth value as the weight of each AGV in the model aggregation. The local network parameters of each AGV are then aggregated to obtain the global model.

[0011] S5: Determine whether the performance of all AGVs in this round of global network update meets the target. If the target is met, the training is completed, the global model is saved, and the process ends. If the target is not met, continue to the next step S6.

[0012] S6: Pass the global model to each AGV, perform a soft update with the local model of the AGV to obtain a new local model, and continue to step S1.

[0013] Furthermore, in step S1, there are K AGVs, and the local network includes 2K Critic networks, K Actor networks, 2K target Critic networks, and K target Actor networks. The network parameters are the Critic network parameters θ. 11,12,…,1K Critic network parameters θ 21,22,…,2K Actor network parameters φ 1,2,...,K Target Critic network parameters θ′ 11,12,…,1K Target Critic network parameters θ′ 21,22,…,2K Target Actor network parameters φ′ 1,2,…,K .

[0014] Furthermore, the interaction between the AGV and the environment in step S includes the following sub-steps:

[0015] S11: Initialize the starting point and target point;

[0016] S12: The AGV starts from the starting point and obtains status information s through the lidar;

[0017] S13: Determine whether a collision has occurred and whether the target point has been reached in the current state. If a collision has occurred or the target point has been reached, end the current round of interactive task, which is defined as completing one round of interactive task, and proceed to step S11. Otherwise, input the state information into the Actor network to obtain AGV action a.

[0018] S14: The AGV executes action a and obtains a new state. It calculates the reward r after executing action a and stores the quadruple (s,a,,r) in the memory playback area.

[0019] S15: Determine whether the number of quadruplets in the memory playback area has reached the predetermined number. If not, return to S12. Otherwise, read quadruplets from the memory playback area, update the AGV local network according to the gradient of the dual-delay depth deterministic strategy, and continue to execute S12.

[0020] Furthermore, in step S1, the criterion for determining whether to perform a global network update is whether all AGVs have completed the M rounds of interactive tasks. If all are completed, a global network update is performed; otherwise, a global network update is not performed.

[0021] Furthermore, the AGV performance in step S2 includes the success rate u and the average reward v in the M-round interactive task, where the average reward v is the average of the reward r obtained in all steps in the M-round interactive task.

[0022] Furthermore, the growth value of the AGV in step S3 is:

[0023]

[0024] In the formula, Let be the growth value of the k-th AGV in the s-th round of global network update, where k represents the k-th AGV, s represents the number of global update rounds, λ∈(0,1) is the adjustment coefficient, u is the success rate in the M-round interactive task, and v is the average reward r obtained for all steps in the M-round interactive task.

[0025] Furthermore, in step S4, the growth value is normalized, and the following formula is used to calculate the normalized growth value:

[0026]

[0027] In the formula, Let be the update weight of the k-th AGV local model in the s-th round of global update, and let e be the natural logarithm. This represents the growth value of the k-th AGV in the s-th round of global update.

[0028] Furthermore, in step S4, the model aggregation is as follows:

[0029]

[0030] In the formula, Here are the global network parameters for the s-th round, K′ is the number of AGVs participating in the model aggregation in this round, and k represents the k-th AGV. For the network parameters of the k-th AGV, This represents the weight coefficient of the k-th AGV in this round of model aggregation.

[0031] Furthermore, the calculation formula for the soft update of the local model in step S6 is as follows:

[0032]

[0033] In the formula, μ∈(0,1) is the soft update parameter. These are the updated local network parameters for the AGV.

[0034] The beneficial effects of this invention compared to the prior art are as follows:

[0035] This invention only requires the transmission of model parameters and growth values, without the need to transmit raw data, thus protecting the data privacy between each AGV and reducing the communication pressure between the AGV and the central server.

[0036] This invention can train multiple AGVs simultaneously in different scenarios, learn obstacle avoidance methods in different environments, accelerate AGV training speed, and improve AGV obstacle avoidance capabilities.

[0037] The present invention uses a growth value update method to quickly select the better performing model parameters, which can accelerate the model convergence speed. Attached image description:

[0038] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0039] Figure 1 This is a flowchart of the AGV self-learning navigation and obstacle avoidance method for large-scale smart factories according to the present invention.

[0040] Figure 2 The image shows a comparison of the training results of the AGV self-learning navigation and obstacle avoidance method for large-scale smart factories provided by this invention with those of traditional deep reinforcement learning algorithms. Detailed implementation method:

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Please see Figure 1An embodiment of the present invention provides a self-learning navigation and obstacle avoidance method for AGVs in large-scale smart factories, comprising the following steps:

[0043] S1: Select K AGVs and initialize the local network of each AGV. The local network includes 2K Critic networks, K Actor networks, 2K target Critic networks, and K target Actor networks. The network parameters are θ, the parameters of the Critic networks. 11,12,…,1K Critic network parameters θ 21,22,…,2K Actor network parameters φ 1,2,…,K Target Critic network parameters θ′ 11,12,…,1K Target Critic network parameters θ′ 21,22,…,2K Target Actor network parameters φ1′ ,2,…,K Each AGV interacts with its environment and updates its local network. This interaction includes the following steps:

[0044] S11: Initialize the starting point and target point.

[0045] S12: The AGV starts from the starting point and obtains status information s through the lidar.

[0046] S13: Determine whether a collision has occurred and whether the target point has been reached in the current state. If a collision has occurred or the target point has been reached, the current round of interactive task ends, which is defined as completing one round of interactive task. Proceed to step S11. Otherwise, input the state information into the Actor network to obtain AGV action a.

[0047] S14: The AGV executes action a and obtains a new state. It calculates the reward r after executing action a and stores the quadruple (s, a, r) ​​in the memory playback area.

[0048] S15: Determine whether the number of quadruplets in the memory playback area has reached the predetermined number. If not, return to S12. Otherwise, read quadruplets from the memory playback area, update the AGV local network according to the gradient of the dual-delay depth deterministic strategy, and continue to execute S12.

[0049] If all AGVs complete the M-round interaction task, a global network update is performed, and the updated local network parameters and AGV performance of each AGV are transmitted to the central server. If not all AGVs complete the M-round interaction task, a global network update is not performed, and the process returns to S11.

[0050] S2: Define AGV performance as including the success rate u and average reward v in M ​​rounds of interactive tasks, where the average reward v is the average of the rewards r obtained for all steps in M ​​rounds of interactive tasks. The central server will compare the AGV performance data passed in this round of global update with that passed in the previous round, and calculate the growth value gv for each AGV:

[0051]

[0052] In the formula, Let be the growth value of the k-th AGV in the s-th round of global network update, where k represents the k-th AGV, s represents the number of global update rounds, λ∈(0,1) is the adjustment coefficient, u is the success rate in the M-round interactive task, and v is the average reward r obtained for all steps in the M-round interactive task. When the global update round is the first round, the same growth value is assigned to each AGV.

[0053] S3: Set a growth threshold M, then determine whether the growth value of each AGV is greater than the threshold M. If it is greater than M, the local network of this AGV participates in model aggregation; otherwise, it does not participate. The threshold M is selected as a real number greater than zero to prevent harmful models from participating in model aggregation.

[0054] S4: The central server normalizes the growth value of each AGV participating in the model aggregation:

[0055]

[0056] In the formula, Let be the update weight of the k-th AGV local model in the s-th round of global update, and let e be the natural logarithm. This represents the growth value of the k-th AGV in the s-th round of global update.

[0057] The normalized growth value is used as the weight for each AGV in the model aggregation, and the local network parameters of each AGV are aggregated in the model:

[0058]

[0059] In the formula, Here are the global network parameters for the s-th round, K′ is the number of AGVs participating in the model aggregation in this round, and k represents the k-th AGV. For the network parameters of the k-th AGV, This represents the weight coefficient of the k-th AGV in this round of model aggregation.

[0060] Model aggregation yields a global model

[0061] S5: Determine whether the performance of all AGVs in this round of global network update meets the target. If the target is met, the training is completed, the global model is saved, and the process ends. If the target is not met, continue to the next step S6.

[0062] S6: Pass the global model to each AGV and perform soft updates with the local models of the AGVs:

[0063]

[0064] In the formula, μ∈(0,1) is the soft update parameter. These are the updated local network parameters for the AGV.

[0065] AGV soft updates yield new local models. Continue with step S1;

[0066] Please see Figure 2 The self-learning navigation and obstacle avoidance method for AGVs in large-scale smart factories provided by this invention is compared with the training results of traditional deep reinforcement learning algorithms.

[0067] Traditional deep reinforcement learning algorithms require approximately 175 rounds for the model to converge, with a cumulative reward of around 3100 after convergence. In contrast, the method described in this invention only requires 110 rounds for convergence, with a cumulative reward of around 3920 after convergence. Comparatively, this method improves model training efficiency by approximately 59.1% and obstacle avoidance performance by approximately 26.5% compared to traditional deep reinforcement learning algorithms.

[0068] In summary, the self-learning navigation and obstacle avoidance method for AGVs (Automated Guided Vehicles) in large-scale smart factories provided by this invention involves the following steps: First, each mobile robot interacts with its environment and updates its local network parameters. After updating, the local network parameters and the mobile robot's performance are transmitted to a central server. The central server compares the performance of the mobile robots in the current and previous rounds of global network updates, assigning different weights to the local networks to update the global network. Then, the updated global network parameters are soft-updated with the local networks of each mobile robot to obtain the updated local networks. Finally, each mobile robot interacts with its environment again and updates its local network, repeating the above steps until the obstacle avoidance success rate of all mobile robots reaches a specified threshold. Through this process, federated adaptive learning can accelerate mobile robot training, increase the obstacle avoidance capability of mobile robots, and ensure data privacy among mobile robots by only transmitting parameters to the central server.

[0069] In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0070] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A self-learning navigation and obstacle avoidance method for AGVs in large-scale smart factories, characterized in that, Includes the following steps: S1: Each AGV interacts with the environment and updates its local network. It determines whether to perform a global network update. If a global network update is performed, the updated local network parameters of each AGV and the performance of the AGV are transmitted to the central server. Otherwise, it continues to interact with the environment. S2: The central server will compare the performance of the AGVs passed in this round of global update with that passed in the previous round, and calculate the growth value of each AGV. When the global update round is the first round, the same growth value will be assigned to the AGVs. S3: Set a growth threshold M, and then determine whether the growth value of each AGV is greater than the threshold M. If it is greater than M, the local network of this AGV participates in the model aggregation; otherwise, it does not participate. The growth value of AGV in step S3 is: In the formula, Let be the growth value of the k-th AGV in the s-th round of global network update, where k represents the k-th AGV and s represents the number of global update rounds. The adjustment coefficient is u, which is the success rate in the M-round interactive task, and v is the average reward r obtained for all steps in the M-round interactive task. S4: The central server normalizes the growth value of each participating AGV and uses the normalized growth value as the weight of each AGV in the model aggregation. The local network parameters of each AGV are then aggregated to obtain the global model. In step S4, the model aggregation is as follows: In the formula, These are global network parameters. This represents the number of AGVs participating in the model aggregation in this round, where k represents the k-th AGV. For the network parameters of the k-th AGV, This refers to the weight coefficient of the k-th AGV in this round of model aggregation; S5: Determine whether the performance of all AGVs in this round of global network update meets the target. If the target is met, the training is completed, the global model is saved, and the process ends. If the target is not met, continue to the next step S6. S6: Pass the global model to each AGV, perform a soft update with the local model of the AGV to obtain a new local model, and continue to step S1.

2. The AGV self-learning navigation and obstacle avoidance method for large-scale smart factories according to claim 1, characterized in that, In step S1, there are K AGVs, and the local network includes 2K Critic networks, K Actor networks, 2K target Critic networks, and K target Actor networks. The network parameters are as follows: Critic network parameters... Critic network parameters Actor network parameters Target Critic network parameters Target Critic network parameters Target Actor network parameters .

3. The AGV self-learning navigation and obstacle avoidance method for large-scale smart factories according to claim 1, characterized in that, The interaction between the AGV and the environment in step S1 includes the following steps: S11: Initialize the starting point and target point; S12: The AGV starts from the starting point and obtains status information s through the lidar; S13: Determine whether a collision has occurred and whether the target point has been reached in the current state. If a collision has occurred or the target point has been reached, end the current round of interactive task, which is defined as completing one round of interactive task, and proceed to step S11. Otherwise, input the state information into the Actor network to obtain AGV action a. S14: The AGV executes action a and obtains a new state. Calculate the reward r after performing action a, and combine the quadruple (s, a, ... (r) Stored in the memory playback area; S15: Determine whether the number of quadruplets in the memory playback area has reached the predetermined number. If not, return to S12. Otherwise, read quadruplets from the memory playback area, update the AGV local network according to the gradient of the dual-delay depth deterministic strategy, and continue to execute S12.

4. The AGV self-learning navigation and obstacle avoidance method for large-scale smart factories according to claim 1, characterized in that, In step S1, the criterion for determining whether to perform a global network update is whether all AGVs have completed the M rounds of interactive tasks. If all AGVs have completed the task, a global network update is performed; otherwise, a global network update is not performed.

5. The AGV self-learning navigation and obstacle avoidance method for large-scale smart factories according to claim 1, characterized in that, The AGV performance in step S2 includes the success rate u and the average reward v in the M-round interactive task, where the average reward v is the average of the reward r obtained in all steps in the M-round interactive task.

6. The AGV self-learning navigation and obstacle avoidance method for large-scale smart factories according to claim 1, characterized in that, The formula for normalizing the growth value in step S4 is as follows: In the formula, Let be the update weight of the k-th AGV local model in the s-th round of global update, and let e be the natural logarithm. This represents the growth value of the k-th AGV in the s-th round of global update.

7. The AGV self-learning navigation and obstacle avoidance method for large-scale smart factories according to claim 1, characterized in that, The calculation formula for soft updating of the local model in step S6 is as follows: In the formula, For soft update parameters, For the updated AGV local network parameters, These are global network parameters. Let be the network parameters of the k-th AGV.

Images