Distributed cloud robot collaboration method and device based on reinforcement learning
Through distributed reinforcement learning and information sharing between cloud robots, the problem of cloud robots being insensitive to local environmental information and low work efficiency is solved, achieving more accurate environment characterization and efficient work.
Patent Information
- Application Number
- CN202310887295.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-18
- Publication Date
- 2025-01-21
AI Technical Summary
The existing reinforcement learning methods of cloud robots mostly adopt the centralized learning distributed execution method, which leads to mutual influence between robots and low frequency of uploading local environmental information, resulting in cloud robots being insensitive to local environmental information, inaccurate portrayal and low work efficiency.
Using a distributed cloud robot collaboration method based on reinforcement learning, the first cloud robot performs reinforcement learning, updates its local environment information and reinforcement learning results, and sends this information to the second cloud robot to achieve distributed reinforcement learning and information shared.
It improves the sensitivity and characterization accuracy of cloud robots to local environmental information, avoids repeated exploration, improves work efficiency, and enables multiple robots to perform distributed reinforcement learning at the same time.
Smart Images
Figure CN119342419A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to robotics technology, and in particular to a distributed cloud robot collaboration method, device and storage medium based on reinforcement learning. Background Art
[0002] With the further development of artificial intelligence, robots, as important service equipment, play an irreplaceable role in smart life. Mobile robots are widely used in modern life. For example, they can be found in various fields such as home, agriculture, industry, and military. The three core areas in the research field of controlling robot movement are robot positioning, task allocation, and path planning technology. Among them, path planning is the primary condition for mobile robots to reach task targets and complete task content. For example: household service cleaning robots need to make reasonable path planning for indoor environments to complete cleaning tasks; agricultural picking robots need path planning to travel between crops to complete picking tasks; industrial robots also need path planning to complete given tasks in shared workspaces.
[0003] Robot system path planning refers to finding a path from the starting state to the target state in its working environment that can avoid all obstacles. This requires the use of machine learning. In related technologies, the most commonly used learning method is reinforcement learning. The Q-Learning algorithm is an important algorithm in reinforcement learning. In related technologies, the Q-Learning algorithm is applied to the path planning of robot systems. The learning process of the Q-Learning algorithm is an iterative process. It needs to gradually update the reinforcement learning results, that is, the Q-value table (action value function) through continuous trial and error and action selection. That is: a reward function is set, the robot selects action instructions according to the greedy strategy (ε-greedy, ε is the exploration factor (0≤ε≤1)), executes the action instructions and updates the Q-value table according to the reward, then generates state parameters, selects the next action according to the state parameters and the greedy strategy, and then continues to execute the action instructions and updates the Q-value table until the final Q-value table is updated, and finally the optimal path is obtained according to the Q-value table.
[0004] Existing cloud robot reinforcement learning methods mostly adopt the method of centralized learning and distributed execution. Centralized learning leads to mutual influence between robots, and the low-frequency local environment information uploaded by the robots is lost in the centralized reinforcement learning process, which makes the cloud robot insensitive to local environment information. There are technical problems such as inaccurate characterization of the local environment by the cloud robot and low work efficiency. Summary of the invention
[0005] The present application provides a distributed cloud robot collaboration method, device and storage medium based on reinforcement learning, which are used to solve the technical problems of inaccurate local environment characterization and low working efficiency of cloud robots.
[0006] On the one hand, the present application provides a distributed cloud robot collaboration method based on reinforcement learning, which is applied to a first cloud robot side, including:
[0007] Performing reinforcement learning based on the first cloud robot, updating the local environment information and reinforcement learning results of the first cloud robot;
[0008] When there is a second cloud robot within the communication range of the first cloud robot, the updated local environment information and reinforcement learning results of the first cloud robot are sent to the second cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot;
[0009] When there is no second cloud robot within the communication range of the first cloud robot, the first cloud robot sends the updated local environment information and reinforcement learning results of the first cloud robot to the second cloud robot through the wireless access point AP associated with the first cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot.
[0010] Optionally, performing reinforcement learning based on the first cloud robot and updating the local environment information and reinforcement learning results of the first cloud robot include:
[0011] The first cloud robot is abstracted as an intelligent agent, the process of the first cloud robot taking actions in the local environment and obtaining feedback information about the actions is modeled as an abstract Markov decision process, and the task requirements of the first cloud robot in the local environment are described as signal timing logic tasks; wherein the Markov decision process model is expressed as: (S, A, P, R, γ), S is the state set of the first cloud robot, A is the action set of the first cloud robot, P is the state transition probability, which indicates the probability that the first cloud robot changes from the first state to the second state after taking action a, R is the reward function, which indicates the reward value obtained after the first cloud robot changes from the first state to the second state, γ is the attenuation factor, γ≤1;
[0012] Based on the Markov decision process model and signal sequential logic tasks, the local environment information and reinforcement learning results of the First Cloud Robot are updated through the reinforcement learning algorithm.
[0013] Optionally, the reinforcement learning algorithm is a Q learning algorithm, the reinforcement learning result is a Q value table, the input parameters of the Q learning algorithm include S, A, P, R and γ, and based on the Markov decision process model and the signal timing logic task, the local environment information and reinforcement learning result of the first cloud robot are updated through the reinforcement learning algorithm, including:
[0014] Step 1: Initialize the Q value table, which is used to store state-action pairs Q(S t ,a t ), let Q(S t ,a t )=Q0, time step t=0, initial state S=S0; where a t Used to represent the action selected by the Agent at time t; S t Used to characterize the state of the Agent at time t; Q(s t ,a t ) is used to characterize the state S t Next Agent performs action a t The cumulative weighted reward returned after
[0015] Step 2: Select action a based on the greedy strategy ∈-greedy t ; Among them, action a t Belongs to action set A, A={a t ~π(S t )},π(S t ) is used to represent the current state at time t as S t When the agent selects an action based on the ∈-greedy strategy;
[0016] Step 3: Execute action a t , get real-time reward return r t+1 and state S t+1 ; Among them, the real-time reward return r t+1 Used to represent the Agent performing action a at time t+1 t Reward return; state S t+1 Used to represent the Agent performing action a t The next state after
[0017] Step 4: Based on S t , a t , r t+1 , S t+1 Update the Q value table, that is, update the state-action pair Q(S t ,a t );
[0018] Step 5: Update the greedy strategy;
[0019] Step 6: Let t = t + 1, S t =S t+1 , if t mod N B =0 means t can divide the preset threshold N B Then execute step 7, otherwise execute step 2;
[0020] Step 7: Execute the Grassfire algorithm, start the width search from the current position of the agent, until the position that has not been explored by the agent is found, and set the state of the position that has not been explored by the agent as the current state of the robot S t , proceed to step 2.
[0021] Optionally, based on S t , a t , r t+1 , S t+1 And the following formula updates the state-action pair Q(S t ,a t ):
[0022]
[0023] In the formula, Q(S t ,a t ) means in state S t Next Agent performs action a t The cumulative weighted reward obtained after the agent is trained; α refers to the learning efficiency of the agent; a t+1 Refers to the next action after the Agent performs an action.
[0024] Optionally, when there is a second cloud robot within the communication range of the first cloud robot, the updated local environment information and Q value table of the first cloud robot are sent to the second cloud robot, so that the second cloud robot updates the local environment information of the second cloud robot based on the local environment information of the first cloud robot and updates the Q value table of the second cloud robot based on the Q value table of the first cloud robot;
[0025] When there is no second cloud robot within the communication range of the first cloud robot, the first cloud robot sends the updated local environment information and Q value table of the first cloud robot to the second cloud robot through the wireless access point AP associated with the first cloud robot, so that the second cloud robot updates the local environment information and Q value table of the second cloud robot based on the local environment information and Q value table of the first cloud robot.
[0026] In the second aspect, the present application proposes a distributed cloud robot collaboration method based on reinforcement learning, which is applied to the second cloud robot side, including:
[0027] In the case where the first cloud robot exists within the communication range of the second cloud robot, the second cloud robot receives the updated local environment information and reinforcement learning results of the first cloud robot sent by the first cloud robot, and updates the local environment information and reinforcement learning results of the second cloud robot;
[0028] When the first cloud robot does not exist within the communication range of the second cloud robot, the second cloud robot receives the updated local environment information and reinforcement learning results of the first cloud robot sent by the first cloud robot through the wireless access point AP associated with the first cloud robot, and updates the local environment information and reinforcement learning results of the second cloud robot.
[0029] Optionally, the present application proposes a distributed cloud robot collaboration method based on reinforcement learning, which is applied to the second cloud robot side, the reinforcement learning algorithm is a Q learning algorithm, and the reinforcement learning result is a Q value table for storing state-action pairs. The local environment information of the second cloud robot and the reinforcement learning result are updated, including:
[0030] The second cloud robot updates the local environment information of the second cloud robot based on the updated local environment information of the first cloud robot;
[0031] The second cloud robot updates the Q value table of the second cloud robot based on the updated Q value table of the first cloud robot.
[0032] Optionally, the second cloud robot updates the Q value table of the second cloud robot based on the updated Q value table of the first cloud robot, including:
[0033] When the updated Q value table of the first cloud robot has the first state-action pair, and the Q value table of the second cloud robot does not have the first state-action pair, updating the first state-action pair to the Q value table of the second cloud robot;
[0034] When there is a second state-action pair with an initial value in the Q value table of the second cloud robot, and the value of the second state-action pair in the Q value table of the first cloud robot is not the initial value, the value of the second state-action pair in the Q value table of the second cloud robot is updated to the value of the second state-action pair in the Q value table of the first cloud robot.
[0035] In a third aspect, the present application provides a distributed cloud robot collaborative device based on reinforcement learning, which is applied to a first cloud robot side, including:
[0036] An updating module, configured to perform reinforcement learning based on the first cloud robot and update the local environment information and reinforcement learning results of the first cloud robot;
[0037] A first sending module is used to send the updated local environment information and reinforcement learning results of the first cloud robot to the second cloud robot when there is a second cloud robot within the communication range of the first cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot;
[0038] The second sending module is used for sending the updated local environment information and reinforcement learning results of the first cloud robot to the second cloud robot through the wireless access point AP associated with the first cloud robot when there is no second cloud robot within the communication range of the first cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot.
[0039] In a fourth aspect, the present application provides a distributed cloud robot collaborative device based on reinforcement learning, which is applied to the second cloud robot side, including:
[0040] A first processing module is configured to, when the first cloud robot exists within the communication range of the second cloud robot, cause the second cloud robot to receive the updated local environment information and reinforcement learning results of the first cloud robot sent by the first cloud robot, and update the local environment information and reinforcement learning results of the second cloud robot;
[0041] The second processing module is used for, when the first cloud robot does not exist within the communication range of the second cloud robot, the second cloud robot receives the updated local environment information and reinforcement learning results of the first cloud robot sent by the first cloud robot through the wireless access point AP associated with the first cloud robot, and updates the local environment information and reinforcement learning results of the second cloud robot.
[0042] In a fifth aspect of the present application, a distributed cloud robot collaborative device based on reinforcement learning is provided, comprising:
[0043] Processor and memory;
[0044] Memory stores computer-executable instructions;
[0045] The processor executes the computer-executable instructions stored in the memory, so that the distributed cloud robot collaboration device based on reinforcement learning performs the distributed cloud robot collaboration method based on reinforcement learning of any one of the first aspect or the second aspect.
[0046] In a sixth aspect, the present application provides a computer-readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed by a processor, they are used to implement a distributed cloud robot collaboration method based on reinforcement learning as described in any one of the first aspect or the second aspect.
[0047] The distributed cloud robot collaboration method, device and storage medium based on reinforcement learning provided by the present application perform reinforcement learning based on the first cloud robot to update the local environment information and reinforcement learning results of the first cloud robot; when there is a second cloud robot within the communication range of the first cloud robot, the updated local environment information and reinforcement learning results of the first cloud robot are sent to the second cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot; when there is no second cloud robot within the communication range of the first cloud robot, the first cloud robot sends the updated local environment information and reinforcement learning results of the second cloud robot through the wireless access point AP associated with the first cloud robot. The local environment information and reinforcement learning results of one cloud robot are sent to the second cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot, thereby realizing distributed reinforcement learning based on the first cloud robot, updating the local environment information and reinforcement learning results of the first cloud robot, and sending the updated local environment information and reinforcement learning results directly or through an associated AP to the second cloud robot within the wireless communication range, thereby achieving the technical effect of multiple robots performing distributed reinforcement learning at the same time, accurately characterizing the local environment, integrating the local environment and reinforcement learning results of the robots, and improving the work efficiency of the cloud robots. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0049] Figure 1 Distributed cloud robot collaboration method process based on reinforcement learning provided in the embodiment of the present application Figure 1 ;
[0050] Figure 2 Distributed cloud robot collaboration method process based on reinforcement learning provided in the embodiment of the present application Figure 2 ;
[0051] Figure 3 Distributed cloud robot collaboration method process based on reinforcement learning provided in the embodiment of the present application Figure 3 ;
[0052] Figure 4Distributed cloud robot collaboration method process based on reinforcement learning provided in the embodiment of the present application Figure 4 ;
[0053] Figure 5 A schematic diagram of a distributed cloud robot collaborative device based on reinforcement learning provided in an embodiment of the present application Figure 1 ;
[0054] Figure 6 A schematic diagram of a distributed cloud robot collaborative device based on reinforcement learning provided in an embodiment of the present application Figure 2 ;
[0055] Figure 7 Hardware structure diagram of the distributed cloud robot collaborative device based on reinforcement learning provided in an embodiment of the present application.
[0056] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0057] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0058] First, the terms involved in this application are explained:
[0059] Cloud robotics technology: refers to the centralized processing of data using the resources provided by cloud computing. In the process of interaction between the robot and the environment, the specific interactive data processing process is as follows: the robot sends the collected environmental information to the cloud platform of the manufacturer to which the robot belongs; the cloud platform determines the action instructions corresponding to the environmental information sent by the robot based on the pre-stored collaboration strategy, and sends the action instructions to the robot.
[0060] Collaboration strategy: refers to the rules that the cloud platform uses to determine the corresponding action instructions based on environmental information.
[0061] The robot performs corresponding actions according to the action instructions sent by the cloud platform, and obtains the action feedback information generated by the robot itself after the action is executed. The robot uploads the action information, action feedback information and environmental information of the executed action to the cloud platform so that the cloud platform can perform reinforcement learning. The collaborative strategy stored in the cloud platform is obtained after the cloud platform regularly conducts centralized reinforcement learning on the action information, action feedback information and environmental information uploaded by multiple robots under it. Existing cloud robot reinforcement learning methods mostly adopt the method of centralized learning and distributed execution. Centralized learning has mutual influence between robots, and the low-frequency local environmental information uploaded by the robot is lost in the centralized reinforcement learning process, resulting in the cloud robot being insensitive to local environmental information. There are technical problems that the cloud robot is not accurate in portraying the local environment and has low work efficiency.
[0062] The distributed cloud robot collaboration method, device and storage medium based on reinforcement learning provided by the present application perform reinforcement learning based on the first cloud robot to update the local environment information and reinforcement learning results of the first cloud robot; when there is a second cloud robot within the communication range of the first cloud robot, the updated local environment information and reinforcement learning results of the first cloud robot are sent to the second cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot; when there is no second cloud robot within the communication range of the first cloud robot, the first cloud robot sends the updated local environment information and reinforcement learning results of the second cloud robot through the wireless access point AP associated with the first cloud robot. The local environment information and reinforcement learning results of one cloud robot are sent to the second cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot, thereby realizing distributed reinforcement learning based on the first cloud robot, updating the local environment information and reinforcement learning results of the first cloud robot, and sending the updated local environment information and reinforcement learning results directly or through an associated AP to the second cloud robot within the wireless communication range, thereby achieving the technical effect of multiple robots performing distributed reinforcement learning at the same time, accurately characterizing the local environment, integrating the local environment and reinforcement learning results of the robots, and improving the work efficiency of the cloud robots.
[0063] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0064] Figure 1 Distributed cloud robot collaboration method process based on reinforcement learning provided in the embodiment of the present application Figure 1 .like Figure 1 As shown, the method of this embodiment is applied to the first cloud robot side, and the method includes:
[0065] S101, performing reinforcement learning based on the first cloud robot, updating the local environment information and reinforcement learning results of the first cloud robot;
[0066] In this embodiment, the first cloud robot is any cloud robot in the distributed cloud robots that perform reinforcement learning; reinforcement learning can use the Q-Learning algorithm combined with the Grassfire algorithm for path planning of the robot system. The learning process of the Q-Learning algorithm is an iterative process, which requires continuous trial and error and action selection to gradually update the reinforcement learning results, that is, update the Q value table (action value function). That is: a reward function is set, the robot selects action instructions according to a greedy strategy (ε-greedy, ε is the exploration factor (0≤ε≤1)), executes the action instructions and updates the Q value table according to the reward, then generates state parameters, selects the next action according to the state parameters and the ε-greedy strategy, and then continues to execute the action instructions and updates the Q value table, and finally obtains the optimal path according to the Q value table; in this application, the Q-Learning algorithm is combined with the Grassfire algorithm. After updating the Q value table, a width search is performed from the current position of the robot based on the Grassfire algorithm until an unexplored position is found, and the state of the unexplored position is set as the current state of the robot, and then the action instructions are selected and executed according to the greedy strategy, which reduces the number of repeated explorations of the cloud robot and improves the work efficiency of the cloud robot.
[0067] S102: When there is a second cloud robot within the communication range of the first cloud robot, send the updated local environment information and reinforcement learning results of the first cloud robot to the second cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot;
[0068] In this embodiment, the first cloud robot can directly send local environment information and reinforcement learning results to the second cloud robot, so that the first cloud robot and the second cloud robot share updated local environment information and reinforcement learning results, avoiding the second cloud robot from repeatedly exploring the location that has been explored by the first cloud robot, reducing the number of repeated explorations by the cloud robot, and improving the work efficiency of the cloud robot.
[0069] S103. When there is no second cloud robot within the communication range of the first cloud robot, the first cloud robot sends the updated local environment information and reinforcement learning results of the first cloud robot to the second cloud robot through the wireless access point AP associated with the first cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot.
[0070] In this embodiment, the first cloud robot can send local environment information and reinforcement learning results to the second cloud robot through the wireless access point AP associated with the first cloud robot. Even if the first cloud robot is far away from the second cloud robot, they can share the updated local environment information and reinforcement learning results, thereby avoiding the second cloud robot from repeatedly exploring the location that has been explored by the first cloud robot, reducing the number of repeated explorations by the cloud robot and improving the work efficiency of the cloud robot.
[0071] The distributed cloud robot collaboration method, device and storage medium based on reinforcement learning provided in the present application perform reinforcement learning based on the first cloud robot to update the local environment information and reinforcement learning results of the first cloud robot; when there is a second cloud robot within the communication range of the first cloud robot, the updated local environment information and reinforcement learning results of the first cloud robot are sent to the second cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot; when there is no second cloud robot within the communication range of the first cloud robot, the first cloud robot sends the updated local environment information and reinforcement learning results of the first cloud robot to the second cloud robot through the wireless access point AP associated with the first cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot, thereby realizing distributed reinforcement learning based on the first cloud robot, updating the local environment information and reinforcement learning results of the first cloud robot, achieving the technical effect that multiple robots perform distributed reinforcement learning at the same time, integrate the local environment and reinforcement learning results of the robots, and improve the work efficiency of the cloud robots.
[0072] Figure 2 The process of the distributed cloud robot collaboration method based on reinforcement learning provided in the embodiment of the present application Figure 2 ,like Figure 2 As shown, the distributed cloud robot collaboration method based on reinforcement learning provided in the embodiment of the present application is applied to the first cloud robot side, the reinforcement learning algorithm is a Q learning algorithm, and the reinforcement learning result is a Q value table, including:
[0073] S201: Abstract the first cloud robot as an intelligent agent, model the process of the first cloud robot taking action in the local environment and obtaining feedback information about the action as an abstract Markov decision process, and describe the task requirements of the first cloud robot in the local environment as a signal timing logic task; wherein the Markov decision process model is expressed as: (S, A, P, R, γ), S is the state set of the first cloud robot, A is the action set of the first cloud robot, P is the state transition probability, indicating the probability that the first cloud robot changes from the first state to the second state after taking action a, R is the reward function, indicating the reward value obtained after the first cloud robot changes from the first state to the second state, γ is the attenuation factor, γ≤1;
[0074] In this embodiment, the attenuation factor γ is used to reflect the valuation ratio of future rewards at the current moment. The attenuation factor determines the degree of influence of the distance of time on the reward, indicating the degree of sacrificing current interests in exchange for long-term benefits. When γ approaches 0, it indicates an immediate return, and when γ approaches 1, it indicates a future return.
[0075] In this embodiment, the Agent receives the input state S in the external environment. t , and output the corresponding action a through the internal reasoning mechanism t ; in a t Under the action of t+1 , and generate real-time reward returns r t+1 ; r t+1 It is the condition of the external environment being S t Agent performs action a in the case t If the behavior strategy obtains positive returns and gets rewards from the external environment, the Agent's tendency to choose action will increase, otherwise the tendency will decrease.
[0076] S202: Based on the Markov decision process model and the signal sequential logic task, the local environment information and reinforcement learning results of the First Cloud Robot are updated through the reinforcement learning algorithm, including:
[0077] Step 1: Initialize the Q value table, which is used to store state-action pairs Q(S t ,a t ), let Q(S t ,a t )=Q0, time step t=0, initial state S=S0; where a t Used to represent the action selected by the Agent at time t; S t Used to characterize the state of the Agent at time t; Q(s t ,a t ) is used to characterize the state S t Next Agent performs action at The cumulative weighted reward returned after
[0078] Step 2: Select action a based on the greedy strategy ∈-greedy t ; Among them, action a t Belongs to action set A, A={a t ~π(S t )},π(S t ) is used to represent the current state at time t as S t When the agent selects an action based on the ∈-greedy strategy;
[0079] In this embodiment, in the Q learning algorithm, the greedy strategy means that the Agent always selects the action with the highest Q value in the current state.
[0080] Step 3: Execute action a t , get real-time reward return r t+1 and state S t+1 ; Among them, the real-time reward return r t+1 Used to represent the Agent performing action a at time t+1 t Reward return; state S t+1 Used to represent the Agent performing action a t The next state after
[0081] Step 4: Based on S t , a t , r t+1 , S t+1 And the following formula updates the Q value table, that is, updates the state-action pair Q(S t ,a t );
[0082]
[0083] In the formula, Q(S t ,a t ) means in state S t Next Agent performs action a t The cumulative weighted reward obtained after the agent is trained; α refers to the learning efficiency of the agent; a t+1 It means that the Agent performs action a t The next action after
[0084] Step 5: Update the greedy strategy;
[0085] Step 6: Let t = t + 1, S t =S t+1 , if t mod N B =0 means t can divide the preset threshold NB Then execute step 7, otherwise execute step 2;
[0086] Step 7: Execute the Grassfire algorithm, start the width search from the current position of the agent, until the position that has not been explored by the agent is found, and set the state of the position that has not been explored by the agent as the current state of the robot S t , execute step 2;
[0087] In this embodiment, S202 involves updating the local environment information of the first cloud robot and the reinforcement learning results through the reinforcement learning algorithm, which is a cyclic reinforcement learning process. By combining the Q-Learning algorithm with the Grassfire algorithm, the cloud robot freely learns the preset threshold N B After the step, the Grassfire algorithm starts a width search from the current position of the robot until an unexplored position is found, and the state of the unexplored position is set as the current state of the robot. Then, the action command is selected and executed according to the greedy strategy, which reduces the number of repeated explorations of the cloud robot and improves the work efficiency of the cloud robot. Among them, the preset threshold N B Is a positive integer.
[0088] S203, when there is a second cloud robot within the communication range of the first cloud robot, sending the updated local environment information and Q value table of the first cloud robot to the second cloud robot, so that the second cloud robot updates the local environment information of the second cloud robot based on the local environment information of the first cloud robot and updates the Q value table of the second cloud robot based on the Q value table of the first cloud robot;
[0089] S204: When there is no second cloud robot within the communication range of the first cloud robot, the first cloud robot sends the updated local environment information and Q value table of the first cloud robot to the second cloud robot through the wireless access point AP associated with the first cloud robot, so that the second cloud robot updates the local environment information and Q value table of the second cloud robot based on the local environment information and Q value table of the first cloud robot.
[0090] In this embodiment, after the Q value of the distributed cloud robot collaboration method based on reinforcement learning provided by the present application is updated during the first cloud robot's cyclic reinforcement learning process, the information integration process involved in S203 and S204 is executed based on the updated local environment information and Q value table.
[0091] The distributed cloud robot collaboration method, device and storage medium based on reinforcement learning provided by the present application perform reinforcement learning based on the first cloud robot to update the local environment information and reinforcement learning results of the first cloud robot; when there is a second cloud robot within the communication range of the first cloud robot, the updated local environment information and reinforcement learning results of the first cloud robot are sent to the second cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot; when there is no second cloud robot within the communication range of the first cloud robot, the first cloud robot sends the updated local environment information and reinforcement learning results of the first cloud robot through the wireless access point AP associated with the first cloud robot. The environmental information and reinforcement learning results are sent to the second cloud robot, so that the second cloud robot updates the local environmental information and reinforcement learning results of the second cloud robot based on the local environmental information and reinforcement learning results of the first cloud robot, thereby realizing distributed reinforcement learning based on the first cloud robot, updating the local environmental information and reinforcement learning results of the first cloud robot, and sending the updated local environmental information and reinforcement learning results directly or through the associated AP to the second cloud robot within the wireless communication range, so as to achieve the technical effect of multiple robots performing distributed reinforcement learning at the same time, accurately characterizing the local environment to achieve the global optimum, integrating the local environment and reinforcement learning results of the cloud robots, realizing efficient group collaboration, and improving the work efficiency of the cloud robots.
[0092] Figure 3 Distributed cloud robot collaboration method process based on reinforcement learning provided in the embodiment of the present application Figure 3 .like Figure 3 As shown, the method of this embodiment is applied to the second cloud robot side, and the method includes:
[0093] S301: When the first cloud robot exists within the communication range of the second cloud robot, the second cloud robot receives the updated local environment information and reinforcement learning results of the first cloud robot sent by the first cloud robot, and updates the local environment information and reinforcement learning results of the second cloud robot;
[0094] S302. When the first cloud robot does not exist within the communication range of the second cloud robot, the second cloud robot receives the updated local environment information and reinforcement learning results of the first cloud robot sent by the first cloud robot through the wireless access point AP associated with the first cloud robot, and updates the local environment information and reinforcement learning results of the second cloud robot.
[0095] The distributed cloud robot collaboration method, device and storage medium based on reinforcement learning provided by the present application are as follows: when the first cloud robot exists within the communication range of the second cloud robot, the second cloud robot receives the updated local environment information and reinforcement learning results of the first cloud robot sent by the first cloud robot, and updates the local environment information and reinforcement learning results of the second cloud robot; when the first cloud robot does not exist within the communication range of the second cloud robot, the second cloud robot receives the updated local environment information and reinforcement learning results of the first cloud robot sent by the first cloud robot through the wireless access point AP associated with the first cloud robot, and updates the local environment information and reinforcement learning results of the second cloud robot, thereby achieving the technical effect of receiving the updated local environment information and reinforcement learning results of the first cloud robot sent by the first cloud robot or the wireless access point AP within the communication range, and updating the local environment information and reinforcement learning results of the second cloud robot, so that the second cloud robot integrates the local environment information and reinforcement learning results of the first cloud robot updated through distributed learning, avoids multiple cloud robots from repeatedly exploring the same position, and thus achieves the technical effect of improving the working efficiency of the cloud robot.
[0096] Figure 4 Distributed cloud robot collaboration method process based on reinforcement learning provided in the embodiment of the present application Figure 4 .like Figure 4 As shown, the method of this embodiment is applied to the second cloud robot side, the reinforcement learning algorithm is a Q learning algorithm, and the reinforcement learning result is a Q value table for storing state-action pairs. The method includes:
[0097] S401, when the first cloud robot exists within the communication range of the second cloud robot, the second cloud robot receives the updated local environment information and Q value table of the first cloud robot sent by the first cloud robot; when the first cloud robot does not exist within the communication range of the second cloud robot, the second cloud robot receives the updated local environment information and Q value table of the first cloud robot sent by the first cloud robot through the wireless access point AP associated with the first cloud robot;
[0098] S402, the second cloud robot updates the local environment information of the second cloud robot based on the updated local environment information of the first cloud robot;
[0099] S403, when the updated Q value table of the first cloud robot has a first state-action pair, and the Q value table of the second cloud robot does not have the first state-action pair, update the first state-action pair to the Q value table of the second cloud robot;
[0100] S404. When there is a second state-action pair with an initial value in the Q value table of the second cloud robot, and the value of the second state-action pair in the Q value table of the first cloud robot is not the initial value, update the value of the second state-action pair in the Q value table of the second cloud robot to the value of the second state-action pair in the Q value table of the first cloud robot.
[0101] The distributed cloud robot collaboration method, device and storage medium based on reinforcement learning provided by the present application are as follows: when the first cloud robot exists within the communication range of the second cloud robot, the second cloud robot receives the updated local environment information and reinforcement learning results of the first cloud robot sent by the first cloud robot, and updates the local environment information and reinforcement learning results of the second cloud robot; when the first cloud robot does not exist within the communication range of the second cloud robot, the second cloud robot receives the updated local environment information and reinforcement learning results of the first cloud robot sent by the first cloud robot through the wireless access point AP associated with the first cloud robot, and updates the local environment information and reinforcement learning results of the second cloud robot The learning results are updated, thereby achieving the goal of receiving the updated local environment information and reinforcement learning results of the first cloud robot sent by the first cloud robot through the first cloud robot or the wireless access point AP within the communication range, and updating the local environment information and reinforcement learning results of the second cloud robot, so that the second cloud robot integrates the local environment information and reinforcement learning results updated by the first cloud robot through distributed learning, avoiding local optimality while avoiding multiple cloud robots from repeatedly exploring the same position, thereby achieving the technical effect of accurately characterizing the local environment to achieve global optimality, integrating the local environment and reinforcement learning results of the cloud robots, achieving efficient group collaboration, and improving the work efficiency of the cloud robots.
[0102] Figure 5 A schematic diagram of a distributed cloud robot collaborative device based on reinforcement learning provided in an embodiment of the present application Figure 1 The device of this embodiment may be in the form of software and / or hardware. Figure 5 As shown, a distributed cloud robot collaboration device 500 based on reinforcement learning provided in an embodiment of the present application is applied to a first cloud robot side, and the device includes: an update module 501, a first sending module 502 and a second sending module 503.
[0103] An updating module 501 is used to perform reinforcement learning based on the first cloud robot and update the local environment information and reinforcement learning results of the first cloud robot;
[0104] A first sending module 502 is used to send the updated local environment information and reinforcement learning results of the first cloud robot to the second cloud robot when there is a second cloud robot within the communication range of the first cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot;
[0105] The second sending module 503 is used for sending the updated local environment information and reinforcement learning results of the first cloud robot to the second cloud robot through the wireless access point AP associated with the first cloud robot when there is no second cloud robot within the communication range of the first cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot.
[0106] In a possible implementation, the update module 501 is used to:
[0107] The first cloud robot is abstracted as an intelligent agent, the process of the first cloud robot taking actions in the local environment and obtaining feedback information about the actions is modeled as an abstract Markov decision process, and the task requirements of the first cloud robot in the local environment are described as signal timing logic tasks; wherein the Markov decision process model is expressed as: (S, A, P, R, γ), S is the state set of the first cloud robot, A is the action set of the first cloud robot, P is the state transition probability, which indicates the probability that the first cloud robot changes from the first state to the second state after taking action a, R is the reward function, which indicates the reward value obtained after the first cloud robot changes from the first state to the second state, γ is the attenuation factor, γ≤1;
[0108] Based on the Markov decision process model and signal sequential logic tasks, the local environment information and reinforcement learning results of the First Cloud Robot are updated through the reinforcement learning algorithm.
[0109] In a possible implementation, the reinforcement learning algorithm is a Q learning algorithm, the reinforcement learning result is a Q value table, the input parameters of the Q learning algorithm include S, A, P, R and γ, and based on the Markov decision process model and the signal timing logic task, the update module 501 is used to:
[0110] Step 1: Initialize the Q value table, which is used to store state-action pairs Q(S t ,a t ), let Q(S t ,a t )=Q0, time step t=0, initial state S=S0; where a t Used to represent the action selected by the Agent at time t; S tUsed to characterize the state of the Agent at time t; Q(s t ,a t ) is used to represent the Agent in state S t Next Agent performs action a t The cumulative weighted reward returned after
[0111] Step 2: Select action a based on the greedy strategy ∈-greedy t ; Among them, action a t Belongs to action set A, A={a t ~π(S t )},π(S t ) is used to represent the current state at time t as S t When the agent selects an action based on the ∈-greedy strategy;
[0112] Step 3: Execute action a t , get real-time reward return r t+1 and state S t+1 ; Among them, the real-time reward return r t+1 Used to represent the Agent performing action a at time t+1 t Reward return; state S t+1 Used to represent the Agent performing action a t The next state after
[0113] Step 4: Based on S t , a t , r t+1 , S t+1 Update the Q value table, that is, update the state-action pair Q(S t ,a t );
[0114] Step 5: Update the greedy strategy;
[0115] Step 6: Let t = t + 1, S t =S t+1 , if t mod N B =0 means t can divide the preset threshold N B Then execute step 7, otherwise execute step 2;
[0116] Step 7: Execute the Grassfire algorithm, start the width search from the current position of the agent, until the position that has not been explored by the agent is found, and set the state of the position that has not been explored by the agent as the current state of the robot S t , proceed to step 2.
[0117] In a possible implementation, the reinforcement learning algorithm is a Q learning algorithm, the reinforcement learning result is a Q value table, and the update module 501 is used to:
[0118] Based on S t , a t , r t+1 , S t+1 And the following formula updates the state-action pair Q(S t ,a t ):
[0119]
[0120] In the formula, Q(S t ,a t ) means in state S t Next Agent performs action a t The cumulative weighted reward obtained after the agent is trained; α refers to the learning efficiency of the agent; a t+1 It means that the Agent performs action a t The next action after.
[0121] In a possible implementation, the reinforcement learning algorithm is a Q learning algorithm, the reinforcement learning result is a Q value table, and the distributed cloud robot collaborative device 500 based on reinforcement learning is also used for:
[0122] When there is a second cloud robot within the communication range of the first cloud robot, the updated local environment information and Q value table of the first cloud robot are sent to the second cloud robot, so that the second cloud robot updates the local environment information of the second cloud robot based on the local environment information of the first cloud robot and updates the Q value table of the second cloud robot based on the Q value table of the first cloud robot;
[0123] When there is no second cloud robot within the communication range of the first cloud robot, the first cloud robot sends the updated local environment information and Q value table of the first cloud robot to the second cloud robot through the wireless access point AP associated with the first cloud robot, so that the second cloud robot updates the local environment information and Q value table of the second cloud robot based on the local environment information and Q value table of the first cloud robot.
[0124] Figure 6 A schematic diagram of a distributed cloud robot collaborative device based on reinforcement learning provided in an embodiment of the present application Figure 2 The device of this embodiment may be in the form of software and / or hardware. Figure 6 As shown, a distributed cloud robot collaboration device 600 based on reinforcement learning provided in an embodiment of the present application is applied to the second cloud robot side. The device includes: a first processing module 601 and a second processing module 602.
[0125] The first processing module 601 is used for, when the first cloud robot exists within the communication range of the second cloud robot, the second cloud robot receives the updated local environment information and reinforcement learning result of the first cloud robot sent by the first cloud robot, and updates the local environment information and reinforcement learning result of the second cloud robot;
[0126] The second processing module 602 is used for receiving the updated local environment information and reinforcement learning results of the first cloud robot sent by the first cloud robot through the wireless access point AP associated with the first cloud robot when the first cloud robot does not exist within the communication range of the second cloud robot, and updating the local environment information and reinforcement learning results of the second cloud robot.
[0127] In a possible implementation, the reinforcement learning algorithm is a Q learning algorithm, the reinforcement learning result is a Q value table for storing state-action pairs, and the distributed cloud robot collaborative device 600 based on reinforcement learning is further used for:
[0128] Based on the updated local environment information of the first cloud robot, updating the local environment information of the second cloud robot;
[0129] Based on the updated Q value table of the first cloud robot, the Q value table of the second cloud robot is updated.
[0130] In a possible implementation, the reinforcement learning algorithm is a Q learning algorithm, the reinforcement learning result is a Q value table for storing state-action pairs, and the distributed cloud robot collaborative device 600 based on reinforcement learning is further used for:
[0131] When the updated Q value table of the first cloud robot has the first state-action pair, and the Q value table of the second cloud robot does not have the first state-action pair, updating the first state-action pair to the Q value table of the second cloud robot;
[0132] When there is a second state-action pair with an initial value in the Q value table of the second cloud robot, and the value of the second state-action pair in the Q value table of the first cloud robot is not the initial value, the value of the second state-action pair in the Q value table of the second cloud robot is updated to the value of the second state-action pair in the Q value table of the first cloud robot.
[0133] Figure 7 The hardware structure diagram of the distributed cloud robot collaborative device based on reinforcement learning provided in the embodiment of the present application. Figure 7 As shown, the distributed cloud robot collaborative device 700 based on reinforcement learning includes:
[0134] Processor 701 and memory 702;
[0135] Memory stores computer-executable instructions;
[0136] The processor executes the computer execution instructions stored in the memory 702, so that the distributed cloud robot collaboration device based on reinforcement learning executes the distributed cloud robot collaboration method based on reinforcement learning as described above.
[0137] It should be understood that the processor 701 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASIC). A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor. The memory 702 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk, or an optical disk.
[0138] The embodiment of the present application also provides a computer-readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed by a processor, they are used to implement the distributed cloud robot collaboration method based on reinforcement learning as described above.
[0139] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0140] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A distributed cloud robot collaboration method based on reinforcement learning, applied to a first cloud robot side, characterized in that: include: Performing reinforcement learning based on the first cloud robot, updating local environment information and reinforcement learning results of the first cloud robot; In the case where there is a second cloud robot within the communication range of the first cloud robot, sending the updated local environment information and reinforcement learning results of the first cloud robot to the second cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot; When the second cloud robot does not exist within the communication range of the first cloud robot, the first cloud robot sends the updated local environment information and reinforcement learning results of the first cloud robot to the second cloud robot through the wireless access point AP associated with the first cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot.
2. The collaborative method according to claim 1, applied to the first cloud robot side, is characterized in that: The performing reinforcement learning based on the first cloud robot and updating the local environment information and reinforcement learning result of the first cloud robot includes: Abstract the first cloud robot as an intelligent agent, model the process of the first cloud robot taking action in the local environment and obtaining feedback information about the action as an abstract Markov decision process, and describe the task requirements of the first cloud robot in the local environment as a signal timing logic task; wherein the Markov decision process model is expressed as: (S, A, P, R, γ), wherein S is the state set of the first cloud robot, A is the action set of the first cloud robot, P is the state transition probability, indicating the probability that the first cloud robot changes from the first state to the second state after taking action a, R is the reward function, indicating the reward value obtained by the first cloud robot after changing from the first state to the second state, and γ is the attenuation factor, wherein γ≤1; Based on the Markov decision process model and the signal timing logic task, the local environment information and reinforcement learning results of the first cloud robot are updated through a reinforcement learning algorithm.
3. The collaborative method according to claim 2, applied to the first cloud robot side, is characterized in that: The reinforcement learning algorithm is a Q learning algorithm, the reinforcement learning result is a Q value table, the input parameters of the Q learning algorithm include the S, the A, the P, the R and the γ, and the local environment information and the reinforcement learning result of the first cloud robot are updated by the reinforcement learning algorithm based on the Markov decision process model and the signal timing logic task, including: Step 1: Initialize the Q value table, which is used to store state-action pairs Q(S t ,a t ), let Q(S t ,a t )=Q0, time step t=0, initial state S=S0; wherein, the a t It is used to represent the action selected by the Agent at time t; t It is used to characterize the state of the Agent at time t; t ,a t ) is used to characterize the Agent in the state S t Next, the Agent performs the action a t The cumulative weighted reward returned after Step 2: Select the action a based on the greedy strategy ∈-greedy t ; wherein the action a t Belongs to action set A, where A={a t ~π(S t )}, the π(S t ) is used to represent the current state at time t as S t When the agent selects an action based on the ∈-greedy strategy; Step 3: Execute the action a t , get real-time reward return r t+1 and state S t+1 ; Wherein, the real-time reward return r t+1 Used to represent that the Agent performs the action a at time t+1 t The reward return; the state S t+1 Used to represent the Agent performing the action a t The next state after Step 4: Based on S t , a t , r t+1 , S t+1 Update the Q value table, that is, update the state-action pair Q(S t ,a t ); Step 5: Update the greedy strategy; Step 6: Let t = t + 1, S t =S t+1 , if t mod N B =0 means t can divide the preset threshold N B Then execute step 7, otherwise execute step 2; Step 7: Execute the Grassfire algorithm, start a width search from the current position of the Agent, until a position that has not been explored by the Agent is found, and set the state of the position that has not been explored by the Agent as the current state S of the robot. t , proceed to step 2.
4. The collaborative method according to claim 3, applied to the first cloud robot side, is characterized in that: Based on S t , a t , r t+1 , S t+1 and the following formula updates the state-action pair Q(S t ,a t ): Wherein, the Q(S t ,a t ) refers to the state S t Next, the Agent performs the action a t The cumulative weighted reward return obtained after the agent is trained; the α refers to the learning efficiency of the agent; the a t+1 It means that the Agent performs the action a t The next action after.
5. The collaborative method according to claim 3, applied to the first cloud robot side, is characterized in that: The collaborative method further comprises: In the case where the second cloud robot exists within the communication range of the first cloud robot, sending the updated local environment information and Q value table of the first cloud robot to the second cloud robot, so that the second cloud robot updates the local environment information of the second cloud robot based on the local environment information of the first cloud robot and updates the Q value table of the second cloud robot based on the Q value table of the first cloud robot; When the second cloud robot does not exist within the communication range of the first cloud robot, the first cloud robot sends the updated local environment information and Q value table of the first cloud robot to the second cloud robot through the wireless access point AP associated with the first cloud robot, so that the second cloud robot updates the local environment information and Q value table of the second cloud robot based on the local environment information and Q value table of the first cloud robot.
6. A distributed cloud robot collaboration method based on reinforcement learning, applied to the second cloud robot side, characterized in that: include: In the case where the first cloud robot exists within the communication range of the second cloud robot, the second cloud robot receives the updated local environment information and reinforcement learning results of the first cloud robot sent by the first cloud robot, and updates the local environment information and reinforcement learning results of the second cloud robot; When the first cloud robot does not exist within the communication range of the second cloud robot, the second cloud robot receives the updated local environment information and reinforcement learning results of the first cloud robot sent by the first cloud robot through the wireless access point AP associated with the first cloud robot, and updates the local environment information and reinforcement learning results of the second cloud robot.
7. The collaborative method according to claim 6, applied to the second cloud robot side, characterized in that: The reinforcement learning algorithm is a Q learning algorithm, and the reinforcement learning result is a Q value table for storing state-action pairs. The updating of the local environment information of the second cloud robot and the reinforcement learning result includes: Based on the updated local environment information of the first cloud robot, updating the local environment information of the second cloud robot; Based on the updated Q value table of the first cloud robot, the Q value table of the second cloud robot is updated.
8. The collaborative method according to claim 7, applied to the second cloud robot side, is characterized in that: The updating of the Q value table of the second cloud robot based on the updated Q value table of the first cloud robot includes: When the updated Q value table of the first cloud robot contains a first state-action pair, and the Q value table of the second cloud robot does not contain the first state-action pair, updating the first state-action pair to the Q value table of the second cloud robot; When there is a second state-action pair with an initial value in the Q-value table of the second cloud robot, and the value of the second state-action pair in the Q-value table of the first cloud robot is not the initial value, the value of the second state-action pair in the Q-value table of the second cloud robot is updated to the value of the second state-action pair in the Q-value table of the first cloud robot.
9. A distributed cloud robot collaborative device based on reinforcement learning, applied to a first cloud robot side, characterized in that: include: An updating module, configured to perform reinforcement learning on the first cloud robot and update the local environment information and reinforcement learning results of the first cloud robot; A first sending module, configured to send updated local environment information and reinforcement learning results of the first cloud robot to the second cloud robot when there is a second cloud robot within the communication range of the first cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot; The second sending module is used for sending the updated local environment information and reinforcement learning results of the first cloud robot to the second cloud robot through the wireless access point AP associated with the first cloud robot when the second cloud robot does not exist within the communication range of the first cloud robot, so that the second cloud robot updates the local environment information and reinforcement learning results of the second cloud robot based on the local environment information and reinforcement learning results of the first cloud robot.
10. A distributed cloud robot collaborative device based on reinforcement learning, applied to the second cloud robot side, characterized in that: include: A first processing module, configured to receive, by the second cloud robot, updated local environment information and reinforcement learning results of the first cloud robot sent by the first cloud robot, and update the local environment information and reinforcement learning results of the second cloud robot when the first cloud robot exists within the communication range of the second cloud robot; The second processing module is used for receiving the updated local environment information and reinforcement learning results of the first cloud robot sent by the first cloud robot through the wireless access point AP associated with the first cloud robot when the first cloud robot does not exist within the communication range of the second cloud robot, and updating the local environment information and reinforcement learning results of the second cloud robot.
11. A distributed cloud robot collaborative device based on reinforcement learning, comprising: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the distributed cloud robot collaboration device based on reinforcement learning executes the distributed cloud robot collaboration method based on reinforcement learning described in any one of claims 1 to 5 or 6 to 8.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the distributed cloud robot collaboration method based on reinforcement learning as described in any one of claims 1 to 5 or 6 to 8.