A Q-learning-based method for on-orbit maintenance task allocation for multi-robot arms
By optimizing the allocation of on-orbit maintenance tasks for multiple robotic arms using the Q-learning algorithm, the problem of unclear task allocation was solved, cooperation and efficiency were improved, and the reliability and stability of the spacecraft were ensured.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF AERONAUTICS & ASTRONAUTICS
- Filing Date
- 2023-12-22
- Publication Date
- 2026-05-26
Smart Images
Figure CN117754571B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of task allocation technology for space multi-arm robots, and in particular to a method for on-orbit maintenance task allocation for multi-arm robots based on Q-learning. Background Technology
[0002] The introduction of multi-armed space robots has become an important measure to improve the efficiency of on-orbit maintenance. These robots possess the ability to cooperate and coordinate, enabling them to complete complex tasks more efficiently and providing crucial support for on-orbit servicing missions. Their flexibility and versatility make them essential in handling a wide variety of maintenance tasks.
[0003] When performing on-orbit maintenance tasks on spacecraft, multi-armed robots often encounter multiple points requiring repair. Simultaneous repair by multiple robotic arms can significantly improve efficiency. However, in this scenario with multiple robotic arms and multiple target points, unclear task allocation can lead to resource waste and increased task execution time. Furthermore, some sub-tasks (such as removing screws or obstructing cables) require collaborative work among the robotic arms. Therefore, ensuring clear task allocation during on-orbit maintenance with multiple robotic arms, and that the allocation reflects the collaborative nature of the robotic arms, is crucial. Summary of the Invention
[0004] The technical problem to be solved by this invention is to provide a multi-robot on-orbit maintenance task allocation method based on Q-learning, which can effectively solve the problem of unclear task allocation in the case of multiple robots and multiple objectives, improve the cooperation between robots, reduce task completion time, and thus give full play to the advantages of multiple robots.
[0005] To address the aforementioned technical problems, this invention provides a multi-robot on-orbit maintenance task allocation method based on Q-learning, comprising the following steps:
[0006] Step 1: Rasterize the simulated on-orbit maintenance task space into a three-dimensional array model;
[0007] Step 2: Initialize the positions of the end effectors of each robotic arm of the space robot and the positions of multiple target points for maintenance tasks;
[0008] Step 3: Based on the Q-learning algorithm, obtain the ideal path for each robotic arm that starts simultaneously, completes all tasks in the shortest time, and has the highest degree of cooperation among the robotic arms.
[0009] Step 4: Obtain the task allocation and task execution order for each robotic arm based on the ideal path of each robotic arm.
[0010] Preferably, in step 1, the simulated on-orbit maintenance task space is rasterized into... A three-dimensional array model.
[0011] Preferably, in step 2, the end effectors of each robotic arm and the multiple maintenance targets of the space robot are simplified into specific spatial points; the initial position of each end effector is set to... n is the number of robotic arms in the environment; multiple target points for maintenance tasks are set as follows: , m is the number of target points in the environment to be repaired; the position coordinates of the end effector of the i-th robotic arm are set as follows:
[0012]
[0013] in, , , .
[0014] Preferably, in step 3, the state space of the Q-learning algorithm Set the coordinates of the robotic arm's end effector and the completion status of the task points:
[0015]
[0016] in, For the task point The completion status, (0 indicates the task point) Not completed, 1 indicates that the task is incomplete. (Completed).
[0017] Preferably, in step 3, a joint action is set. As the action space for the Q-learning algorithm, the action of the i-th robotic arm is set as follows:
[0018]
[0019] Where n is the number of robotic arms, and the actions UP, ..., STAY, from left to right, represent the robotic arm end effector moving up, down, left, right, forward, backward, and stopping to begin maintenance work in space.
[0020] Preferably, in step 3, obtaining the ideal path for each robotic arm based on the Q-learning algorithm, which optimizes the time for all robotic arms to start simultaneously and complete all tasks while maximizing the degree of cooperation among the robotic arms, specifically includes the following steps:
[0021] Step 31: Set the maximum number of training rounds and the maximum number of movements for each robotic arm;
[0022] Step 32: Initialize the robotic arm joint Q-table as follows ;
[0023] Step 33: Initialize the state space as follows ;
[0024] Step 34: Set the greedy coefficient G, select the joint action A according to the greedy strategy, and obtain the next state. ;
[0025] Step 35: Obtain reward value according to the set reward and punishment mechanism. ;
[0026] Step 36, Update ;
[0027] Step 37: Determine whether all maintenance tasks at all task points have been completed;
[0028] Step 38: If all maintenance tasks at all points are completed, the training ends; if not all are completed, the current status is changed. Set as In the new cycle, select joint action A again until the number of training rounds equals the preset maximum number of rounds and then end the training.
[0029] Step 39, according to The ideal paths for each robotic arm are obtained under the planned space, where all robotic arms start simultaneously, complete all tasks in the optimal time, and achieve the highest degree of cooperation among the robotic arms.
[0030] Preferably, in step 33, the state space is initialized. .
[0031] Preferably, in step 35, The updated formula is:
[0032]
[0033] in, In the next state All actions at that time The maximum Q value, This is the return coefficient.
[0034] Preferably, in step 35, based on the state classification, to minimize the time required for task allocation and maximize collaboration, the reward value R is set as follows:
[0035]
[0036] in, This represents the reward value for the k-th task point; The settings are as follows:
[0037]
[0038] in, This represents the state of the k-th point. This indicates that the task at this location has been completed. This indicates that multiple robotic arms are collaborating on maintenance at this location. This indicates that a single robotic arm is being repaired at this location. This indicates that the robotic arm is undergoing ineffective maintenance at this location. This indicates that there is no robotic arm under maintenance at this location, and n1 is the number of robotic arms that are working together to maintain this location.
[0039] Preferably, in step 36, the criterion for determining whether all maintenance tasks at all task points have been completed is that all... .
[0040] Preferably, in step 4, the task allocation and execution order of each robotic arm are obtained based on the encounter order between the ideal path of each robotic arm and the target point of the task to be repaired.
[0041] The beneficial effects of this invention are as follows: This invention adopts a multi-robot on-orbit maintenance task allocation method based on Q-learning. The coordinates of the robot arm's end effector and the completion status of the task points are set as the state space, and the movement and dwelling maintenance actions in the task space are set as the action space. A reward function is set to encourage collaborative maintenance among multiple robots, penalize ineffective maintenance by robots, and optimize task completion time. The method is trained to obtain a task allocation and execution order where all robots start simultaneously, the time to complete all tasks is optimal, and the collaboration among robots is highest. This effectively solves the problem of unclear task allocation in multi-robot, multi-objective scenarios, improves the collaboration between robots, reduces task completion time, and leverages the advantages of multiple robots. Therefore, this multi-robot on-orbit maintenance task allocation method enables multiple robots to more efficiently handle complex on-orbit maintenance tasks, ensuring the reliability and stability of spacecraft operation in orbit. Attached Figure Description
[0042] Figure 1 This is a schematic diagram illustrating the implementation process of the present invention.
[0043] Figure 2 This is a schematic diagram of the method flow of the present invention. Detailed Implementation
[0044] like Figure 1 As shown, a multi-robot on-orbit maintenance task allocation method based on Q-learning includes the following steps:
[0045] Step 1: Rasterize the simulated on-orbit maintenance task space into a three-dimensional array model;
[0046] Step 2: Initialize the positions of the end effectors of each robotic arm of the space robot and the positions of multiple target points for maintenance tasks;
[0047] Step 3: Based on the Q-learning algorithm, obtain the ideal path for each robotic arm that starts simultaneously, completes all tasks in the shortest time, and has the highest degree of cooperation among the robotic arms.
[0048] Step 4: Obtain the task allocation and task execution order for each robotic arm based on the ideal path of each robotic arm.
[0049] like Figure 2 As shown in the embodiment of the present invention, the multi-robot on-orbit maintenance task allocation method based on Q-learning specifically includes the following steps:
[0050] (1) Rasterize the simulated on-orbit maintenance task space into A three-dimensional array model. The task space is defined separately. , , The range of values for the three axes is as follows: , , .
[0051] (2) Simplify the four end effectors of the space robot's robotic arms and the eight maintenance targets into specific spatial points. Initialize the positions of the four end effectors of the space robot as follows: , , , The locations of the eight target points for maintenance tasks are as follows: , , , , , , , .
[0052] (3) Based on the Q-learning algorithm, the ideal path of each robotic arm is obtained, which is the optimal time for each robotic arm to start at the same time and complete all tasks, and the highest degree of cooperation among each robotic arm.
[0053] In the Q-learning algorithm, the state space, action space, and reward function are set as follows:
[0054] Set the position coordinates of the end effector of the i-th robotic arm as follows:
[0055]
[0056] in, , , .
[0057] The state space of the Q-learning algorithm is set as the coordinates of the robotic arm's end effector and the completion status of the task points:
[0058]
[0059] in, For the task point The completion status, (0 indicates the task point) Not completed, 1 indicates that the task is incomplete. (Completed).
[0060] The motion of the i-th robotic arm is set as follows:
[0061]
[0062] Among them, the actions UP, ..., STAY, from left to right, represent the robotic arm end effector moving up, down, left, right, forward, backward, and stopping to begin maintenance work in space.
[0063] Set up joint actions , which serves as the action space for the Q-learning algorithm.
[0064] The state S in the environment is divided into three categories:
[0065]
[0066] Among them, the failure state (FS) indicates that the end effector of the robotic arm has left the task space, the success state (WS) indicates that the maintenance work of all task points has been completed, and the half-success state (MS) indicates that there are still unfinished task points in the environment.
[0067] Based on the state classification, to minimize the time required for task allocation and maximize collaboration, the reward value R is set as follows:
[0068]
[0069] in, This represents the reward value for the k-th task point.
[0070] The settings are as follows:
[0071]
[0072] in, This represents the state of the k-th point. This indicates that the task at this location has been completed. This indicates that multiple robotic arms are collaborating on maintenance at this location. This indicates that a single robotic arm is being repaired at this location. This indicates that the robotic arm is undergoing ineffective maintenance at this location. This indicates that there is no robotic arm under maintenance at this location, and n1 is the number of robotic arms that are working together to maintain this location.
[0073] Obtaining the ideal path based on the Q-learning algorithm includes the following steps:
[0074] Step 3.1: Set the maximum number of training rounds to 3000 and the maximum number of movements for each robotic arm to 200.
[0075] Step 3.2, initialize the robotic arm joint Q-table as follows: ;
[0076] Step 3.3, Initialize the state space ;
[0077] Step 3.4: Set the greedy coefficient G=0.2, select the joint action A according to the greedy strategy, and obtain the next state. ;
[0078] Step 3.5: Obtain the reward value according to the set reward and punishment mechanism. ;
[0079] Step 3.6, Update The updated formula is:
[0080]
[0081] in, In the next state All actions at that time Maximum Q value, return coefficient .
[0082] Step 3.7: Determine whether all maintenance tasks at all task points have been completed. The determination is marked as complete. ;
[0083] Step 3.8: If all maintenance tasks at all points are completed, the training ends; if not all are completed, the current state is updated. Set as In the new cycle, select joint action A again until the number of training rounds equals the preset maximum number of rounds and then end the training.
[0084] Step 3.9, according to The ideal paths for each robotic arm are obtained under the planned space, where all robotic arms start simultaneously, complete all tasks in the optimal time, and achieve the highest degree of cooperation among the robotic arms.
[0085] (4) Based on the order in which each robotic arm’s ideal path meets the target point of the task to be repaired, the task allocation and execution order of each robotic arm are obtained.
[0086] This invention can effectively solve the problem of unclear task allocation in the case of multiple robotic arms and multiple targets, improve the cooperation between robotic arms, reduce task completion time, and thus give full play to the advantages of multiple robotic arms.
Claims
1. A method for allocating on-orbit maintenance tasks for multiple robotic arms based on Q-learning, characterized in that, Includes the following steps: Step 1: Rasterize the simulated on-orbit maintenance task space into a three-dimensional array model; Step 2: Initialize the positions of the end effectors of each robotic arm of the space robot and the positions of multiple target points for maintenance tasks; Step 3: Based on the Q-learning algorithm, obtain the ideal paths for each robotic arm that start simultaneously, complete all tasks in the shortest time, and have the highest degree of cooperation. The state space of the Q-learning algorithm... Set the coordinates of the robotic arm's end effector and the completion status of the task target point: in, For the mission objective point The completion status, among which , 0 indicates the target point of the task. Not completed, 1 indicates the objective point of this task. Completed; Let be the position coordinates of the end effector of the i-th robotic arm. Let n be the number of target points for maintenance tasks in the environment, and n be the number of robotic arms in the environment. Set up joint actions As the action space for the Q-learning algorithm, the action of the i-th robotic arm is set as follows: in, ,action The sequence represents the robotic arm's end effector moving upwards, downwards, leftwards, rightwards, forwards, backwards, and then stopping to begin maintenance work. Step 4: Obtain the task allocation and execution order of each robotic arm based on the ideal path of each robotic arm; The ideal path for each robotic arm, based on the Q-learning algorithm, which optimizes the time for all robotic arms to start simultaneously and complete all tasks, and maximizes the cooperation among the robotic arms, includes the following steps: Step 31: Set the maximum number of training rounds and the maximum number of movements for each robotic arm; Step 32: Initialize the robotic arm joint Q-table as follows ; Step 33: Initialize the state space as follows ; Step 34: Set the greedy coefficient G, select the joint action A according to the greedy strategy, and obtain the next state. ; Step 35: Obtain reward value according to the set reward and punishment mechanism. Based on the state classification, to minimize the time required for task allocation and maximize collaboration, the reward value R is set as follows: in, This represents the reward value for the k-th task objective. The settings are as follows: In this context, FS indicates that the robotic arm's end effector has moved out of the task space, WS indicates that all task target points have been repaired, and MS indicates that there are still unfinished task target points in the environment. This represents the state of the k-th task target point. This indicates that the task at the target point has been completed. This indicates that the target point of the task involves collaborative maintenance by multiple robotic arms. This indicates that a single robotic arm is under maintenance at the target location for this task. This indicates that the robotic arm is not performing any maintenance at the target location of the task. This indicates that there is no robotic arm under maintenance at the target point of the task, and n1 is the number of robotic arms that are being repaired collaboratively at the target point of the task. Step 36, Update ; Step 37: Determine whether all maintenance tasks at all task target points have been completed; Step 38: If the maintenance tasks at all target points are completed, the training ends; if not all are completed, the current state is changed. Set as In the new cycle, select joint action A again until the number of training rounds equals the preset maximum number of rounds and then end the training. Step 39, according to The ideal paths for each robotic arm are obtained under the planned space, where all robotic arms start simultaneously, complete all tasks in the optimal time, and achieve the highest degree of cooperation among the robotic arms.
2. The multi-robot on-orbit maintenance task allocation method based on Q-learning as described in claim 1, characterized in that, In step 1, the simulated on-orbit maintenance task space is rasterized into... A three-dimensional array model.
3. The multi-robot on-orbit maintenance task allocation method based on Q-learning as described in claim 1, characterized in that, In step 2, the end effectors of each robotic arm and multiple target points for maintenance tasks are simplified into specific spatial points; the initial positions of each robotic arm end effector are set to... The position coordinates of the end effector of the i-th robotic arm are set as follows: in, , , .
4. The multi-robot on-orbit maintenance task allocation method based on Q-learning as described in claim 1, characterized in that, In step 36, The updated formula is: in, In the next state All actions at that time The maximum Q value, This is the return coefficient.
5. The multi-robot on-orbit maintenance task allocation method based on Q-learning as described in claim 1, characterized in that, In step 37, the criterion for determining whether the maintenance tasks at all task target points have been completed is that all... .