Space truss multi-robot collaborative assembly method and device based on reinforcement learning
Through a multi-agent system based on reinforcement learning, combined with deep neural network and proximity optimization strategy network, intelligent and autonomous decision-making of multi-argister arms in complex spatial truss assembly is achieved, and the low efficiency of multi-argister arms assembly in the existing technology is solved. and the problem of relying on manual experience improves assembly efficiency and scalability.
Patent Information
- Application Number
- CN202411101667.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-12
AI Technical Summary
The prior art is difficult to enable multiple robot arms to flexibly and intelligently complete complex spatial truss assembly tasks, especially in non-connected structured environments, where effective assembly sequence decisions and multi-robot collaborative operating systems are lacking.
Using reinforcement learning methods, through a multi-agent system that perceives, parses, extracts and decision-making, the decision-making process of truss assembly can be realized intelligent and autonomous learning, integrates deep neural networks and near-end optimization strategy networks, and generates assembly sequences and operation instructions. , realize the coordinated assembly of multiple robotic arms.
Reliance on manual experience is reduced, assembly efficiency and intelligence is improved, the system's adaptability and fault tolerance are enhanced, and the system is achieved higher scalability and robustness are achieved.
Smart Images

Figure CN119017378A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of space robots, and in particular to a method and device for collaborative assembly of space trusses by multiple robots based on reinforcement learning. Background Art
[0002] With the development of space robot technology and artificial intelligence technology, the assembly of a single robotic arm is limited by operational capabilities. In view of the complex problem of space truss structure, the assembly efficiency of a single space robotic arm is limited. Compared with the assembly operation of a single robotic arm, the collaborative operation of multiple robotic arm systems in the process of executing tasks will have higher efficiency, adaptability and fault tolerance.
[0003] On the other hand, for the complex truss assembly process, processing it through manual experience will consume a lot of manpower costs and have a lot of repetitive work, which is not conducive to the scalability of the space truss. Therefore, having a multi-space manipulator collaborative assembly system with assembly sequence decision-making and task target classification is of great significance to the on-orbit assembly technology of the space truss.
[0004] At present, although there are some preliminary studies on multi-robot reinforcement learning based on reinforcement learning, there is no public report on the research on complex truss assembly problems in non-structured environments, especially multi-robot collaborative operation systems that include complete environment and truss recognition, truss assembly sequence generation, and multi-robot operation instruction generation.
[0005] In summary, in order to enable multiple robotic arms to complete complex truss assembly tasks flexibly and intelligently, it is necessary to comprehensively utilize the multi-robotic arm system based on multi-agent reinforcement learning to realize intelligent and autonomous learning of the truss assembly decision-making process through perception, analysis, extraction, and decision-making, and map the full state information of the assembly sequence, truss unit category, and dynamic environment end-to-end to the assembly operation decision of the multi-robotic arm truss assembly system. Summary of the invention
[0006] In order to solve the technical problem of how to enable multiple robotic arms to flexibly and intelligently complete complex truss assembly tasks in the prior art, an embodiment of the present invention provides a spatial truss multi-robot collaborative assembly method and device based on reinforcement learning. The technical solution is as follows:
[0007] On the one hand, a reinforcement learning-based space truss multi-robot collaborative assembly method is provided, the method is implemented by a space truss multi-robot collaborative assembly device, and the method includes:
[0008] S1. Determine the truss unit to be assembled.
[0009] S2. Acquire status information of multiple space manipulators, input the status information into a strategy model of an optimal space manipulator, and obtain motion decision control instructions of the multiple space manipulators.
[0010] S3. According to the motion decision control instructions, multiple space manipulators are controlled to assemble the truss units, completing the multi-robot collaborative truss assembly task.
[0011] Optionally, the process of constructing the strategy model of the optimal space manipulator in S2 includes:
[0012] S21. Determine the truss unit.
[0013] S22, constructing a strategy model of the space robot arm, collecting state information of multiple space robot arms in a simulation environment as training data to train the strategy model of the space robot arm, and obtaining a trained strategy model of the space robot arm.
[0014] S23. Input the state information into the trained strategy model of the space robot arm to obtain sample motion decision control instructions.
[0015] S24. According to the sample motion decision control instructions, control multiple space manipulators to assemble the truss units to obtain a reward value and new state information.
[0016] S25. Perform decision correction on the trained strategy model of the space robot according to the reward value and the new state information to obtain a corrected strategy model of the space robot.
[0017] S26: Determine whether the set termination condition is met.
[0018] If the set termination condition is met, the truss unit assembly is completed, and it is determined whether the set number of iterations is met; if the set number of iterations is met, the strategy model of the optimal spatial robot is output; if the set number of iterations is not met, the next truss unit is obtained and step S23 is executed.
[0019] If the set termination condition is not met, the process proceeds to step S24.
[0020] Optionally, determining the truss unit in S21 includes:
[0021] S211, obtaining the assembly sequence of the truss unit, and collecting assembly sequence state information of the assembly sequence using a proximal optimization strategy network.
[0022] S212: Extract features from assembly sequence state information to generate an assembly sequence of the truss unit.
[0023] S213: Obtain target truss unit.
[0024] S214: Identify the target truss unit to obtain the target truss unit characteristics.
[0025] S215 : judging whether the target truss unit is a truss unit to be assembled according to the assembly sequence of the truss unit and the characteristics of the target truss unit.
[0026] If so, the truss elements are output.
[0027] If not, the next truss unit is obtained as the target truss unit, and the process goes to step S214.
[0028] Optionally, identifying the target truss unit in S214 to obtain the target truss unit feature includes:
[0029] A representation neural network is constructed to identify the target truss unit and obtain the characteristics of the target truss unit.
[0030] Optionally, the training of the strategy model of the space manipulator in S22 includes:
[0031] The policy model for a spatial manipulator is trained using a deep deterministic policy gradient framework.
[0032] Optionally, the state information of multiple space manipulators in the collected simulation environment in S22 is used as training data to train the strategy model of the space manipulator to obtain a trained strategy model of the space manipulator, including:
[0033] The base status information of multiple space manipulators in the simulation environment is collected, and the neural network parameters are trained according to the base status information to obtain the optimal formation decision model.
[0034] The robot state information of multiple spatial manipulators in the simulation environment is collected, and the neural network parameters are trained according to the robot state information to obtain the optimal collaborative motion decision.
[0035] According to the optimal formation decision model and the optimal collaborative motion decision, a strategy model of the trained space manipulator is obtained.
[0036] On the other hand, a multi-robot collaborative assembly device for a space truss based on reinforcement learning is provided, and the device is applied to a multi-robot collaborative assembly method for a space truss based on reinforcement learning, and the device comprises:
[0037] The determination module is used to determine the truss elements to be assembled.
[0038] The input module is used to obtain the state information of multiple space manipulators, input the state information into the strategy model of the optimal space manipulator, and obtain the motion decision control instructions of the multiple space manipulators.
[0039] The output module is used to control multiple spatial manipulators to assemble truss units according to motion decision control instructions, thereby completing the multi-robot collaborative truss assembly task.
[0040] Optionally, the input module is further configured to:
[0041] S21. Determine the truss unit.
[0042] S22, constructing a strategy model of the space robot arm, collecting state information of multiple space robot arms in a simulation environment as training data to train the strategy model of the space robot arm, and obtaining a trained strategy model of the space robot arm.
[0043] S23. Input the state information into the trained strategy model of the space robot arm to obtain sample motion decision control instructions.
[0044] S24. According to the sample motion decision control instructions, control multiple space manipulators to assemble the truss units to obtain a reward value and new state information.
[0045] S25. Perform decision correction on the trained strategy model of the space robot according to the reward value and the new state information to obtain a corrected strategy model of the space robot.
[0046] S26: Determine whether the set termination condition is met.
[0047] If the set termination condition is met, the truss unit assembly is completed, and it is determined whether the set number of iterations is met; if the set number of iterations is met, the strategy model of the optimal spatial robot is output; if the set number of iterations is not met, the next truss unit is obtained and step S23 is executed.
[0048] If the set termination condition is not met, the process proceeds to step S24.
[0049] Optionally, the input module is further configured to:
[0050] S211, obtaining the assembly sequence of the truss unit, and collecting assembly sequence state information of the assembly sequence using a proximal optimization strategy network.
[0051] S212: Extract features from assembly sequence state information to generate an assembly sequence of the truss unit.
[0052] S213: Obtain target truss unit.
[0053] S214: Identify the target truss unit to obtain the target truss unit characteristics.
[0054] S215 : judging whether the target truss unit is a truss unit to be assembled according to the assembly sequence of the truss unit and the characteristics of the target truss unit.
[0055] If so, the truss elements are output.
[0056] If not, the next truss unit is obtained as the target truss unit, and the process goes to step S214.
[0057] Optionally, the input module is further configured to:
[0058] A representation neural network is constructed to identify the target truss unit and obtain the characteristics of the target truss unit.
[0059] Optionally, the input module is further configured to:
[0060] The policy model for a spatial manipulator is trained using a deep deterministic policy gradient framework.
[0061] Optionally, the input module is further configured to:
[0062] The base status information of multiple space manipulators in the simulation environment is collected, and the neural network parameters are trained according to the base status information to obtain the optimal formation decision model.
[0063] The robot state information of multiple spatial manipulators in the simulation environment is collected, and the neural network parameters are trained according to the robot state information to obtain the optimal collaborative motion decision.
[0064] According to the optimal formation decision model and the optimal collaborative motion decision, a strategy model of the trained space manipulator is obtained.
[0065] On the other hand, a space truss multi-robot collaborative assembly device is provided, and the space truss multi-robot collaborative assembly device comprises: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned reinforcement learning-based space truss multi-robot collaborative assembly methods is implemented.
[0066] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned reinforcement learning-based spatial truss multi-robot collaborative assembly methods.
[0067] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:
[0068] In the embodiment of the present invention, the overall assembly process is split to realize an autonomous truss assembly system through multiple robotic arms, which effectively reduces the reliance on manual experience, reduces the consumption of human resources, and improves the intelligence of operation.
[0069] The present invention introduces a deep neural network throughout the entire system, which can fully approximate complex nonlinear mapping relationships and learning capabilities.
[0070] The present invention combines proximal optimization strategy, target recognition network and reinforcement learning. In the overall system, it includes perception process and decision-making process, which is more similar to the manual assembly process of the truss. It has a certain adaptability to emergencies in the task environment and can effectively correct the collaborative operation decisions of multiple robotic arms.
[0071] The present invention adopts a space base to provide effective space capabilities for multiple robotic arms, so that the working space of the robotic arms can be effectively expanded, the overall task space can be effectively covered, and the fault tolerance of the assembly process can be improved.
[0072] The truss collaborative assembly system based on multi-agent reinforcement learning of the present invention adopts a collaborative operation mode, has higher scalability, robustness and higher load capacity, and cooperates with the system proposed by the present invention to make the assembly process have better intelligence and adaptability.
[0073] The introduction of neural networks makes the decision-making process of the truss collaborative assembly system based on multi-agent reinforcement learning more robust and capable of real-time correction. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0075] Figure 1 This is a flow chart of a multi-robot collaborative assembly method for a space truss based on reinforcement learning provided by an embodiment of the present invention;
[0076] Figure 2 This is an overall architecture diagram of a multi-robot collaborative assembly method for a space truss based on reinforcement learning provided by an embodiment of the present invention;
[0077] Figure 3 It is an overall task flow chart of a multi-robot collaborative assembly method of a space truss based on reinforcement learning provided by an embodiment of the present invention;
[0078] Figure 4is a flow chart of an assembly sequence generation system provided by an embodiment of the present invention;
[0079] Figure 5 is a flow chart of a multiple robot arm operation decision system provided by an embodiment of the present invention;
[0080] Figure 6 It is a schematic diagram of offline scenario training of multiple robot arm operations provided by an embodiment of the present invention;
[0081] Figure 7 It is a block diagram of a multi-robot collaborative assembly device for a space truss based on reinforcement learning provided by an embodiment of the present invention;
[0082] Figure 8 It is a structural schematic diagram of a space truss multi-robot collaborative assembly device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0083] The technical solution of the present invention is described below in conjunction with the accompanying drawings.
[0084] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.
[0085] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.
[0086] In the embodiments of the present invention, sometimes the subscripts such as W 1 It may be written in non-subscript form such as W1. When the difference is not emphasized, the meaning is the same.
[0087] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0088] The embodiment of the present invention provides a method for multi-robot collaborative assembly of a space truss based on reinforcement learning. The method can be implemented by a multi-robot collaborative assembly device for a space truss, and the multi-robot collaborative assembly device for a space truss can be a terminal or a server. Figure 1The flowchart of the multi-robot collaborative assembly method of a space truss based on reinforcement learning is shown. The processing flow of the method may include the following steps:
[0089] S1. Determine the truss unit to be assembled.
[0090] S2. Acquire status information of multiple space manipulators, input the status information into a strategy model of an optimal space manipulator, and obtain motion decision control instructions of the multiple space manipulators.
[0091] Optionally, the overall architecture of the spatial truss multi-robot collaborative assembly method based on reinforcement learning is as follows: Figure 2 As shown in the figure, the overall task flow chart of the multi-robot collaborative assembly method of spatial trusses based on reinforcement learning is as follows: Figure 3 As shown, the process of constructing the strategy model of the optimal space manipulator in S2 may include the following steps S21-S26:
[0092] S21. Determine the truss unit.
[0093] Optionally, the above step S21 may include the following steps S211-S215:
[0094] S211, obtaining the assembly sequence of the truss unit, and collecting assembly sequence state information of the assembly sequence using a proximal optimization strategy network.
[0095] In one feasible implementation, a proximal optimization strategy network is used to collect assembly sequence state information from a sequence environment.
[0096] Specifically, a neural network is constructed, and assembly sequence information and space robot body information are input from the first layer of neurons of the neural network.
[0097] S212: Extract features from assembly sequence state information to generate an assembly sequence of the truss unit.
[0098] In a feasible implementation, the truss assembly sequence of the truss collaborative assembly system based on multi-agent reinforcement learning will be Figure 4 The proximal optimization strategy algorithm shown in the figure is used for training, the assembly sequence of the truss is constructed as a directed graph, and a set of neural network parameters is found to generate an effective assembly sequence. The specific implementation details of the algorithm are as follows:
[0099] S211.1: Initialize strategy parameters and the value function parameters ;
[0100] S211.2: Strategies through space manipulators Collect running tracks ;
[0101] S211.3: Calculate reward value ;
[0102] S211.4: Based on existing value functions Calculating the odds estimate ;
[0103] S211.5: Update the policy by maximizing the PPO-Clip objective:
[0104]
[0105] In the formula, Indicates strategy, Indicates the current iteration number. represents the time step, Indicates action, Indicates the status, represents the advantage function, Represents the clipping threshold in the PPO algorithm.
[0106] S211.6: Correction of the value function by regressing the mean squared errors:
[0107]
[0108] In the formula, Represents a value function.
[0109] S211.7: Repeat steps S211.2 to S211.6 until the algorithm converges and generates a truss assembly sequence.
[0110] S213: Obtain target truss unit.
[0111] S214. Construct a representation neural network to identify the target truss unit and obtain the target truss unit characteristics. The data obtained by truss unit identification is the original high-dimensional signal.
[0112] In a feasible implementation, a representation neural network is constructed, and the truss image signal is input from the first layer of neurons of the representation neural network after feature extraction.
[0113] The truss unit recognizes multiple spatial manipulators, truss units and environmental status information, and the signal recognized by the truss unit includes an RGB image.
[0114] S215 : judging whether the target truss unit is a truss unit to be assembled according to the assembly sequence of the truss unit and the characteristics of the target truss unit.
[0115] If so, the truss elements are output.
[0116] If not, the next truss unit is obtained as the target truss unit, and the process goes to step S214.
[0117] In a feasible implementation, according to the generated truss assembly sequence, a target detection network is used to identify and classify the target object, thereby providing constraints on whether to grasp the target object. The specific implementation details of the algorithm are as follows:
[0118] Step 1: Initialize feature extraction network parameters ;
[0119] Step 2: Input the truss element data set;
[0120] Step 3: Extract and classify the features of the truss unit data set. The loss function for updating the feature extraction network parameters is: loss function = box position loss + category loss + confidence loss;
[0121] Step 4: Repeat steps 2 to 4 until the algorithm meets the termination condition.
[0122] S22, constructing a strategy model of the space robot arm, collecting state information of multiple space robot arms in a simulation environment as training data to train the strategy model of the space robot arm, and obtaining a trained strategy model of the space robot arm.
[0123] Optionally, the training of the strategy model of the space manipulator in S22 includes:
[0124] The policy model for a spatial manipulator is trained using a deep deterministic policy gradient framework.
[0125] Optionally, the state information of multiple space manipulators in the collected simulation environment in S22 is used as training data to train the strategy model of the space manipulator to obtain a trained strategy model of the space manipulator, including:
[0126] S221. Collect the base status information of multiple space manipulators in the simulation environment, train the neural network parameters according to the base status information, and obtain the optimal formation decision model.
[0127] In a feasible implementation, based on the obtained truss unit identification and classification results, it is determined whether to grasp and collaboratively assemble the mission target, and the state information of the space robot arm base is obtained from the mission space as input to train the neural network parameters, thereby obtaining the optimal formation decision model.
[0128] S222. Collect the robot state information of multiple spatial robot arms in the simulation environment, train the neural network parameters according to the robot state information, and obtain the optimal collaborative motion decision.
[0129] In a feasible implementation, the decoupling relationship of the space manipulator provides a prerequisite for the partial control of the space manipulator. Step S221 is first performed on the base of the space manipulator to make it reach the mission target point, and then the manipulator end of the space manipulator is collaboratively operated. The manipulator state information of the space manipulator is used as input to train the neural network parameters, thereby obtaining the optimal collaborative motion decision.
[0130] S223. According to the optimal formation decision model and the optimal collaborative motion decision, a strategy model of a trained space manipulator is obtained.
[0131] In a feasible implementation, the strategy model of the space manipulator learned in step S221 and step S222 is a mapping from the state space to the action space. The action is the controller's operating instruction to the space manipulator, which controls the angle increments of the six joints of the space manipulator by sending joint instructions, as well as the position control of the space end of the space manipulator base. In theory, there is a decoupling relationship between the base and the manipulator end of a space manipulator, and the action space dimension of the manipulator is , and the action space dimension of the base is .
[0132] S23. Input the state information into the trained strategy model of the space robot arm to obtain sample motion decision control instructions.
[0133] S24. According to the sample motion decision control instructions, control multiple space manipulators to assemble the truss units to obtain a reward value and new state information.
[0134] S25. Perform decision correction on the trained strategy model of the space robot according to the reward value and the new state information to obtain a corrected strategy model of the space robot.
[0135] In a feasible implementation, the space robot performs corresponding actions according to the motion decision instruction of step S23, so that the working environment changes. After the environment changes, the space robot will be able to obtain the reward value in the current state, and feed it back to the decision model in step S22 to correct the model. The reward value is set manually, mainly in the form of a composite reward, and the rewards of multiple robots in the assembly process are compositely set to ensure that the space robot can obtain different rewards in various situations. The reward value setting principle is as follows: the reward value is the largest when the truss is fully assembled, and the reward value gradually decreases as it is further away from this state. The state of being away from the task here is mainly the distance. At the same time, in the working environment, if the space robot collides, the task will be terminated.
[0136] Furthermore, the completion of steps S22 to S25 is defined as a complete interaction, and the completion of an assembly task by the space manipulator is defined as a round. The space manipulator needs to interact with the environment multiple times in a round, that is, continuously repeating steps S22 to S25 until the task termination condition is met. The termination condition includes three aspects: reaching the maximum number of interactions, collision, and completing the truss assembly.
[0137] S26: Determine whether the set termination condition is met.
[0138] If the set termination condition is met, the truss unit assembly is completed, and it is determined whether the set number of iterations is met; if the set number of iterations is met, the strategy model of the optimal spatial robot is output; if the set number of iterations is not met, the next truss unit is obtained and step S23 is executed.
[0139] If the set termination condition is not met, the process proceeds to step S24.
[0140] In a feasible implementation, the space robot system that meets the termination conditions enters the next round and records the sum of the reward values obtained by the space robot in the current round, and feeds the reward value back to step S22 for the neural network to update parameters along the gradient direction of maximizing the reward value.
[0141] Furthermore: After many iterations of training, the multi-agent reinforcement learning algorithm converges, and multiple spatial robotic arms can collaborate with each other to complete the truss assembly task. The neural network obtained at this time is the optimal decision model for sending motion instructions to the spatial robotic arms to achieve the truss assembly task.
[0142] The collaborative operating system of multiple robotic arms perceives the status information of multiple robotic arms, truss units and the environment, and builds a centralized training and distributed execution MADDPG (Multi-Agent Deep Deterministic Policy Gradient) multi-agent reinforcement learning model offline. It selects the optimal decision according to the real-time status of multiple robotic arms and sends control instructions to multiple robotic arms to control the effective movement of the spatial robotic arms for positioning and adjusting the posture, thereby obtaining the maximum reward value and completing the collaborative assembly of multiple robotic arms.
[0143] The present invention is achieved by Figure 5 The multi-agent deterministic gradient framework training shown in the figure seeks a set of neural network parameters that enables the space robot to complete the truss assembly task in the simulation training environment. The rewards during the training process are as follows: Figure 6 The specific implementation details of the system are as follows:
[0144] Step 1: Randomly initialize the spatial manipulator joint angles and base pose motion exploration values ;
[0145] Step 2: Record and initialize the joint angles of the space robot and the position of the base ;
[0146] Step 3: For each space robot , select the joint angles and base poses of the space manipulator That is, the current strategy and exploration value; Indicates the robot arm serial number, Indicates the current strategy, Represents a robotic arm observation status.
[0147] Step 4: Execute the space robot arm joint angle and base posture action Simultaneous Observation Reward And the new space robot joint angle and base posture status information ;
[0148] Step 5: Save to playback buffer ;
[0149] Step 6: Update the joint angle and base pose status information of the spatial manipulator:
[0150] ;
[0151] Step 7: Randomly select the playback buffer Minibatch ;
[0152] Step 8: Calculate the target value ,in, is the discount factor, is a deterministic policy function:
[0153] ; j represents the sample number, represents the target Q function, Indicates the new joint angle and base posture state information of the space manipulator corresponding to the jth sample, Indicates the next action. represents the updated policy function, represents the observed state in sample j.
[0154] Step 9: Minimize the loss function Updated Critique Network:
[0155] ;
[0156] In the formula, Indicates the number of samples in a mini-batch.
[0157] Step 10: Update the actor network using sampled policy gradients:
[0158] ;
[0159] In the formula, Represents the parameters of the actor network relative to The gradient of represents the objective function, Relative to the action gradient.
[0160] Step 11: Existence A space robot arm, Repeat steps 7 to 10;
[0161] Step 12: For Each Space Robot Update the target network parameters:
[0162] ;
[0163] Step 13: Repeat steps 3 to 12 until the termination condition is met;
[0164] Step 14: Repeat steps 1 to 13 until the algorithm converges.
[0165] The data acquired by the assembly sequence generation, truss unit recognition and robot arm decision-making of the present invention are original high-dimensional signals.
[0166] The truss unit identification system signal includes an RGB image.
[0167] Assembly sequence generation and robot arm decision-making rely on fully connected neural networks to extract features, and image signals for truss unit recognition rely on convolutional neural networks for feature extraction.
[0168] The fully connected neural network and convolutional neural network are trained using the autoencoder framework, and feature extraction is performed after convergence.
[0169] S3. According to the motion decision control instructions, multiple space manipulators are controlled to assemble the truss units, completing the multi-robot collaborative truss assembly task.
[0170] The method provided by the embodiment of the present invention can change the spatial capability requirements of the manipulator according to the specific scenario, especially when certain special circumstances occur or when the working environment changes, so that multiple spatial manipulators have higher adaptability, robustness and safety. On the one hand, the present invention aims to improve the assembly efficiency, reliability and anti-interference of the spatial truss, and on the other hand, to develop a new complex truss assembly system with a complete assembly process.
[0171] In the embodiment of the present invention, the overall assembly process is split to realize an autonomous truss assembly system through multiple robotic arms, which effectively reduces the reliance on manual experience, reduces the consumption of human resources, and improves the intelligence of operation.
[0172] The present invention introduces a deep neural network throughout the entire system, which can fully approximate complex nonlinear mapping relationships and learning capabilities.
[0173] The present invention combines proximal optimization strategy, target recognition network and reinforcement learning. In the overall system, it includes perception process and decision-making process, which is more similar to the manual assembly process of the truss. It has a certain adaptability to emergencies in the task environment and can effectively correct the collaborative operation decisions of multiple robotic arms.
[0174] The present invention adopts a space base to provide effective space capabilities for multiple robotic arms, so that the working space of the robotic arms can be effectively expanded, the overall task space can be effectively covered, and the fault tolerance of the assembly process can be improved.
[0175] The truss collaborative assembly system based on multi-agent reinforcement learning of the present invention adopts a collaborative operation mode, has higher scalability, robustness and higher load capacity, and cooperates with the system proposed by the present invention to make the assembly process have better intelligence and adaptability.
[0176] The introduction of neural networks makes the decision-making process of the truss collaborative assembly system based on multi-agent reinforcement learning more robust and capable of real-time correction.
[0177] Figure 7 is a block diagram of a multi-robot collaborative assembly device for a space truss based on reinforcement learning according to an exemplary embodiment, and the device is used in a multi-robot collaborative assembly method for a space truss based on reinforcement learning. Figure 7 The device includes a determination module 310, an input module 320 and an output module 330. Among them:
[0178] The determination module 310 is used to determine the truss units to be assembled.
[0179] The input module 320 is used to obtain the status information of the multiple space manipulators, input the status information into the strategy model of the optimal space manipulator, and obtain the motion decision control instructions of the multiple space manipulators.
[0180] The output module 330 is used to control multiple spatial manipulators to assemble the truss units according to the motion decision control instructions, thereby completing the multi-robot collaborative truss assembly task.
[0181] Optionally, the input module 320 is further configured to:
[0182] S21. Determine the truss unit.
[0183] S22, constructing a strategy model of the space robot arm, collecting state information of multiple space robot arms in a simulation environment as training data to train the strategy model of the space robot arm, and obtaining a trained strategy model of the space robot arm.
[0184] S23. Input the state information into the trained strategy model of the space robot arm to obtain sample motion decision control instructions.
[0185] S24. According to the sample motion decision control instructions, control multiple space manipulators to assemble the truss units to obtain a reward value and new state information.
[0186] S25. Perform decision correction on the trained strategy model of the space robot according to the reward value and the new state information to obtain a corrected strategy model of the space robot.
[0187] S26: Determine whether the set termination condition is met.
[0188] If the set termination condition is met, the truss unit assembly is completed, and it is determined whether the set number of iterations is met; if the set number of iterations is met, the strategy model of the optimal spatial robot is output; if the set number of iterations is not met, the next truss unit is obtained and step S23 is executed.
[0189] If the set termination condition is not met, the process proceeds to step S24.
[0190] Optionally, the input module 320 is further configured to:
[0191] S211, obtaining the assembly sequence of the truss unit, and collecting assembly sequence state information of the assembly sequence using a proximal optimization strategy network.
[0192] S212: Extract features from assembly sequence state information to generate an assembly sequence of the truss unit.
[0193] S213: Obtain target truss unit.
[0194] S214: Identify the target truss unit to obtain the target truss unit characteristics.
[0195] S215 : judging whether the target truss unit is a truss unit to be assembled according to the assembly sequence of the truss unit and the characteristics of the target truss unit.
[0196] If so, the truss elements are output.
[0197] If not, the next truss unit is obtained as the target truss unit, and the process goes to step S214.
[0198] Optionally, the input module 320 is further configured to:
[0199] A representation neural network is constructed to identify the target truss unit and obtain the characteristics of the target truss unit.
[0200] Optionally, the input module 320 is further configured to:
[0201] The policy model for a spatial manipulator is trained using a deep deterministic policy gradient framework.
[0202] Optionally, the input module 320 is further configured to:
[0203] The base status information of multiple space manipulators in the simulation environment is collected, and the neural network parameters are trained according to the base status information to obtain the optimal formation decision model.
[0204] The robot state information of multiple spatial manipulators in the simulation environment is collected, and the neural network parameters are trained according to the robot state information to obtain the optimal collaborative motion decision.
[0205] According to the optimal formation decision model and the optimal collaborative motion decision, a strategy model of the trained space manipulator is obtained.
[0206] In the embodiment of the present invention, the overall assembly process is split to realize an autonomous truss assembly system through multiple robotic arms, which effectively reduces the reliance on manual experience, reduces the consumption of human resources, and improves the intelligence of operation.
[0207] The present invention introduces a deep neural network throughout the entire system, which can fully approximate complex nonlinear mapping relationships and learning capabilities.
[0208] The present invention combines proximal optimization strategy, target recognition network and reinforcement learning. In the overall system, it includes perception process and decision-making process, which is more similar to the manual assembly process of the truss. It has a certain adaptability to emergencies in the task environment and can effectively correct the collaborative operation decisions of multiple robotic arms.
[0209] The present invention adopts a space base to provide effective space capabilities for multiple robotic arms, so that the working space of the robotic arms can be effectively expanded, the overall task space can be effectively covered, and the fault tolerance of the assembly process can be improved.
[0210] The truss collaborative assembly system based on multi-agent reinforcement learning of the present invention adopts a collaborative operation mode, has higher scalability, robustness and higher load capacity, and cooperates with the system proposed by the present invention to make the assembly process have better intelligence and adaptability.
[0211] The introduction of neural networks makes the decision-making process of the truss collaborative assembly system based on multi-agent reinforcement learning more robust and capable of real-time correction.
[0212] Figure 8 is a schematic diagram of the structure of a space truss multi-robot collaborative assembly device provided by an embodiment of the present invention, such as Figure 8 As shown, the space truss multi-robot collaborative assembly equipment may include the above Figure 7 The illustrated reinforcement learning-based space truss multi-robot collaborative assembly device. Optionally, the space truss multi-robot collaborative assembly device 410 may include a first processor 2001 .
[0213] Optionally, the space truss multi-robot collaborative assembly equipment 410 may further include a memory 2002 and a transceiver 2003 .
[0214] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.
[0215] Combine the following Figure 8 The various components of the space truss multi-robot collaborative assembly equipment 410 are specifically introduced:
[0216] The first processor 2001 is the control center of the space truss multi-robot collaborative assembly device 410, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).
[0217] Optionally, the first processor 2001 may execute various functions of the space truss multi-robot collaborative assembly device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .
[0218] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 8 CPU0 and CPU1 are shown in FIG.
[0219] In a specific implementation, as an embodiment, the space truss multi-robot collaborative assembly device 410 may also include multiple processors, such as Figure 8 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).
[0220] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.
[0221] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and access the memory 2002 through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.
[0222] The transceiver 2003 is used to communicate with a network device or a terminal device.
[0223] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 8 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.
[0224] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and communicate with the first processor 2001 through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.
[0225] It should be noted that Figure 8 The structure of the spatial truss multi-robot collaborative assembly device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0226] In addition, the technical effects of the space truss multi-robot collaborative assembly equipment 410 can refer to the technical effects of the space truss multi-robot collaborative assembly method based on reinforcement learning described in the above method embodiment, and will not be repeated here.
[0227] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0228] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).
[0229] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.
[0230] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.
[0231] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0232] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0233] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0234] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0235] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0236] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0237] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0238] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.
[0239] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A multi-robot collaborative assembly method for spatial trusses based on reinforcement learning. It is characterized in that The method comprises: S1. Determine the truss unit to be assembled; S2, obtaining state information of multiple space manipulators, inputting the state information into a strategy model of an optimal space manipulator, and obtaining motion decision control instructions of multiple space manipulators; S3. According to the motion decision control instructions, control multiple spatial manipulators to assemble the truss units, thereby completing the multi-robot collaborative truss assembly task.
2. The multi-robot collaborative assembly method for spatial trusses based on reinforcement learning according to claim 1, It is characterized in that The construction process of the optimal spatial manipulator strategy model in S2 includes: S21, determine the truss unit; S22, constructing a strategy model of the space manipulator, collecting state information of multiple space manipulators in a simulation environment as training data to train the strategy model of the space manipulator, and obtaining a trained strategy model of the space manipulator; S23, inputting the state information into the trained strategy model of the space manipulator to obtain a sample motion decision control instruction; S24, according to the sample motion decision control instruction, controlling multiple space manipulators to assemble the truss unit, and obtaining a reward value and new state information; S25, making decision corrections to the trained strategy model of the space manipulator according to the reward value and the new state information to obtain a corrected strategy model of the space manipulator; S26, judging whether the set termination condition is met; If the set termination condition is met, the truss unit assembly is completed, and it is determined whether the set number of iterations is met; if the set number of iterations is met, the strategy model of the optimal spatial manipulator is output; if the set number of iterations is not met, the next truss unit is obtained, and step S23 is executed; If the set termination condition is not met, the process proceeds to step S24.
3. The multi-robot collaborative assembly method for spatial trusses based on reinforcement learning according to claim 2, It is characterized in that The determining of the truss unit in S21 includes: S211, obtaining an assembly sequence of truss units, and collecting assembly sequence state information of the assembly sequence using a proximal optimization strategy network; S212, extracting features from the assembly sequence state information to generate an assembly sequence of a truss unit; S213, obtaining a target truss unit; S214, identifying the target truss unit to obtain the target truss unit feature; S215, judging whether the target truss unit is a truss unit to be assembled according to the assembly sequence of the truss unit and the characteristics of the target truss unit; If so, the truss element is output; If not, the next truss unit is obtained as the target truss unit, and the process goes to step S214.
4. The multi-robot collaborative assembly method for a spatial truss based on reinforcement learning according to claim 3, It is characterized in that The step of identifying the target truss unit in S214 to obtain the target truss unit features includes: A representation neural network is constructed to identify the target truss unit and obtain the characteristics of the target truss unit.
5. The multi-robot collaborative assembly method for spatial trusses based on reinforcement learning according to claim 2, It is characterized in that The step S22 of training the strategy model of the space robot arm includes: The policy model for a spatial manipulator is trained using a deep deterministic policy gradient framework.
6. The multi-robot collaborative assembly method for a spatial truss based on reinforcement learning according to claim 2, It is characterized in that The state information of multiple space manipulators in the simulation environment is collected as training data in S22 to train the strategy model of the space manipulator to obtain a trained strategy model of the space manipulator, including: Collecting the base state information of multiple space manipulators in the simulation environment, training the neural network parameters according to the base state information, and obtaining the optimal formation decision model; Collecting the manipulator state information of multiple space manipulators in the simulation environment, training the neural network parameters according to the manipulator state information, and obtaining the optimal collaborative motion decision; According to the optimal formation decision model and the optimal collaborative motion decision, a strategy model of a trained space manipulator is obtained.
7. A multi-robot collaborative assembly device for a space truss based on reinforcement learning, wherein the multi-robot collaborative assembly device for a space truss based on reinforcement learning is used to implement the multi-robot collaborative assembly method for a space truss based on reinforcement learning as described in any one of claims 1 to 7, It is characterized in that The device comprises: A determination module, used for determining the truss units to be assembled; An input module, used to obtain state information of multiple space manipulators, input the state information into a strategy model of an optimal space manipulator, and obtain motion decision control instructions of the multiple space manipulators; The output module is used to control multiple spatial manipulators to assemble the truss units according to the motion decision control instructions, so as to complete the truss assembly task of multi-robot collaboration.
8. The multi-robot collaborative assembly device for spatial trusses based on reinforcement learning according to claim 7, It is characterized in that The input module is further used for: S21, determine the truss unit; S22, constructing a strategy model of the space manipulator, collecting state information of multiple space manipulators in a simulation environment as training data to train the strategy model of the space manipulator, and obtaining a trained strategy model of the space manipulator; S23, inputting the state information into the trained strategy model of the space manipulator to obtain a sample motion decision control instruction; S24, according to the sample motion decision control instruction, controlling multiple space manipulators to assemble the truss unit, and obtaining a reward value and new state information; S25, making decision corrections to the trained strategy model of the space manipulator according to the reward value and the new state information to obtain a corrected strategy model of the space manipulator; S26, judging whether the set termination condition is met; If the set termination condition is met, the truss unit assembly is completed, and it is determined whether the set number of iterations is met; If the set number of iterations is met, the strategy model of the optimal spatial manipulator is output; if the set number of iterations is not met, the next truss unit is obtained and the process goes to step S23; If the set termination condition is not met, the process proceeds to step S24.
9. A space truss multi-robot collaborative assembly equipment, It is characterized in that The space truss multi-robot collaborative assembly equipment comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Multi-robot path collaborative planning method for on-orbit assembly of space truss structure
CN111273664A
Multi-mechanical-arm collaborative assembly method and system based on deep reinforcement learning
CN111881772A
Compliant control method and device for on-orbit assembly of double-arm robot
CN117182929A
Intelligent flexible assembly method for 3C products
CN118013838A
System and Method for Optimizing Radiofrequency Power Delivery based on Direction of Electric Field
KR103034672B1
Cited By
Multi-mechanical-arm cooperative motion planning method
CN119260743A
Multi-agent reinforcement learning-based conduit assembly sequence optimization method
CN119861667A
Catheter assembly sequence optimization method based on multi-agent reinforcement learning
CN119861667B
