Double-arm hierarchical control method and system for spatial non-cooperative target capture

Abstract

The invention relates to a double-arm hierarchical control method and system for capturing a spatial non-cooperative target, and the method improves the control precision through a double-arm hierarchical control architecture, and provides basic trajectory tracking and stable response at the initial stage of disturbance at a lower layer through a PD controller. And an SAC reinforcement learning strategy network is introduced into an upper layer to realize online residual error compensation of model errors and environment disturbance. According to the upper-layer SAC strategy, a multi-target reward function constructed based on a tail end position error and an attitude error is adopted, and a dynamic priority experience playback mechanism is introduced, so that key state samples are sampled more frequently in the training process, the strategy convergence speed is effectively increased, the self-adaptive control capability of the strategy in a key state is improved, and the self-adaptive performance of the strategy in the key state is improved. According to the method, the dependence on the system parameter modeling precision is weakened, the control system still has the stable operation capability under the disturbance condition, and the good engineering realizability is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of dual-arm robot control, and particularly to a dual-arm hierarchical control method and system for capturing a space non-cooperative target. BACKGROUND

[0002] Current control methods for capturing a space non-cooperative target mainly include three categories: traditional model-driven control, rule-based intelligent control, and deep reinforcement learning methods that have emerged in recent years. However, most researches focus on single-arm systems, and in space non-cooperative target capture tasks with high control accuracy requirements, a single control strategy often cannot provide a solution that takes into account both control accuracy and adaptability.

[0003] The existing control methods have the following disadvantages:

[0004] (1) Traditional control methods rely on high-precision dynamic modeling, and have poor adaptability to model errors and environmental disturbances. In particular, when a non-cooperative target has different disturbances, the control accuracy significantly decreases.

[0005] (2) Intelligent control methods usually rely on pre-set task rules and control logic, and lack self-adaptability to complex dynamic environments, making it difficult to deal with the uncertainty of non-cooperative targets.

[0006] (3) Reinforcement learning methods have the ability of self-learning strategy, but have problems such as low efficiency of training samples, slow convergence speed, and unstable early strategy, making it difficult to deploy in actual space applications.

[0007] In summary, space non-cooperative targets (such as failed satellites, debris, etc.) have great uncertainty and disturbance interference in the capture process due to the lack of pose information, and traditional single-arm control methods and single control strategies cannot meet the dual requirements of control accuracy and robustness. Although existing reinforcement learning methods have self-learning ability, their training process is easily affected by unstable initial strategies and low sample efficiency, making it difficult to be directly deployed in actual space tasks.

[0008] There is currently a lack of a dual-arm hierarchical control method and system to solve or partially solve the aforementioned problems. SUMMARY

[0009] The purpose of the present application is to overcome the defects of the prior art and provide a dual-arm hierarchical control method and system for capturing a space non-cooperative target, to solve or partially solve the technical problem that existing single-arm or single control strategies cannot simultaneously consider control accuracy, robustness, and dual-arm collaboration capability in capturing a space non-cooperative target.

[0010] The purpose of the present application can be achieved by the following technical solutions:

[0011] In one aspect of the present application, a dual-arm hierarchical control method for capturing a space non-cooperative target is provided, comprising the following steps:

[0012] Based on the pose of the space non-cooperative target, a desired trajectory is generated, and joint angles and angular velocities of the dual arms are obtained through inverse kinematics;

[0013] The lower PD controller outputs a basic torque according to the joint angles and angular velocities of the dual arms, and realizes stable tracking in the initial stage of strategy insufficient convergence or disturbance;

[0014] The upper SAC reinforcement learning controller outputs a compensation torque according to the joint angles and angular velocities of the dual arms, and corrects the output of the lower PD controller, thereby realizing compensation for modeling errors and external disturbances;

[0015] For samples generated by the interaction between the robotic arm and the environment during the reinforcement learning process, the priority is calculated based on the deviation of the samples from the task target and written into an experience replay pool;

[0016] Based on the samples in the experience replay pool, the SAC reinforcement learning controller is trained through dynamic priority experience replay;

[0017] Based on the basic torque and the compensation torque, dual-arm hierarchical control is realized.

[0018] As a preferred technical solution, in the SAC reinforcement learning controller, the state is the observation at each time, including joint angles, angular velocities and end pose errors, and the action is the compensation torque output by the policy network.

[0019] As a preferred technical solution, in the SAC reinforcement learning controller, the policy generates continuous control quantities by using a Gaussian policy, and the policy exploration is realized by maximizing the entropy regularization objective.

[0020] As a preferred technical solution, in the SAC reinforcement learning controller, the state value function is:

[0021]

[0022] Wherein, V π (s t ) is the state value function, s t is the state at time t, represents the expectation, Q π (s t ,a t ) is the state-action value function, a is the temperature coefficient, and a t is the action at time t.

[0023] As a preferred technical solution, in the SAC reinforcement learning controller, the state-action value function is:

[0024]

[0025] wherein Q π (s t ,a t ) is a state-action value function, denotes expectation, and gamma is a discount factor, r t+k is the reward at time t+k.

[0026] As a preferred technical solution, the priority based on the deviation degree of the sample from the task target comprises the following steps:

[0027] The deviation degree of the sample from the task target is represented by weighted summation of the end position error and the attitude error, and the priority of the sample is calculated.

[0028] As a preferred technical solution, the SAC reinforcement learning controller comprises an Actor network and a Critic network, the Critic network calculates a TD error based on a time difference and guides the Actor network to update a policy to obtain a compensation torque.

[0029] Another aspect of the present application provides a dual-arm hierarchical control system for capturing a non-cooperative target in space, characterized in that it comprises a space base and a dual-arm system carried by the space base, and the dual-arm system realizes capturing of a non-cooperative target in space through the aforementioned dual-arm hierarchical control method for capturing a non-cooperative target in space.

[0030] Another aspect of the present application provides an electronic device, comprising one or more processors and a memory, the memory storing one or more programs, the one or more programs comprising instructions for executing the aforementioned dual-arm hierarchical control method for capturing a non-cooperative target in space.

[0031] Another aspect of the present application provides a computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, the one or more programs comprising instructions for executing the aforementioned dual-arm hierarchical control method for capturing a non-cooperative target in space.

[0032] Compared with the prior art, the present application has at least one of the following beneficial effects:

[0033] (1) Reduce the dependence on high-precision dynamic modeling: In order to solve the problem that the traditional control method is not suitable for model error and environmental disturbance, the lower PD controller is used to realize the basic trajectory tracking, and the upper SAC strategy network output residual compensation torque is used to realize the hierarchical control architecture, which improves the precision and stability of the control system in the dynamic disturbance environment, and reduces the dependence on high-precision dynamic modeling.

[0034] (2) Improve adaptive control ability: In order to solve the problem that the training sample efficiency of reinforcement learning method is low and the convergence speed is slow, the multi-objective reward function based on end position error and attitude error and dynamic priority experience replay mechanism are introduced, which accelerates the policy convergence speed and improves the adaptive control ability in key state.

[0035] (3) High success rate of capturing task: In order to solve the problem that single arm or single control strategy is difficult to meet the demand of dual-arm cooperative capture, the dual-arm hierarchical control design is adopted, and the synergistic effect of PD control and SAC reinforcement learning is combined to enhance the dual-arm pose adjustment and coordination ability, and improve the success rate of space non-cooperative target capture task. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The flow chart of the dual-arm hierarchical control method for space non-cooperative target capture in the embodiment;

[0037] Figure 2 The schematic diagram of the space dual-arm system in the embodiment;

[0038] Figure 3 The principle diagram of the dual-arm hierarchical control in the embodiment;

[0039] Figure 4 The schematic diagram of the electronic device in the embodiment. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0041] Embodiment 1

[0042] In view of the problems of low control accuracy and unstable strategy commonly existing in the space non-cooperative target capturing task, and in consideration of the comprehensive requirements of high-precision control, strategy stability and dual-arm cooperation ability in the space non-cooperative target capturing process, the embodiment provides a dual-arm hierarchical control method for space non-cooperative target capturing, aiming to maintain control accuracy in a dynamic environment, so as to realize stable and reliable capturing of the non-cooperative target.

[0043] Referring to Figure 1 and Figure 3 , the method comprises the following steps:

[0044] Step S1, generating an expected trajectory based on the pose of the space non-cooperative target, and obtaining the joint angle and angular velocity of the dual arm through inverse kinematics.

[0045] Step S2, the PD controller of the lower layer outputs the basic torque according to the joint angle and angular velocity of the dual arm, so as to realize stable tracking in the initial stage of strategy insufficient convergence or disturbance.

[0046] Step S3, the SAC reinforcement learning controller of the upper layer outputs the compensation torque according to the joint angle and angular velocity of the dual arm, corrects the output of the lower layer PD controller, and realizes compensation for modeling errors and external disturbances.

[0047] The Critic network calculates the TD error based on the time difference and guides the Actor network to update the strategy, so as to obtain the compensation torque and cope with the modeling errors and external disturbances online.

[0048] Specifically, the state s t of reinforcement learning is composed of the observation of the system at time t, including the joint angle q, angular velocity and end pose error e p ,e r The action a t is the compensation torque vector output by the strategy network, which is used to correct the output of the lower layer PD controller and realize compensation for modeling errors and external disturbances. The strategy π adopts a Gaussian strategy to generate continuous control, and the state value function is The state-action value function is Wherein, α is the temperature coefficient, and γ is the discount factor.

[0049] Step S4, for the samples generated by the interaction between the robot arm and the environment in the reinforcement learning process, the priority is calculated based on the deviation of the samples from the task target and written into the experience replay pool.

[0050] To improve the training efficiency of large error state samples, a dynamic priority experience replay mechanism is introduced. When the samples generated by the interaction between the robot arm and the environment are written into the experience pool, the priority of the sample is calculated according to the weighted sum of the end position error and the attitude error, which reflects the deviation of the sample from the task target.

[0051] During the running process, the collected (s t ,a t ,r t ,s t+1 ) samples are written into the experience replay pool to provide samples for later training.

[0052] Step S5, based on the samples in the experience replay pool, the SAC reinforcement learning controller is trained through dynamic priority experience replay.

[0053] Step S6, based on the base torque and the compensation torque, the dual-arm hierarchical control is realized.

[0054] The mechanical arm system is driven by the control torques output by the PD controller and the SAC policy network in the upper and lower layers respectively, realizing high-precision pose alignment and cooperative capture of the non-cooperative target.

[0055] The method completes modeling and simulation training in the MuJoCo 3.2.0 physical engine environment, and the policy training is realized based on the Stable-Baselines3 reinforcement learning framework. The method combines the advantages of model-driven and data-driven control in the hierarchical control architecture, realizes the basic tracking control of the target trajectory through the lower PD controller, and the upper SAC policy network outputs the residual compensation torque to respond to modeling errors and external disturbances in real time, thereby improving the precision and stability of the control system. At the same time, the method weakens the dependence on complex modeling and a large amount of prior knowledge in design, and improves the usability of the control system in complex space scenarios.

[0056] Embodiment 2

[0057] Based on embodiment 1, the dual-arm hierarchical control system for space non-cooperative target capture is provided, which includes a space platform and a dual-arm system carried by the space platform. The dual-arm system realizes the capture of the space non-cooperative target through the dual-arm hierarchical control method for space non-cooperative target capture as in embodiment 1. Referring to Figure 2 , the dual-arm system is the basis, which is composed of two six-degree-of-freedom UR5e robot arms, respectively fixed on the two sides of the platform, as shown in Figure 1 , where ∑ O is the world coordinate system, ∑ LE and ∑ RE represent the coordinate systems of the left and right end effectors respectively, and the base coordinate system ∑ BThe relative position and attitude between the target and the world coordinate system are described, and the cooperative capture operation of the space non-cooperative target can be completed, and the pose adjustment and coordination ability required for executing the high-precision capture task is possessed.

[0058] The application adopts a double-arm hierarchical design in a control architecture, and a PD controller is configured for the left and right mechanical arms respectively to realize tracking of a target trajectory. Under the condition that the dynamic model has uncertainty and insufficient precision, the lower-layer control strategy can still maintain the stability of the pose control, and effectively reduces the dependence of the system on high-precision modeling under the condition of interference.

[0059] In order to realize online adaptive compensation for modeling errors and external disturbances, a residual compensation learning strategy based on the SAC algorithm is designed in the upper-layer control module, and a multi-objective reward function r = - lambda1e p - lambda2e r is constructed, wherein lambda1 and lambda2 are self-defined weights, e p is the end position error, e r is the end attitude error. Compared with other capture methods using only position error, the attitude is considered at the same time, which promotes the realization of stable capture, and a dynamic priority experience replay mechanism is introduced to improve the training efficiency and key state coverage of the strategy. The strategy cooperates with the lower-layer PD controller, effectively enhances the pose adjustment ability of the system when the double arms cooperatively execute the capture task, improves the task success rate, and overcomes the limitations of existing single strategy or single-arm control method in control precision and convergence efficiency.

[0060] Embodiment 3

[0061] The embodiment provides an electronic device, including one or more processors and memories, the memories store one or more programs, the one or more programs include instructions for executing the double-arm hierarchical control method for space non-cooperative target capture as described in embodiment 1.

[0062] As Figure 4 described, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course, other hardware required by the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to realize the above Figure 1 described method. Of course, in addition to the software implementation, the application does not exclude other implementation manners, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0063] Memory can include, without being limited to, non- transitory computer-readable media, random access memory (RAM), and / or read-only memory (ROM), such as flash memory, etc. Memory is an example of computer-readable media.

[0064] Computer-readable media includes permanent and non- permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, without being limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic disks storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media, such as modulated data signals and carrier waves.

[0065] The above descriptions are only specific embodiments of the present application, and the protection scope of the present application is not limited thereto. Any modifications or replacements within the technical scope disclosed by the present application can be easily conceived by those skilled in the art, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A dual-arm hierarchical control method for spatial non-cooperative target capture, characterized in that, The method comprises the following steps: The desired trajectory is generated based on the pose of the space non-cooperative target, and the joint angle and angular velocity of the dual-arm are obtained through inverse kinematics; The lower PD controller outputs the basic torque according to the joint angle and angular velocity of the dual-arm, and realizes stable tracking in the initial stage of disturbance or when the strategy is not fully converged; The upper SAC reinforcement learning controller outputs the compensation torque according to the joint angle and angular velocity of the dual-arm, and corrects the output of the lower PD controller to realize compensation for modeling errors and external disturbances; The priority is calculated based on the deviation of the sample from the task target and written into the experience replay pool; The SAC reinforcement learning controller is trained based on the samples in the experience replay pool through dynamic priority experience replay; The dual-arm hierarchical control is realized based on the basic torque and the compensation torque.

2. The dual-arm hierarchical control method for capturing a space non-cooperative target according to claim 1, wherein, In the SAC reinforcement learning controller, the state is the observation at each time, including the joint angle, angular velocity and end pose error, and the action is the compensation torque output by the policy network.

3. The dual-arm hierarchical control method for capturing a space non-cooperative target according to claim 1, wherein, In the SAC reinforcement learning controller, the policy generates continuous control quantity through Gaussian policy, and realizes policy exploration by maximizing the entropy regularization target.

4. The dual-arm hierarchical control method for capturing a space non-cooperative target according to claim 1, wherein, In the SAC reinforcement learning controller, the state value function is: where V π (s t ) is the state value function, s t is the state at time t, denotes the expectation, Q π (s t ,a t ) is the state-action value function, and a t is the action at time t.

5. The dual-arm hierarchical control method for capturing a space non-cooperative target according to claim 1, wherein, In the SAC reinforcement learning controller, the state-action value function is: where Q π (s t , a t ) is the state-action value function, denotes expectation, and γ is a discount factor. r t+k is the reward at time t+k.

6. The dual-arm hierarchical control method for capturing a space non-cooperative target according to claim 1, wherein, The priority is calculated based on the deviation of the sample from the task target and written into the experience replay pool, which comprises the following steps: The deviation of the sample from the task target is represented by the weighted sum of the end position error and the attitude error, and the priority of the sample is calculated.

7. The dual-arm hierarchical control method for capturing a space non-cooperative target according to claim 1, wherein, The SAC reinforcement learning controller comprises an Actor network and a Critic network, and the Critic network calculates the TD error based on the time difference and guides the Actor network to update the policy to obtain the compensation torque.

8. A dual-arm hierarchical control system for space non-cooperative target capture, characterized in that, The dual-arm hierarchical control method for capturing space non-cooperative targets comprises a space base and a dual-arm system carried by the space base, and realizes the capture of space non-cooperative targets through the dual-arm hierarchical control method for capturing space non-cooperative targets.

9. An electronic device, comprising: It comprises: One or more processors and a memory, the memory storing one or more programs, the one or more programs comprising instructions for executing the dual-arm hierarchical control method for capturing space non-cooperative targets.

10. A computer-readable storage medium, characterized in that, It comprises one or more programs for one or more processors of an electronic device, the one or more programs comprising instructions for executing the dual-arm hierarchical control method for capturing space non-cooperative targets.

Images