Control method of space continuous type rope driving arm

Abstract

The invention discloses a control method for a space continuous type rope driving arm. The control method comprises the steps that a simulation environment is built for the space continuous type rope driving arm; constructing a reinforcement learning framework of the spatial continuous rope drive arm, and initializing an experience playback pool and a reinforcement learning model of the reinforcement learning framework; acquiring a plurality of expert tracks, and training a behavior cloning model according to the plurality of expert tracks; prefilling the experience playback pool with high-quality data; carrying out online interaction with a simulation environment by utilizing a reinforcement learning model, and storing generated interaction data into an experience playback pool; training data are extracted from the experience playback pool, the reinforcement learning model is updated under the assistance of the behavior cloning model, and after updating is completed, the reinforcement learning model after updating is completed is adopted to control the space continuous type rope drive arm. According to the method, the advantage of simple design of a sparse reward function can be reserved, and the problems of undefined strategy updating direction, difficulty in convergence and the like caused by sparse reward signals can be effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of rope-driven manipulator control, and particularly relates to a space continuous rope-driven arm control method. BACKGROUND

[0002] With the rapid development of space technology, space exploration has become an important direction of national development, and the number of artificial spacecraft is showing a rapid growth trend. Under this background, it is particularly important to establish a perfect spacecraft on-orbit maintenance system, which includes regular detection and maintenance of on-orbit spacecraft, and removal of failed spacecraft, which is crucial to ensure the safety of the space environment and maintain the operation of existing artificial spacecraft.

[0003] In the face of the challenges of complex spacecraft structure and limited operation space in the space environment, the traditional manipulator system often fails to perform fine operation tasks due to size limitations. On the contrary, the space continuous rope-driven arm can avoid the problems of traditional manipulators. It uses cable tension driving instead of traditional motor direct driving, and realizes joint angle control by accurately regulating the length of the rope stretch through the motor. At the same time, the space continuous rope-driven arm has a slender shape, and all driving motors are concentrated in the control box of the base, realizing the separation of mechanical and electrical systems. Therefore, the space continuous rope-driven arm has excellent motion flexibility and lightweight characteristics, and is suitable for fine operation in complex external environment. For this reason, this technology has broad application prospects in satellite on-orbit maintenance, space debris removal and other space tasks.

[0004] In recent years, in the field of control of space continuous rope-driven arms, reinforcement learning (RL) methods have gradually surpassed traditional control methods based on kinematics and dynamics modeling, especially after the combination of reinforcement learning algorithms and deep neural networks, the application range of reinforcement learning in the field of robots has been greatly expanded, and it has broad application prospects. Currently, space robot control methods based on reinforcement learning are mainly divided into two categories: deep reinforcement learning based on dense reward function (DRLDR) and deep reinforcement learning based on sparse reward function (DRLSR). The former designs a fine reward function, so that the robot can obtain immediate feedback closely related to the environment state at each step, thereby significantly improving training efficiency and final performance, but this method faces challenges such as complex reward function design and the need for domain knowledge support; the latter only provides reward signals at key task nodes (such as task success or failure), which simplifies the reward function design process, but due to the lack of fine-grained training guidance, it often has problems such as low learning efficiency and convergence difficulty.

[0005] Most of the current reinforcement learning methods for space continuous rope-driven arms are based on dense reward functions. The reward value of this method is closely related to the real-time state of the robot arm. Therefore, by adjusting the structure of the reward function step by step, the learning direction of the agent can be finely guided, thereby improving the control effect of the robot arm. However, this method has the following defects: 1) Designing an effective dense reward function often requires combining task characteristics. The function form usually contains multiple weighted terms and complex parameter settings, which not only requires the designer to have deep professional knowledge, but also needs to fine-tune through a large number of trial-and-error experiments. This highly dependent on expert experience design process not only significantly increases the technical threshold, but also significantly increases the algorithm development cycle and deployment cost, making it difficult for this method to be quickly popularized and popularized in practical engineering applications, severely restricting the widespread application of intelligent control technology in high-end fields such as aerospace. 2) The space environment has high dynamics and uncertainty. The traditional dense reward function often contains too many complex constraints related to the state of the environment and the parameters of the device. Once the task scenario or the execution mechanism changes, the reward function structure needs to be redesigned and the hyperparameters need to be adjusted. This repetitive work not only has low efficiency, but also is prone to reward function failure due to environmental disturbances, which seriously affects the stability and adaptability of the control algorithm.

[0006] Another technical solution is to combine the sparse reward-based reinforcement learning method with Hindsight Experience Replay (HER) or Random Network Distillation (RND) to improve the control effect of the sparse reward-based reinforcement learning method. However, this method has the following defects: 1) This technology improves the control effect of the sparse reward-based reinforcement learning algorithm by improving the utilization efficiency of online interaction data. When the reinforcement learning strategy is weak, it is difficult to interact with high-quality data, so HER and RND cannot fundamentally solve the problem of insufficient data quality, and the effect of improving the sparse reward-based reinforcement learning is weak. 2) This technology requires structural modification of the neural network architecture of reinforcement learning to effectively integrate HER and RND, which not only involves complex algorithm integration work, but also requires re-adjustment of network hyperparameters, which increases the implementation difficulty and significantly improves the complexity of engineering deployment.

[0007] The disclosure of the above background art content is only used to assist in understanding the concept and technical solutions of the present application, and it does not necessarily belong to the prior art of the present patent application. In the absence of explicit evidence that the above content has been disclosed before the filing date of the present patent application, the above background art should not be used to evaluate the novelty and inventiveness of the present application. SUMMARY

[0008] To solve the above technical problems, the application provides a control method for a spatial continuous rope-driven arm, which can retain the simple advantage of sparse reward function design and effectively solve the problems of unclear policy update direction and convergence difficulty caused by sparse reward signal.

[0009] To achieve the above object, the application adopts the following technical scheme: In a first aspect, the application discloses a control method for a spatial continuous rope-driven arm, comprising the following steps: S1: building a simulation environment for the spatial continuous rope-driven arm; S2: constructing a reinforcement learning framework for the spatial continuous rope-driven arm and initializing an experience replay pool and a reinforcement learning model of the reinforcement learning framework; S3: obtaining multiple expert trajectories and training a behavior cloning model according to the multiple expert trajectories; S4: pre-filling high-quality data into the experience replay pool of the reinforcement learning framework; S5: performing online interaction with the simulation environment by using the reinforcement learning model and storing the generated interaction data into the experience replay pool of the reinforcement learning framework; S6: extracting training data from the experience replay pool of the reinforcement learning framework, updating the reinforcement learning model with the assistance of the behavior cloning model, and controlling the spatial continuous rope-driven arm by using the updated reinforcement learning model.

[0010] Preferably, the simulation environment in S1 is constructed based on a MuJoCo platform, wherein the spatial continuous rope-driven arm comprises a floating base and at least two arm segments, each arm segment is controlled to deform by a plurality of groups of adjustable tension driving units, and the driving units realize the bending deformation of the arm segments by continuously adjusting the tension values.

[0011] Preferably, S3 specifically comprises: obtaining multiple expert trajectories and training a behavior cloning model according to the multiple expert trajectories in a supervised learning manner.

[0012] Preferably, the high-quality data in S4 comprises the multiple expert trajectories and successful trajectories obtained by the interaction between the behavior cloning model and the simulation environment.

[0013] Preferably, the updating of the reinforcement learning model with the assistance of the behavior cloning model in S6 specifically comprises: generating an auxiliary action by the behavior cloning model, constructing a time difference target by combining the auxiliary action and the action generated by the reinforcement learning model, and updating the reinforcement learning model based on the time difference target.

[0014] Preferably, the step of combining the auxiliary action with the action generated by the reinforcement learning model to construct a temporal difference target includes: Using a soft maximum function to evaluate the value of the auxiliary action and the action generated by the reinforcement learning model; The final action selection is determined based on probability distribution sampling to construct the temporal difference target.

[0015] Preferably, updating the reinforcement learning model based on the temporal difference objective includes: The network parameters of the reinforcement learning model are updated by the gradient descent loss function, where the loss function is expressed as:

[0016] Where, It is a value network The loss function is updated respectively. A value network, It is a strategic network The loss function of is the temporal difference target, For the experience replay pool; yes t The state of the time-space continuous rope-driven arm, yes t The movement of the time-space continuous rope-driven arm, yes t Reward of the moment.

[0017] Preferably, in S6, during the process of updating the reinforcement learning model with the assistance of the behavior cloning model, a random batch distillation method is used to simultaneously copy multiple value networks, and two value networks are randomly selected as the value networks used for each update.

[0018] Preferably, the reinforcement learning model adopts a double-delayed deep deterministic policy gradient algorithm.

[0019] In a second aspect, the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program is configured to be executable by a processor to execute the control method of the spatial continuous rope-driven arm described in the first aspect.

[0020] Compared with the prior art, the control method of the spatial continuous rope-driven arm has the beneficial effects that: the control method of the spatial continuous rope-driven arm combines a behavior cloning model and a reinforcement learning model, the strategy of the behavior cloning model is used as an auxiliary optimization target, and the target function in the reinforcement learning model is used to guide the reinforcement learning model to update, the advantages of simple design of a sparse reward function can be retained, and problems such as unclear strategy updating direction and convergence difficulty caused by the sparse reward signal can be effectively solved. The hybrid learning paradigm of cooperative optimization of the strategy of the behavior cloning model and the reinforcement learning model not only avoids the disadvantages of complex artificial design and parameter adjustment of deep reinforcement learning (DRLDR) based on a dense reward function, but also fully utilizes limited expert knowledge to guide the learning process, realizes efficient reinforcement learning control in a sparse reward environment, and finally realizes control of the spatial continuous rope-driven arm in the sparse reward environment.

[0021] In further schemes, the present application has the following beneficial effects: (1) In the present application, the behavior cloning model is trained by using multiple expert trajectories in a supervised learning manner, which can effectively reduce the gap between the output actions of the agent and the expert trajectories, and provide high-quality training data for subsequent reinforcement learning, thereby significantly improving the problem of low sample efficiency in a sparse reward environment.

[0022] (2) In the present application, multiple expert trajectories and successful trajectories obtained by interaction between the behavior cloning model and the simulation environment are pre-filled into the experience replay pool, which can fundamentally solve the problem of insufficient data quality and significantly improve the training effect.

[0023] (3) In the present application, the random batch distillation method is used in the process of auxiliary updating of the reinforcement learning model by the behavior cloning model, which improves the sample efficiency and further improves the training effect. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a control method flowchart of the spatial continuous rope-driven arm of the embodiment one of the present application; Figure 2 is a general flow framework diagram of the sparse reward reinforcement learning control method of the spatial continuous rope-driven arm based on behavior cloning assistance in the embodiment one of the present application; Figure 3 is an illustration of a simulation environment and a structure diagram of a spatial continuous rope-driven arm in the embodiment of the present application; Figure 4 is a schematic diagram of the process of the spatial continuous rope-driven arm approaching a target under the control of the algorithm of the embodiment of the present application; Figure 5 is a result diagram of comparative experiments of the algorithm of the embodiment of the present application and a baseline algorithm in anchoring space, floating space and sputtering space. Figure 6 is the ablation experiment result graph of the algorithm proposed by the embodiment of the present application. DETAILED DESCRIPTION

[0025] The embodiments of the present application will be described in detail below. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present application and its applications.

[0026] It should be noted that when an element is referred to as being "fixed" or "set" on another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, the connection can be for fixing or for circuit / signal communication.

[0027] It should be understood that the terms "length", "width", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the embodiments of the present application and simplify the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0028] In addition, the terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0029] In view of the problems existing in the prior art, the application innovatively proposes a control framework combining behavior cloning (BC) and reinforcement learning (RL). As an important branch of imitation learning, behavior cloning can effectively reduce the gap between the output action of the agent and the expert trajectory by using expert demonstration data to construct a policy model through supervised learning. In the application, first, the BC model is obtained through supervised learning to provide high-quality training data for subsequent reinforcement learning, thereby significantly improving the problem of low sample efficiency in a sparse reward environment. Second, in the RL update stage, the behavior cloning strategy is used as an auxiliary optimization goal to guide the RL model update together with the reinforcement learning objective function, thereby retaining the advantage of simple design of the sparse reward function and effectively solving the problems of unclear strategy update direction and convergence difficulty caused by the sparsity of the reward signal. This hybrid learning paradigm not only avoids the disadvantages of complex manual design and parameter tuning of DRLDR, but also fully utilizes limited expert knowledge to guide the learning process, and finally realizes the control of complex tasks such as space continuous rope-driven arms in a sparse reward environment.

[0030] As shown in Figure 1 and Figure 2 , the preferred embodiment of the application discloses a control method of a space continuous rope-driven arm, comprising the following steps: S1: building a simulation environment for the space continuous rope-driven arm; The simulation environment is built based on a MuJoCo platform, and the space continuous rope-driven arm includes a floating base and at least two arm segments, each arm segment is controlled to deform through a plurality of groups of adjustable tension driving units, and the driving units realize the bending deformation of the arm segment by continuously adjusting the tension value.

[0031] In one embodiment, the simulation environment is built by a MuJoCo (Multi-Joint Dynamics with Contact) platform, the gravity parameter in the MuJoCo model is set to 0 to simulate a space microgravity environment, and the interaction force between the floating base and the arm segment is modeled by a dynamic coupling equation to ensure that the base pose is updated in real time with the movement of the arm segment. As shown in Figure 3As shown, the simulation environment simulates the ropes by using the tendon module unique to MuJoCo, avoiding the equivalence of the spatial continuous rope-driven arm to a joint-driven system, and significantly improving the accuracy of the simulation environment. In this embodiment, the spatial continuous mechanical arm model includes one floating base and two connected arm segments (arm segment I and arm segment II), each arm segment being driven by four tendons (for example, the four tendons of each arm segment are evenly distributed in a circumferential direction and are located in four positions of front, rear, left and right, and the bending deformation of the arm segment in the horizontal and vertical directions is realized by coordinating the tension difference of adjacent tendons, for example, by independently adjusting the contraction length of the four tendons to generate differential tension, making the adjacent arm rods rotate around the cross joint to form an equal curvature bending), and the end of the arm segment II is provided with an end effector. Each arm segment includes six arm rods, and the arm rods are connected by cross joints. By setting the damping and stiffness of the cross joints, the horizontal rotation joint bending angles of adjacent arm rods in a single arm segment are ensured to be the same, and the vertical rotation joint bending angles of adjacent arm rods in a single arm segment are ensured to be the same, so that a single arm segment can be bent into an equal curvature arc, and the bending modes of different arm segments are independent of each other, which is the characteristic of the spatial continuous rope-driven arm that is "coupled within a segment and independent between segments". The floating base can move freely and has six degrees of freedom; each arm segment has two equivalent degrees of freedom, so the system of the spatial continuous rope-driven arm has a total of ten degrees of freedom.

[0032] S2: Constructing a reinforcement learning framework of the spatial continuous rope-driven arm, and initializing an experience replay pool of the reinforcement learning framework and a reinforcement learning model; In one embodiment, the reinforcement learning model adopts a double-delay deep deterministic policy gradient algorithm, and updates the policy by a target network to ensure the stability of training. The constructed reinforcement learning framework is based on sparse rewards. In the sparse reward reinforcement learning framework, the sparse reward function only provides a reward signal when the distance between the end effector and the target point is less than a preset threshold, and the reward is 0 at other times.

[0033] Specifically, reinforcement learning is used to make sequential decisions in the framework of Markov Decision Process (MDP) and can be described as the following tuple: , wherein represents a state space, represents an action space, represents a state transition function , i.e., the probability of transitioning from the current state-action pair to the next time state , and represent a reward function and a discount factor.

[0034] Specifically, the state may be represented as follows:

[0035] wherein, and represent the position and velocity of the end effector, represent the position of the target point, represent the distance of the end effector relative to the target point, represent the pose vector of the floating base, represent the center of mass velocity and rigid body angular velocity vector of the floating base, represent the moving distance of the rope, represent the number of arm segments.

[0036] Action may be represented as follows:

[0037] Since each rope is controlled by a linear motor, represent the control signal of the i-th rope at time t, i.e., the control signal of the i-th linear motor; represent the maximum control signal of the i-th sliding motor, since the rope has a one-way deformation property (tension is generated in the tensile state, otherwise it is relaxed), located in the negative interval . .

[0038] The reward at time t can be represented as follows:

[0039] wherein, is the distance of the end effector of the spatial continuous rope-driven arm relative to the target point at time t, i.e., the distance after the action is completed; is a threshold distance. When , it can be considered that the end effector has reached the target point, at which time the task completion reward is 1, otherwise the task has not been completed, and the reward is 0.

[0040] S3: Obtain multiple expert trajectories, and train a behavior cloning model according to the multiple expert trajectories; This step specifically includes: obtaining multiple expert trajectories, and training a behavior cloning (BC) model according to the multiple expert trajectories in a supervised learning manner.

[0041] ​​​​Wherein, each expert trajectory contains the state, action and reward information of the successful trajectory, and before training the BC model, all the state and action information is stored in the BC experience replay pool , and the behavior cloning model is , , the model parameters, are trained through a neural network, and the following loss function is gradually reduced:

[0042] When is close to 0, is the same as , and at this time it can be considered that the BC model can output the corresponding action for the state contained in the expert trajectory.

[0043] The BC model has extremely high precision in imitating expert actions, which enables it to effectively support the subsequent training stage. Secondly, BC only needs a small amount of demonstration samples to complete training without online environment interaction. Moreover, BC has the advantages of high precision, high sample efficiency and stability, and is very suitable as an auxiliary model.

[0044] S4: pre-fill high-quality data into the experience replay pool of the reinforcement learning framework; The high-quality data in this step includes multiple expert trajectories and successful trajectories obtained by interacting with the simulation environment through the behavior cloning model. That is, this step specifically includes: S4.1: storing the pre-collected expert trajectories into the experience replay pool of the reinforcement learning framework; S4.2: using the behavior cloning model to interact with the simulation environment online, and storing the successful trajectories obtained by the interaction into the experience replay pool of the reinforcement learning framework.

[0045] The reinforcement learning framework based on sparse rewards is difficult to produce high-quality data in the interaction process with the environment due to the lack of accurate reward guidance, and therefore often has poor training effect or even no effect. In order to solve this problem, before training the reinforcement learning (RL) model, the RL experience replay pool is pre-filled with high-quality data . Wherein, the high-quality data includes the following two sources: (1) expert trajectories (all successful trajectories); (2) successful trajectories obtained by interacting with the simulation environment using the BC model. Because the cost of obtaining space environment data is high, the number of available expert trajectories is usually small, therefore, the BC model is used to interact with the environment and collect successful trajectories to expand the scale of high-quality data.

[0046] S5: using the reinforcement learning model to interact with the simulation environment online, and storing the interaction data generated into the experience replay pool of the reinforcement learning framework; The training process of the RL model can be divided into an interaction stage and an update stage, the present step S5 is the interaction stage, and S6 is the update stage. In the interaction stage, the RL model interacts with the environment alone without the participation of the BC model, and the data obtained by the interaction is stored in the RL experience replay pool .

[0047] S6: extracting training data from the experience replay pool of the reinforcement learning framework, updating the reinforcement learning model with the assistance of the behavior cloning model, and obtaining the reinforcement learning model capable of controlling the spatial continuous rope-driven arm in the sparse reward environment after the update is completed.

[0048] Wherein, updating the reinforcement learning model with the assistance of the behavior cloning model specifically includes: generating an auxiliary action through the behavior cloning model, constructing a time difference target combining the auxiliary action and the action generated by the reinforcement learning model, and updating the reinforcement learning model based on the time difference target.

[0049] Combining the auxiliary action and the action generated by the reinforcement learning model to construct a time difference target includes: using a soft maximum function to evaluate the value of the auxiliary action and the action generated by the reinforcement learning model; determining the final action selection based on the probability distribution sampling to construct the time difference target. Wherein, when the soft maximum function is introduced, the value evaluation weights of the BC generated action and the RL generated action are normalized to ensure that the probability distribution of the action selection is consistent with the value evaluation, and to avoid the sampling deviation caused by the Q value scale difference.

[0050] Updating the reinforcement learning model based on the time difference target includes: updating the network parameters of the reinforcement learning model through gradient descent loss function.

[0051] In the process of updating the reinforcement learning model with the assistance of the behavior cloning model, a random batch distillation method is used to copy N value networks at the same time, and two value networks are randomly selected from the N value networks as the value networks used for this update each time.

[0052] Specifically, the update stage includes: (1) extracting data from the experience replay pool; (2) generating actions and constructing a time difference target; (3) updating the neural network.

[0053] The present embodiment extracts data from the RL experience replay pool in batches, wherein the pre-filled data and the online interaction data have the same probability of being extracted, and the mean network of the BC model and the target policy network of the RL model respectively generate actions:

[0054] Wherein, is The state of the time-space continuous rope-driven arm, and The BC model and the RL model are To improve the efficiency of sample utilization, the present invention uses Random Ensemble Distillation (RED), which is to copy Value networks, each time an update is made, 2 of them are randomly selected as the value networks used in this update. For example, . Then, using the target value network of the RL model The value of two actions can be evaluated:

[0055] Since the softmax function is continuously differentiable and suitable for gradient descent when updating the neural network, the following discrete probability distribution can be constructed:

[0056] in, is the proportional coefficient of the soft maximum function, is an exponential function. By randomly sampling in this probability distribution, the action is determined , and finally get the time series difference target :

[0057] in, for Reward at all times, is the discount factor.

[0058] In some embodiments, The recommended value range of is [0.1, 1.0], which adjusts the randomness of action selection by balancing exploration and exploitation; For example, setting it to 0.99 meets the standard reinforcement learning requirement for discounting long-term rewards.

[0059] After constructing the temporal difference target, the parameters of the reinforcement learning neural network can be updated through the gradient descent loss function, where It is a value network The loss function can be updated separately A value network, It is a strategic network The loss function of .

[0060]

[0061] Then update the target value network and a target policy network :

[0062] wherein, a network parameter of the target value network , a network parameter of the value network , a soft update parameter, a network parameter of the target policy network , a network parameter of the policy network .

[0063] In the early stage of the RL training process, the target value network is difficult to accurately evaluate and , but at this time, the RL experience replay pool is mostly pre-filled high-quality data, and the newly filled interaction data is less, so the data quality used for RL model updating is still high; in the later stage of the RL training process, can select better actions from and to further train the RL model. In summary, the BC model can replace poor in the process of updating the RL model, thereby improving the training effect of the RL.

[0064] In one specific embodiment, the RL model adopts TD3 (Twin Delayed Deep Deterministic Policy Gradient), which has the advantages of fast training speed, simple structure, and low sensitivity to hyperparameters. In the present application, TD3 is combined with BC to ultimately construct BATD3 (Behavior Cloning assisted Twin Delayed Deep Deterministic Policy Gradient).

[0065] The control method of the spatial continuous rope-driven arm disclosed in Embodiment One of the present application solves the following problems existing in the existing control method of the spatial continuous rope-driven arm: (1) The present application is based on a sparse reward reinforcement learning algorithm, which avoids the complex reward function shaping process of a dense reward function, and also avoids the repeated workload of algorithm transplantation between different devices.

[0066] (2) The application introduces expert trajectories into the experience replay pool of reinforcement learning, and generates successful trajectories by interacting with the environment using behavior cloning, thereby fundamentally solving the problem of insufficient data quality and significantly improving the training effect.

[0067] (3) The application avoids using methods that require structural adjustment of the original network of reinforcement learning, but instead integrates behavior cloning with reinforcement learning, which is simple to integrate and has good training effect.

[0068] The effectiveness of the control method of the spatial continuous rope-driven arm provided by the embodiment one of the application is verified as follows.

[0069] The control task completed by the embodiment of the application is to drive the rope to change the arm type of the spatial continuous rope-driven arm, so that the end effector gradually approaches and reaches the target point, and the approaching process is as shown in Figure 4 To verify the effectiveness of the algorithm BATD3 proposed by the embodiment of the application, comparative experiments and ablation experiments are performed in three different workspaces. Among them, the anchor space is a cuboid space containing all expert trajectory target points, representing large space debris; the floating space is the union of the 0.1m neighborhood of all expert trajectory target points, representing the possible space where small space debris may float; and the sputtering space is the union of the 0.15m neighborhood of all expert trajectory target points, representing the possible space where small space debris may be sputtered when operating large space debris.

[0070] (1) The training process of the comparative experiment is as shown in Figure 5 The verification result is shown in Table 1. Whether in the test stage during the training process or in the verification process, BATD3 only applies the RL model when interacting with the environment, without using the BC model. Among them, the method proposed by the embodiment of the application is BATD3, the baseline method for comparison is TD3, BC, TD3wD (TD3 with Demonstrations, TD3 algorithm with expert trajectories), RBL (TD3 regularized with the BC loss function, TD3 algorithm regularized with the BC loss function), TD3+HER (TD3 algorithm combined with HER), RLPD (Reinforcement Learning with Prior Data, reinforcement learning algorithm with prior data) and DRLDR. From Figure 5 It can be seen that the success rate and training speed of BATD3 in the training stage are higher than those of the remaining baseline algorithms, which shows the superior performance of BATD3 in multiple aspects; from Table 1, it can be seen that BATD3 has a higher success rate and a lower episode length in the verification stage, indicating that the verification effect is consistent with the training effect, and BATD3 can complete the task in a shorter time.

[0071] Table 1 Comparison of the results of the model verification experiment of the algorithm of the embodiment of the application and the baseline algorithm

[0072] (2) The training process of the ablation experiment is shown in Figure 6 The pre-filled high-quality data, the auxiliary role of BC and the sample efficiency improvement role of RED are removed respectively, and the effects of the three design items are viewed. As shown in Figure 6 , after removing the pre-filled data, the success rate is reduced to 0, proving its decisive role in the sparse reward environment; after removing BC or RED, the convergence speed and success rate of the algorithm are significantly reduced, proving that the auxiliary effect of BC indeed enhances the learning effect of RL, and RED also improves the sample utilization efficiency. The experiment proves the necessity of each module and its unique value in the overall architecture.

[0073] In summary, the main advantages of the application are as follows: (1) The application uses behavior cloning to assist the reinforcement learning algorithm, which significantly improves its effect in the sparse reward environment; (2) The application avoids the complex dense reward function shaping, reduces the implementation difficulty of the method, and increases its generalizability; (3) The application uses the behavior cloning model to expand the pre-filled high-quality data, so that the method can still run efficiently in the space environment with less expert data; (4) The application combines random batch distillation with the reinforcement learning basic framework to improve the sample efficiency, thereby further improving the training effect; (5) The application builds a simulation environment of a space continuous rope-driven arm on the MuJoCo platform, realizes the characteristics of independent space continuous rope-driven arm segments, and uses the tendon module to drive joint movement, which can simulate the rope with high precision.

[0074] Another preferred embodiment of the application discloses a computer readable storage medium, which stores a computer program, wherein the computer program is arranged to be run by a processor to execute the control method of the space continuous rope-driven arm in the above-mentioned preferred embodiment.

[0075] Optionally, the above-mentioned storage medium can include but is not limited to: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk and various storage media that can store computer programs.

[0076] The background section of this document can contain information about the problem or environment of the invention not described by other people as prior art. Therefore, the content contained in the background section is not an admission that the prior art is prior art to the present invention.

[0077] The above further describes the present application in connection with specific / preferred embodiments but is not to be taken as a restriction on the overall scope of the application. Any such changes and modifications that come within the meaning and range of equivalents of the claims are to be embraced within the notion of the application. In the description of the specification, the terms "one embodiment", "some embodiments", "preferred embodiment", "an example", "a specific example", or "some examples" and the like mean that a particular feature, structure, material or characteristic is included in at least one embodiment or example of the present application. The illustrative examples of the specification are not meant to be limiting to a same embodiment or example; but are meant to be exemplary embodiments or examples. Furthermore, such terms can comprise a singular, a plural, or both, unless specifically stated otherwise. Also, the terms "comprises", "comprising", "includes", "including", "contains", "containing" or variations thereof do not specify an exhaustive or complete list of elements or steps as they can pertain to a specific embodiment or example. Further, no element or step can be present in one or more embodiments or examples of the application, not expressly described herein. One skilled in the art will further understand that terms such as "front", "back", "top", "bottom", "over", "under" and the like are used for description only and do not indicate relative position of the elements or steps.

Claims

1. A control method for a spatial continuous rope-driven arm, characterized in that: The following steps are involved: S1: Building a simulation environment for the spatial continuous rope-driven arm; S2: constructing a reinforcement learning framework for the spatial continuous rope-driven arm, and initializing the experience replay pool and reinforcement learning model of the reinforcement learning framework; S3: Acquire multiple expert trajectories and train a behavior cloning model based on the multiple expert trajectories; S4: Pre-filling the experience replay pool of the reinforcement learning framework with high-quality data; S5: Using the reinforcement learning model to interact with the simulation environment online, and storing the generated interaction data in the experience replay pool of the reinforcement learning framework; S6: Extracting training data from the experience replay pool of the reinforcement learning framework, updating the reinforcement learning model with the assistance of the behavior cloning model, and after the updating is completed, using the updated reinforcement learning model to control the spatial continuous rope-driven arm.

2. The control method of the spatial continuous rope-driven arm according to claim 1, characterized in that: The simulation environment in S1 is built based on the MuJoCo platform, wherein the spatial continuous rope-driven arm includes a floating base and at least two arm segments, and each arm segment is controlled to deform by multiple groups of drive units with adjustable tension, and the drive unit realizes the bending deformation of the arm segment by continuously adjusting the tension value.

3. The control method of the spatial continuous rope-driven arm according to claim 1, characterized in that: S3 specifically includes: obtaining multiple expert trajectories, and training a behavior cloning model according to the multiple expert trajectories based on a supervised learning approach.

4. The control method of the spatial continuous rope-driven arm according to claim 1, characterized in that: The high-quality data in S4 includes the multiple expert trajectories and successful trajectories obtained by the interaction between the behavior cloning model and the simulation environment.

5. The control method of the spatial continuous rope-driven arm according to claim 1, characterized in that: Updating the reinforcement learning model with the assistance of the behavior cloning model in S6 specifically includes: generating an auxiliary action through the behavior cloning model, combining the auxiliary action with the action generated by the reinforcement learning model to construct a temporal difference target, and updating the reinforcement learning model based on the temporal difference target.

6. The control method of the spatial continuous rope-driven arm according to claim 5, characterized in that: Combining the auxiliary action with the action generated by the reinforcement learning model to construct a temporal difference target includes: Using a soft maximum function to evaluate the value of the auxiliary action and the action generated by the reinforcement learning model; The final action selection is determined based on probability distribution sampling to construct the temporal difference target.

7. The control method of the spatial continuous rope-driven arm according to claim 5, characterized in that: Updating the reinforcement learning model based on the temporal difference objective includes: The network parameters of the reinforcement learning model are updated by the gradient descent loss function, where the loss function is expressed as: Where, It is a value network The loss function is updated respectively. A value network, It is a strategic network The loss function of is the temporal difference target, For the experience replay pool; yes t The state of the time-space continuous rope-driven arm, yes t The movement of the time-space continuous rope-driven arm, yes t Reward of the moment.

8. The control method of a spatial continuous rope-driven arm according to claim 1, characterized in that: In S6, during the process of updating the reinforcement learning model with the assistance of the behavior cloning model, a random batch distillation method is used to simultaneously copy multiple value networks, and two value networks are randomly selected as the value networks used for each update.

9. The control method of a spatial continuous rope-driven arm according to claim 1, characterized in that: The reinforcement learning model adopts a dual-delay deep deterministic policy gradient algorithm.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program is configured to be executed by a processor to perform the control method of the spatial continuous rope-driven arm according to any one of claims 1 to 9.

Images