A deformable object robot manipulation method based on demonstration enhanced deep deterministic policy gradient

Abstract

The application discloses a deformable object robot operation method based on demonstration enhanced deep deterministic policy gradient, and belongs to the field of deformable object robot operation. The method adopts the combination of feature points and grabbing points of deformable objects in demonstration data sets to train a high-dimensional fuzzy system model, and further utilizes the output grabbing point information of the high-dimensional fuzzy system model to guide the training of a reinforcement learning model after the training is completed, so that the placing point can be quickly and accurately found, and the efficiency of the whole training process is remarkably improved. In addition, the application allows flexible grabbing operation around the grabbing feature point determined by the high-dimensional fuzzy system, so that the output of the reinforcement learning includes the offset vector from the grabbing feature point to the actual grabbing point in addition to the placing point, and the flexibility and adaptability of the operation are further enhanced.

Description

Technical Field

[0001] This invention belongs to the field of robot manipulation of deformable objects, and more specifically, relates to a robot manipulation method for deformable objects based on a demonstration-enhanced depth deterministic policy gradient. Background Technology

[0002] The manipulation of deformable objects (i.e., flexible objects) is involved in many important industries and fields related to economic development and improving people's quality of life. In the industrial sector, the assembly and processing of deformable objects in industries such as food processing, textile manufacturing, and packaging largely require manual handling for rapid and accurate processing, which is time-consuming and labor-intensive. The introduction of automated operating systems will save labor costs, improve production efficiency, and increase assembly accuracy. In the field of medical surgery, the manipulation of deformable objects such as soft tissues and organs by surgical robots, such as shape control, needle insertion, cutting, and suturing, is an important step in surgical procedures. However, most current surgical systems have a low degree of automation, such as the da Vinci system. Robotic surgical systems employ remote operation by the surgeon from a control console. The STAR autonomous soft tissue surgical robot requires manual adjustment for 40% of the surgical time during suturing. Further improvements in surgical robot automation technology can reduce the adverse effects of human factors such as hand tremors, overcome physiological limitations such as human eye resolution, reduce surgical time, improve surgical quality, and lower infection and bleeding rates. In the field of service robots, the automated manipulation technology of deformable objects also has important applications. Service robots autonomously perform tasks such as food cutting, clothing folding, and window cleaning, promoting standardized service practices and improving service quality and efficiency. Therefore, the automated manipulation of deformable objects has attracted widespread attention and research. Among these, the manipulation of fabrics, as a fundamental operation and key step, has become a research topic for scholars both domestically and internationally due to its application value in home assistance, industrial fabric cutting, and medical care.

[0003] Most state-of-the-art work in robotic manipulation focuses on handling rigid objects that do not deform or deform negligibly when gripped. However, robots face many new challenges when dealing with deformable objects, including high-dimensional state representation; accurate modeling of object behavior; and complex dynamics, physical properties, and geometric features (viscoelasticity, nonlinearity, high-dimensional deformation, anisotropy, etc.).

[0004] Taking fabric as an example, current research on fabric manipulation mainly falls into two categories: model-based and data-driven. Model-based methods have many limitations and shortcomings in areas such as fabric deformation calculation and robotic arm motion control. For example, the contradiction between the accuracy and real-time performance of the object deformation model leads to slow updates of high-precision models, and the system does not consider the robustness and stability issues of model errors. Data-driven methods mainly utilize imitation learning, reinforcement learning, and supervised learning to train agents. Their limitations include the need for a large database and long training time, and low generalization and accuracy for different objects and physical parameters.

[0005] In summary, overcoming the shortcomings of model- and data-driven approaches to control robots for effective, robust, and highly generalizable manipulation of deformable objects has always been a major challenge in robotics. Summary of the Invention

[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a deformable robot manipulation method based on demonstration-enhanced deep deterministic policy gradient. Within the DDPG framework, through more scientific problem modeling, demonstration acquisition, training methods, and higher efficiency in utilizing demonstration data, a reliable and effective deformable robot manipulation strategy can be quickly obtained.

[0007] To achieve the above objectives, according to a first aspect of the present invention, a method for manipulating a deformable robot based on a demonstration of enhanced depth deterministic policy gradients is provided, comprising:

[0008] Training phase:

[0009] Obtain demonstration data for the grasping task, which includes feature points of the deformable object, grasping points, placement points, and offset vectors; wherein, the offset vector is the vector from the grasping point to the actual grasping point;

[0010] The HTSK model is trained under supervision using the aforementioned feature points as samples and the grasp points as labels.

[0011] Using the feature points and grasping points as states, and the offset vector and placement points as actions, the DDPG model is trained using the demonstration data.

[0012] Application phase:

[0013] The feature points of the deformable object to be manipulated are input into the trained HTSK model to obtain its grasping points.

[0014] The feature points and grasping points of the deformable object to be manipulated are input into the current Actor network of the trained DDPG model to obtain its placement point and offset vector.

[0015] The robot's motion trajectory is obtained based on the grasping point, placement point, and offset vector of the deformable object to be operated, so that the robot can operate on the deformable object.

[0016] According to a second aspect of the present invention, a deformable object robot operating system based on a demonstration of enhanced depth deterministic policy gradient is provided, comprising: a computer-readable storage medium and a processor;

[0017] The computer-readable storage medium is used to store executable instructions;

[0018] The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in the first aspect.

[0019] According to a third aspect of the invention, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores computer instructions for causing a processor to perform the method as described in the first aspect.

[0020] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0021] 1. The method provided by this invention simplifies and decomposes the manipulating task of a deformable robot and uses conditional policy learning. Then, it combines imitation learning within the framework of Deep Deterministic Policy Gradient (DDPG) and fully utilizes demonstration data to accelerate the training of the entire system, thereby quickly obtaining a reliable and effective manipulating strategy for the deformable robot. Specifically, this invention uses a combination of feature points and grasping points of deformable objects in the demonstration dataset to train the HTSK model. After training, it further utilizes the output grasping point information to guide the training of the reinforcement learning model, so as to quickly and accurately find the placement point, thereby significantly improving the efficiency of the entire training process. In addition, this invention allows flexible grasping operations around the grasping feature points determined by the High-dimensional Takagi-Sugeuo-Kang (HTSK) system. Therefore, the output of the reinforcement learning includes not only the placement point but also the offset vector from the grasping feature point to the actual grasping point, further enhancing the flexibility and adaptability of the operation.

[0022] 2. As a further preferred embodiment of the present invention, the present invention introduces a behavioral cloning loss term L into the loss function of the current actor network of DDPG. bc It can use demonstration data to initialize the current Actor network, thus accelerating the convergence speed.

[0023] 3. As a further preferred embodiment of the present invention, the present invention adds a new loss term Q to the current Critic's loss function. diff This allows the current Critic network to assign higher Q values ​​to demonstration actions and lower Q values ​​to actions output by the placement policy μ, thereby accelerating the training of the current Critic. In addition, thanks to the more reasonable Q values ​​output by the current Critic, the utilization rate of human demonstrations in behavior cloning will also increase, further improving the training efficiency of the current Actor.

[0024] 4. As a further preferred embodiment of the present invention, this invention employs nonlinear model predictive control (NMPC) to collect demonstration data in a simulation environment. A deformable object model based on spring-mass points is constructed, with each mass point connected by a spring, simulating the physical characteristics of the deformable object. Then, using the position of each mass point in the model as the state variable and the force applied to each mass point as the control variable, an NMPC problem is constructed and transformed into a nonlinear rolling optimization problem. By solving this nonlinear problem, a series of control variables are obtained, which are then transformed into specific actions that the robotic arm can recognize and execute. Finally, the robotic arm performs tasks such as grasping and moving the deformable object according to these actions, achieving highly precise and flexible manipulation of the deformable object. Furthermore, considering that although NMPC can theoretically complete deformable object manipulation tasks, it requires a detailed description of the state of each mass point as a state vector, but in practical applications, accurate tracking of each mass point is a highly challenging task. Therefore, this invention establishes a deformable object model based on spring-mass points to collect demonstration data using NMPC in a simulation environment, and uses this data to train a deep reinforcement learning model. In this method, the input to deep reinforcement learning is processed visual features, which can be understood as a simplified version of the state vector of NMPC, rather than direct point states. This makes the algorithm more efficient and practical when dealing with real-world scenarios. Specifically, this invention establishes a more realistic finite element model of a deformable object in the Pybullet simulation environment for the robotic arm to collect interaction data in the simulation environment, thereby facilitating the training of the reinforcement learning agent in the simulation. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a spring-mass model established for an embodiment of the present invention.

[0026] Figure 2 This is a schematic diagram illustrating NMPC-based data acquisition as provided in an embodiment of the present invention.

[0027] Figure 3The diagram illustrates three types of tasks provided in the embodiments of the present invention.

[0028] Figure 4 This is a schematic diagram of conditional strategy learning provided in an embodiment of the present invention.

[0029] Figure 5 This diagram illustrates a comparison of the rewards obtained by the method provided in this embodiment of the invention with those obtained by various algorithms in performing tasks under different experimental scenarios. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0031] This invention provides a method for manipulating a deformable robot based on a demonstration of enhanced depth deterministic policy gradients, comprising:

[0032] Training phase:

[0033] Obtain demonstration data for the grasping task, which includes feature points of the deformable object, grasping points, placement points, and offset vectors; wherein, the offset vector is the vector from the grasping point to the actual grasping point;

[0034] The HTSK model is trained under supervision using the feature points as samples and the grab points as labels to obtain a trained HTSK model.

[0035] Using the feature points and grasping points as states, and the offset vector and placement points as actions, the DDPG model is trained using the demonstration data to obtain a trained DDPG model.

[0036] Application phase:

[0037] The feature points of the deformable object to be manipulated are input into the trained HTSK model to obtain its grasping points.

[0038] The feature points and grasping points of the deformable object to be manipulated are input into the current Actor network of the trained DDPG model to obtain its placement point and offset vector.

[0039] The robot's motion trajectory is obtained based on the grasping point, placement point, and offset vector of the deformable object to be operated, so that the robot can operate on the deformable object.

[0040] It is understood that the deformable object can be any deformable object, such as fabric, rope, flexible tube, sponge, wire, etc. The method provided by the present invention will be described below using fabric as an example.

[0041] First, this invention proposes a conditional policy learning method based on HTSK, dividing the action space into two action primitives: grasping and placing, and then performing conditional policy learning. In this method, the output of the grasping policy H is used as the input to the placing policy μ, ultimately yielding the total policy π:

[0042]

[0043] This study found that in most cases, manipulating specific feature points of the fabric or their vicinity is sufficient to complete the intended task. Therefore, we consider the feature points of the fabric as candidate grasping points, and treat the determination of these grasping points as a classification problem, which is solved using the HTSK fuzzy classification model. This model uses the position s of the fabric feature points as the basis for classification. t The input is the optimal feature point ID to be grasped (i.e., the grasp point ID). Feature points differ across tasks, but are generally easy to specify; for example, feature points in a folding task are the endpoints of the fabric, while feature points in a flattening task are several points on the contour. This study uses a human demonstration dataset. This study trains an HTSK model using a combination of fabric state and gripping points, and then further utilizes the output gripping point information to guide the training of a reinforcement learning model after training. This enables the model to quickly and accurately find placement points, significantly improving the efficiency of the entire training process. Furthermore, this research allows for flexible gripping operations around the gripping feature points determined by HTSK; therefore, the reinforcement learning output includes not only the placement point but also the offset vector from the gripping feature points to the actual gripping point. This further enhances the flexibility and adaptability of the operation.

[0044] It is understood that the feature points, gripping points, and placement points in the demonstration data of this invention all refer to the locations of the feature points, gripping points, and placement points, that is, the location data of the feature points, gripping points, and placement points are used.

[0045] Secondly, this invention introduces a demo dataset (hereinafter referred to as demo) within the framework of DDPG. To initialize the policy, that is, to introduce a behavior cloning loss term L into the loss function of the current actor network of DDPG. bc Furthermore, pre-training is set up, and during the pre-training process, the policy μ is forced to learn from the demonstration.

[0046] Suppose Q is the current critic network of DDPG, and the action performed at time i is... This study will Let it be bi And set Since the quality of the offset vector and placement point is closely related to their corresponding gripping point, the action vector evaluated by the Q-factor of the critic network in this study includes not only... It also includes grab points, such as w i a i The loss function L of the current actor network μ in DDPG. re The following formula can be used to calculate:

[0047]

[0048]

[0049] Where, λ bc and λ Q These are the weighting coefficients of the two loss terms.

[0050] Given the original placement strategy μ of DDPG, the loss function -Q(s) i ,w i ;θ q ) is the output Q(s) of Critic i ,w i ;θ q This invention will demonstrate how to train the current Critic, and ultimately accelerate the training of the placement policy μ through an accurate current Critic. Specifically, based on the fact that the expected reward obtained from the demonstrated action is significantly greater than the action output by the placement policy μ at the beginning of training, during the pre-training phase, the current Critic is forced to give a higher Q value to the demonstrated action and a lower Q value to the action output by the placement policy μ. This invention adds a new loss term Q to the loss function of the current Critic. diff To realize this idea:

[0051]

[0052] Therefore, we can define the complete loss function for the critic as follows:

[0053] L critic =λ 1step L 1s +λ nstep L ns +λ diff Q diff (1.5)

[0054] Where, λ 1step ,λ nstep ,λ diff It is the full-weighted coefficient, L 1s ,L nsThese are the first-order and nth-order temporal difference error terms, respectively. Here, we define Q′ and μ′ as the target critic network and the target actor network w′, respectively. i+k =(H(s) i+k ;θ h ),μ′(s i+k ,H(s i+k ;θ h );θ μ k = 1, 2, ..., N, r i For the reward, γ is the discount factor, then L 1s ,L ns They can be defined as follows:

[0055] L 1s =(Q(s) i ,a i ;θ q )-(r i +γQ′(s i+1 ,w' i+1 ;θ q' ))) 2 (1.6)

[0056] L ns =(Q(s) i ,a i )-y ns ) 2 (1.7)

[0057]

[0058] The aforementioned demonstration dataset can be extracted from manual operations. Considering the efficiency of obtaining demonstration data and the labor cost, this invention preferably uses NMPC to obtain the demonstration data. That is, preferably, the grab points, placement points, and offset vectors in the demonstration data are obtained through nonlinear model prediction control, including:

[0059] A spring-mass model of the deformable object is established. Under constraints, the optimal control sequence of each mass at each moment within the prediction time range is obtained by minimizing the sum of the differences between the distance between any two masses in the spring-mass model at each moment in the corresponding state within the prediction time range and their distance at the target state. The control quantity in the optimal control sequence is the external force applied to the mass.

[0060] The particle corresponding to the maximum control quantity in the optimal control sequence is taken as the actual grasping point, the feature point closest to the actual grasping point is taken as the grasping point, and the point reached by the actual grasping point under the action of the maximum control quantity is taken as the placement point. The vector from the grasping point to the actual grasping point is the offset vector.

[0061] like Figure 1 As shown. The spring-mass model established in this invention consists of a point set including N mass points. and a set of springs containing M springs Composition. This paper defines the neighbor set of the i-th mass point, that is, the set of mass points connected to mass point i via a spring, as... And it is assumed that the neighbor set of each mass point is fixed, i.e. It is a set of constants. For a square piece of cloth, N = n 2 , where n is the number of particles on one edge of the cloth. The mass of each particle i is m. i It is usually assumed that all point masses have equal mass, that is...

[0062] In the model used in this paper, the particles are first subjected to the elastic force of a spring, which depends on the relative positions of the particles. For any two particles i and j connected by a spring, the elastic force at time k is... It can be represented as:

[0063]

[0064] in, Let be the actual distance between particles i and j at time t. Besides the elastic force, each particle is also subjected to gravity G = mg and external forces. and damping force The effect of damping force. The damping force is proportional to the particle velocity and can be expressed as:

[0065]

[0066] Where c is the damping coefficient. Let be the velocity vector of particle i at time t.

[0067] In summary, the net force acting on particle i at time t It can be represented as:

[0068]

[0069] According to Newton's second law, the acceleration of particle i at time t is... It can be represented as:

[0070]

[0071] Furthermore, we use the following formula to update the velocity of the mass:

[0072]

[0073] Then we put If we consider it as a function of time t, then the position of particle i at time t0+Δt is... This can be obtained by performing a Taylor expansion at t = t0:

[0074]

[0075] Through analysis, we can find that the higher-order term O(Δt) 3 The magnitude of ) is mainly related to the time step and acceleration. When our time step Δt is small enough and the damping is sufficient... When chosen appropriately, the influence of higher-order terms can be ignored. Therefore, we will use a smaller time step Δt and set the damping coefficient c to a large positive number to ensure the stability of the model. In the following text, we will simplify the subscript by writing t+n·Δt as t+n and use the following formula to update the position of the particle:

[0076]

[0077] In the study, the goal of NMPC control was to predict the time range H. p Find an optimal control sequence within. This minimizes the objective function J. The system state (i.e., the state of the particles) is the coordinates of each particle. The control quantity is the external force applied to each particle. in These are the components of the external force in the x, y, and z directions, respectively.

[0078] Loss function L(X) t The aim is to quantify the current state X. t With target state X ref The differences between them. In this study, considering that task completion can be characterized by the distance between feature points of the cloth, we carefully designed the loss function in the NMPC problem, defining it based on the distance between each pair of points. This method not only improves the efficiency of problem solving but also more intuitively reflects the impact of the cloth state on task completion. For a given task objective, the loss function can be defined as:

[0079]

[0080] in, It is the set of all point mass pairs that need to be considered. In target state X ref The distance between lower particles i and j.

[0081] In summary, the NMPC control proposed in this paper can be achieved by solving the following optimization problem:

[0082]

[0083] And satisfy the following constraints:

[0084] X k+1|t =f(X) k|t U k ), k=t,...,t+H p -1; X t|t =X t (1.18)

[0085]

[0086]

[0087] Among them, J(X) t U list ) is the entire prediction range H p The cumulative loss function within the system. f(·) represents the system dynamics model, that is, predicting the state of the fabric at the next moment through equations (1.9)-(1.15), i.e., X k|t and X k+1|t They represent the current state X respectively. t and a series of control variables U list The predicted states at times k and k+1. Specifically, when k = t, X is defined. t|t =X t . Indicates according to X k|t The distance between particles i and j at time k is calculated. Let represent the set of state constraints within the robot's workspace. By solving the above optimization problem, we can obtain the optimal control sequence that minimizes the loss function within the prediction range.

[0088] After collecting demonstration data through NMPC, this study aims to make full use of this data. This study first... The forces in the equation are applied to the corresponding particles to obtain their states. Then, find the given The mass i corresponding to the maximum control quantity g and distance i g Recent fabric feature point p g , will p g As the capture point at this moment, i g As the actual crawling point, i g to p g The vector is used as the offset vector. And Mid-range ig The position is used as the placement point p p That is, applying the maximum control quantity to the established spring-mass model, i g The position of the point reached under the action of the maximum control variable is taken as the position of the placement point, forming an action vector a. t This motion vector is then input into the robotic arm for actual operation.

[0089] It is understandable that, based on action vector a t The robot's motion trajectory is generated using existing technology. For example: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] t Input the data into Moveit! to obtain the robot's motion trajectory.

[0090] Figure 2 This demonstrates the experimental scenario built in the PyBullet simulation environment, which includes a UR5e robot, a highly realistic square piece of cloth, and a table. When using NMPC to collect data, the state of the cloth in the next moment is simulated by the robotic arm performing an operation via PyBullet. t The state of the fabric was obtained afterward. This design aims to reduce the computational burden on NMPC by employing a simplified spring-mass model, while leveraging the high simulation fidelity provided by PyBullet to compensate for errors that may be introduced by the simplified model, thus achieving a balance between computational efficiency and simulation realism.

[0091] like Figure 3 As shown, this invention has been tested for three tasks: folding fabric diagonally, folding fabric along the central axis, and flattening folded fabric. The robotic arm interacts with the fabric by setting an anchor point between itself and a certain point on the fabric.

[0092] like Figure 4 As shown, this invention employs conditional policy learning combined with HTSK. Specifically, the task is decomposed into two motion primitives: grasping and placing, resulting in two policies: a grasping policy H and a placing policy μ. This invention uses HTSK as the grasping policy H, and incorporates environmental observation information s t As input, crawl points As output. Since in most cases the task can be completed by operating on a limited number of feature points of the cloth or points near those feature points, the grab point... This can be simplified to the selection of gripping points, which is essentially a classification problem. We use demo data to train HTSK, enabling it to select appropriate gripping points based on environmental information, just like in the demo. This invention employs a neural network as the placement strategy μ, which processes the gripping points output by HTSK. and environmental observation information t The splicing is used as input to the placement strategy μ, and the placement point is... and capture offset vector As output, the invention will finally capture the points. and placement point and capturing offset vectors Merging as an action

[0093] In summary, the method provided by this invention, within the DDPG framework, employs conditional policy learning combined with HTSK and optimizes the model training process to fully utilize demonstration data to enhance the training effect of DDPG. Furthermore, the proposed use of NMPC to collect demonstration data can improve the efficiency of demonstration data acquisition and reduce labor costs.

[0094] like Figure 5 As shown, this invention compares the scores obtained by four algorithms on three tasks through simulation. The four algorithms are:

[0095] 1. The improved Rainbow-DDPG algorithm of this invention has not been incorporated.

[0096] 2. The RDI algorithm is trained and optimized using demonstrations but does not employ conditional learning strategies.

[0097] 3. Compare with RDH, an algorithm that combines HTSK for conditional policy learning but does not use demonstrations for training and optimization.

[0098] 4. The algorithm obtained by adding all the above improvements to the DDPG algorithm is called HDE-DDPG.

[0099] The three tasks are as follows: Figure 3 As shown: Fold the fabric diagonally, fold the fabric along the central axis, and flatten the folded fabric.

[0100] The results show that the algorithm trained using the demonstration optimization method achieves better scores on simpler tasks (folding fabric diagonally and folding fabric along the central axis), but still cannot solve more complex tasks (flattening wrinkled fabric). Combining conditional policy learning with HTSK yields better results on all tasks. Optimal results can be achieved by incorporating both improvements simultaneously.

[0101] This invention provides an operating system for a deformable robot based on a demonstration of enhanced depth deterministic policy gradients, comprising: a computer-readable storage medium and a processor;

[0102] The computer-readable storage medium is used to store executable instructions;

[0103] The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any of the above embodiments.

[0104] This invention provides a computer-readable storage medium storing computer instructions that cause a processor to perform the method described in any of the above embodiments.

[0105] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for manipulating deformable robot based on demonstration-enhanced depth deterministic policy gradient, characterized in that, include: Training phase: Obtain demonstration data for the grasping task, which includes feature points of the deformable object, grasping points, placement points, and offset vectors; wherein, the offset vector is the vector from the grasping point to the actual grasping point; The HTSK model is trained under supervision using the aforementioned feature points as samples and the grasp points as labels. Using the feature points and grasping points as states, and the offset vector and placement points as actions, the DDPG model is trained using the demonstration data. Application phase: The feature points of the deformable object to be manipulated are input into the trained HTSK model to obtain its grasping points. The feature points and grasping points of the deformable object to be manipulated are input into the current Actor network of the trained DDPG model to obtain its placement point and offset vector. The robot's motion trajectory is obtained based on the gripping point, placement point, and offset vector of the deformable object to be operated, so that the robot can operate on the deformable object to be operated. The grab points, placement points, and offset vectors in the demonstration data are obtained through nonlinear model predictive control, including: A spring-mass model of the deformable object is established. Under constraints, the optimal control sequence of each mass at each moment within the prediction time range is obtained by minimizing the sum of the differences between the distance between any two masses in the spring-mass model at each moment in the corresponding state within the prediction time range and their distance at the target state. The control quantity in the optimal control sequence is the external force applied to the mass. The particle corresponding to the maximum control quantity in the optimal control sequence is taken as the actual grasping point, the feature point closest to the actual grasping point is taken as the grasping point, and the point reached by the actual grasping point under the action of the maximum control quantity is taken as the placement point. The vector from the grasping point to the actual grasping point is the offset vector. The constraints include: Particle state constraints: ; External constraints: ; Robot workspace constraints: ; in, , , N For the number of particles, For the time range of the forecast; in particular, when k = t hour, , This represents a system dynamics model used to predict the state of a deformable object in the next moment: ;in, To preset the step size, It is the velocity of each particle. It is the spring force acting on each particle. It is the gravitational force acting on the point mass. It is the damping force acting on each particle. It is the mass of each point mass; and These represent the values ​​based on the current state of the particle. and control sequence The predicted first and The set of particle states at time t, , , , , They are external forces Components in direction; To determine the current state of the particle and a series of control quantities The predicted first Momentary point i state, This represents the set of state constraints within the robot's workspace.

2. The method as described in claim 1, characterized in that, The loss function of the current Actor network is: in, , and These are the weighting coefficients of the two loss terms. To demonstrate the dataset, , , , For the current Actor network with network parameters of Input is and The combination of the output offset vector and the placement point. For the current critic network with network parameters of At that time, in the state Next action The predicted total reward to be obtained in the future. For the current critic network with network parameters of At that time, in the state Next action The predicted total reward to be obtained in the future. The location of the feature points of the deformable object. i For a moment, For the capture point, For placement point, This is the offset vector.

3. The method as described in claim 1 or 2, characterized in that, The loss function of the current Critic in the DDPG model also includes a loss term. ; ,in, To demonstrate the dataset, , , For the current critic network with network parameters of At that time, in the state Next action The predicted total reward to be obtained in the future. For the current critic network with network parameters of At that time, in the state Next action The predicted total reward to be obtained in the future. For the current Actor network with network parameters of Input is and The combination of the output offset vector and the placement point. For placement point, This is the offset vector.

4. The method as described in claim 1, characterized in that, Set an objective function, and obtain the optimal control sequence for each particle at each time point within the prediction time range by minimizing the objective function; The objective function is: in, Indicates according to The calculated first Point mass at time and The distance between them Indicates the target state lower particle and The distance between them For the set of particles in the spring-mass model, These are the weighting coefficients.

5. An operating system for deformable robot based on demonstration-enhanced depth deterministic policy gradient, characterized in that, include: Computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a processor to perform the method as described in any one of claims 1-4.

Images