Soft robot control method and system based on DSAC-T algorithm

Abstract

The application discloses a soft robot control method and system based on a DSAC-T algorithm, and comprises the following steps: performing mechanical modeling on a soft robot, driving the soft robot to bend and twist by equidistantly arranging torque control points on the soft robot, dynamically generating a target ball trajectory, and collecting state space data of the soft robot in real time; based on a DSAC-T reinforcement learning algorithm, a soft robot control model is constructed, the state space of the soft robot is taken as input, the action space of the soft robot is taken as output, a reward function is established, the soft robot control model is trained by continuously interacting with the environment, the trained network parameters are applied to the soft robot control model, and target tracking control of the soft robot is completed; and a system for implementing the method is designed correspondingly. The method and system designed by the application have high stability, strong robustness and good generalization ability, and can realize efficient control of the soft robot in complex tasks.

Description

Technical Field

[0001] This invention relates to the field of robot control, specifically to a soft robot control method based on the DSAC-T algorithm, and also to a soft robot control system based on the DSAC-T algorithm. Background Technology

[0002] With the development of soft robotics technology, its applications in fields such as medicine, service, and exploration are becoming increasingly widespread. Compared to traditional rigid robots, soft robots offer greater flexibility, safety, and environmental adaptability. However, soft robots typically possess infinite degrees of freedom, nonlinear dynamics, and strong coupling characteristics, making it difficult to establish accurate mathematical models and posing significant challenges to control.

[0003] Currently, model-based control methods (such as PID, LQR, and MPC) have limited performance in soft robot control and struggle to handle complex tasks. In recent years, reinforcement learning (RL) has become a research hotspot in soft robot control due to its advantages such as not requiring prior models and strong adaptability. However, traditional RL algorithms (such as DDPG, PPO, and SAC) still have shortcomings in training stability, convergence speed, and reward sensitivity, making it difficult to meet the high-precision control requirements of soft robots. Therefore, there is an urgent need for a control method with high stability, strong robustness, and good generalization ability to achieve efficient control of soft robots in complex tasks. Summary of the Invention

[0004] The purpose of this invention is to provide a soft robot control method and system based on the DSAC-T algorithm, aiming to solve the following problems: improving the training stability of reinforcement learning algorithms in soft robot control; reducing the sensitivity to the scale of the reward function; improving the policy convergence speed and control accuracy; and realizing target tracking and posture control of soft robots in three-dimensional space.

[0005] To achieve the above functions, this invention designs a soft robot control method based on the DSAC-T algorithm. For a soft robot composed of multiple rod-shaped equidistant joints, the following steps S1-S4 are executed to complete the motion control of the soft robot:

[0006] Step S1: Perform mechanical modeling on the soft robot, and set multiple control points for force and torque on the soft robot. The control points are located at the head end of the soft robot base and the tip of the soft robot arm. Other positions are set at equal intervals according to the number of control points, so as to drive the soft robot to bend and twist in three-dimensional space.

[0007] Step S2: Generate a three-dimensional target ball. Use the target ball as the target for the soft robot to track its movements. Dynamically generate the trajectory of the target ball. The target ball has a fixed starting position and speed. It runs along a random direction at a fixed speed and changes direction at preset time intervals.

[0008] Step S3: Real-time acquisition of the soft robot's state, construction of the soft robot's state space, and tracking of the target ball's motion; Based on the DSAC-T algorithm, construction of a soft robot control model, using the soft robot's state space as input and the soft robot's motion space as output, and establishing a reward function to train the soft robot control model;

[0009] Step S4: Apply the trained soft robot network parameters to the soft robot control model to complete the motion control of the soft robot.

[0010] As a preferred technical solution of the present invention: In step S1, the soft robot is composed of 11 rod-shaped equidistant joints and 6 control points are set, of which two control points are located at the head and end of the soft robot respectively, and the other four are equidistantly distributed in the middle of the soft robot; each control point is only allowed to be driven and apply torque on the normal and subnormal planes, generating a total of 12 degrees of freedom.

[0011] As a preferred technical solution of the present invention: the dynamic generation of the trajectory of the target ball in step S2 includes setting the initial position, moving speed, change of direction time, change of direction angle, total simulation time, and time step.

[0012] As a preferred technical solution of the present invention, step S3 is as follows:

[0013] Step S3.1: Real-time acquisition of the soft robot's state, construction of the soft robot's state space, the soft robot's state includes: the spatial coordinates of each equidistant segment, the velocity and direction of the end point, the target position, and the target velocity and direction;

[0014] Step S3.2: Based on the DSAC-T algorithm, construct a soft robot control model, including:

[0015] The actor network is an MLP network with two fully connected hidden layers. It uses the input state and output action as Gaussian distribution parameters to generate a random policy. The output action includes the torques of the six control points of the soft robot along the normal and the binormal, respectively.

[0016] The dual-criterion network is structured as two independent MLP networks with two fully connected hidden layers. The inputs are the state and the action, and the two MLP networks output value distributions, each including the mean. and standard deviation ;

[0017] The target network has the same structure as the dual-commenter network, and a soft update rate is set to track the parameters of the dual-commenter network through soft updates.

[0018] Step S3.3: Collect experience samples and store them in the experience replay buffer. The experience samples include the state space, the executed actions, and the reward value.

[0019] Step S3.4: Update the dual-critic network and generate the target Q-value based on the next state and the target policy.

[0020] ;

[0021] in, As a discount factor, For temperature coefficient, Indicates the reward value; Indicates the target Q value. Represents the mean of the value distribution. Indicates the next state. Indicates the next action. Represents the policy distribution. Indicates the state Take action below The probability of;

[0022] For the two value distributions output by the dual-critic network, select the value distribution with the smaller mean and calculate its KL divergence loss with the target distribution; then, based on the standard deviation of the value distribution... , use 3 Rules dynamically set gradient clipping boundaries;

[0023] Step S3.5: Update the actor network according to the following formula:

[0024] ;

[0025] In the formula, Represents the mathematical expectation. Indicates the temperature coefficient; Indicates the current state. This represents the current action; the policy parameters are updated through gradient ascent, with the goal of maximizing the entropy increase Q value. , These are the means of the two value distributions output by the dual-critic network. This indicates the state. Take action below The probability of;

[0026] Temperature coefficient Adjustments were made, setting the target entropy as the action dimension, and the temperature coefficient was optimized using gradient descent. This makes the actual strategy entropy close to the target entropy.

[0027] As a preferred embodiment of the present invention, the reward function established in step S3 is as follows:

[0028] ;

[0029] in, Represents the reward function, This represents the L2 norm distance between the tip of the soft robot arm and the target location, i.e. ,in This refers to the end effector position of the soft robot arm. For the target location; It is the preset reward range distance; Indicates a penalty item; Indicates the current action. Represents the sum of squares of actions. Let i represent action i, and 0.01 be the action smoothing penalty coefficient.

[0030] As a preferred embodiment of the present invention: when a non-numerical value is detected in the state of the soft robot, the penalty term in the reward function... The value is -1000.

[0031] As a preferred technical solution of the present invention: during the training process of the soft robot control model, the first 200 steps are random exploration to fill the experience replay buffer. After that, experience samples are collected at each step, and training is performed once every 7 steps. At the same time, the average reward is calculated every 10 rounds to evaluate the training progress.

[0032] This invention also designs a soft robot control system based on the DSAC-T algorithm, comprising a three-layer architecture: a simulation environment layer, an algorithm core layer, and a training support layer. Each layer interacts with other layers via code interfaces to implement the aforementioned soft robot control method based on the DSAC-T algorithm.

[0033] The simulation environment layer includes a soft robot modeling module, a 3D target generation module, and a state acquisition interface module, among which:

[0034] Soft robot modeling module: Performs mechanical modeling of the soft robot, sets multiple force and torque control points on the soft robot, the control point positions include the head end of the soft robot base, the end of the soft robot arm tip, and other positions are set at equal intervals according to the number of control points, driving the soft robot to bend and twist in three-dimensional space;

[0035] 3D target generation module: Generates a three-dimensional target ball and dynamically generates the target ball's trajectory;

[0036] State acquisition interface module: Real-time acquisition of the soft robot's state to construct the soft robot's state space;

[0037] The core layer of the algorithm takes the state space of the soft robot as input and the motion space of the soft robot as output. Based on the DSAC-T algorithm, a control model for the soft robot is constructed, taking the state space of the soft robot as input and the motion space of the soft robot as output.

[0038] The training support layer establishes a reward function to train the soft robot control model.

[0039] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0040] This invention combines the DSAC-T reinforcement learning algorithm with the Elastica compliant mechanics simulation environment to solve the problems of poor stability, reward sensitivity and insufficient accuracy of traditional algorithms in soft robot control. It can realize the development and verification of control strategies without relying on physical hardware.

[0041] This invention improves the training stability of reinforcement learning algorithms in soft robot control; reduces the sensitivity to the scale of the reward function; enhances policy convergence speed and control accuracy; and enables target tracking and attitude control of soft robots in three-dimensional space. Attached Figure Description

[0042] Figure 1 This is a decision-making process diagram of a soft robot control model provided according to an embodiment of the present invention;

[0043] Figure 2 This is a flowchart of the update and training process of a soft robot control model according to an embodiment of the present invention;

[0044] Figure 3 This is a simulation diagram of a training task provided according to an embodiment of the present invention;

[0045] Figure 4 This is a comparison chart of round rewards for different RL algorithms provided in an embodiment of the present invention;

[0046] Figure 5 This is a bar chart comparing performance metrics and analyzing training stability according to embodiments of the present invention. Detailed Implementation

[0047] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0048] The soft robot control method based on the DSAC-T algorithm provided in this embodiment of the invention, for a soft robot composed of multiple rod-shaped equidistant joints, executes the following steps S1-S4 to complete the motion control of the soft robot:

[0049] Step S1: Perform mechanical modeling on the soft robot, and set multiple force and torque control points on the soft robot. The control points are located at the head end of the soft robot base and the tip of the soft robot arm. Other positions are set at equal intervals according to the number of control points, driving the soft robot to bend and twist in three-dimensional space. In this embodiment, the bending angle range of the soft robot is [-30°, 30°]. The specific physical parameters of the soft robot are shown in Table 1.

[0050] Table 1. Physical parameters of soft robots

[0051]

[0052] Based on the Elastica simulation platform, a soft robot is modeled according to the Cosserat link theory. The soft robot consists of 11 equidistant link segments and has 6 control points. Two control points are located at the head and end of the soft robot, respectively, and the other four are equidistantly distributed in the middle of the soft robot. Each control point is only allowed to drive and apply torque on the normal and subnormal planes, resulting in a total of 12 degrees of freedom.

[0053] Step S2: Generate a three-dimensional target ball. Use the target ball as the target for the soft robot to track its movements. Dynamically generate the trajectory of the target ball. The target ball has a fixed starting position and speed. It runs along a random direction at a fixed speed and changes direction at preset time intervals.

[0054] The dynamic generation of the target ball's trajectory includes setting the initial position, moving speed, change-of-direction time, change-of-direction angle, total simulation duration, and time step.

[0055] In this embodiment, the initial position coordinates of the soft robot are set to x=0.5m, y=0.5m, z=0.8m. It moves at a constant speed of 0.5m / s and changes direction randomly every 0.7 seconds (the direction change angle is ±20°). The total simulation time is 10 seconds and the time step is 0.2 milliseconds.

[0056] Step S3: Real-time acquisition of the soft robot's state, construction of the soft robot's state space, and construction of a soft robot control model based on the DSAC-T (Distributional Soft Actor-Critic with Target entropy auto-tuning) algorithm. The soft robot's state space is used as input, the soft robot's motion space is used as output, and a reward function is established to train the soft robot control model.

[0057] Reference Figure 1 , Figure 2 The specific steps of step S3 are as follows:

[0058] Step S3.1: Real-time acquisition of the soft robot's state to construct its state space. The soft robot's state includes: spatial coordinates of 11 equidistant segments. (33-dimensional) is used to sample the shape information of the soft robot, as well as the velocity and orientation of the end point. (4D), target location (3D), target velocity and direction (4D) Samples are taken every 1.4 milliseconds;

[0059] Step S3.2: Based on the DSAC-T algorithm, construct a soft robot control model, including:

[0060] The actor network is a two-layer fully connected MLP network that uses the input state and output action as Gaussian distribution parameters to generate a random policy. The output action has 12 dimensions, including the torques of the six control points of the soft robot along the normal. And the torque of the binormal line Torque control quantities with a total of 12 degrees of freedom;

[0061] The dual-critic network is structured as two independent MLP networks with two fully connected hidden layers. Inputting states and actions, each MLP network outputs a value distribution, preventing overestimation by a single network. Both value distributions include the mean. and standard deviation ;

[0062] The target network has the same structure as the dual-commentator network. A soft update rate is set to track the parameters of the dual-commentator network through soft updates. Soft updates can achieve slow tracking of the parameters of the main network, ensuring training stability.

[0063] Step S3.3: Collect experience samples and store them in the experience replay buffer; randomly sample a batch of experience samples from the experience replay buffer for subsequent network updates, break the temporal correlation of samples, and realize the interaction between the agent and the environment. The experience samples include the state space, the executed actions, and the reward value.

[0064] Step S3.4: Update the dual-critic network and generate the target Q-value based on the next state and the target policy.

[0065] ;

[0066] in, As a discount factor, For temperature coefficient, Indicates the reward value; Indicates the target Q value. Represents the mean of the value distribution. Indicates the next state. Indicates the next action. Represents the policy distribution. Indicates the state Take action below The probability of;

[0067] For the two value distributions output by the dual-critic network, the distribution with the smaller mean is selected, and its KL divergence loss with the target distribution is calculated to suppress value overestimation; and the standard deviation of the value distribution is used as a basis for further analysis. , use 3 The rules dynamically set gradient clipping boundaries to prevent gradient explosion and vanishing.

[0068] Step S3.5: Update the actor network according to the following formula:

[0069] ;

[0070] In the formula, Represents the mathematical expectation. Indicates the temperature coefficient; Indicates the current state. This represents the current action; the policy parameters are updated through gradient ascent, with the goal of maximizing the entropy increase Q value. , These are the means of the two value distributions output by the dual-critic network. This indicates the state. Take action below The probability; the policy parameters are updated through gradient ascent, with the goal of maximizing the entropy increase Q value; the action is shifted towards high-value areas while retaining a certain degree of exploratory nature.

[0071] Temperature coefficient Adjust it, set the target entropy to the action dimension, and optimize the temperature coefficient through gradient descent , so that the actual policy entropy approaches the target entropy, so that the actual policy entropy approaches the target entropy, and automatically balance exploration (high ) and exploitation (low ).

[0072] The specific parameter settings of the DSAC-T algorithm are shown in Table 2:

[0073] Table 2. Parameter settings of the DSAC-T algorithm

[0074]

[0075] Construct a reward function, which aims to reward the distance between the tip of the soft robotic arm and the target position, as follows:

[0076] ;

[0077] where represents the L2 norm distance between the tip of the soft robotic arm and the target position, that is , where is the position of the end of the soft robotic arm, is the target position; is the preset reward range distance. In the embodiment, set = 5 cm; represents the penalty term. Since the allowed action space may cause the simulation to become unstable, whenever a non-numeric value (NaN) is detected in the state of the soft robot, it means that the simulation has become unstable, and the penalty term in the reward function takes a value of -1000, that is, detecting a non-numeric value will cause the episode to end; represents the current action, represents the sum of squares of the actions, represents action i, and 0.01 is the action smoothing penalty coefficient.

[0078] The above reward function contains a two-fold mechanism: action smoothing penalty is used to suppress violent actions; reward for reaching the standard. When the distance < d, in the embodiment, when the distance < 5 cm, the reward is +2.0 to accelerate convergence. During the training process of the soft robot control model, random exploration is performed in the first 200 steps to fill the experience replay buffer. After that, experience samples are collected in each step, and training and policy update are performed every 7 steps; at the same time, the average reward is calculated every 10 episodes. The average reward can evaluate whether the reward value is rising and converging as the training progresses, and is used to select hyperparameters, decide whether to stop training, and compare different models.

[0079] Such as Figure 3The diagram shown illustrates a simulation of the training task. In the training task, a small ball serves as the tracking target for the soft robot. The ball moves at a speed of 0.5 m / s and randomly changes direction every 0.7 s. The end effector of the soft robot actively follows the target's movement; the solid line represents the trajectory of the ball. Higher precision training is achieved by adjusting the ball's radius, thus improving its target tracking performance.

[0080] The parameters of the DSAC-T algorithm involved in the training task are defined in Table 3 below:

[0081] Table 3. Parameter Definition Table of DSAC-T Algorithm

[0082]

[0083] Step S4: Obtain the trained soft robot control model, and apply the soft robot control model to complete the motion control of the soft robot.

[0084] This invention also provides a soft robot control system based on the DSAC-T algorithm, comprising a three-layer architecture: a simulation environment layer, an algorithm core layer, and a training support layer. Each layer interacts with other layers via code interfaces to implement the aforementioned soft robot control method based on the DSAC-T algorithm.

[0085] The simulation environment layer includes a soft robot modeling module, a 3D target generation module, and a state acquisition interface module, among which:

[0086] Soft robot modeling module: Performs mechanical modeling of the soft robot, sets multiple force and torque control points on the soft robot, the control point positions include the head end of the soft robot base, the end of the soft robot arm tip, and other positions are set at equal intervals according to the number of control points, driving the soft robot to bend and twist in three-dimensional space;

[0087] 3D target generation module: Generates a three-dimensional target ball and dynamically generates the target ball's trajectory;

[0088] State acquisition interface module: Real-time acquisition of the soft robot's state to construct the soft robot's state space;

[0089] The core layer of the algorithm takes the state space of the soft robot as input and the motion space of the soft robot as output. Based on the DSAC-T algorithm, a control model for the soft robot is constructed, taking the state space of the soft robot as input and the motion space of the soft robot as output.

[0090] The training support layer establishes a reward function to train the soft robot control model.

[0091] like Figure 4 The image shows a comparison of round rewards for different RL algorithms. Figure 4The study compares three offline reinforcement learning algorithms—SAC, DDPG, and TD3—with the DSAC-T algorithm. DDPG, SAC, and TD3 are all well-known algorithms in the field, and their parameter settings follow the OpenAI Baselines standard. Figure 4 In the diagram, the X-axis represents the number of training epochs, ranging from 0 to 1000; the Y-axis represents the epoch reward, ranging from -5000 to 10000. Regarding convergence speed, DSAC-T (purple) converges quickly to the high reward range (close to 10000) within 200 epochs; DDPG (brown) converges slightly slower, stabilizing after about 300 epochs; SAC and TD3 require more than 400 epochs to gradually converge, with TD3 exhibiting negative rewards in the initial stages (0-100 epochs), showing the weakest initial performance. In terms of reward stability, DSAC-T exhibits minimal curve fluctuations after convergence, consistently maintaining a high reward range; DDPG, SAC, and TD3 show more pronounced curve oscillations (especially TD3's dramatic fluctuations in the early stages). This demonstrates that DSAC effectively suppresses training oscillations through mechanisms such as "value distribution modeling" and "gradient adaptive adjustment."

[0092] like Figure 5 As shown, the performance of four reinforcement learning algorithms, DDPG, DSAC, SAC, and TD3, is compared and analyzed from two dimensions: reward performance and training stability. Figure 5 The left side is a bar chart comparing performance metrics, with the vertical axis representing the reward value and the horizontal axis representing the algorithm. The DDPG algorithm has a final reward of 6548.78, a maximum reward of 12246.15, and an average reward of 4492.69. The DSAC algorithm has a final reward of 8915.98, a maximum reward of 13163.54, and an average reward of 6319.04. The SAC algorithm has a final reward of 1617.39 (the lowest among the four algorithms), a maximum reward of 13448.02 (the highest among the four algorithms), and an average reward of 4888.92. TD3 has a final reward of 5126.53, a maximum reward of 13081.85, and an average reward of 4302.76. DSAC leads in both "final reward" and "average reward," exhibiting the best overall performance; SAC has a high "maximum reward" but a very low "final reward," indicating significant policy degradation in the later stages of training.

[0093] Figure 5 The right side is a bar chart of training stability analysis. The vertical axis represents the coefficient of variation (the ratio of standard deviation to mean; the smaller the value, the less training fluctuation and the stronger the stability). The horizontal axis represents the algorithm. Figure 5 It can be seen that DSAC-T has better training stability than other algorithms; TD3 has the largest fluctuation and is the least stable during training.

[0094] In summary, DSAC-T performs best in both reward performance and training stability, making it the preferred choice among the four algorithms. While SAC has the potential for the "highest reward," its final performance and stability are insufficient. DDPG and TD3 are significantly weaker than DSAC-T in overall performance.

[0095] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A soft robot control method based on the DSAC-T algorithm, characterized in that, For a soft robot composed of multiple rod-shaped equidistant joints, perform the following steps S1-S4 to complete the motion control of the soft robot: Step S1: Perform mechanical modeling on the soft robot, and set multiple control points for force and torque on the soft robot. The control points are located at the head end of the soft robot base and the tip of the soft robot arm. Other positions are set at equal intervals according to the number of control points, so as to drive the soft robot to bend and twist in three-dimensional space. Step S2: Generate a three-dimensional target ball. Use the target ball as the target for the soft robot to track its movements. Dynamically generate the trajectory of the target ball. The target ball has a fixed starting position and speed. It runs along a random direction at a fixed speed and changes direction at preset time intervals. Step S3: Real-time acquisition of the soft robot's state, construction of the soft robot's state space, and tracking of the target ball's motion; Based on the DSAC-T algorithm, construction of a soft robot control model, using the soft robot's state space as input and the soft robot's motion space as output, and establishing a reward function to train the soft robot control model; The specific steps of step S3 are as follows: Step S3.1: Real-time acquisition of the soft robot's state, construction of the soft robot's state space, the soft robot's state includes: the spatial coordinates of each equidistant segment, the velocity and direction of the end point, the target position, and the target velocity and direction; Step S3.2: Based on the DSAC-T algorithm, construct a soft robot control model, including: The actor network is an MLP network with two fully connected hidden layers. It uses the input state and output action as Gaussian distribution parameters to generate a random policy. The output action includes the torques of the six control points of the soft robot along the normal and the binormal, respectively. The dual-criterion network is structured as two independent MLP networks with two fully connected hidden layers. The inputs are the state and the action, and the two MLP networks output value distributions, each including the mean. and standard deviation ; The target network has the same structure as the dual-commenter network, and a soft update rate is set to track the parameters of the dual-commenter network through soft updates. Step S3.3: Collect experience samples and store them in the experience replay buffer. The experience samples include the state space, the executed actions, and the reward value. Step S3.4: Update the dual-critic network and generate the target Q-value based on the next state and the target policy. ； in, As a discount factor, For temperature coefficient, Indicates the reward value; Indicates the target Q value. Represents the mean of the value distribution. Indicates the next state. Indicates the next action. Represents the policy distribution. Indicates the state Take action below The probability of; For the two value distributions output by the dual-critic network, select the value distribution with the smaller mean and calculate its KL divergence loss with the target distribution; then, based on the standard deviation of the value distribution... , use 3 Rules dynamically set gradient clipping boundaries; Step S3.5: Update the actor network according to the following formula: ； In the formula, Represents the mathematical expectation. Indicates the temperature coefficient; Indicates the current state. This represents the current action; the policy parameters are updated through gradient ascent, with the goal of maximizing the entropy increase Q value. , These are the means of the two value distributions output by the dual-critic network. This indicates the state. Take action below The probability of; Temperature coefficient Adjustments were made, setting the target entropy as the action dimension, and the temperature coefficient was optimized using gradient descent. This makes the actual strategy entropy close to the target entropy; The reward function established in step S3 is as follows: ； in, Represents the reward function, This represents the L2 norm distance between the tip of the soft robot arm and the target location, i.e. ,in This refers to the end effector position of the soft robot arm. For the target location; It is the preset reward range distance; Indicates a penalty item; Indicates the current action. Represents the sum of squares of actions. Let i represent action i, and 0.01 be the action smoothing penalty coefficient; Step S4: Apply the trained soft robot network parameters to the soft robot control model to complete the motion control of the soft robot.

2. The soft robot control method based on the DSAC-T algorithm according to claim 1, characterized in that, In step S1, the soft robot consists of 11 rod-shaped equidistant joints and has 6 control points. Two of the control points are located at the head and end of the soft robot, respectively, and the other four are equidistantly distributed in the middle of the soft robot. Each control point is only allowed to drive and apply torque on the normal and subnormal planes, resulting in a total of 12 degrees of freedom.

3. The soft robot control method based on the DSAC-T algorithm according to claim 1, characterized in that, Step S2 involves dynamically generating the trajectory of the target ball, including setting the initial position, moving speed, change-of-direction time, change-of-direction angle, total simulation duration, and time step.

4. The soft robot control method based on the DSAC-T algorithm according to claim 1, characterized in that, When a non-numeric value is detected in the state of the soft robot, the penalty term in the reward function... The value is -1000.

5. The soft robot control method based on the DSAC-T algorithm according to claim 1, characterized in that, During the training of the soft robot control model, the first 200 steps are random exploration to fill the experience replay buffer. After that, experience samples are collected at each step, and training is performed once every 7 steps. At the same time, the average reward is calculated every 10 rounds to evaluate the training progress.

6. A soft robot control system based on the DSAC-T algorithm, characterized in that, The system comprises a three-layer architecture: a simulation environment layer, an algorithm core layer, and a training support layer. Each layer interacts with the other through code interfaces to implement the soft robot control method based on the DSAC-T algorithm as described in any one of claims 1-5. The simulation environment layer includes a soft robot modeling module, a 3D target generation module, and a state acquisition interface module, among which: Soft robot modeling module: Performs mechanical modeling of the soft robot, sets multiple force and torque control points on the soft robot, the control point positions include the head end of the soft robot base, the end of the soft robot arm tip, and other positions are set at equal intervals according to the number of control points, driving the soft robot to bend and twist in three-dimensional space; 3D target generation module: Generates a three-dimensional target ball and dynamically generates the target ball's trajectory; State acquisition interface module: Real-time acquisition of the soft robot's state to construct the soft robot's state space; The core layer of the algorithm takes the state space of the soft robot as input and the motion space of the soft robot as output. Based on the DSAC-T algorithm, a control model for the soft robot is constructed, taking the state space of the soft robot as input and the motion space of the soft robot as output. The training support layer establishes a reward function to train the soft robot control model.

Images