Sensing, decision-making and execution integrated intelligent control method and system for redundant rope-driven mechanical arm

Abstract

The invention discloses a sensing, decision-making and execution integrated intelligent control method and system for a redundant rope-driven mechanical arm, and relates to the technical field of robot motion planning. The method comprises the following steps: processing an environment point cloud to obtain a three-dimensional environment model; generating a plurality of groups of collision-free expert track sequences based on the three-dimensional environment model and a simplified rigid connecting rod model of the mechanical arm; training a motion control model based on an improved enhancement algorithm by adopting a three-stage learning strategy, wherein the first stage is based on barrier-free and static target environment training, the second stage is based on barrier-free and dynamic target environment training, and the third stage is based on random barrier and dynamic target environment training; a collision-free expert track sequence is introduced into the improved strengthening algorithm to serve as expert experience; and acquiring expert experience corresponding to the real-time pose, the target position and the obstacle category of the mechanical arm, and performing real-time motion control on the mechanical arm by utilizing the trained motion control model. According to the method, collision-free motion planning of the redundant rope-driven mechanical arm in the shielding scene can be achieved.

Description

Technical Field

[0001] This invention relates to the field of robot motion planning technology, and in particular to an integrated intelligent control method and system for perception, decision-making and execution of a redundant rope-driven robotic arm. Background Technology

[0002] The shortcomings of traditional manual labor in performing complex tasks are particularly pronounced in outdoor, structurally complex environments. Due to the characteristics of these environments, such as severe occlusion, diverse and irregularly distributed targets, higher demands are placed on the robot's safety, accessibility, operational flexibility, and multiphysics coupling capabilities.

[0003] In terms of perception, multimodal sensors are relatively mature, capable of outputting color images and depth maps and generating 3D point clouds. This supports robots in constructing 3D environmental representations in complex environments, enabling target localization and obstacle detection, and providing support for robot localization and basic obstacle avoidance. However, in structural environments containing irregular shapes, large scale differences, or complex occlusion relationships, such as branches, cables, supports, shelves, pipes, equipment corners, and human bodies, relying solely on coarse-grained point clouds or simple geometric approximations makes it difficult to accurately analyze the spatial relationships between key obstacles and manipulated objects, as well as their corresponding risks and operational constraints.

[0004] In decision-making and control, traditional kinematics, static path planning, and local obstacle avoidance methods are mostly based on the premise of environmental geometric rules, rigid manipulators, and accurate models. For redundant flexible mechanisms such as tethered continuum manipulators, due to their nonlinear strong coupling, complex deformation, and high state dimension, the computational burden of planning and control based on accurate models is large, and they are sensitive to modeling errors and noise, making it difficult to balance real-time performance and robustness. Although deep reinforcement learning can provide new ideas for the control of flexible redundant manipulators by interactively learning complex policies in a high-dimensional state-action space, it usually starts with random policies, relies on a large number of trial and error samples, resulting in low sample efficiency, high training costs, and safety and wear risks on real robots. Introducing expert demonstrations or prior trajectories can improve convergence speed and stability to some extent. However, existing research mostly focuses on simplified environments or rigid manipulator platforms, and has not yet formed a unified perception, decision-making, and control framework for complex redundant mechanisms such as tethered continuums that fully integrates high-quality environmental perception information.

[0005] Therefore, in unstructured environments, for redundant rope-driven robotic arms, how to achieve an integrated perception, decision-making, and execution method that combines active perception capabilities, path planning / obstacle avoidance, and motion functions through hardware and software co-design, and how to achieve efficient, safe, and robust operation in complex outdoor dynamic environments, is a key issue that current technology urgently needs to solve. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides an integrated intelligent control method and system for sensing, decision-making, and execution of a redundant rope-driven robotic arm.

[0007] According to one aspect of the present invention, a redundant rope-driven robotic arm with integrated sensing, decision-making, and execution intelligent control method is proposed, the method comprising:

[0008] An environmental point cloud is acquired using a depth camera, and the environmental point cloud is processed to obtain a three-dimensional environment model; the three-dimensional environment model includes multiple obstacle category models;

[0009] Parametric 3D modeling of the robotic arm is performed, and an equivalent simplified rigid link model is established;

[0010] Multiple sets of collision-free expert trajectory sequences are generated based on the simplified rigid link model and the three-dimensional environment model;

[0011] A three-stage learning strategy is employed to train a motion control model based on an improved reinforcement algorithm. The first stage trains a first motion control model based on the reinforcement algorithm in an accessible, static target environment. The second stage trains a second motion control model based on the improved reinforcement algorithm in an accessible, dynamic target environment, using the weight parameters of the first motion control model obtained in the first stage as initialization parameters. The third stage trains a third motion control model based on the improved reinforcement algorithm in a random obstacle, dynamic target environment, using the weight parameters of the second motion control model obtained in the second stage as initialization parameters. The improvements to the reinforcement algorithm include the introduction of collision-free expert trajectory sequences as expert experience during the training process.

[0012] The system acquires the real-time pose, target position, and obstacle category of the robotic arm; determines the corresponding expert experience based on the obstacle category; inputs the real-time pose, target position, and expert experience into a trained third motion control model, and achieves real-time motion control of the robotic arm through the output control signal.

[0013] Furthermore, the process of processing the environmental point cloud to obtain a 3D environment model includes: removing outliers and downsampling voxel meshes from the environmental point cloud; segmenting the point cloud into independent objects using clustering or region growing algorithms, wherein the independent objects include targets and obstacles; calculating the point cloud center for each segmented point cloud fragment and performing shape analysis using PCA principal component analysis; estimating the size of the independent objects based on the shape analysis results and combining them with obstacle categories to obtain an obstacle category model; and reconstructing the multi-frame point cloud into a complete 3D environment model.

[0014] Furthermore, the obstacle categories are predetermined as follows: multiple images containing different categories of obstacles are acquired, preprocessed and feature extracted to obtain high-dimensional feature vectors that can characterize obstacle information; the high-dimensional feature vectors are processed to obtain low-dimensional feature representations; the low-dimensional feature representations are input into a pre-trained classifier to obtain the obstacle categories.

[0015] Furthermore, the structure of the robotic arm consists of multiple rigid links connected sequentially by universal joints, and the drive mechanism of the robotic arm includes multiple cooperatively driven ropes; the simplified rigid link model is constructed using the DH parameter method.

[0016] Furthermore, the generation of multiple collision-free expert trajectory sequences based on the simplified rigid link model and the three-dimensional environment model includes: generating collision-free expert trajectory sequences using the artificial potential field method, including: calculating the virtual force acting on the end effector of the robotic arm according to the target position and obstacle category model, and then mapping the virtual force to the desired velocity of the movable joint on the robotic arm through the Jacobian matrix.

[0017] Furthermore, the improvements to the enhancement algorithm also include: adding a reward shaping function to the reward function, which is calculated based on the expert experience, and its expression is as follows:

[0018] ;

[0019] In the formula, This represents the reward shaping function; This represents the hyperparameters used to adjust reward sensitivity; Indicates the current state The minimum Euclidean distance to all expert states in the expert experience buffer.

[0020] Furthermore, the training process of the second motion control model based on the improved reinforcement algorithm or the third motion control model based on the improved reinforcement algorithm includes:

[0021] State vector in state space This includes the angles and angular velocities of multiple movable joints on the robotic arm, the speeds of multiple drive motors, the length of the rope, the three-dimensional coordinates of the end effector, and the three-dimensional coordinates of the target point; among which, the state vector during the training process of the third motion control model is... It also includes the three-dimensional coordinates of all obstacles;

[0022] Action vectors in action space Includes speed signals from multiple drive motors;

[0023] Environmental reward function Including rewards provided by the environment With reward shaping function The formula is:

[0024] ;

[0025] In the formula, This represents the hyperparameter used to adjust the intensity of expert experience guidance; environmental rewards. for: , Indicates the base distance penalty. , The Euclidean distance from the current end effector to the target. This is the distance weight parameter, with a value range of [0.5, 10]. This indicates a form of punishment, used to punish excessive or forceful actions. , For the first The acceleration of each joint, To control the weighting parameter, the value range is [0.01, 0.00001]. This represents a success reward, which is a sparse reward based on logical judgment. , To successfully determine the distance, The success weight parameter has a value range of [0, 3000]. This represents path-aware reward. , The path-aware weight parameter has a value range of [0, 50].

[0026] During the online interaction and data collection phase, the agent interacts with the environment to generate training data; during the model training and parameter update phase, a batch of experience tuples is randomly sampled from the main experience buffer, and for each state in the batch... Calculation based on expert experience Reward Shaping Function In turn, you will receive a combination reward. ;

[0027] Then use combined rewards To construct the target Q value Subsequently, based on the target Q value Gradient updates are performed on the value network and policy network. The value network parameters are updated by minimizing the mean square Bellman error, while the policy network parameters are updated by maximizing the Q-value of the value network evaluation and the policy entropy. Then, a soft update is performed on the target network.

[0028] Furthermore, the model networks in the first and second stages are multilayer perceptron networks; the model network in the third stage is a long short-term memory network; during the training process, the third motion control model, for the state vector... First, normalization is performed; then, feature processing and dimensionality increase are performed on the normalized state vector; then, temporal dependency capture and dynamic feature extraction are performed through a two-layer unidirectional long short-term memory network to obtain the hidden state sequence H; then, the hidden state of the last time step of the hidden state sequence H is extracted; then, the hidden state of the last time step is input into a four-layer fully connected network for deep nonlinear mapping and deep feature fusion processing to obtain the input feature vectors of the policy network and the value network.

[0029] Furthermore, the proportion of expert experience samples in the total training sample ranges from 0% to 30%.

[0030] According to another aspect of the present invention, a redundant rope-driven robotic arm integrated intelligent control system for perception, decision-making, and execution is proposed. The system is implemented based on the aforementioned redundant rope-driven robotic arm integrated intelligent control method for perception, decision-making, and execution. The system includes:

[0031] The model building module includes an environment model building submodule and a robotic arm model building submodule. The environment model building submodule is configured to use a depth camera to collect environmental point clouds, process the environmental point clouds, and obtain a three-dimensional environment model, which includes multiple obstacle category models. The robotic arm model building submodule is configured to perform parametric three-dimensional modeling of the robotic arm and establish an equivalent simplified rigid link model.

[0032] An expert trajectory generation module is configured to generate multiple sets of collision-free expert trajectory sequences based on the simplified rigid link model and the three-dimensional environment model.

[0033] The motion control model training module is configured to train a motion control model based on an improved reinforcement algorithm using a three-stage learning strategy. The first stage trains a first motion control model based on the reinforcement algorithm in an unobstructed, static target environment. The second stage trains a second motion control model based on the improved reinforcement algorithm in an unobstructed, dynamic target environment, using the weight parameters of the first motion control model obtained in the first stage as initialization parameters. The third stage trains a third motion control model based on the improved reinforcement algorithm in a random obstacle, dynamic target environment, using the weight parameters of the second motion control model obtained in the second stage as initialization parameters. The improvements to the reinforcement algorithm include the introduction of collision-free expert trajectory sequences as expert experience during the training process.

[0034] The motion intelligent control module is configured to acquire the real-time pose, target position, and obstacle category of the robotic arm; determine the corresponding expert experience based on the obstacle category; input the real-time pose, target position, and expert experience into a trained third motion control model; and realize real-time motion control of the robotic arm through the output control signal.

[0035] The embodiments of the present invention have the following technical effects:

[0036] This invention proposes an integrated intelligent control method and system for redundant rope-driven robotic arms, encompassing perception, decision-making, and execution. The invention constructs an end-to-end control framework from depth perception to the output of control commands to the drive rope or motor, achieving integrated intelligent control of the hardware and software algorithms without the need for traditional hierarchical programming or inverse kinematics solutions.

[0037] At the perception level, multimodal sensors are used to acquire color images and depth data. Point cloud processing and feature analysis are used to obtain the geometric and spatial distribution information of the target object and surrounding obstacles. At the decision-making level, firstly, the acquired scene data is processed using feature extraction and dimensionality reduction methods, and a classifier is used to classify obstacles in typical work scenarios, constructing corresponding 3D environment models and intelligent control strategies for different obstacles. Secondly, a planning algorithm is used to generate large-scale collision-free trajectories on an equivalent simplified model, constructing an expert experience buffer, which is more efficient than traditional data generation relying on real robot data. Thirdly, a reward shaping function based on the distance between the current state and the expert trajectory is designed, integrating expert experience into the soft actor-critic algorithm. This significantly improves sample efficiency and accelerates training convergence. Furthermore, by integrating a long short-term memory module into the policy network, combined with dual experience replay and automatic entropy adjustment mechanisms, the temporal coupling problem of the rope-driven system is solved, improving the stability and generalization ability of the policy in dynamic occlusion environments. The training strategy of the reinforcement learning model adopts a three-stage learning process from easy to difficult, training policies in environments with no obstacles and fixed targets, no obstacles and dynamic targets, and environments with random obstacles and dynamic targets, respectively. Policies are trained separately for different obstacle categories and stored in the policy library, achieving rapid scheduling and high success rate in complex scenarios. At the execution level, the trained policies are deployed to complex outdoor environments. Real-world testing shows that this invention can run in real-time in complex environments with a high success rate.

[0038] This invention improves the obstacle avoidance success rate and control accuracy of rope-driven redundant robotic arms in unstructured environments, and has good robustness and engineering application value. Attached Figure Description

[0039] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0040] Figure 1 This is a flowchart of an integrated intelligent control method for perception, decision-making, and execution of a redundant rope-driven robotic arm, provided in an embodiment of the present invention.

[0041] Figure 2 This is another flowchart of an integrated intelligent control method for perception, decision-making, and execution of a redundant rope-driven robotic arm, provided in an embodiment of the present invention.

[0042] Figure 3 This is a schematic diagram of the obstacle category determination process in an embodiment of the present invention.

[0043] Figure 4 These are schematic diagrams of the structure (corresponding to the left figure) and the simulation structure (corresponding to the right figure) of the rope-driven robotic arm in this embodiment of the invention.

[0044] Figure 5 This is a schematic diagram of the three-stage learning strategy training process of the motion control model based on the improved SAC algorithm in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0046] This invention proposes an integrated intelligent control method and system for redundant rope-driven robotic arms, encompassing perception, decision-making, and execution. It aims to solve obstacle avoidance and control problems in complex dynamic environments. Active perception is achieved through a hardware-software collaborative mechanism, and deep reinforcement learning guided by expert knowledge is used to solve high-dimensional nonlinear control challenges. This integration of perception, decision-making, and execution enables safe operation in complex outdoor environments.

[0047] This invention proposes an integrated intelligent control method for perception, decision-making, and execution of a redundant rope-driven robotic arm, such as... Figures 1-2 As shown, the method includes:

[0048] S1. Use a depth camera to collect environmental point clouds, process the environmental point clouds, and obtain a three-dimensional environment model; the three-dimensional environment model includes multiple obstacle category models;

[0049] S2. Perform parametric 3D modeling on the robotic arm and establish an equivalent simplified rigid link model;

[0050] S3. Generate multiple sets of collision-free expert trajectory sequences based on the simplified rigid link model and the three-dimensional environment model;

[0051] S4. A three-stage learning strategy is adopted to train a motion control model based on an improved reinforcement algorithm. The first stage trains a first motion control model based on the reinforcement algorithm in an unobstructed, static target environment. The second stage trains a second motion control model based on the improved reinforcement algorithm in an unobstructed, dynamic target environment, using the weight parameters of the first motion control model obtained in the first stage as initialization parameters. The third stage trains a third motion control model based on the improved reinforcement algorithm in a random obstacle, dynamic target environment, using the weight parameters of the second motion control model obtained in the second stage as initialization parameters. The improvements to the reinforcement algorithm include the introduction of collision-free expert trajectory sequences as expert experience during the training process.

[0052] S5. Obtain the real-time pose, target position, and obstacle category of the robotic arm; determine the corresponding expert experience based on the obstacle category; input the real-time pose, target position, and expert experience into the trained third motion control model, and realize real-time motion control of the robotic arm through the output control signal.

[0053] The method begins with S1. In S1, a depth camera is used to acquire environmental point clouds, which are then processed to obtain a three-dimensional environment model; the three-dimensional environment model contains multiple obstacle category models.

[0054] According to embodiments of the present invention, such as Figure 2As shown, a depth camera is used to perform a 360-degree scan of the robotic arm's working environment to acquire environmental point clouds. The point clouds are then processed as follows: First, outlier removal and voxel mesh downsampling are performed to reduce noise and unify resolution. Then, clustering or region growing algorithms are used to segment the point cloud into independent objects, including targets and obstacles. For example, in an orchard environment, the target is fruit, and obstacles are leaves and branches. For each segmented point cloud fragment, its center is calculated, and shape analysis is performed using Principal Component Analysis (PCA). This includes centering the point cloud, solving the covariance matrix, and performing eigenvalue decomposition. The eigenvectors corresponding to the largest, second-largest, and smallest eigenvalues ​​describe the principal axis direction of the independent object in 3D space, and the magnitude of the eigenvalue reflects the extent of the independent object along that direction. Based on the shape analysis results, the size of the independent objects is estimated, and combined with obstacle categories, an approximate ellipsoidal or cuboid model is obtained. Finally, based on the Poisson reconstruction-based TSDF fusion method, multiple frames of point clouds are reconstructed into a complete 3D environment model. This model is used for obstacle category setting in the MuJoCo simulation environment.

[0055] like Figure 3 As shown, obstacle categories are determined as follows. In a structural environment containing various irregular shapes, large scale differences, and complex occlusion relationships, such as branches, cables, supports, shelves, pipes, and equipment corners, the categories of obstacles encountered during operations are determined. Specifically, before executing the task, multiple obstacle images of different categories are acquired and preprocessed for feature extraction (including color, texture, shape, etc.) to obtain high-dimensional feature vectors that can represent obstacle information. Next, the high-dimensional feature vectors are processed by studying the inherent, non-linear low-dimensional manifold structure in the high-dimensional data to transform them into low-dimensional feature representations. Finally, the low-dimensional feature representations are input into a pre-trained supervised classifier (such as support vector machine, random forest, k-nearest neighbors, etc.). This classifier discriminates the low-dimensional feature representations and outputs their respective obstacle categories. This classification result is not only used for visual understanding but also serves as an index to directly drive subsequent policy library selection and motion planning processes, achieving integrated linkage between perception, decision-making, and execution.

[0056] As an example, when the work environment is an orchard, the corresponding obstacle categories include: leaf occlusion, fruit occlusion, branch occlusion, leaf and fruit occlusion, leaf and branch occlusion, branch and fruit occlusion, and occlusion caused by a combination of leaves, branches, and fruit. For other complex facilities and human-machine collaborative work environments, the corresponding obstacle categories may include, but are not limited to: strong leaf occlusion, strong branch occlusion, non-target fruit occlusion, cable / wire occlusion, support / column occlusion, shelf / pallet structure occlusion, pipe occlusion, equipment corner occlusion, protective netting or covering occlusion, human body occlusion, mixed human body and tool occlusion, and mixed occlusion caused by a combination of tree body (leaves, branches, or fruit) and external components (cables, supports, shelves, pipes, equipment, etc.), etc.

[0057] This step utilizes a depth camera to scan the working environment and reconstruct occlusion obstacle models, achieving accurate environmental modeling. Combining manifold learning and a traditional classifier, occlusion situations are further categorized and associated with a policy library, enabling targeted decision-making. Compared to existing methods that commonly employ single policies or simplified obstacle handling, the multi-policy index significantly improves the accuracy of obstacle avoidance and operational procedures. The obstacle categories output by the classifier serve as indexes, enabling rapid matching and retrieval of the policy library and environmental model. This allows the system to select more suitable policies based on perception results in various complex obstacle combinations and complex outdoor application scenarios, thereby improving the overall accuracy and robustness of decision-making.

[0058] Then execute S2, in which parametric 3D modeling of the robotic arm is performed, and an equivalent simplified rigid link model is established.

[0059] According to embodiments of the present invention, such as Figure 4 As shown, the robotic arm is a redundant cable-driven robotic arm. Its specific structure consists of four rigid links connected sequentially by universal joints, with a total length of 598.5 mm. The drive system consists of 12 cooperatively driven cables. The cables are led out from the base platform and pass through 12 evenly distributed guide holes on the end disc of each link, which is 153 mm long.

[0060] The redundant cable-driven manipulator was parametrically modeled using 3D modeling software to obtain its dynamic model. During modeling, the materials and physical parameters of each component needed to be clearly defined: the universal joint, platform, and rigid linkage were made of aluminum alloy with a density of 2700 kg / m³, Poisson's ratio of 0.33, and elastic modulus of 70 GPa; the rope was made of stainless steel with a density of 8000 kg / m³, Poisson's ratio of 0.28, and elastic modulus of 193 GPa. After completing the 3D modeling, the model was converted to the MJCF format supported by the MuJoCo simulation environment, and all material parameters, linkage mass, and inertial properties were configured in MuJoCo. The drive rope was modeled using a point-to-point tendon modeling method to accurately simulate force transmission along the rope and its elastic expansion and contraction.

[0061] Furthermore, to avoid the inverse kinematics problem of solving the complex dynamic model, the solution is performed on a simplified rigid link model with kinematic equivalents. Specifically, the dynamic model of the redundant cable-driven manipulator is equivalent to a simplified rigid link model. This simplified model is constructed using the Denavit-Hartenberg (DH) parametric method, and is a simplified model with four links and eight joints. The range of motion of all eight joints in this model is limited to... .

[0062] Then, S3 is executed, in which multiple sets of collision-free expert trajectory sequences are generated based on the simplified rigid link model and the three-dimensional environment model.

[0063] According to an embodiment of the present invention, the motion planning of the robotic arm is decoupled into two sub-problems: trajectory generation and motion control. This step solves the problem of trajectory generation.

[0064] Within the workspace of the simplified rigid linkage model, the target point of the end effector lies within a predefined three-dimensional spatial range. Based on the obstacle categories determined in S1 above, a three-dimensional environment model containing multiple obstacle category models is imported, where the obstacle positions and radii remain consistent with the subsequent training environment.

[0065] Trajectory generation employs the Artificial Potential Field (APF) method. This method calculates the virtual force acting on the robotic arm's end effector based on the target position and obstacle category model, and then maps it to motion in joint space using the Jacobian matrix. The mathematical model used is as follows: Virtual Total Force By gravity and repulsive force composition: Among them, gravity Generated from the target point, pointing towards the target point, and its magnitude is proportional to the distance from the current position of the endpoint to the target. Its calculation formula is:

[0066] ;

[0067] Where p is the current position of the end effector; The location of the target point; The gravitational gain coefficient, which physically means adjusting the speed of the end effector toward the target point, can be set to 0.3.

[0068] The repulsive force points from the obstacle towards the end effector, its magnitude is inversely proportional to the distance, and increases sharply as the obstacle approaches. Its calculation formula is as follows:

[0069] ;

[0070] in, This represents the shortest distance between the end effector and the obstacle. This is the repulsive force gain coefficient, which physically adjusts the magnitude of the repulsive force during obstacle avoidance and can be set to 0.2. The range of repulsive force influence, which physically means the maximum distance at which the obstacle begins to generate repulsive force, can be set to 0.05m.

[0071] when When the repulsive force is zero, it does not affect long-distance movement; when near The repulsive force is generated smoothly when Time repulsion This ensures the effectiveness of obstacle avoidance. The total virtual force in Cartesian space is calculated from this. Then, through the position (including target position and obstacle position) Jacobian matrix ( 3 The inverse pseudomatrix of an 8-matrix , will the total virtual force Mapped to the desired velocity of 8 movable joints The calculation formula is as follows: .

[0072] As an example, the specific logic for trajectory generation is as follows: iterate through the set total number of scenarios and randomly generate target points and obstacles in each attempt; then, call the APF path planning function to generate the trajectory; check whether the planning is successful and whether the length of the generated trajectory is within the set minimum and maximum length range; if the above logic is true (i.e., the planning is successful and the length of the generated trajectory is within the set minimum and maximum length range), then format and save the trajectory data of the scenario as a file to generate a dataset containing a large number of collision-free expert trajectories. This dataset is stored in the expert experience buffer and used to guide the reinforcement learning agent in subsequent steps.

[0073] This step decouples trajectory planning from dynamic control, enabling the rapid generation of expert trajectories using a planning algorithm on a lightweight kinematic model. This ensures both the authenticity and diversity of the data while significantly reducing generation time. Compared to existing technologies that rely on data collection from real robots or complex model computations, the expert trajectory data generation speed in this invention is faster and more easily scalable.

[0074] It should be noted that other trajectory generation methods can also be used to generate collision-free expert trajectories.

[0075] Then, S4 is executed, in which a three-stage learning strategy is used to train a motion control model based on an improved reinforcement algorithm. The first stage trains a first motion control model based on a barrier-free, static target environment. The second stage trains a second motion control model based on an improved reinforcement algorithm in a barrier-free, dynamic target environment, using the weight parameters of the first motion control model obtained in the first stage as initialization parameters. The third stage trains a third motion control model based on an improved reinforcement algorithm in a random obstacle, dynamic target environment, using the weight parameters of the second motion control model obtained in the second stage as initialization parameters. The improvements to the reinforcement algorithm include the introduction of collision-free expert trajectory sequences as expert experience during the training process.

[0076] According to embodiments of the present invention, such as Figure 5 As shown, training is conducted in three stages according to a learning strategy that progresses from easy to difficult: the first stage trains basic motor skills in an accessible environment with fixed targets; the second stage introduces expert trajectory experience to accelerate learning in an accessible environment with random targets; and the third stage trains complex obstacle avoidance strategies using an LSTM architecture in an environment with random obstacles and random targets, or simply introduces expert trajectory experience for training. Strategies are trained separately for various obstacle categories and stored in a strategy library to achieve targeted control.

[0077] Specifically, in the first stage, the target point is set to be static, and an obstacle-free simulation environment is established. The goal of this stage is to enable the agent to understand the kinematic and dynamic characteristics of the rope-driven redundant manipulator, establish a basic mapping from the configuration space to the task space, and achieve accurate arrival at the static target point. A standard reinforcement learning method is adopted (the soft actor-critic (SAC) algorithm is used in this embodiment), without introducing external expert experience (i.e., without introducing collision-free expert trajectory sequences). In reinforcement learning, the agent learns only through autonomous interaction with the environment, and a multilayer perceptron (MLP) network is selected. As an example, the total training time step in this stage is set to 300,000. The trained reinforcement learning model serves as the first motion control model, and its weight parameters are used as the initial weight parameters for the subsequent second learning stage.

[0078] In the second stage, the target points are set to be dynamically random, and an obstacle-free simulation environment is established, allowing the robotic arm to perform uniform sampling within a predetermined workspace. The goal of this stage is to generalize the model obtained in the static target point stage to dynamically random target points and guide the agent to learn the motion trajectory. An improved reinforcement learning method (i.e., an improved soft actor-critic (SAC) algorithm) is used as the second motion control model. In reinforcement learning, the agent learns through autonomous interaction with the environment and incorporates expert experience. An MLP network is still used, and the state distribution of expert experience is incorporated into the Bellman update equation of reinforcement learning in a defined reward shaping form. The proportion of expert experience samples in the total training samples is set to range from 0% to 30%. As an example, the total training time step in this stage is set to 300,000.

[0079] The improved soft actor-critic algorithm used in the second stage establishes two replay buffers: expert experience and online experience, and utilizes reward shaping based on the minimum distance between the state and the expert trajectory. 1) Constructing dual experience (expert experience and master experience) replay buffers: Expert experience buffer Used to load and store the collision-free expert trajectory sequence generated in S3, i.e., the state in expert experience. It contains the joint angles, velocities, end-effector positions, target positions, and obstacle information of the robotic arm corresponding to the collision-free expert trajectory sequence; the main experience buffer is used to store experience tuples generated by the online interaction between the agent and the environment. ,in For state, For action, As a reward value, This serves as the termination marker. 2) Design of the reward shaping function and Q-value update: In order to integrate expert experience into the Bellman update equation of the SAC algorithm in the form of reward shaping, a reward shaping function was designed. The design and derivation of this function are as follows:

[0080] Calculate the current state With expert experience buffer All experts in China minimum Euclidean distance :

[0081] ;

[0082] Reward Shaping Function The exponential decay function designed as the minimum Euclidean distance mentioned above is calculated as follows:

[0083] ;

[0084] in This is a hyperparameter used to adjust the sensitivity of the reward. The design of the reward shaping function provides a non-sparse guiding signal: when the agent's current state... When approaching the state of any expert trajectory Provides strong positive guidance; when the agent is in its current state When the state deviates from the expert trajectory, the reward decays exponentially. The design of a reward shaping function based on the distance between the state and the expert trajectory significantly reduces invalid samples introduced by random exploration, thereby improving the convergence speed and accuracy of deep reinforcement learning.

[0085] The improved soft actor-critic algorithm process specifically includes: initialization phase, online interaction and data collection phase, and model training and parameter update phase.

[0086] 1) In the initialization phase, the control problem is abstracted into a Markov decision process, the core elements of which are as follows:

[0087] State space (s): states Defined as a 52-dimensional vector containing angles at 8 joints. Angular velocity of 8 joints, speed of 12 drive motors The length of 12 ropes The three-dimensional coordinates of the end effector The three-dimensional coordinates of the mission objective point If the environment contains obstacles, then the state is... It also includes the 3D coordinates of all obstacles in the scene. ;

[0088] Action space (A): Action It is defined as a 12-dimensional continuous vector, corresponding to the speed signals of 12 drive motors;

[0089] Environment reward function (R): To guide the agent to complete the task, design a multi-component reward function. as follows:

[0090] ;

[0091] In the formula, Indicates the base distance penalty. , This represents the Euclidean distance from the current end effector to the target. This is the distance weighting parameter, with a value range of [0.5, 10]. This indicates a punishment for excessive or forceful actions. , For the first The acceleration of each joint, To control the weighting parameter, its value range is [0.01, 0.00001]; This represents a success reward, which is a sparse reward based on logical judgment. , To successfully determine the distance, a range of [0.04, 0] m can be set, where... The success weight parameter has a value range of [0, 3000]. This represents the path-aware reward, which is a logically based reward given when the agent perceives that the path is blocked. ,in The path-aware weight parameter has a value range of [0, 50].

[0092] Initializing the hyperparameters of the SAC algorithm - discount factor This hyperparameter is used to balance the importance of immediate rewards and future rewards, and can be set to 0.99.

[0093] 2) In the online interaction and data collection phase, the agent interacts with the environment to generate training data;

[0094] 3) During the model training and parameter update phase, a batch of empirical tuples is randomly sampled from the main empirical buffer. Then for each state in the batch Reward-based shaping, i.e., calculation Reward Shaping Function The rewards provided by the environment With shaping rewards Perform linear combinations to form the final combined reward. The calculation formula is as follows:

[0095] ;

[0096] in This is a hyperparameter used to adjust the intensity of expert experience guidance.

[0097] Then use combined rewards To construct the target Q value The calculation formula is as follows:

[0098] ;

[0099] in, For the target Q network, For policy networks, This refers to the temperature parameter.

[0100] Subsequently, the value network and policy network are updated, with the value network parameters... The update is performed by minimizing the mean square Bellman error (MSBE), which is calculated as follows:

[0101] ;

[0102] in, The mean squared Bellman variance, The mean of a small batch of data randomly sampled from the master experience replay buffer; The learnable parameters of the value network; This represents the value network. This update forces the value network... Learn a Q-value that simultaneously maximizes environmental rewards and mimics expert experience.

[0103] The policy network is updated using the Q-value and policy entropy evaluated by maximizing the value network, and its calculation formula is as follows:

[0104] ;

[0105] in, The loss value of the policy network. Indicates the sampling state from the main experience replay buffer. Expectations To adjust the hyperparameters of the entropy weights, Output actions for the policy network The logarithm of the probability, This represents the actions of two value networks in response to the current policy generation. The minimum score is taken.

[0106] Due to value networks The policy network, having already been shaped, indirectly learns and imitates expert experience during the update process to generate higher Q-value actions. After completing the gradient updates of the policy and value networks, the algorithm executes the final step of this iteration: a soft update of the target network. To ensure the stability of the training process and suppress oscillations in Q-value estimation, hyperparameters are introduced. The parameters of the main value network are synchronized to the target network, and the calculation formula is as follows:

[0107] ;

[0108] in, For target network parameters; This is the soft update coefficient. The retention factor.

[0109] The above process completes one full parameter learning iteration. Algorithm training will continuously repeat the closed-loop process of environment interaction-data sampling-gradient update-soft update, allowing the agent's strategy to gradually converge in a balance between expert experience and self-exploration, ultimately mastering the optimal strategy for precise and compliant control of the rope-driven robotic arm in complex unstructured environments.

[0110] In the third stage, the target points are set to be dynamically randomized, and obstacles are introduced to learn complex obstacle avoidance strategies. The goal of this stage is to enable the agent to learn complex obstacle avoidance strategies in a dynamic target environment with introduced obstacles. This stage is based on the model trained in the second stage (i.e., the weight parameters of the second motion control model are used as the initialization parameters of the third motion control model), and two different learning strategies are designed. Strategy 1: Continue the hybrid learning mode, that is, introduce corresponding expert experience trajectories (obstacle avoidance trajectories in an environment with obstacles) into the environment of the third stage, and train it based on the improved soft actor-critic algorithm used in the second stage. Strategy 1 accelerates the learning of complex obstacle avoidance strategies by forcibly guiding the agent to tend towards expert experience. The network architecture still uses MLP, and the proportion of expert experience samples ranges from 0% to 30%. Strategy 2: Change the network architecture from MLP to a Long Short-Term Memory (LSTM) based architecture. Strategy 2 forces the agent to learn obstacle avoidance strategies by relying on its own memory ability and direct interaction with the environment. As an example, the training time step for both strategies in the third stage is set to 200,000.

[0111] Strategy two is as follows. To address the hysteresis effect of the robotic arm, in the third phase of training, the observed states... After passing through the LSTM feature extractor, the processing flow of the LSTM feature extractor includes: for the input observed state sequence Normalization is performed: , This represents the normalization function; subsequently, feature processing and dimensionality increase are performed on the normalized observation sequence: , Represents a fully connected linear transformation. This represents the activation function. During training, a portion of neurons are randomly set to zero to prevent overfitting. Subsequently, a two-layer unidirectional long short-term memory network is used for temporal dependency capture and dynamic feature extraction to obtain the hidden state sequence H. LSTM represents a Long Short-Term Memory network; subsequently, the hidden state of the last time step of the hidden state sequence H is extracted: Then, hide the state of the last time step. Deep nonlinear mapping and feature fusion are performed on the input four-layer fully connected network. ,in is the feature vector, which serves as the input to the subsequent policy network and value network.

[0112] After the third stage of training is completed, the proportion of expert experience and the parameters of the reinforcement learning agent are configured to obtain the optimal collision-free motion planning strategy (i.e., the third motion control model).

[0113] Then, S5 is executed. In S5, the real-time pose, target position, and obstacle category of the robotic arm are obtained; the corresponding expert experience is determined according to the obstacle category; the real-time pose, target position, and expert experience are input into the trained third motion control model, and the real-time motion control of the robotic arm is realized through the output control signal.

[0114] According to an embodiment of the present invention, the trained strategy is deployed in a physical robotic arm system. During operation, the system senses joint states, target positions, and obstacle information in real time, selects appropriate strategies based on obstacle categories, and outputs the drive rope speed through a trained third motion control model, achieving end-to-end collision-free path planning and control. Specifically, the system acquires environmental point clouds in real time using a depth camera, obtains the robotic arm's state in real time using displacement sensors, angle sensors, and rope displacement sensors, and identifies obstacle categories in the environmental point clouds using an obstacle category recognition model deployed on a host computer. Based on the robotic arm's real-time state, and by calling the obstacle category model corresponding to the obstacle category and the pre-deployed third motion control model, the system outputs the desired motion commands for the corresponding drive motors of the robotic arm in real time. This forms a perception-decision-execution closed loop, enabling the redundant rope-driven robotic arm to actively perceive and operate stably in complex dynamic environments.

[0115] This invention constructs an end-to-end control framework through deep visual perception and deep reinforcement learning, realizing a direct mapping from 3D environment perception to drive rope speed output. It eliminates the need to solve complex inverse kinematics or design hierarchical planning modules, and can adapt to the highly nonlinear dynamics of the rope-driven robotic arm. The dense point cloud generated by the depth camera enables the system to construct a 3D environment representation in real time and assists the robot in achieving robust localization and active obstacle avoidance in dynamic scenes.

[0116] This invention also proposes an integrated intelligent control system for perception, decision-making, and execution of a redundant rope-driven robotic arm. The system is implemented based on the integrated intelligent control method for perception, decision-making, and execution of a redundant rope-driven robotic arm described in the above embodiments. The system includes:

[0117] The model building module includes an environment model building submodule and a robotic arm model building submodule. The environment model building submodule is configured to use a depth camera to collect environmental point clouds, process the environmental point clouds, and obtain a three-dimensional environment model, which includes multiple obstacle category models. The robotic arm model building submodule is configured to perform parametric three-dimensional modeling of the robotic arm and establish an equivalent simplified rigid link model.

[0118] An expert trajectory generation module is configured to generate multiple sets of collision-free expert trajectory sequences based on the simplified rigid link model and the three-dimensional environment model.

[0119] The motion control model training module is configured to train a motion control model based on an improved reinforcement algorithm using a three-stage learning strategy. The first stage trains a first motion control model based on the reinforcement algorithm in an unobstructed, static target environment. The second stage trains a second motion control model based on the improved reinforcement algorithm in an unobstructed, dynamic target environment, using the weight parameters of the first motion control model obtained in the first stage as initialization parameters. The third stage trains a third motion control model based on the improved reinforcement algorithm in a random obstacle, dynamic target environment, using the weight parameters of the second motion control model obtained in the second stage as initialization parameters. The improvements to the reinforcement algorithm include the introduction of collision-free expert trajectory sequences as expert experience during the training process.

[0120] The motion intelligent control module is configured to acquire the real-time pose, target position, and obstacle category of the robotic arm; determine the corresponding expert experience based on the obstacle category; input the real-time pose, target position, and expert experience into a trained third motion control model; and realize real-time motion control of the robotic arm through the output control signal.

[0121] The function of the integrated intelligent control system for perception, decision-making and execution of a redundant rope-driven robotic arm described in this embodiment of the invention can be explained by the aforementioned integrated intelligent control method for perception, decision-making and execution of a redundant rope-driven robotic arm. Therefore, for the parts not described in detail in the system embodiment, please refer to the above method embodiment, and they will not be repeated here.

[0122] The technical effects of the present invention were further verified through experiments.

[0123] To verify the impact of expert experience ratio on model training and convergence performance, the experiment first compared and analyzed models with different expert experience ratio configurations in a simplified scenario containing only two small-scale obstacles. This stage of the experiment aimed to isolate the role of the expert guidance mechanism; therefore, no obstacle perception reward function was introduced. Instead, the study investigated the effect of expert experience on improving sample efficiency and final control accuracy.

[0124] As shown in Table 1, with the introduction of approximately 10% expert experience samples, the average final distance is reduced to approximately 0.029m, while the average final distance of the baseline model without expert experience exceeds 0.27m, verifying the significant advantage of the present invention in terms of sample efficiency.

[0125] Table 1

[0126]

[0127] Based on determining the optimal expert ratio, the performance of this invention in complex dynamic environments was further verified. This invention employs a three-stage learning strategy, gradually transitioning from training on unobstructed static targets to training on dynamic targets with obstacles, and introduces an LSTM module into the policy network to handle the inherent temporal dependencies of the rope-driven system. Experiments were conducted in a large-scale obstacle environment, and the test results are shown in Table 2. The SAC-10%-LSTM model of this invention exhibits excellent robustness, achieving an average final distance of 0.033m without collisions.

[0128] Table 2

[0129]

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A redundant rope-driven robotic arm integrated intelligent control method for perception, decision-making, and execution, characterized in that: include: An environmental point cloud is acquired using a depth camera, and the environmental point cloud is processed to obtain a three-dimensional environment model; the three-dimensional environment model includes multiple obstacle category models; Parametric 3D modeling of the robotic arm is performed, and an equivalent simplified rigid link model is established; Multiple sets of collision-free expert trajectory sequences are generated based on the simplified rigid link model and the three-dimensional environment model; A three-stage learning strategy is employed to train a motion control model based on an improved reinforcement algorithm. The first stage trains a first motion control model based on the reinforcement algorithm in an accessible, static target environment. The second stage trains a second motion control model based on the improved reinforcement algorithm in an accessible, dynamic target environment, using the weight parameters of the first motion control model obtained in the first stage as initialization parameters. The third stage trains a third motion control model based on the improved reinforcement algorithm in a random obstacle, dynamic target environment, using the weight parameters of the second motion control model obtained in the second stage as initialization parameters. The improvements to the reinforcement algorithm include the introduction of collision-free expert trajectory sequences as expert experience during the training process. The system acquires the real-time pose, target position, and obstacle category of the robotic arm; determines the corresponding expert experience based on the obstacle category; inputs the real-time pose, target position, and expert experience into a trained third motion control model, and achieves real-time motion control of the robotic arm through the output control signal.

2. The integrated intelligent control method for perception, decision-making, and execution of a redundant rope-driven robotic arm according to claim 1, characterized in that, The process of processing the environmental point cloud to obtain a 3D environment model includes: removing outliers and downsampling voxel grids from the environmental point cloud; segmenting the point cloud into independent objects, including targets and obstacles, using clustering or region growing algorithms; calculating the point cloud center for each segmented point cloud fragment and performing shape analysis using PCA principal component analysis; estimating the size of the independent objects based on the shape analysis results and combining them with obstacle categories to obtain an obstacle category model; and reconstructing the multi-frame point cloud into a complete 3D environment model.

3. The integrated intelligent control method for perception, decision-making, and execution of a redundant rope-driven robotic arm according to claim 2, characterized in that, The obstacle categories are predetermined as follows: multiple images containing different categories of obstacles are acquired, preprocessed, and feature extracted to obtain high-dimensional feature vectors that can represent obstacle information; the high-dimensional feature vectors are processed to obtain low-dimensional feature representations; the low-dimensional feature representations are input into a pre-trained classifier to obtain the obstacle categories.

4. The integrated intelligent control method for perception, decision-making, and execution of a redundant rope-driven robotic arm according to claim 1, characterized in that, The robotic arm is constructed by connecting multiple rigid links sequentially via universal joints. The drive mechanism of the robotic arm includes multiple ropes that drive in concert. The simplified rigid link model is constructed using the DH parameter method.

5. The integrated intelligent control method for perception, decision-making, and execution of a redundant rope-driven robotic arm according to claim 1, characterized in that, The generation of multiple collision-free expert trajectory sequences based on the simplified rigid link model and the three-dimensional environment model includes: generating collision-free expert trajectory sequences using the artificial potential field method, including: calculating the virtual force acting on the end effector of the robotic arm according to the target position and obstacle category model, and then mapping the virtual force to the desired velocity of the movable joint on the robotic arm through the Jacobian matrix.

6. The integrated intelligent control method for perception, decision-making, and execution of a redundant rope-driven robotic arm according to claim 1, characterized in that, The improvements to the enhancement algorithm also include: adding a reward shaping function to the reward function, which is calculated based on the expert experience, and its expression is as follows: ； In the formula, This represents the reward shaping function; This represents the hyperparameters used to adjust reward sensitivity; Indicates the current state The minimum Euclidean distance to all expert states in the expert experience buffer.

7. The integrated intelligent control method for perception, decision-making, and execution of a redundant rope-driven robotic arm according to claim 6, characterized in that, The training process of the second motion control model based on the improved reinforcement algorithm or the third motion control model based on the improved reinforcement algorithm includes: State vector in state space This includes the angles and angular velocities of multiple movable joints on the robotic arm, the speeds of multiple drive motors, the length of the rope, the three-dimensional coordinates of the end effector, and the three-dimensional coordinates of the target point; among which, the state vector during the training process of the third motion control model is... It also includes the three-dimensional coordinates of all obstacles; Action vectors in action space Includes speed signals from multiple drive motors; Environmental reward function Including rewards provided by the environment With reward shaping function The formula is: ； In the formula, This represents the hyperparameter used to adjust the intensity of expert experience guidance; environmental rewards. for: , Indicates the base distance penalty. , This represents the Euclidean distance from the current end effector to the target. This is the distance weight parameter, with a value range of [0.5, 10]. This indicates a form of punishment, used to punish excessive or forceful actions. , For the first The acceleration of each joint, To control the weighting parameter, the value range is [0.01, 0.00001]. This represents a success reward, which is a sparse reward based on logical judgment. , To successfully determine the distance, The success weight parameter has a value range of [0, 3000]. This represents path-aware reward. , The path-aware weight parameter has a value range of [0, 50]. During the online interaction and data collection phase, the agent interacts with the environment to generate training data; during the model training and parameter update phase, a batch of experience tuples is randomly sampled from the main experience buffer, and for each state in the batch... Calculation based on expert experience Reward Shaping Function In turn, you will receive a combination reward. ; Then use combined rewards To construct the target Q value Subsequently, based on the target Q value Gradient updates are performed on the value network and policy network. The value network parameters are updated by minimizing the mean square Bellman error, while the policy network parameters are updated by maximizing the Q-value of the value network evaluation and the policy entropy. Then, a soft update is performed on the target network.

8. The integrated intelligent control method for perception, decision-making, and execution of a redundant rope-driven robotic arm according to claim 1, characterized in that, The model networks in the first and second stages are multilayer perceptron networks; the model network in the third stage is a long short-term memory network; during the training process, the third motion control model, for the state vector... First, normalization is performed; then, feature processing and dimensionality increase are performed on the normalized state vector; then, temporal dependency capture and dynamic feature extraction are performed through a two-layer unidirectional long short-term memory network to obtain the hidden state sequence H; then, the hidden state of the last time step of the hidden state sequence H is extracted; then, the hidden state of the last time step is input into a four-layer fully connected network for deep nonlinear mapping and deep feature fusion processing to obtain the input feature vectors of the policy network and the value network.

9. The integrated intelligent control method for perception, decision-making, and execution of a redundant rope-driven robotic arm according to claim 1, characterized in that, The proportion of expert experience samples in the total training sample ranged from 0% to 30%.

10. A redundant rope-driven robotic arm integrated intelligent control system for perception, decision-making, and execution, characterized in that: The system is implemented based on the integrated intelligent control method for perception, decision-making, and execution of a redundant rope-driven robotic arm as described in any one of claims 1-9; the system includes: The model building module includes an environment model building submodule and a robotic arm model building submodule. The environment model building submodule is configured to use a depth camera to collect environmental point clouds, process the environmental point clouds, and obtain a three-dimensional environment model, which includes multiple obstacle category models. The robotic arm model building submodule is configured to perform parametric three-dimensional modeling of the robotic arm and establish an equivalent simplified rigid link model. An expert trajectory generation module is configured to generate multiple sets of collision-free expert trajectory sequences based on the simplified rigid link model and the three-dimensional environment model. The motion control model training module is configured to train a motion control model based on an improved reinforcement algorithm using a three-stage learning strategy. The first stage trains a first motion control model based on the reinforcement algorithm in an unobstructed, static target environment. The second stage trains a second motion control model based on the improved reinforcement algorithm in an unobstructed, dynamic target environment, using the weight parameters of the first motion control model obtained in the first stage as initialization parameters. The third stage trains a third motion control model based on the improved reinforcement algorithm in a random obstacle, dynamic target environment, using the weight parameters of the second motion control model obtained in the second stage as initialization parameters. The improvements to the reinforcement algorithm include the introduction of collision-free expert trajectory sequences as expert experience during the training process. The motion intelligent control module is configured to acquire the real-time pose, target position, and obstacle category of the robotic arm; determine the corresponding expert experience based on the obstacle category; input the real-time pose, target position, and expert experience into a trained third motion control model; and realize real-time motion control of the robotic arm through the output control signal.

Images