Mechanical arm dynamic tracking and obstacle avoidance system and method based on reinforcement learning

Abstract

The invention discloses a mechanical arm dynamic tracking and obstacle avoidance system and method based on reinforcement learning. The mechanical arm dynamic tracking and obstacle avoidance system comprises a sensing module, a target motion prediction module, a reinforcement learning decision module, a safety control and constraint module, an execution module and a training and simulation module. According to the invention, a complete system including environment perception, target motion prediction, reinforcement learning decision and execution control is constructed, so that the mechanical arm can predictively track a moving target in an unknown or dynamic environment, and real-time obstacle avoidance is carried out on a static or dynamic obstacle; and the requirements of safety, robustness and real-time performance of an industrial site are met in the whole control process.

Description

Technical Field

[0001] This invention relates to the fields of robot control, machine learning and intelligent manufacturing technology, and in particular to a system and method for dynamic tracking and obstacle avoidance of a robotic arm based on reinforcement learning. Background Technology

[0002] As a core execution unit in scenarios such as intelligent manufacturing, smart warehousing, human-robot collaborative robots, and service robots, the robotic arm's flexibility, safety, and autonomy directly determine the overall system performance. With the increasing flexibility and automation of industrial production, robotic arms have evolved from traditional repetitive operations at fixed points to performing complex tasks in dynamic, unknown, and semi-structured environments, such as dynamically grasping moving objects, collaborative handling with humans, medical assistance, intelligent sorting, and logistics transportation. These tasks, with their extremely high real-time requirements, present the following key challenges:

[0003] 1) The target has characteristics of position change, velocity change, and even random motion, and the robotic arm needs to have continuous real-time tracking capability; 2) Unmodeled, randomly appearing, or moving obstacles may appear in the environment, and the robotic arm must have the ability to safely avoid obstacles; 3) High-degree-of-freedom robotic arms have high-dimensional and strongly coupled motion spaces, which are difficult to solve efficiently using traditional analytical models.

[0004] Traditional model-based motion planning methods are insufficient. Existing robotic arm motion planning often employs inverse kinematics analytical methods, sampling-based path planning such as RRT and PRM, search-based algorithms such as A*, or optimal control methods such as MPC. While these methods have some effectiveness, they suffer from significant drawbacks in dynamic environments and unknown scenarios: First, they rely on static or quasi-static environment models. Traditional methods assume that the environment geometry and obstacle positions remain fixed within the planning period. However, in real-world scenarios, phenomena such as high-speed target movement, workers entering the vicinity of the robotic arm, temporary placement of tools or boxes, and difficulty in predicting obstacle trajectories occur, making these algorithms unable to quickly adapt to dynamic changes. Second, they lack real-time performance, failing to meet high-frequency control requirements. Methods such as RRT, PRM, and A* require global or local searches, especially in obstacle-dense, high-dimensional motion spaces, leading to unpredictable planning times, single-planning delays potentially reaching hundreds of milliseconds, and difficulties in achieving real-time motion control at 20-50Hz.

[0005] For example, CN114378827A describes a dynamic target tracking and grasping method based on the overall control of a mobile robotic arm. This method, for mobile robotic arms, proposes: training a target detection model using labeled image datasets; calculating the target's spatial coordinates and depth using monocular vision; establishing a kinematic model and Jacobian matrix for the mobile chassis and robotic arm; using visual servoing technology to dynamically track the target and predict its trajectory; and finally using PID closed-loop control to achieve dynamic grasping. The problems with this application are that the control method is still based on traditional models and PID controllers, lacking the adaptive strategy capability of reinforcement learning. The entire control framework relies on explicit kinematic models, Jacobian matrices, and PID controllers, making it more suitable for structured scenarios. When the environment changes significantly (e.g., dense obstacles or complex target motion patterns), manual parameter retuning or controller reconstruction is required, lacking self-learning and adaptive strategy capabilities. Furthermore, the patent application focuses on "dynamic target tracking and grasping," relying more on pre-modeling or simple constraints for obstacles in the environment, without constructing a real-time obstacle avoidance mechanism based on point cloud / depth information as input, and particularly lacking reinforcement learning obstacle avoidance strategies for unknown or moving obstacles.

[0006] In addition, traditional deep reinforcement learning methods have shortcomings. While deep reinforcement learning has been used for robot control, it still suffers from problems such as poor policy stability and sensitivity to environmental changes. Most deep reinforcement learning methods only use the current state and do not consider the future trajectory prediction of dynamic targets, making them prone to problems such as tracking lag, control oscillations, and incompatibility with target acceleration and sharp turns. For example, CN113232016A proposes a path planning method for a robotic arm based on the fusion of reinforcement learning and fuzzy obstacle avoidance. It proposes to combine reinforcement learning algorithms with fuzzy obstacle avoidance methods. When the distance between the robotic arm and the obstacle is less than a certain threshold, fuzzy control is used for obstacle avoidance; when the distance is greater than the threshold, a reinforcement learning algorithm based on DDPG+HER is used for path planning. The overall goal is to generate a collision-free path for the robotic arm, improving the obstacle avoidance capability of redundant robotic arms in three-dimensional space. However, its problems are that it is only for static / quasi-static obstacle scenarios. The method uses the "robotic arm-obstacle distance threshold" as the switching condition and assumes that the obstacle position is relatively stable during the planning process. It mainly focuses on obstacle avoidance in static environments and does not conduct system modeling and experimental verification for dynamic obstacles or rapidly changing environments. Moreover, its goal is to "plan a collision-free path", which is more of a one-time or segmented path planning problem. It does not involve the continuous tracking and grasping of moving targets, and the future motion information of the target is not introduced into the state space.

[0007] In summary, existing robotic arms exhibit the following shortcomings when performing target tracking and obstacle avoidance tasks in dynamic environments: First, control methods based solely on the current target position suffer from significant lag and jitter. Second, traditional visual servoing or trajectory planning methods mostly utilize only the target's pose information in the current frame, ignoring the target's motion trend and future state. This leads to problems such as tracking lag and large end-effector swing when facing fast-moving or suddenly changing targets, affecting motion efficiency. Third, there is a lack of robust obstacle avoidance capabilities for unknown or dynamic obstacles. Existing obstacle avoidance methods largely rely on pre-built environmental models or static obstacle information. For temporarily appearing or constantly changing obstacles (such as people, moving vehicles, or randomly placed objects), path replanning is often required, resulting in high computational time and sensitivity to sensor noise, making it difficult to meet real-time and safety requirements. Finally, it is difficult to comprehensively balance tracking accuracy, obstacle avoidance performance, and motion smoothness in a high-dimensional motion space. Robotic arms typically have more than 6 degrees of freedom. In a high-dimensional motion space, traditional control or simple reward design can often only optimize a single objective, such as optimizing only the end-effector tracking error or only the obstacle avoidance distance. It is impossible to achieve a balance between tracking accuracy, obstacle avoidance safety, motion smoothness, and joint safety margin, resulting in low training efficiency and unstable strategies. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings and defects of the prior art and to provide a system and method for dynamic tracking and obstacle avoidance of a robotic arm based on reinforcement learning.

[0009] One objective of this invention is to provide a system for dynamic tracking and obstacle avoidance of a robotic arm based on reinforcement learning, comprising a perception module, a target motion prediction module, a reinforcement learning decision module, a safety control and constraint module, and an execution module. The perception module acquires dynamic target information, obstacle information, and robotic arm state information within the robotic arm's workspace, converting unstructured sensor data into structured environmental state vectors. The target motion prediction module predicts the target's trajectory over a short period based on its historical motion information. The reinforcement learning decision module generates robotic arm joint control motion vectors based on the current environmental state vector and the predicted target trajectory, enabling tracking of the dynamic target and obstacle avoidance. The safety control and constraint module performs safety judgment and correction on the robotic arm joint control motion vectors output by the reinforcement learning decision module through range and collision prediction detection, outputting safe motion control vectors to ensure that the executed actions meet preset safety constraints and avoid potential collisions and over-limit behaviors. The execution module converts the safe motion control vectors output by the safety control and constraint module into robotic arm drive signals and outputs them to a servo system to control the robotic arm's movement.

[0010] Preferably, the perception module includes a visual acquisition and target detection unit, a point cloud acquisition and obstacle feature extraction unit, and a robotic arm body state acquisition unit; the visual acquisition and target detection unit includes an RGB-D camera and a target detection network: the RGB-D camera is mounted above the robotic arm's workspace to acquire color images and depth maps; the target detection network uses a pre-trained YOLO target detection network to infer the RGB images acquired by the RGB-D camera, outputting the target's two-dimensional bounding box, category, and confidence score; based on the obtained two-dimensional pixel coordinates of the target, the depth value of the bounding box region is extracted from the corresponding depth map, filtered, and averaged to obtain the target's three-dimensional coordinates (x, y, z) in the camera coordinate system; through camera calibration parameters, the three-dimensional coordinates are transformed from the camera coordinate system to the robotic arm base coordinate system to obtain the target's absolute position p in the robotic arm's workspace. target,t .

[0011] Preferably, the point cloud acquisition and obstacle feature extraction unit includes a structured light depth sensor and a point cloud feature extraction network. The structured light depth sensor acquires three-dimensional point cloud data within the robotic arm's workspace, performs voxel mesh filtering on the original three-dimensional point cloud data, performs statistical filtering on the filtered point cloud to remove outliers and noise points, obtains a smooth point cloud, uses Euclidean clustering to cluster and segment the point cloud data, divides the point clouds of different obstacles, inputs each obstacle point cloud block into the point cloud feature extraction network to extract its high-dimensional feature representation, and concatenates the high-dimensional feature representations of each obstacle to form an obstacle encoding feature vector F. obs,t .

[0012] Preferably, the robotic arm body state acquisition unit includes a robotic arm controller, which reads the joint angles q of the robotic arm in real time. t and joint angular velocity The current pose p of the robotic arm's end effector is calculated using a forward kinematics model. ee,t .

[0013] Preferably, the safety control and constraint module controls the joint increment Δq of the robotic arm. i Perform range detection when the joint increment Δq i Exceeding the preset maximum allowable range Δq max At this time, a clipping function is used to restrict the overall motion vector of the robotic arm and constrain its norm to prevent excessive single-cycle motion. The clipping function is as follows:

[0014] Δq′ i =clip(Δq) i ,-Δa max ,Δa max ).

[0015] Preferably, the safety control and constraint module performs collision prediction and detection, including using the current joint angle α of the robotic arm. t With the initial action vector A t Predicting joint angles after the action is performed Based on predicted joint angles Calculate the key set L of each link of the robotic arm; calculate the minimum distance d from the key set L of the robotic arm to the obstacle set based on the current obstacle point cloud and distance field. min,t+1 If d min,t+1 Less than the preset safe distance threshold d safe If the action is deemed to have a potential collision risk, it is entered into the action safety projection. When the action is determined to have a collision risk or exceed the safe operation limits, an action safety feasible domain is constructed:

[0016]

[0017] Among them, C safe Let represent the safe and feasible region of the action, 'a' represent the candidate motion vectors of the robotic arm joints, and 'd(a)' represent the minimum distance from each key point of the robotic arm link to the set of obstacles after executing the candidate motion vectors of the robotic arm joints; i The motion value of the i-th joint in candidate motion vector a;

[0018] The optimization problem is solved using the following formula to obtain the optimized motion vector, i.e., the incremental joint command A′. t As a replacement for the final safety action vector:

[0019]

[0020] Preferably, the execution module will increment the joint command A′ t The target joint position is converted, and a continuous control trajectory is generated through an interpolation algorithm to control the smooth movement of the robotic arm.

[0021] Preferably, the reinforcement learning decision module sets a reward function to calculate a reward for each time step; including tracking error reward, obstacle avoidance reward, smoothing reward, and safety reward, each reward is assigned a corresponding weight, and their weighted sum constitutes the reward function of this application.

[0022] Preferably, the tracking error reward is R. track =-||p ee,t -p target,t ||;

[0023] The obstacle avoidance reward is λ represents the obstacle avoidance penalty coefficient.

[0024] The smoothing reward is R. smooth =-η||A t -At-1 ||, where η represents the smoothing penalty coefficient.

[0025] The security reward is γ represents the safety penalty coefficient. This represents the real-time angular velocity of the i-th joint;

[0026] The reward function is R = αR track +βR obs +γR smooth +δR safe

[0027] Where α, β, γ, and δ represent the corresponding weights.

[0028] Another objective of this invention is to provide a reinforcement learning-based method for dynamic tracking and obstacle avoidance of a robotic arm, comprising the following steps:

[0029] The sensing module acquires dynamic target information, obstacle information, and robotic arm body state information within the robotic arm's workspace, and converts unstructured sensing data into structured environmental state vectors.

[0030] The target motion prediction module predicts the target's trajectory over a short period of time based on the target's historical motion information.

[0031] The reinforcement learning decision-making module generates joint control motion vectors for the robotic arm based on the current environmental state vector and the predicted target motion trajectory, thereby enabling the tracking of dynamic targets and the avoidance of obstacles.

[0032] The safety control and constraint module performs safety judgment and correction on the robotic arm joint control motion vector output by the reinforcement learning decision module through range and collision prediction detection, and outputs a safe motion control vector to ensure that the executed action meets the preset safety constraints and avoid potential collisions and over-limit behaviors.

[0033] The execution module converts the safety action control vector output by the safety control and constraint module into a robotic arm drive signal and outputs it to the servo system to control the movement of the robotic arm.

[0034] This invention application constructs a complete system that includes environmental perception, target motion prediction, reinforcement learning decision-making, and execution control, enabling robotic arms to predictively track moving targets and avoid static or dynamic obstacles in real time in unknown or dynamic environments, while meeting the safety, robustness, and real-time requirements of industrial sites throughout the entire control process.

[0035] This invention employs a deep reinforcement learning algorithm to automatically learn the control strategy of the robotic arm through interaction with the environment. It does not rely on a precise dynamic model and exhibits better adaptability to complex and variable environments. Furthermore, this invention inputs the output of the target future trajectory prediction module along with the environmental state into the reinforcement learning decision network, achieving "prediction-enhanced RL." This allows the robotic arm to plan its actions in advance based on the target's future position, reducing response lag and improving dynamic tracking accuracy.

[0036] This invention application targets the joint task of "dynamic tracking + obstacle avoidance". It designs a target motion prediction module and uses the future trajectory of the target as the input for reinforcement learning, enabling the robotic arm to plan its actions in advance and achieve smooth tracking of high-speed moving targets. Furthermore, it constructs a multi-dimensional composite reward function that includes tracking accuracy reward, obstacle avoidance penalty, motion smoothness reward, and safety constraint reward. This strategy ensures obstacle avoidance and efficiency while taking into account motion smoothness and safety, which is more in line with the application requirements of industrial robots. Attached Figure Description

[0037] Figure 1 This is a diagram showing the overall hardware composition of the reinforcement learning-based robotic arm dynamic tracking and obstacle avoidance system of the present invention.

[0038] Figure 2 This is a diagram showing the overall module structure of the system software for dynamic tracking and obstacle avoidance of a robotic arm based on reinforcement learning, as described in this invention.

[0039] Figure 3 This is a flowchart of the reinforcement learning-based robotic arm dynamic tracking and obstacle avoidance method of the present invention.

[0040] Figure 4 This is a data flow diagram of the perception layer of the system for dynamic tracking and obstacle avoidance of a robotic arm based on reinforcement learning, as presented in this invention.

[0041] Figure 5 This is a structural diagram of the LSTM target trajectory prediction module of the reinforcement learning-based robotic arm dynamic tracking and obstacle avoidance system of the present invention.

[0042] Figure 6 The reinforcement learning network strategy structure of the TD3 reinforcement learning decision module of the reinforcement learning-based robotic arm dynamic tracking and obstacle avoidance system of the present invention is shown.

[0043] Figure 7 This is a flowchart of the motion safety control module of the system for dynamic tracking and obstacle avoidance of a robotic arm based on reinforcement learning, according to the present invention. Detailed Implementation

[0044] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0045] In an exemplary embodiment of this application, the reinforcement learning-based robotic arm dynamic tracking and obstacle avoidance system includes: a perception module 10 (or perception processing module), a target motion prediction module 20 (or LSTM target prediction module, LSTM trajectory prediction module), a reinforcement learning decision module 30 (or TD3 decision module, TD3 decision generation module), a safety control and constraint module 40 (or safety control module), and an execution module 50; the perception module 10 is used to acquire dynamic target information, obstacle information, and robotic arm body state information within the robotic arm's workspace, and convert unstructured sensor data into structured environmental state vectors for use by subsequent modules; the target motion prediction module 20 is used to predict the target's historical motion... The information predicts the target's trajectory over a short period of time (i.e., the target's future trajectory) so that the reinforcement learning strategy can consider the target's movement trend when making decisions, reducing tracking lag. The reinforcement learning decision module 30 generates robotic arm joint control motion vectors based on the current environment state vector and the target's future trajectory, enabling tracking of dynamic targets and obstacle avoidance. The safety control and constraint module 40 performs safety checks and corrections on the robotic arm joint control motion vectors output by the reinforcement learning decision module, outputting safe motion vectors to ensure that the executed actions meet various safety constraints and avoid potential collisions and over-limit behaviors. The execution module 50 is used to process the safe motion vectors output by the safety control and constraint module (such as the optimized motion vector A′ mentioned later). t The signal is converted into a drive signal for the robotic arm and output to the servo system to control the movement of the robotic arm.

[0046] For example, in this application, each of the above modules can be implemented by one or more industrial computers, embedded controllers and external sensors. The modules communicate with each other via industrial Ethernet, and the operating cycle is preferably 50Hz to meet the real-time control requirements in dynamic environments.

[0047] In this application, for example, the perception module 10 includes a visual acquisition and target detection unit, a point cloud acquisition and obstacle feature extraction unit, and a robotic arm body state acquisition unit.

[0048] For example, the visual acquisition and target detection unit includes an RGB-D camera and a target detection network: wherein, the RGB-D camera is mounted above the workspace of the robotic arm to acquire color images and depth maps, and then performs visual preprocessing, target detection and 3D positioning to obtain perception-related information; for example, the working frame rate of the RGB-D camera is preferably set to 30 frames / second.

[0049] For example, the target detection network uses a pre-trained YOLO target detection network to infer the RGB images captured by the RGB-D camera, outputting the target's two-dimensional bounding box, category, and confidence score. Based on the obtained two-dimensional pixel coordinates of the target, the depth values ​​of the bounding box region are extracted from the corresponding depth map, filtered, and averaged to obtain the target's three-dimensional coordinates (x, y, z) in the camera coordinate system. Through camera calibration parameters, the three-dimensional coordinates are transformed from the camera coordinate system to the robot arm base coordinate system to obtain the target's absolute position p in the robot arm's workspace. target,t .

[0050] In this application, exemplarily, the point cloud acquisition and obstacle feature extraction unit includes a structured light depth sensor (such as a lidar or depth module) and a point cloud feature extraction network. The structured light depth sensor acquires three-dimensional point cloud data within the robotic arm's workspace. The original three-dimensional point cloud data is then filtered using a voxel grid to reduce the data volume; the filtering resolution can be set to 0.03m. Statistical filtering is performed on the filtered point cloud to remove outliers and noise points, resulting in a smooth point cloud. Euclidean clustering is then used to cluster and segment the point cloud data, dividing it into different obstacle point clouds. Each obstacle point cloud block is input into the point cloud feature extraction network to extract its high-dimensional feature representation. The high-dimensional feature representations of each obstacle are then concatenated to form an obstacle encoding feature vector F. obs,t .

[0051] In terms of hardware, the RGB-D camera and structured light depth sensor serve as the perception layer, while the perception module, target motion prediction module, reinforcement learning decision-making module, and safety control and constraint module form the computation and control layer. Together with the execution module, which serves as the execution layer, they constitute the system hardware structure. The computation and control processing layer and the execution layer communicate via wired methods such as EtherCAT or TCP / IP.

[0052] In this application, for example, the robotic arm body state acquisition unit includes a robotic arm controller and sensors connected to the robotic arm controller, such as a torque sensor to read the angular velocity of the robotic arm joints, and the robotic arm controller to read the angle q of each joint of the robotic arm acquired by the sensors in real time. t and joint angular velocity The current pose p of the robotic arm's end effector is calculated using a forward kinematics model (such as the MujocoXM model). ee,t .

[0053] For example, the safety control and constraint module 40 performs motion boundary checks, including checking the increment Δq of each joint. i Perform a range check; if it exceeds the preset maximum allowable range Δq max At that time, restrictions are imposed using a clipping function:

[0054] Δq′ i =clip(Δq) i ,-Δq max ,Δq max (1)

[0055] Similarly, the norm of the overall motion vector of the robotic arm is constrained to prevent excessively large single-cycle movements. Where Δq i For the joint vector before clipping, Δq i ′ represents the joint vector after clipping;

[0056] The safety control and constraint module 40 also needs to perform collision prediction checks, including using the current joint angle q of the robotic arm. t With action vector A t Predicting joint angles after the action is performed Based on predicted joint angles Calculate the key set L of each link of the robotic arm; calculate the minimum distance d from the key set L of the robotic arm to the obstacle set based on the current obstacle point cloud and distance field. min,t+1 If d min,t+1 Less than the preset safe distance threshold d safe If an action is deemed to pose a potential collision risk, it enters the action safety projection. The action safety projection primarily constructs a safe and feasible domain for an action when it is determined to have a collision risk or exceed safe operating limits.

[0057]

[0058] Among them, C safe Let represent the safe and feasible region of the action, 'a' represent the candidate motion vectors of the robotic arm joints, and 'd(a)' represent the minimum distance from each key point of the robotic arm link to the set of obstacles after executing the candidate motion vectors of the robotic arm joints; i This represents the action value of the i-th joint in the candidate action vector a;

[0059] Solve the optimization problem to obtain the optimized motion vector, which is then used as the incremental joint command:

[0060]

[0061] The above optimization can be achieved by making a linear approximation near the boundary and using an analytical projection method for fast solution, thus obtaining A′. t This is the final safety action, used to replace the original action A. t .

[0062] For example, in execution module 50, the incremental joint command A′ is... tThe target joint position is converted, and a continuous control trajectory is generated through an interpolation algorithm to ensure smooth robotic arm movement. The execution module, as the execution layer, includes servo drivers, robot controllers, and end effectors.

[0063] In one embodiment, a training and simulation module 60 is also provided for training and simulating the reinforcement learning decision module 30 to obtain the network model of the reinforcement learning decision module 30. For the specific network processing of the training and simulation module, please refer to the relevant content of step S3 of the reinforcement learning-based robotic arm dynamic tracking and obstacle avoidance method described later.

[0064] This application also provides a reinforcement learning-based method for dynamic tracking and obstacle avoidance of a robotic arm, which can be implemented in a host computer via software. The steps include:

[0065] Step S1: Environmental Status Acquisition

[0066] The sensing module 10 collects images, depth images, point clouds, and the state of the robotic arm itself within the workspace. Specifically, this includes: acquiring color images and depth information via an RGB-D camera; acquiring obstacle point clouds via LiDAR or a depth module; and obtaining the robotic arm joint angles q based on the robotic arm controller. t Joint angular velocity The current pose p of the end effector of the robotic arm is obtained using the MujocoXM model. ee,t The above information is then converted into an environmental state vector S. t ;

[0067]

[0068] Where, p target,t ,F obs,t ,I sensor,t These represent the absolute position of the target in the workspace of the robotic arm, the high-dimensional feature representations of each obstacle assembled into an obstacle encoding feature vector, and the sensor's own state data.

[0069] Step S2: Target trajectory prediction

[0070] Using the target motion prediction module 20, based on the input historical target position sequence X t Using time series prediction models, such as LSTM, the trajectory of a target is predicted at several future time steps, thus obtaining the target's future trajectory vector. The prediction of the target's future trajectory mainly includes the following steps:

[0071] Step B1: From the environment state vector S tExtract the target positions from the most recent k frames, typically k=7, which corresponds to the motion trajectory of the most recent 7 control cycles, and perform historical trajectory extraction and caching.

[0072] X t ={p target,t-k ,…,p target,t}(5)

[0073] Step B2: Transfer the historical trajectory sequence X t Converted to a fixed-length time series, as follows:

[0074] X t =[f t-k+1 ,f t-k+2 ,…,f t ],X t ∈R k×d (6)

[0075] Among them, f t =[x t ,y t ,z t ,v x,t ,v t,t ,v z,t ,a x,t ,a y,t ,a z,t ];

[0076] Among them, position, velocity, and acceleration satisfy v x,t =x t -x t-1 a x,t =v x,t -v x,t-1 ;

[0077] This allows for the construction of a fixed-length target historical trajectory time series that meets the input requirements of the prediction model.

[0078] Step B3: Using an improved LSTM time series prediction model, the spatial trajectory of the target is predicted for a future period of time based on the target's historical trajectory time series, thus obtaining the target's predicted motion trajectory.

[0079] In this embodiment of the application, for example, the step of predicting the spatial trajectory of a target over a future period using an improved LSTM time series prediction model includes the following steps:

[0080] Step B31: Construct multi-dimensional input features, i.e., construct a single-step time feature vector f t :

[0081] f t =[x t ,y t,z t ,v x,t ,v y,t ,v z,t ,a x,t ,a y,t ,a z,t (7)

[0082] Among them, (x t ,y t ,z t (v) represents the three-dimensional position of the target at time t; x,t ,v y,t ,v z,t The velocity of the target at time t is calculated from its historical position using finite difference; (a x,t ,a y,t ,a z,t ) represents the acceleration of the target at time t, estimated by velocity difference.

[0083] For the above single-step time feature vector f t Reconstructing, we obtain the input sequence of time feature vectors of length k:

[0084] X t =[f t-k+1 ,f t-k+2 ,…,f t (8)

[0085] In conventional methods, time series prediction typically uses only the target's historical position (x, y, z) as input. In this embodiment, multi-dimensional features such as position, velocity, and acceleration are introduced to construct a single-step time feature vector f. t It can improve the ability to depict complex movements such as violent acceleration, sudden stop, and turning:

[0086] Step B32: Use an LSTM network to process the time feature vector input sequence X. t Feature extraction is performed to obtain the hidden state {h} at each time step. t-k+1 ,...,h t};

[0087] For example, in this application, the LSTM network structure uses a stacked LSTM network as the core for time series modeling, and its structure includes:

[0088] The first LSTM layer has the same input dimension as the aforementioned feature dimension and 128 hidden units. The second LSTM layer has 64 hidden units and is used to further abstract the hidden state of the previous layer. A Bi-LSTM structure is used to enhance the representation ability of long-short-term dependencies. Finally, a network layer maps the hidden state of the LSTM at the last moment to a future trajectory prediction vector.

[0089] At each time step τ, the LSTM unit update formula is:

[0090]

[0091] Among them, i τ f τ o τ These are the input gate, forget gate, and output gate, respectively. τ In cellular state, h τ It is in a hidden state.

[0092] Step B33: Utilize the interstitial force mechanism to analyze the hidden state {h} at each time step. t-k+1 ,...,h t Different weights are assigned, and historical information is aggregated to obtain a weighted context vector;

[0093] Considering that the recent motion state usually reflects the current motion trend better than the historical state at a more distant time, in order to further improve the prediction accuracy, this embodiment may optionally introduce a time attention mechanism into the LSTM output to assign different weights to the hidden states at different time steps.

[0094] Specifically, after obtaining the hidden state {h} at each time step t-k+1 ,...,h t After that, a set of learnable attention weights {α} are introduced. t-k+1 ,...,α t The calculation method is as follows:

[0095]

[0096] Among them, e τ e represents the attention energy at time step τ. μ These are all the attention energy values ​​used for weighting during the entire softmax normalization process; thus, a weighted context vector (temporal representation vector) is obtained:

[0097]

[0098] Then the weighted context vector The input is fed into a subsequent fully connected network (FC) to generate prediction results.

[0099] Step B34: Output of future trajectory prediction and multi-step trajectory prediction results

[0100] The final time representation vector obtained through steps B32 and B33 Input to a top-level fully connected network, output the position information for the next k' time steps. That is, predicting the trajectory;

[0101]

[0102] Each of them Let be the predicted position at time t+j.

[0103] For example, the output layer of the top-level fully connected network is designed to directly output a vector of length 3k′, and sequentially split it into positions for each future time step; the predicted k′ ​​time steps are coordinated with the control period to correspond to a future time window of 300ms.

[0104] Step B36: Interface output with the reinforcement learning decision module

[0105] In this embodiment of the application, step B3 involves the predicted trajectory output by the target motion prediction module. As data, it is incorporated into subsequent reinforcement learning strategy decision-making methods.

[0106] Step S3: Reinforcement Learning Decision:

[0107] Reinforcement learning observation construction and action generation

[0108] The predicted trajectory output in step B3 With the environment state vector S t Obstacle encoding feature vector F obs,t Together they form the observation vector O t As input to the reinforcement learning decision-making module, the reinforcement learning policy network makes decisions using "current state + future trajectory" rather than relying solely on the current position, thus exhibiting stronger foresight and control smoothness in dynamic tracking and obstacle avoidance tasks.

[0109] The reinforcement learning strategy exemplified in this application was trained in the Mujoco simulation environment.

[0110] First, the environment state vector S t With predicted trajectory Obstacle encoding feature vector F obs,t Fusion, constructing observation vector O t As the input vector for the reinforcement learning decision module, it is as follows:

[0111]

[0112] Specifically, the environment state vector S t Predicted trajectory Directly concatenate the points cloud obstacles to encode the feature vector F. obs After dimensionality reduction by MLP, the parts are stitched together.

[0113] The reinforcement learning decision-making module obtains action A through policy network inference. t The constructed observation vector O t The input is the Actor network, which is a three-layer fully connected MLP with 256-256-128 neurons in each layer and ReLU activation function. The output dimension is the incremental motion of each joint of the robotic arm.

[0114] A t =[Δq1,Δq2,…,Δq n (13)

[0115] A reward function is set up, and the strategy is run in the Mujoco environment to calculate the reward for each time step. The reward includes tracking error reward, obstacle avoidance reward, smoothing reward, and safety reward. Each reward is assigned a corresponding weight, and the weighted sum of these rewards constitutes the reward function of this application.

[0116] The tracking error reward is as follows:

[0117] R track =-||p ee,t -p target,t ||(14)

[0118] The obstacle avoidance reward is calculated based on the Mujoco collision detection distance field as follows:

[0119]

[0120] λ represents the obstacle avoidance penalty coefficient; d min This represents the minimum distance from a robotic arm joint to an obstacle.

[0121] The smoothing reward is as follows:

[0122] R smooth =-η||A t -A t-1 ||(16)

[0123] η represents the smoothing penalty coefficient, A t f t-1 Represents the joint control motion vectors at times t and t-1;

[0124] The security rewards are as follows:

[0125]

[0126] γ represents the safety penalty coefficient. v represents the real-time angular velocity of the i-th joint; max This indicates the maximum permissible angular velocity.

[0127] The sum of the reward functions is as follows:

[0128] R = αR track +βR obs +γR smooth +δR safe (18)

[0129] Wherein, α, β, γ, and δ represent the corresponding weights, that is, α represents the weight of the tracking error reward; β represents the weight of the obstacle avoidance reward; γ represents the weight of the motion smoothness reward; and δ represents the weight of the safety constraint reward.

[0130] In this application, by way of example, step S3, the training and simulation processing steps of the TD3 algorithm of the reinforcement learning decision module specifically include the following:

[0131] Step C1: Initialize parameters

[0132] Randomly initialize the joint states of the robotic arm and the initial position of the target p target,0 This serves as the initial training state; and initializes the policy network parameters θ in the TD3 algorithm. π Two sets of Q-value network parameters and the corresponding target network parameters

[0133] Step C2: The core network structure of the TD3 algorithm includes:

[0134] (1) Dual-Q network: employing two independent Q value networks and Used to mitigate the overestimation bias of Q value;

[0135] (2) Delayed policy update: The policy network updates less frequently than the Q network, allowing the Q network to converge first and then adjust the policy, thus improving training stability.

[0136] (3) Target policy smoothing: When calculating the target Q value, add a small Gaussian noise to the target policy action and trim it to a specified range to reduce the overfitting problem caused by the policy being too sharp.

[0137] The above structure, combined with the deterministic policy gradient in the continuous action space, makes training the robotic arm's dynamic tracking and obstacle avoidance strategies in Mujoco more stable and reliable.

[0138] Step C3: Construction of Interactive Sampling and Experience Replay Buffer

[0139] Step C3-1: In each training cycle, based on the current state s of the Mujoco environment... t (i.e., the aforementioned observation vector O) t The policy network outputs the current deterministic action.

[0140] Step C3-2: To facilitate exploration, during the training phase, guide action a... t Adding zero-mean Gaussian noise (∈) yields the actual action executed:

[0141]

[0142] And on Perform trimming to ensure it does not exceed the action boundaries;

[0143] Step C3-3: Perform the action Input the Mujoco simulation environment to perform a simulation step and obtain the next state s. t+1 Instant rewards t And a marker indicating whether the process has ended;

[0144] Step C3-4: Combine the quadruple (s) t ,a t ,r t ,s t+1 (done) Store in the experience replay buffer This is used for subsequent batch training.

[0145] Step C4: Batch Sampling and Target Action Generation

[0146] Step C4-1: From the experience replay buffer A batch of empirical samples was randomly and uniformly sampled.

[0147] (s i ,a i ,r i ,s′ i ,done i ), i=1,…,N (20)

[0148] Step C4-2: Utilize the target policy network Calculate the next state s′ i Corresponding target action:

[0149]

[0150] Step C4-3: To achieve smooth target policy, add a small amount of noise to the target action and then trim it.

[0151]

[0152] Where, σ target Let a be the noise intensity of the target strategy. min ,a max This represents the upper and lower bounds of the action.

[0153] Step C5: Calculation of target value for dual-Q network

[0154] Step C5-1: Utilize the target Q-network respectively and Calculate the Q-value corresponding to the target action:

[0155]

[0156] Step C5-2: Take the smaller of the two values ​​as the target Q value benchmark for TD3 to reduce overestimation.

[0157]

[0158] Step C5-3: Construct the TD objective:

[0159]

[0160] Where γ is the discount factor.

[0161] Step C6: Update Dual-Q Network Parameters

[0162] Minimize the mean squared error loss function for each of the two Q-networks:

[0163]

[0164] Through backpropagation and gradient update This allows the dual-Q network to approach the target Q value.

[0165] Step C7: Delay Policy Network Update

[0166] To improve training stability, TD3 employs a "policy delayed update mechanism," meaning that each Q-network update is counted as one step, and a policy network update is only performed every d steps. When the update conditions are met, the following steps are executed:

[0167] Step C7-1, utilize the current Q network Evaluate the value of the actions output by the strategy and construct the strategy objective:

[0168]

[0169] Step C7-2: Minimize the above loss using gradient descent and update the policy network parameters θ. π .

[0170] Step C8: Target network soft update

[0171] After updating the policy network and Q network, perform a soft update on the target network parameters:

[0172]

[0173] Where τ∈(0,1) is the soft update coefficient, which is set to 0.008 to ensure smooth changes in the target network and improve training stability.

[0174] Step C9: Training Termination Conditions and Strategy Consolidation

[0175] By iteratively executing steps C3 to C8 multiple times, the TD3 policy training can be determined to have converged or achieved the expected results when one of the following conditions is met:

[0176] (1) The average return value reaches the preset threshold within a certain number of evaluation rounds;

[0177] (2) The success rate of tracking and obstacle avoidance in the scenario meets the requirements;

[0178] (3) The number of training rounds or simulation steps reaches the preset upper limit.

[0179] At this point, the trained policy network parameters θ π The model is solidified and exported as a deployable model for real-time execution of the strategy reasoning process of step S3 in an actual robotic arm control system.

[0180] Step S4: Safety Check and Action Correction

[0181] Motion boundary constraints: Incremental motion A of the robotic arm joints t Each joint movement in the code is trimmed to ensure it does not exceed joint limits:

[0182] A t ′(i)=clip(A t (i),-Δq max ,Δq max (29)

[0183] Based on Mujoco collision prediction, and utilizing the Geom-Geom distance calculation function provided by Mujoco, the execution action A is predicted. t Minimum distance d after ′ min , such as d min <d safe If a collision risk is identified, a safe action E is generated by projecting the action using gradient descent, and the following optimization is performed:

[0184]

[0185] Step S5: Action Execution

[0186] The safety action G is sent to the robotic arm controller for execution via the execution module. During the execution, feedback data is collected and the robotic arm status is updated to provide input for the next control cycle.

[0187] Step S6: Run in a loop

[0188] Repeat steps S1 to S5 until the predetermined tracking and operation task is completed.

[0189] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention.

[0190] Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of the equivalents of the claims be included within the invention.

[0191] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A dynamic tracking and obstacle avoidance system for a robotic arm based on reinforcement learning, characterized in that, include: The system comprises a perception module, a target motion prediction module, a reinforcement learning decision-making module, a safety control and constraint module, and an execution module. The perception module acquires dynamic target information, obstacle information, and the robot arm's own state information within the workspace, converting unstructured sensor data into structured environmental state vectors. The target motion prediction module predicts the target's trajectory over a short period based on its historical motion information. The reinforcement learning decision-making module generates joint control motion vectors for the robot arm based on the current environmental state vector and the predicted target trajectory, enabling tracking of dynamic targets and obstacle avoidance. The safety control and constraint module performs safety checks and corrections on the joint control motion vectors output by the reinforcement learning decision-making module, using range and collision prediction detection, and outputs safe motion control vectors to ensure that the executed actions meet preset safety constraints, avoiding potential collisions and over-limit behaviors. The execution module converts the safe motion control vectors output by the safety control and constraint module into robot arm drive signals and outputs them to the servo system to control the robot arm's movement.

2. The reinforcement learning-based robotic arm dynamic tracking and obstacle avoidance system according to claim 1, characterized in that, The perception module includes a visual acquisition and target detection unit, a point cloud acquisition and obstacle feature extraction unit, and a robotic arm body state acquisition unit. The visual acquisition and target detection unit includes an RGB-D camera and a target detection network: the RGB-D camera is mounted above the robotic arm's workspace and is used to acquire color images and depth maps; the target detection network uses a pre-trained YOLO target detection network to infer the RGB images acquired by the RGB-D camera and output the target's two-dimensional bounding box, category, and confidence score. Based on the obtained 2D pixel coordinates of the target, the depth values ​​of the bounding box region are extracted from the corresponding depth map, filtered, and averaged to obtain the 3D coordinates (x, y, z) of the target in the camera coordinate system. Using camera calibration parameters, the 3D coordinates are transformed from the camera coordinate system to the robot arm base coordinate system to obtain the absolute position p of the target in the robot arm's workspace. target,t .

3. The reinforcement learning-based robotic arm dynamic tracking and obstacle avoidance system according to claim 2, characterized in that, The point cloud acquisition and obstacle feature extraction unit includes a structured light depth sensor and a point cloud feature extraction network. The structured light depth sensor acquires three-dimensional point cloud data in the workspace of the robotic arm and performs voxel mesh filtering on the original three-dimensional point cloud data. Statistical filtering is applied to the filtered point cloud to remove outliers and noise points, resulting in a smooth point cloud. Euclidean clustering is then used to segment the point cloud data, dividing it into different obstacle point clouds. Each obstacle point cloud block is input into a point cloud feature extraction network to extract its high-dimensional feature representation. The high-dimensional feature representations of each obstacle are then concatenated to form the obstacle encoding feature vector F. obs,t .

4. The reinforcement learning-based robotic arm dynamic tracking and obstacle avoidance system according to claim 3, characterized in that, The robotic arm body state acquisition unit includes a robotic arm controller, which reads the joint angles q of the robotic arm in real time. t and joint angular velocity The current pose p of the robotic arm's end effector is calculated using a forward kinematics model. ee,t .

5. The reinforcement learning-based robotic arm dynamic tracking and obstacle avoidance system according to claim 4, characterized in that, The safety control and constraint module controls the joint increment Δq of the robotic arm. I Perform range detection when the joint increment Δq i Exceeding the preset maximum allowable range Δq max At this time, a clipping function is used to restrict the overall motion vector of the robotic arm and constrain its norm to prevent excessive single-cycle motion. The clipping function is as follows: Δq′ i =clip(Δq I ,-Δq max ,Δq max )。 6. The reinforcement learning-based robotic arm dynamic tracking and obstacle avoidance system according to claim 5, characterized in that, The safety control and constraint module performs collision prediction and detection, including using the current joint angle q of the robotic arm. t With the initial action vector A t Predicting joint angles after the action is performed Based on predicted joint angles Calculate the key set L of each link of the robotic arm; calculate the minimum distance d from the key set L of the robotic arm to the obstacle set based on the current obstacle point cloud and distance field. min,t+1 If d min,t+1 Less than the preset safe distance threshold d safe If the action is deemed to have a potential collision risk, it is entered into the action safety projection. When the action is determined to have a collision risk or exceed the safe operation limits, an action safety feasible domain is constructed: Among them, C safe Let represent the safe and feasible region of the action, 'a' represent the candidate motion vectors of the robotic arm joints, and 'd(a)' represent the minimum distance from each key point of the robotic arm link to the set of obstacles after executing the candidate motion vectors of the robotic arm joints; i The motion value of the i-th joint in candidate motion vector a; The optimization problem is solved using the following formula to obtain the optimized motion vector, i.e., the incremental joint command A. ′ t As a replacement for the final safety action vector:

7. The reinforcement learning-based robotic arm dynamic tracking and obstacle avoidance system according to claim 6, characterized in that, The execution module will increment the joint command A′ t The target joint position is converted, and a continuous control trajectory is generated through an interpolation algorithm to control the smooth movement of the robotic arm.

8. The reinforcement learning-based robotic arm dynamic tracking and obstacle avoidance system according to claim 1, characterized in that, The reinforcement learning decision module sets a reward function to calculate a reward for each time step; it includes tracking error reward, obstacle avoidance reward, smoothing reward, and safety reward, each of which is assigned a corresponding weight, and the weighted sum of these rewards constitutes the reward function of this application.

9. The reinforcement learning-based robotic arm dynamic tracking and obstacle avoidance system according to claim 8, characterized in that, The tracking error reward is R. track =-||p ee,t -p target,t ||; The obstacle avoidance reward is λ represents the obstacle avoidance penalty coefficient. The smoothing reward is R. smooth =-η||A t -A t-1 ||, where η represents the smoothing penalty coefficient. The security reward is γ represents the safety penalty coefficient. This represents the real-time angular velocity of the i-th joint; The reward function is R = αR track +βR obs +γR smooth +δR safe Where α, β, γ, and δ represent the corresponding weights.

10. The reinforcement learning-based robotic arm dynamic tracking and obstacle avoidance method according to any one of claims 1-9, characterized in that, Including the following steps: The sensing module acquires dynamic target information, obstacle information, and robotic arm body state information within the robotic arm's workspace, and converts unstructured sensing data into structured environmental state vectors. The target motion prediction module predicts the target's trajectory over a short period of time based on the target's historical motion information. The reinforcement learning decision-making module generates joint control motion vectors for the robotic arm based on the current environmental state vector and the predicted target motion trajectory, thereby enabling the tracking of dynamic targets and the avoidance of obstacles. The safety control and constraint module performs safety judgment and correction on the robotic arm joint control motion vector output by the reinforcement learning decision module through range and collision prediction detection, and outputs a safe motion control vector to ensure that the executed action meets the preset safety constraints and avoid potential collisions and over-limit behaviors. The execution module converts the safety action control vector output by the safety control and constraint module into a robotic arm drive signal and outputs it to the servo system to control the movement of the robotic arm.

Images