Double-mechanical-arm self-adaptive motor neural network optimization system

Abstract

The invention discloses a double-mechanical-arm adaptive motor neural network optimization system, and relates to the technical field of mechanical arm motion control, and the system is characterized in that a feature fusion sensing module synchronizes multi-sensor data through an FPGA, combines technologies such as YOLOv5s, Poi ntNet + + and Doppler frequency shift, achieves precise extraction and fusion of obstacle 2D / 3D features, and generates a real-time environment state matrix; the kinematic chain prediction module quantifies the obstacle motion probability based on historical trajectory clustering and Bayesian prediction, constructs a collision risk field, and dynamically evaluates the collision risk in the future 2 seconds; the planning module searches a low-risk path by adopting an improved A algorithm, optimizes a joint movement track in combination with quadratic programming, embeds kinematics and dynamics constraints, and ensures the feasibility and safety of the path; in addition, a correction mechanism and an energy consumption optimization mechanism are introduced, motion parameters are dynamically adjusted according to the real-time state of the mechanical arm and environment interference, and the obstacle avoidance performance and the operation efficiency of the mechanical arm are further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robot arm motion control, in particular to a dual robot arm adaptive motion neural network optimization system. Background Art

[0002] In the field of modern industry and automation, robots are increasingly used. Dual-arm systems, with their high efficiency and flexibility, have been widely used in many scenarios such as logistics handling, precision assembly, and complex processing. However, existing dual-arm systems have significant deficiencies in motion control.

[0003] In complex dynamic environments, multi-sensor data fusion technology is not perfect. At present, most dual-arm systems cannot accurately synchronize data from multiple sensors such as binocular cameras, millimeter-wave radars, and arm encoders, resulting in delays and deviations in information acquisition, making it difficult to accurately perceive obstacles and their states in the environment in real time, which puts the robot at risk of collision during movement and makes it impossible to ensure operational safety.

[0004] Traditional robotic arm obstacle avoidance systems have significant limitations. On the one hand, their trajectory prediction accuracy is insufficient, relying only on single-frame environmental data, and the trajectory prediction error for fast-moving obstacles exceeds 30%, making it difficult to meet the precise obstacle avoidance requirements in complex dynamic scenes. On the other hand, path planning lacks consideration of the robotic arm's dynamic constraints, resulting in delayed obstacle avoidance actions and prone to motion jitter, which seriously affects the robotic arm's operating efficiency and stability.

[0005] In order to solve the above defects, a technical solution is now provided. Summary of the invention

[0006] The purpose of the present invention is to solve the problem that the traditional robot arm obstacle avoidance system has large prediction error for the trajectory of fast-moving obstacles and the path planning does not consider the dynamic constraints, and proposes a dual-robot arm adaptive motion neural network optimization system.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] Dual-manipulator adaptive motion neural network optimization system, including:

[0009] The feature fusion perception module synchronizes multi-sensor data through a field programmable gate array, combines the target detection model, the deep learning model of point cloud data and the Doppler frequency shift to extract the 2D and 3D features of obstacles, and uses the spatiotemporal graph neural network to fuse the position and speed information to generate a real-time environment state matrix;

[0010] The motion chain prediction module quantifies the probability of obstacle movement based on historical trajectory clustering and Bayesian prediction, constructs a collision risk field, and dynamically evaluates the collision risk within the next 2 seconds;

[0011] The planning module searches for low-risk paths by improving the A* algorithm, optimizes the joint motion trajectory by combining quadratic programming, and embeds kinematic and dynamic constraints to ensure path feasibility and safety.

[0012] Furthermore, the execution process of the feature fusion perception module is as follows:

[0013] Utilize the FPGA hardware synchronization signal to connect the binocular camera, millimeter-wave radar, and robotic arm encoder respectively;

[0014] Through the hardware circuit and synchronization algorithm, attach specific timestamps to each sensor data;

[0015] Taking a certain fixed time point as the starting reference, perform time calibration on sensor data with different frequencies;

[0016] Cross-modal feature extraction: Use the YOLOv5s model to detect the 2D bounding box of obstacles for the RGB image. Input the RGB image obtained by the binocular camera into the YOLOv5s model;

[0017] The YOLOv5s model first extracts features from the image, and through the convolutional layer and pooling layer network structure, extracts feature information at different levels in the image;

[0018] In the detection head part, use the anchor box mechanism to predict the 2D bounding box position, category, and confidence of the obstacles;

[0019] Use PointNet++ to extract the 3D geometric features of obstacles for the point cloud data. After obtaining the point cloud data from the millimeter-wave radar, input it into the PointNet++ model;

[0020] The model extracts local point cloud features at different scales through sampling, grouping, and feature learning operations; first sample the original point cloud to reduce the data volume while retaining key feature points; then group the sampled points and perform feature learning on the point cloud within each group to obtain its 3D geometric features;

[0021] Calculate the radial velocity of the obstacle through the Doppler frequency shift for the millimeter-wave radar data. The millimeter-wave radar emits a continuous wave signal, and when the signal is reflected back by the obstacle, a Doppler frequency shift is generated;

[0022] According to the Doppler effect formula where f d is the Doppler frequency shift, v is the radial velocity of the obstacle, λ is the radar wavelength, f 0 is the transmitted signal frequency, perform spectral analysis on the echo signal received by the millimeter-wave radar, calculate the Doppler frequency shift, and then obtain the radial velocity of the obstacle;

[0023] Finally, the fused obstacle state matrix is ​​output through attention fusion.

[0024] Furthermore, the specific operation steps of the feature fusion perception module to output the fused obstacle state matrix through attention fusion are as follows:

[0025] Construct a spatiotemporal graph neural network, using the 2D bounding box, 3D geometric features, and radial velocity of each obstacle as node features;

[0026] For edge weight calculation between nodes, the edge weight is dynamically calculated through a function based on the relative distance and speed difference between obstacles;

[0027] The message passing mechanism of graph neural networks is used to fuse node features in the time and space dimensions, and finally the fused obstacle state matrix is ​​output, which contains position, speed, and size confidence information.

[0028] Furthermore, the specific operation steps of the kinematic chain prediction module are as follows:

[0029] Firstly, motion pattern clustering is performed based on the incremental trajectory clustering method of spatiotemporal autoencoder;

[0030] Online Bayesian prediction: real-time matching of the current obstacle motion characteristics and pattern library, and calculation of the posterior probability of each pattern;

[0031] For patterns with probability > 15%, the corresponding dynamic equation is used to generate multiple predicted trajectories within the next 2 seconds; and a set of trajectories with probability weights is output;

[0032] Collision risk field modeling: The predicted trajectory is converted into a probabilistic risk field. The field strength formula is: Where P i is the probability of trajectory i, ∑ is the expansion safety area of ​​the robot body; R i (x, t) represents the risk field strength of trajectory i at point x at time t; x is a point in space; x i (t) represents the position of trajectory i at time t; T is the transposition symbol.

[0033] Furthermore, the specific operation steps of the motion chain prediction module for performing motion pattern clustering are as follows:

[0034] The historical obstacle trajectory data is sliced, and each trajectory is intercepted with a data segment of 0.5 seconds, which contains the time series information of position, speed, and acceleration;

[0035] The coordinates are converted into a relative coordinate system with the base of the robot as the origin, and the velocity and acceleration are normalized so that the value range is limited to [-1, 1]; for each data segment, the motion state label is automatically annotated;

[0036] Construct a deep network model including encoder and decoder. The process is as follows:

[0037] Encoder: A one-dimensional convolutional neural network is used to extract the spatial features of the trajectory, with the convolution kernel size set to 5 and the number of channels set to 64;

[0038] A long short-term memory network layer is connected with 128 hidden units to capture temporal dependencies.

[0039] The encoder output is a 32-dimensional latent vector that represents the core features of the trajectory;

[0040] Decoder: reconstructs the original trajectory data through deconvolution layers and fully connected layers to ensure that the reconstruction error is minimized;

[0041] Joint optimization: During the training process, the trajectory reconstruction error and the clustering separability of the latent space are optimized simultaneously; by introducing a clustering loss function, the latent vector is forced to align with the centroid of the six preset motion modes;

[0042] Real-time detection of the matching degree between new trajectory data and existing patterns: If the similarity between the new trajectory and all existing patterns is less than 85%, the temporary storage mechanism is triggered. After the same trajectory is detected five times in a row, the pattern is automatically added to the feature library; unconfirmed temporary patterns are stored in the buffer, and after manual review, it is decided whether to keep them permanently;

[0043] For usage patterns that have not been matched for more than 72 hours, their confidence weight will be automatically reduced by 20% each week. When the weight is lower than 10%, it will be archived as a historical pattern and will no longer participate in real-time prediction.

[0044] Furthermore, the specific operation steps of the planning module are as follows:

[0045] Kinematic constraints: According to the kinematic principle of the manipulator, the feasible domain of the angular velocity-acceleration of the manipulator joints is established; the Jacobian matrix J is derived through the DH parameters of the manipulator, and the angular velocity and acceleration limits of the joint space are projected to the end space through the Jacobian matrix to obtain the feasible range of the speed and acceleration of the end effector, and then the correction mechanism is used to correct the feasible range of the speed and acceleration of the end effector;

[0046] Dynamic constraints: Using the recursive Newton-Euler algorithm, starting from the base of the robot arm, the forces and moments of each joint are calculated in sequence;

[0047] Analyze the motor torque limit and transmission mechanism load capacity of each joint of the robot arm, determine the torque limit of each joint, generate the torque feasible domain, and prevent the robot arm from damaging the equipment due to excessive torque during movement;

[0048] The hierarchical optimization path generation process is as follows:

[0049] Primary planning: Use the improved A algorithm to search for a low collision probability path in the risk field. The cost function is: Cost = α·R total +β·PathLength, where α, β are weight coefficients, R total is the total risk value on the path, PathLength is the path length, and the path risk and length are balanced by adjusting the weight coefficient; the algorithm starts from the starting point and expands the search node according to the heuristic function until the target point is found or it is determined that there is no feasible path, and the primary planning path is generated;

[0050] Secondary correction: Discretize the primary planned path into 50 waypoints. For each waypoint, treat it as a quadratic programming problem; the objective function is min||Jq-v d || 2 , where q is the joint angle, J is the Jacobian matrix, and v d is the expected terminal velocity;

[0051] The constraint condition is T min ≤M(q)+C(q,φ)≤T max , T min , T max are the upper and lower limits of the joint torque, M(q) is the joint driving torque, and C(q, φ) is the torque generated by the Coriolis force and the centrifugal force;

[0052] By solving the quadratic programming problem, the joint angle at each waypoint is optimized to make the end-of-arm movement smoother and more accurate; and the energy consumption of the joint torque is optimized through the energy consumption optimization mechanism;

[0053] Feedback adjustment: During the movement of the robot arm, the contact force information is obtained in real time through the force sensor installed on the end effector, and the end contact force is adjusted online using the impedance control algorithm;

[0054] Set the trajectory tracking error threshold. When the error between the actual trajectory and the planned trajectory exceeds the threshold, local replanning is triggered.

[0055] By utilizing the dual-constraint modeling and hierarchical optimization path generation method of the planning module, the local path can be replanned within 10ms to ensure that the robotic arm can avoid obstacles or adjust its motion trajectory in time.

[0056] Furthermore, the specific operation steps of using the correction mechanism in the planning module to correct the feasible range of the speed and acceleration of the end effector are as follows:

[0057] The correction mechanism collects the influencing parameters of the robot arm for analysis, including:

[0058] Joint torque load rate: the ratio of the actual torque of each joint to the rated torque;

[0059] Trajectory tracking error: The Euclidean distance between the actual end position and the planned position;

[0060] Instantaneous power consumption: The real-time power calculated according to the motor current;

[0061] Environmental interference intensity: The amplitude of the external contact force obtained through the end force sensor;

[0062] Health index: The comprehensive score of joint wear degree, temperature and dust concentration based on historical data;

[0063] Map all parameters to the interval of 0-1, assign weights according to the importance of the influencing parameters, and then use the formula: SZ = gl×ψ 1 +gj×ψ 2 +nh×ψ 3 +hg×ψ 4 +he×ψ 5 To obtain the corrected score SZ; where gl, gj, nh, hg, he are the mapped values of the joint torque load rate, trajectory tracking error, instantaneous power consumption, environmental interference intensity and health index respectively, and ψ 1 , ψ 2 , ψ 3 , ψ 4 , ψ 5 Are the preset weight coefficients of the joint torque load rate, trajectory tracking error, instantaneous power consumption, environmental interference intensity and health index respectively;

[0064] Then divide different risk intervals according to the corrected score SZ, which are:

[0065] Low risk, that is, SZ ≤ 0.3: Allow the end speed or acceleration to reach the theoretical maximum;

[0066] Medium risk, that is, 0.3 < SZ ≤ 0.6: Limit the motion parameters proportionally: U new = U max ×(1 - SZ), a new = a max ×(1 - SZ); where U new Is the limited end speed, U max Is the theoretical maximum end speed; a new Limited acceleration; a max Is the theoretical maximum acceleration;

[0067] High risk, that is, SZ > 0.6: Trigger a three-level response, including:

[0068] First-level response, that is, 0.6 < SZ ≤ 0.8: Reduce the speed to 50% and perform local replanning;

[0069] Secondary response, i.e., 0.8 < SZ ≤ 0.95: Pause the current action and switch to impedance control mode to absorb external force;

[0070] Tertiary response, i.e., S > 0.95: Emergency stop and lock the joints.

[0071] Furthermore, the specific operation steps of the energy consumption optimization mechanism in the planning module are as follows:

[0072] The objective function is adjusted to minimize the weighted combination of the end-effector velocity tracking error and the sum of the squares of joint torques, where the energy consumption weight coefficient is set to 0.3;

[0073] Through the optimization algorithm, on the premise of satisfying kinematic and dynamic constraints, preferentially select the joint motion scheme with lower torque consumption;

[0074] During the operation of the robotic arm, the current data of each joint motor is collected in real time to calculate the instantaneous power:

[0075] If the power of a certain joint exceeds the safety threshold for 10 consecutive control cycles, the energy consumption weight coefficient is automatically increased to 0.6, forcing the algorithm to reallocate the joint load;

[0076] Trigger the joint task migration mechanism: Transfer part of the motion tasks of the high-load joints to the low-load joints, and adjust the weights of the Jacobian matrix to make the end-effector motion of the robotic arm completed by multiple joints in cooperation;

[0077] Introduce an elastic buffer zone in the joint torque constraint condition: When the torque reaches 90% of the limit value, generate a warning signal and reduce the acceleration limit by 5% to gain time for dynamic adjustment;

[0078] If the torque continues to exceed the limit, trigger the emergency load reduction strategy: Pause the current task and perform reverse torque compensation operation.

[0079] Compared with the prior art, the beneficial effects of the present invention are:

[0080] (1) In the present invention, through the feature fusion perception module, the FPGA is used to synchronize multi-sensor data, and combined with technologies such as YOLOv5s, PointNet++, and Doppler frequency shift, the accurate extraction and fusion of 2D / 3D features of obstacles are realized, and a real-time environmental state matrix is generated; compared with traditional systems, it can perceive the position, speed and other information of dynamic obstacles more comprehensively and accurately, providing a more reliable data basis for subsequent motion prediction and planning; at the same time, based on historical trajectory clustering and Bayesian prediction, the motion chain prediction module can quantify the motion probability of obstacles, construct a collision risk field, and dynamically evaluate the collision risk within the next 2 seconds;

[0081] (2) The present invention adopts an improved A algorithm to search for low-risk paths, and combines quadratic programming to optimize joint motion trajectories, while embedding kinematic and dynamic constraints; the hierarchical optimization path generation method not only considers the risk and length balance of the path, but also fully takes into account the physical limitations of the robot arm to ensure that the generated motion trajectory is both safe and feasible. The correction mechanism is used to dynamically correct the feasible range of the speed and acceleration of the end effector, and an energy consumption optimization mechanism is introduced. The present invention can reasonably adjust the motion parameters according to the real-time state of the robot arm and environmental interference, avoid motion jitter caused by excessive load or improper parameters, and thus ensure the motion stability and safety of the robot arm during obstacle avoidance;

[0082] (3) The present invention introduces intelligent adaptive mechanisms in multiple links. In motion pattern clustering, it can detect the matching degree between new trajectory data and existing patterns in real time, automatically add or archive motion patterns, and realize adaptive learning and updating of complex motion characteristics. In terms of energy consumption optimization, by real-time acquisition of joint motor current data, dynamic adjustment of energy consumption weight coefficient and triggering of joint task migration mechanism, joint load distribution is intelligently optimized to improve the energy efficiency and reliability of the system. At the same time, the feedback adjustment mechanism uses the force sensor installed on the end effector to obtain contact force information in real time, adopts impedance control algorithm to adjust the end contact force online, and quickly triggers local re-planning when the trajectory tracking error exceeds the threshold, ensuring that the robot arm can respond to environmental changes and task requirements in a timely and accurate manner, further improving the intelligence and adaptability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with the accompanying drawings;

[0084] Figure 1 This is the overall system block diagram of the present invention. DETAILED DESCRIPTION

[0085] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0087] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0088] like Figure 1 As shown, the dual-manipulator adaptive motion neural network optimization system includes a feature fusion perception module, a motion chain prediction module and a planning module;

[0089] The feature fusion perception module synchronizes multi-sensor data through FPGA, combines YOLOv5s, PointNet++ and Doppler frequency shift to extract 2D / 3D features of obstacles, and uses the spatiotemporal graph neural network (ST-GNN) to fuse position and speed information to generate a real-time environment state matrix;

[0090] Using FPGA hardware synchronization signals, the binocular camera (100Hz), millimeter-wave radar (50Hz), and robotic arm encoder (1kHz) are connected respectively; through hardware circuits and synchronization algorithms, a specific timestamp is stamped for each sensor data; with a fixed time point as the starting reference, the sensor data of different frequencies are time-calibrated, such as arranging the data collected by the binocular camera every 10ms, the data collected by the millimeter-wave radar every 20ms, and the data collected by the robotic arm encoder every 1ms in chronological order to generate a unified time series data stream, ensuring the consistency of each sensor data in the time dimension, which is convenient for subsequent fusion processing;

[0091] Cross-modal feature extraction: Use the YOLOv5s model to detect the 2D bounding box of obstacles on RGB images. Input the RGB images obtained by the binocular camera into the YOLOv5s model. The YOLOv5s model first extracts features from the image, and extracts feature information at different levels in the image through network structures such as convolutional layers and pooling layers. In the detection head, the anchor box mechanism is used to predict the 2D bounding box position, category, and confidence of the obstacle. For multiple obstacles in complex scenes, the model accurately identifies the 2D position of each obstacle and provides 2D spatial information for subsequent fusion.

[0092] Point cloud data is used to extract the 3D geometric features of obstacles using Poi ntNet++. After obtaining point cloud data from millimeter-wave radar, it is input into the Poi ntNet++ model. The model extracts local point cloud features at different scales through sampling, grouping and feature learning operations. First, the original point cloud is sampled to reduce the amount of data while retaining key feature points. The sampled points are then grouped, and feature learning is performed on the point cloud in each group to obtain its 3D geometric features, such as shape and size, to provide richer information for the spatial description of obstacles. The radial velocity of obstacles is calculated using the Doppler frequency shift of millimeter-wave radar data. Millimeter-wave radar emits continuous wave signals. When the signal is reflected by an obstacle, a Doppler frequency shift is generated. According to the Doppler effect formula where f d is the Doppler frequency shift, v is the radial velocity of the obstacle, λ is the radar wavelength, f 0 To determine the frequency of the transmitted signal, perform spectrum analysis on the echo signal received by the millimeter-wave radar, calculate the Doppler frequency shift, and then obtain the radial velocity of the obstacle;

[0093] Attention Fusion: Construct a spatiotemporal graph neural network (ST-GNN) with the 2D bounding box, 3D geometric features, radial velocity, etc. of each obstacle as node features. For edge weight calculation between nodes, the relative distance between obstacles is used. (x i ,y i , z i is the coordinate of obstacle i, x j ,y j , z j is the coordinate of obstacle j, d ij is the relative distance between the two obstacles) and the speed difference Δv ij =|v i -v j |(v i ,v j are the speed of obstacle ij, Δv ij is the speed difference between the two obstacles), through the function (∈ is to prevent the denominator from being a minimum value of 0) Dynamically calculate the edge weight w ij ; Using the message passing mechanism of graph neural networks, the node features are fused in the time and space dimensions, and finally the fused obstacle state matrix is ​​output, which contains position, speed, and size confidence information.

[0094] The motion chain prediction module quantifies the probability of obstacle movement based on historical trajectory clustering and Bayesian prediction, constructs a collision risk field, and dynamically evaluates the collision risk within the next 2 seconds;

[0095] Motion pattern clustering: Slice the historical obstacle trajectory data, and capture a 0.5-second data segment for each trajectory, including the timing information of position, velocity, and acceleration. Convert the coordinates to a relative coordinate system with the robot base as the origin, and normalize the velocity and acceleration so that their value range is limited to [-1,1]. For each data segment, automatically annotate the motion status label, such as marking whether the trajectory is a straight line motion (curvature is less than 0.1), whether there is significant acceleration (average acceleration absolute value is greater than 0.3m / s 2 ) and other features; build a deep network model including an encoder and a decoder: Encoder: Use a one-dimensional convolutional neural network (1D-CNN) to extract the spatial features of the trajectory, set the convolution kernel size to 5, and the number of channels to 64; then connect to the long short-term memory network (LSTM) layer with 128 hidden units to capture temporal dependencies. The encoder output is a 32-dimensional latent vector that represents the core features of the trajectory; Decoder: Reconstruct the original trajectory data through deconvolution layers and fully connected layers to ensure that the reconstruction error is minimized; Joint optimization: During the training process, the trajectory reconstruction error and the clustering separability of the latent space are optimized simultaneously. By introducing a clustering loss function, the latent vector is forced to align with the centroid of the preset 6 motion modes; Real-time detection of the matching degree between new trajectory data and existing modes:

[0096] If the similarity between the new trajectory and all existing patterns is less than 85%, the temporary storage mechanism is triggered. After the same trajectory is detected for 5 consecutive times, the pattern is automatically added to the feature library (up to 8 patterns). Unconfirmed temporary patterns are stored in the buffer and manually reviewed to decide whether to keep them permanently. For patterns that have not been used for a long time (such as not being matched within 72 hours), their confidence weight is automatically reduced by 20% every week. When the weight is less than 10%, it is archived as a historical pattern and no longer participates in real-time prediction.

[0097] Online Bayesian prediction: real-time matching of the current obstacle motion characteristics and pattern library, calculating the posterior probability of each pattern; for patterns with probability > 15%, using the corresponding dynamic equation to generate multiple predicted trajectories within the next 2 seconds; outputting a set of trajectories with probability weights (e.g., trajectory A probability 42%, trajectory B probability 33%); collision risk field modeling: converting the predicted trajectory into a probabilistic risk field, the field strength formula is: Where P i is the probability of trajectory i, ∑ is the expansion safety area of ​​the robot body; R i (x, t) represents the risk field strength of trajectory i at point x at time t; x is a point in space; x i (t): represents the position of trajectory i at time t; T is the transposition symbol.

[0098] The planning module searches for a low-risk path by improving the A* algorithm, optimizes the joint motion trajectory through quadratic programming, and embeds kinematic and dynamic constraints to ensure the feasibility and safety of the path;

[0099] Kinematic constraints: According to the kinematic principle of the robotic arm, a feasible region of the joint angular velocity-acceleration of the robotic arm is established. Through the D-H parameters of the robotic arm, the Jacobian matrix J is derived, and the angular velocity and acceleration limits in the joint space are projected onto the end-effector space through the Jacobian matrix to obtain the feasible range of the velocity and acceleration of the end-effector, ensuring that the joint motion parameters of the robotic arm are within the safe and operable range during motion; Then, a correction mechanism is used to correct the feasible range of the velocity and acceleration of the end-effector. The correction mechanism analyzes by collecting the influencing parameters of the robotic arm, including:

[0100] Joint torque load ratio: The ratio of the actual torque of each joint to the rated torque; Trajectory tracking error: The Euclidean distance between the actual position of the end-effector and the planned position; Instantaneous power consumption: The real-time power calculated according to the motor current; Environmental interference intensity: The amplitude of the external contact force obtained through the end-effector force sensor; Health index: The comprehensive score of joint wear, temperature, and dust concentration based on historical data;

[0101] Map all parameters to the interval of 0-1, assign weights according to the importance of the influencing parameters, and then use the formula: SZ = gl×ψ 1 +gj×ψ 2 +nh×ψ 3 +hg×ψ 4 +he×ψ 5 To obtain the correction score SZ; where gl, gj, nh, hg, he are the mapped values of the joint torque load ratio, trajectory tracking error, instantaneous power consumption, environmental interference intensity, and health index respectively, and ψ 1 、ψ 2 、ψ 3 、ψ 4 、ψ 5 Are the preset weight coefficients of the joint torque load ratio, trajectory tracking error, instantaneous power consumption, environmental interference intensity, and health index respectively, and are taken as 0.35, 0.25, 0.15, 0.15, 0.1;

[0102] Then, different risk intervals are divided according to the correction score SZ, which are: Low risk (SZ≤0.3): Allow the end-effector velocity / acceleration to reach the theoretical maximum value; Medium risk (0.3 < SZ≤0.6): Limit the motion parameters proportionally: U new =U max ×(1 - SZ), a new =a max ×(1 - SZ); where U new Is the limited end-effector velocity, Umax is the theoretical maximum end velocity; a new the restricted acceleration; a max is the theoretical maximum acceleration; High risk (SZ > 0.6): Trigger a level-three response:

[0103] Level-one response (0.6 < SZ ≤ 0.8): Decelerate to 50% and perform local replanning; Level-two response (0.8 < SZ ≤ 0.95): Pause the current action and switch to impedance control mode to absorb external forces; Level-three response (S > 0.95): Emergency stop and lock the joints.

[0104] Dynamic constraints: Use the recursive Newton-Euler algorithm to calculate the forces and torques of each joint sequentially starting from the base of the robotic arm. Considering factors such as the motor torque limits of each joint of the robotic arm and the load-bearing capacity of the transmission mechanism, determine the torque limits of each joint, generate a torque feasible region, and prevent the robotic arm from damaging the equipment due to excessive torque during movement;

[0105] Hierarchical optimization path generation: Primary planning: Use the improved A* algorithm to search for a path with a low collision probability in the risk field. The cost function is: Cost = α·R total +β·PathLength (α and β are weight coefficients, R total is the total risk value on the path, and PathLength is the path length), and balance the path risk and length by adjusting the weight coefficients. The algorithm starts from the starting point and continuously expands the search nodes according to the heuristic function until the target point is found or it is determined that there is no feasible path, generating the primary planning path; Secondary correction: Discretize the primary planning path into 50 waypoints. For each waypoint, consider it as a quadratic programming problem to solve. The objective function is min||Jq - v d || 2 (q is the joint angle, J is the Jacobian matrix, v d is the desired end velocity), and the constraint condition is T min ≤M(q)+C(q, φ)≤T max , T min , T max are the upper and lower limits of the joint torque, M(q) is the joint driving torque, and C(q, φ) is the torque generated by Coriolis force and centrifugal force; By solving the quadratic programming problem, optimize the joint angles at each waypoint to make the movement of the robotic arm end smoother and more accurate. And perform joint torque energy consumption optimization through the energy consumption optimization mechanism. The process is as follows:

[0106] On the basis of the original trajectory smoothing objective, a new joint torque energy consumption optimization item is added: the objective function is adjusted to minimize the weighted combination of the terminal velocity tracking error and the sum of the squares of the joint torques, where the energy consumption weight coefficient is set to 0.3; through the optimization algorithm, on the premise of satisfying the kinematic and dynamic constraints, the joint motion scheme with lower torque consumption is given priority; during the operation of the robot arm, the current data of each joint motor is collected in real time to calculate the instantaneous power: if the power of a joint exceeds the safety threshold for 10 consecutive control cycles (such as 10ms / cycle), the energy consumption weight coefficient is automatically increased to 0.6, forcing the algorithm to redistribute the joint load;

[0107] Trigger the "joint task migration" mechanism: transfer part of the motion tasks of the high-load joint to the low-load joint. For example, by adjusting the weights of the Jacobian matrix, the movement of the end of the robot arm is completed by multiple joints in coordination to avoid overload of a single joint. Introduce an elastic buffer in the joint torque constraint: when the torque approaches 90% of the limit value, generate a warning signal and slightly relax the acceleration limit (such as reducing it by 5%) to buy time for dynamic adjustment. If the torque continues to exceed the limit, the emergency load reduction strategy is triggered: suspend the current task and perform a reverse torque compensation operation.

[0108] Feedback adjustment: During the movement of the robot arm, the contact force information is obtained in real time through the force sensor installed on the end effector, and the end contact force is adjusted online using the impedance control algorithm. The trajectory tracking error threshold is set, and when the error between the actual trajectory and the planned trajectory exceeds the threshold, local replanning is triggered. The dual-constraint modeling and hierarchical optimization path generation method of the planning module are used to replan the local path within 10ms to ensure that the robot arm can avoid obstacles or adjust the motion trajectory in time to ensure the safety and accuracy of the movement.

[0109] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. Dual-manipulator adaptive motion neural network optimization system, characterized in that: include: The feature fusion perception module is used to synchronize multi-sensor data through the field programmable gate array, extract the 2D and 3D features of obstacles by combining the target detection model, the deep learning model of point cloud data and the Doppler frequency shift, and use the spatiotemporal graph neural network to fuse the position and speed information to generate a real-time environment state matrix; The motion chain prediction module is used to quantify the probability of obstacle movement, construct a collision risk field, and dynamically evaluate the collision risk within the next 2 seconds based on historical trajectory clustering and Bayesian prediction; The planning module is used to search for low-risk paths by improving the A algorithm, optimize joint motion trajectories by combining quadratic programming, and embed kinematic and dynamic constraints to ensure path feasibility and safety.

2. The dual-manipulator adaptive motion neural network optimization system according to claim 1, characterized in that: The execution process of the feature fusion perception module is as follows: Use FPGA hardware synchronization signals to connect the binocular camera, millimeter-wave radar, and robotic arm encoder respectively; Through hardware circuits and synchronization algorithms, each sensor data is stamped with a specific time stamp; Taking a fixed time point as the starting reference, time calibration is performed on sensor data of different frequencies; Cross-modal feature extraction: Use the YOLOv5s model to detect the 2D bounding box of obstacles on the RGB image. Input the RGB image obtained by the binocular camera into the YOLOv5s model; The YOLOv5s model first extracts features from the image, and extracts feature information at different levels in the image through the convolutional layer and pooling layer network structure; In the detection head, the anchor box mechanism is used to predict the 2D bounding box position, category, and confidence of the obstacle; Use PointNet++ to extract the 3D geometric features of obstacles from the point cloud data. After obtaining the point cloud data from the millimeter wave radar, input it into the PointNet++ model; The model extracts local point cloud features at different scales through sampling, grouping and feature learning operations. First, the original point cloud is sampled to reduce the amount of data while retaining key feature points. Then the sampled points are grouped and feature learning is performed on the point cloud in each group to obtain its 3D geometric features. The radial velocity of the obstacle is calculated by Doppler frequency shift of the millimeter wave radar data. The millimeter wave radar transmits a continuous wave signal. When the signal encounters an obstacle and is reflected back, Doppler frequency shift is generated. According to the Doppler effect formula where f d is the Doppler frequency shift, v is the radial velocity of the obstacle, λ is the radar wavelength, f0 is the transmission signal frequency, and the echo signal received by the millimeter-wave radar is analyzed by spectrum analysis to calculate the Doppler frequency shift and then obtain the radial velocity of the obstacle; Finally, the fused obstacle state matrix is ​​output through attention fusion.

3. The dual-manipulator adaptive motion neural network optimization system according to claim 2, characterized in that: The specific operation steps of the feature fusion perception module to output the fused obstacle state matrix through attention fusion are as follows: Construct a spatiotemporal graph neural network, using the 2D bounding box, 3D geometric features, and radial velocity of each obstacle as node features; For edge weight calculation between nodes, the edge weight is dynamically calculated through a function based on the relative distance and speed difference between obstacles; The message passing mechanism of graph neural networks is used to fuse node features in the time and space dimensions, and finally the fused obstacle state matrix is ​​output, which contains position, speed, and size confidence information.

4. The dual-manipulator adaptive motion neural network optimization system according to claim 1, characterized in that: The specific operation steps of the kinematic chain prediction module are as follows: Firstly, motion pattern clustering is performed based on the incremental trajectory clustering method of spatiotemporal autoencoder; Online Bayesian prediction: real-time matching of the current obstacle motion characteristics and pattern library, and calculation of the posterior probability of each pattern; For patterns with probability > 15%, the corresponding dynamic equation is used to generate multiple predicted trajectories within the next 2 seconds; and a set of trajectories with probability weights is output; Collision risk field modeling: The predicted trajectory is converted into a probabilistic risk field. The field strength formula is: Where P i is the probability of trajectory i, ∑ is the expansion safety area of ​​the robot body; R i (x, t) represents the risk field strength of trajectory i at point x at time t; x is a point in space; x i (t): represents the position of trajectory i at time t; T is the transposition symbol.

5. The dual-manipulator adaptive motion neural network optimization system according to claim 4, characterized in that: The specific operation steps of the motion chain prediction module for motion pattern clustering are as follows: The historical obstacle trajectory data is sliced, and each trajectory is intercepted with a data segment of 0.5 seconds, which contains the time series information of position, speed, and acceleration; The coordinates are converted into a relative coordinate system with the base of the robot as the origin, and the velocity and acceleration are normalized so that the value range is limited to [-1, 1]; for each data segment, the motion state label is automatically annotated; Construct a deep network model including encoder and decoder. The process is as follows: Encoder: A one-dimensional convolutional neural network is used to extract the spatial features of the trajectory, with the convolution kernel size set to 5 and the number of channels set to 64; A long short-term memory network layer is connected with 128 hidden units to capture temporal dependencies. The encoder output is a 32-dimensional latent vector that represents the core features of the trajectory; Decoder: reconstructs the original trajectory data through deconvolution layers and fully connected layers to ensure that the reconstruction error is minimized; Joint optimization: During the training process, the trajectory reconstruction error and the clustering separability of the latent space are optimized simultaneously; by introducing a clustering loss function, the latent vector is forced to align with the centroid of the six preset motion modes; Real-time detection of the matching degree between new trajectory data and existing patterns: If the similarity between the new trajectory and all existing patterns is less than 85%, the temporary storage mechanism is triggered. After the same trajectory is detected five times in a row, the pattern is automatically added to the feature library; unconfirmed temporary patterns are stored in the buffer, and after manual review, it is decided whether to keep them permanently; For usage patterns that have not been matched for more than 72 hours, their confidence weight will be automatically reduced by 20% each week. When the weight is lower than 10%, it will be archived as a historical pattern and will no longer participate in real-time prediction.

6. The dual-manipulator adaptive motion neural network optimization system according to claim 1, characterized in that: The specific operation steps of the planning module are as follows: Kinematic constraints: According to the kinematic principle of the manipulator, the feasible domain of the angular velocity-acceleration of the manipulator joints is established; the Jacobian matrix J is derived through the DH parameters of the manipulator, and the angular velocity and acceleration limits of the joint space are projected to the end space through the Jacobian matrix to obtain the feasible range of the speed and acceleration of the end effector, and then the correction mechanism is used to correct the feasible range of the speed and acceleration of the end effector; Dynamic constraints: Using the recursive Newton-Euler algorithm, starting from the base of the robot arm, the forces and moments of each joint are calculated in sequence; Analyze the motor torque limit and transmission mechanism load capacity of each joint of the robot arm, determine the torque limit of each joint, generate the torque feasible domain, and prevent the robot arm from damaging the equipment due to excessive torque during movement; The hierarchical optimization path generation process is as follows: Primary planning: Use the improved A algorithm to search for low collision probability paths in the risk field. The cost function is: Cost = α·R total +β·PathLength, where α and β are weight coefficients, R total is the total risk value on the path, PathLength is the path length, and the path risk and length are balanced by adjusting the weight coefficient; The algorithm starts from the starting point and expands the search nodes according to the heuristic function until the target point is found or it is determined that there is no feasible path, and the primary planning path is generated; Secondary correction: Discretize the primary planned path into 50 waypoints. For each waypoint, treat it as a quadratic programming problem; the objective function is min||Jq-v d || 2 , where q is the joint angle, J is the Jacobian matrix, and v d is the expected terminal velocity; The constraint condition is T min ≤M(q)+C(q,φ)≤T max ,T min ,T max are the upper and lower limits of the joint torque, M(q) is the joint driving torque, and C(q, φ) is the torque generated by the Coriolis force and the centrifugal force; By solving the quadratic programming problem, the joint angle at each waypoint is optimized to make the end-of-arm motion smoother and more accurate; And the energy consumption of joint torque is optimized through energy consumption optimization mechanism; Feedback adjustment: During the movement of the robotic arm, the contact force information is obtained in real time through the force sensor installed on the end effector, and the impedance control algorithm is used to adjust the end contact force online; Set the trajectory tracking error threshold. When the error between the actual trajectory and the planned trajectory is detected to exceed the threshold, local replanning is triggered; Using the double-constraint modeling and hierarchical optimization path generation method of the planning module, the local path is replanned within 10 ms to ensure that the robotic arm can avoid obstacles in time or adjust the motion trajectory.

7. The dual-manipulator adaptive motion neural network optimization system according to claim 6, characterized in that: The specific operation steps for correcting the feasible range of the speed and acceleration of the end effector using the correction mechanism in the planning module are as follows: The correction mechanism analyzes by collecting the influence parameters of the robotic arm. The influence parameters include: Joint torque load rate: The ratio of the actual torque of each joint to the rated torque; Trajectory tracking error: The Euclidean distance between the actual position of the end and the planned position; Instantaneous power consumption: The real-time power calculated based on the motor current; Environmental interference intensity: The amplitude of the external contact force obtained through the end force sensor; Health index: The comprehensive score of joint wear, temperature, and dust concentration based on historical data; Map all parameters to the interval of 0-1, assign weights according to the importance of the influence parameters, and then use the formula: SZ = gl×ψ1 + gj×ψ2 + nh×ψ3 + hg×ψ4 + he×ψ5 to obtain the correction score SZ; where gl, gj, nh, hg, he are the mapped values of the joint torque load rate, trajectory tracking error, instantaneous power consumption, environmental interference intensity, and health index respectively, and ψ1, ψ2, ψ3, ψ4, ψ5 are the preset weight coefficients of the joint torque load rate, trajectory tracking error, instantaneous power consumption, environmental interference intensity, and health index respectively; Then divide different risk intervals according to the correction score SZ, which are: Low risk, i.e., SZ ≤ 0.3: Allow the end speed or acceleration to reach the theoretical maximum; Medium risk, i.e., 0.3 < SZ ≤ 0.6: Limit the motion parameters proportionally: U new = U max ×(1 - SZ), a new = a max ×(1 - SZ); where U new is the limited end velocity, U max is the theoretical maximum end velocity; a new is the limited acceleration; a max is the theoretical maximum acceleration; High risk, i.e., SZ > 0.6: Trigger a three-level response, including: First-level response, i.e., 0.6 < SZ ≤ 0.8: Reduce the speed to 50% and perform local replanning; Second-level response, i.e., 0.8 < SZ ≤ 0.95: Pause the current action and switch to the impedance control mode to absorb external forces; Third-level response, i.e., S > 0.95: Emergency stop and lock the joints.

8. The dual-manipulator adaptive motion neural network optimization system according to claim 6, characterized in that: The specific operation steps of the energy consumption optimization mechanism in the planning module are as follows: The objective function is adjusted to minimize the weighted combination of the end speed tracking error and the sum of the squares of the joint torques, where the energy consumption weight coefficient is set to 0.3; Through the optimization algorithm, on the premise of meeting the kinematic and dynamic constraints, preferentially select the joint motion plan with lower torque consumption; During the operation of the robotic arm, the current data of each joint motor is collected in real time to calculate the instantaneous power: If the power of a certain joint exceeds the safety threshold for 10 consecutive control cycles, automatically increase the energy consumption weight coefficient to 0.6 and force the algorithm to redistribute the joint load; Trigger the joint task migration mechanism: Transfer part of the motion tasks of the high-load joints to the low-load joints, and adjust the weights of the Jacobian matrix to make the end motion of the robotic arm completed by multiple joints in cooperation; An elastic buffer is introduced into the joint torque constraint: when the torque reaches 90% of the limit value, a warning signal is generated and the acceleration limit is reduced by 5% to buy time for dynamic adjustment; If the torque continues to exceed the limit, the emergency load reduction strategy is triggered: the current task is suspended and the reverse torque compensation operation is performed.

Images