Dual robotic arm adaptive motion neural network optimization system

Abstract

The application discloses a double-robot-arm adaptive motion neural network optimization system and relates to the technical field of robot arm motion control.The application comprises the following steps: a feature fusion perception module synchronizes multi-sensor data through FPGA, combines YOLOv5s, PointNet++ and Doppler frequency shift and the like technologies, realizes accurate extraction and fusion of 2D / 3D features of obstacles, and generates a real-time environment state matrix; a motion chain prediction module quantifies the motion probability of obstacles based on historical trajectory clustering and Bayesian prediction, constructs a collision risk field, and dynamically evaluates the collision risk within the next 2 seconds; a planning module adopts an improved A algorithm to search for a low-risk path, combines quadratic programming to optimize joint motion trajectories, embeds kinematics and dynamics constraints, and ensures the feasibility and safety of the path; in addition, a correction mechanism and an energy consumption optimization mechanism are introduced, motion parameters are dynamically adjusted according to the real-time state of the robot arm and environmental interference, and the obstacle avoidance performance and operation efficiency of the robot arm are further improved.

Description

Technical Field

[0001] This invention relates to the field of robotic arm motion control technology, specifically a dual-robotic arm adaptive motion neural network optimization system. Background Technology

[0002] In the field of modern industry and automation, robots are being used more and more widely. Dual-arm robotic systems, with their high efficiency and flexibility, have been widely used in many scenarios such as logistics handling, precision assembly, and complex processing. However, existing dual-arm robotic systems have significant shortcomings in motion control.

[0003] In complex and dynamic environments, multi-sensor data fusion technology is not yet perfect. Currently, most dual-arm robotic systems cannot accurately synchronize data from multiple sensors such as binocular cameras, millimeter-wave radar, and robotic arm encoders, resulting in delays and deviations in information acquisition. This makes it difficult to perceive obstacles and their states in the environment in real time and accurately, exposing the robotic arm to collision risks during movement and failing to ensure operational safety.

[0004] Traditional obstacle avoidance systems for robotic arms have significant limitations. On the one hand, their trajectory prediction accuracy is insufficient, relying solely on single-frame environmental data. The trajectory prediction error for fast-moving obstacles exceeds 30%, making it difficult to meet the precise obstacle avoidance requirements in complex dynamic scenarios. On the other hand, path planning lacks consideration of the robotic arm's dynamic constraints, resulting in delayed obstacle avoidance actions and a tendency to cause motion jitter, severely impacting the robotic arm's operational efficiency and stability.

[0005] To address the aforementioned shortcomings, a technical solution is provided. Summary of the Invention

[0006] The purpose of this invention is to solve the problems of large error in predicting the trajectory of fast-moving obstacles and the lack of consideration for dynamic constraints in path planning of traditional robotic arm obstacle avoidance systems, and to propose a dual-robotic arm adaptive motion neural network optimization system.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] A dual-arm adaptive motion neural network optimization system includes:

[0009] The feature fusion perception module synchronizes multi-sensor data through a field-programmable gate array, combines a target detection model, a deep learning model of point cloud data, and Doppler frequency shift to extract 2D and 3D features of obstacles, and uses a spatiotemporal graph neural network to fuse position and velocity information to generate a real-time environmental state matrix.

[0010] The motion chain prediction module quantifies the probability of obstacle movement based on historical trajectory clustering and Bayesian prediction, constructs a collision risk field, and dynamically assesses the collision risk within the next 2 seconds.

[0011] The planning module searches for low-risk paths by improving the A algorithm, optimizes joint motion trajectories by combining quadratic programming, and embeds kinematic and dynamic constraints to ensure path feasibility and safety.

[0012] Furthermore, the execution process of the feature fusion perception module is as follows:

[0013] Using FPGA hardware synchronization signals, binocular cameras, millimeter-wave radar, and robotic arm encoders are connected respectively.

[0014] By using hardware circuitry and synchronization algorithms, a specific timestamp is added to each sensor data.

[0015] Using a fixed point in time as the starting reference, time calibration is performed on sensor data of different frequencies;

[0016] Cross-modal feature extraction: The YOLOv5s model is used to detect 2D bounding boxes of obstacles in the RGB image. The RGB image acquired by the binocular camera is input into the YOLOv5s model.

[0017] The YOLOv5s model first extracts features from the image by using a network structure of convolutional and pooling layers to extract feature information at different levels in the image.

[0018] In the detection head, the anchor box mechanism is used to predict the 2D bounding box location, category, and confidence level of obstacles;

[0019] Point cloud data is used to extract 3D geometric features of obstacles. Point cloud data is obtained from millimeter-wave radar and then input into the Point Net++ model.

[0020] The model extracts local point cloud features at different scales through sampling, grouping, and feature learning operations. First, the original point cloud is sampled to reduce the amount of data while retaining key feature points. Then, the sampled points are grouped, and feature learning is performed on the point cloud within each group to obtain its 3D geometric features.

[0021] The radial velocity of an obstacle is calculated from millimeter-wave radar data using Doppler frequency shift. The millimeter-wave radar emits a continuous wave signal, and when the signal is reflected back from an obstacle, a Doppler frequency shift is generated.

[0022] According to the Doppler effect formula Where f d Let v be the Doppler frequency shift, λ be the radial velocity of the obstacle, λ be the radar wavelength, and f0 be the transmitted signal frequency. By performing spectral analysis on the echo signal received by the millimeter-wave radar, the Doppler frequency shift is calculated, and thus the radial velocity of the obstacle is obtained.

[0023] Finally, the fused obstacle state matrix is ​​output through attention fusion.

[0024] Furthermore, the specific operation steps of the feature fusion perception module in outputting the fused obstacle state matrix through attention fusion are as follows:

[0025] A spatiotemporal graph neural network is constructed, using the 2D bounding boxes, 3D geometric features, and radial velocities of each obstacle as node features;

[0026] For the calculation of edge weights between nodes, the edge weights are dynamically calculated using a function based on the relative distance and speed difference between obstacles.

[0027] By utilizing the message passing mechanism of graph neural networks, node features are fused in the spatiotemporal dimensions, and the final output is a fused obstacle state matrix containing position, velocity, size confidence information.

[0028] Furthermore, the specific operation steps of the kinetic chain prediction module are as follows:

[0029] First, motion pattern clustering is performed using an incremental trajectory clustering method based on a spatiotemporal autoencoder;

[0030] Online Bayesian prediction: Real-time matching of current obstacle motion features with a pattern library to calculate the posterior probability of each pattern;

[0031] For patterns with a probability >15%, the corresponding dynamic equation is used to generate multiple predicted trajectories within the next 2 seconds; the output is a set of trajectories with probability weights.

[0032] Collision risk field modeling: The predicted trajectory is transformed into a probabilistic risk field, and the field strength formula is: Where P i Let R be the probability of trajectory i, and ∑ be the safe expansion zone of the robotic arm body; i (x, t) represents the risk field strength of trajectory i at point x and time t; x is a point in space; x i (t) represents the position of trajectory i at time t; T is the transpose symbol.

[0033] Furthermore, the specific steps for motion pattern clustering in the motion chain prediction module are as follows:

[0034] The historical obstacle trajectory data is sliced, and each trajectory segment is extracted into a 0.5-second data fragment containing the temporal information of position, velocity, and acceleration.

[0035] The coordinates are converted to a relative coordinate system with the robotic arm base as the origin, and the velocity and acceleration are normalized to limit the numerical range to [-1, 1]. For each data segment, motion status labels are automatically added.

[0036] The process of constructing a deep network model containing an encoder and a decoder is as follows:

[0037] Encoder: A one-dimensional convolutional neural network is used to extract the spatial features of the trajectory, with the kernel size set to 5 and the number of channels to 64;

[0038] It is connected to the Long Short-Term Memory network layer with 128 hidden units to capture temporal dependencies;

[0039] The encoder outputs a 32-dimensional latent vector, which represents the core features of the trajectory.

[0040] Decoder: Reconstructs the original trajectory data through deconvolutional layers and fully connected layers to ensure that the reconstruction error is minimized;

[0041] Joint optimization: During training, both trajectory reconstruction error and cluster separability of the latent space are optimized simultaneously; by introducing a clustering loss function, the latent vector is forced to align with the centroids of the six preset motion patterns;

[0042] Real-time detection of the matching degree between new trajectory data and existing patterns: If the similarity between the new trajectory and all existing patterns is less than 85%, a temporary storage mechanism is triggered. After five consecutive detections of the same type of trajectory, the pattern is automatically added to the feature library. Unconfirmed temporary patterns are stored in a buffer and a decision on whether to retain them permanently is made after manual review.

[0043] For usage patterns that are not matched within 72 hours, their confidence weight is automatically reduced by 20% weekly. When the weight is below 10%, they are archived as historical patterns and no longer participate in real-time prediction.

[0044] Furthermore, the specific operation steps of the planning module are as follows:

[0045] Kinematic constraints: Based on the kinematic principle of the robotic arm, the feasible region of angular velocity-acceleration of the robotic arm joints is established; through the DH parameters of the robotic arm, the Jacobian matrix J is derived, and the angular velocity and acceleration constraints in the joint space are projected onto the end effector space through the Jacobian matrix to obtain the feasible range of velocity and acceleration of the end effector. Then, the feasible range of velocity and acceleration of the end effector is corrected by the correction mechanism.

[0046] Dynamic constraints: The recursive Newton-Euler algorithm is used to calculate the forces and torques of each joint sequentially, starting from the base of the robotic arm.

[0047] Analyze the motor torque limits and transmission capacity factors of each joint of the robotic arm, determine the torque limits of each joint, generate the feasible torque domain, and prevent the robotic arm from being damaged by excessive torque during movement;

[0048] The hierarchical optimization path generation process is as follows:

[0049] Preliminary planning: Use the improved A algorithm to search for low-collision-probability paths in the risk field, with the cost function: Cost = α·R total +β·PathLength, where α and β are weighting coefficients, and R total The total risk value on the path is denoted as PathLength, and the path risk and length are balanced by adjusting the weight coefficients. The algorithm starts from the starting point and expands the search nodes according to the heuristic function until the target point is found or it is determined that there is no feasible path, thus generating a preliminary planned path.

[0050] Secondary correction: Discretize the primary planning path into 50 waypoints. For each waypoint, treat it as a quadratic programming problem to solve; the objective function is min||Jq-v d || 2 Where q is the joint angle, J is the Jacobian matrix, and v d The desired terminal velocity;

[0051] The constraint condition is T min ≤M(q)+C(q,φ)≤T max T min T max Let M(q) be the upper and lower limits of the joint torque, M(q) be the joint driving torque, and C(q, φ) be the torque generated by the Coriolis force and centrifugal force.

[0052] By solving a quadratic programming problem, the joint angles at each waypoint are optimized to make the end effector of the robotic arm move more smoothly and accurately; and the joint torque energy consumption is optimized through an energy consumption optimization mechanism.

[0053] Feedback Adjustment: During the movement of the robotic arm, contact force information is acquired in real time by force sensors installed on the end effector, and the end contact force is adjusted online using an impedance control algorithm;

[0054] Set a trajectory tracking error threshold. When the error between the actual trajectory and the planned trajectory exceeds the threshold, trigger local replanning.

[0055] By utilizing the dual-constraint modeling and hierarchical optimization path generation method of the planning module, the local path is replanned within 10ms, ensuring that the robotic arm can avoid obstacles or adjust its motion trajectory in time.

[0056] Furthermore, the specific steps for correcting the feasible range of speed and acceleration of the end effector using the correction mechanism in the planning module are as follows:

[0057] The correction mechanism analyzes the impact parameters of the robotic arm, including:

[0058] Joint torque load ratio: The ratio of the actual torque of each joint to the rated torque;

[0059] Trajectory tracking error: The Euclidean distance between the actual end position and the planned position;

[0060] Instantaneous power consumption: The real-time power calculated based on the motor current;

[0061] Environmental interference intensity: The amplitude of the external contact force obtained through the end force sensor;

[0062] Health index: The comprehensive score of joint wear degree, temperature and dust concentration based on historical data;

[0063] Map all parameters to the interval of 0 - 1, assign weights according to the importance of the influencing parameters, and then use the formula: SZ = gl×ψ1 + gj×ψ2 + nh×ψ3 + hg×ψ4 + he×ψ5 to obtain the corrected score SZ; where gl, gj, nh, hg, he are the mapped values of the joint torque load rate, trajectory tracking error, instantaneous power consumption, environmental interference intensity and health index respectively, and ψ1, ψ2, ψ3, ψ4, ψ5 are the preset weight coefficients of the joint torque load rate, trajectory tracking error, instantaneous power consumption, environmental interference intensity and health index respectively;

[0064] Then divide different risk intervals according to the corrected score SZ, which are:

[0065] Low risk, that is, SZ ≤ 0.3: Allow the end speed or acceleration to reach the theoretical maximum value;

[0066] Medium risk, that is, 0.3 < SZ ≤ 0.6: Limit the motion parameters proportionally: U new = U max ×(1 - SZ), a new = a max ×(1 - SZ); where U new is the limited end speed, U max is the theoretical maximum end speed; a new is the limited acceleration; a max is the theoretical maximum acceleration;

[0067] High risk, that is, SZ > 0.6: Trigger a three-level response, including:

[0068] First-level response, that is, 0.6 < SZ ≤ 0.8: Reduce the speed to 50% and perform local replanning;

[0069] Second-level response, that is, 0.8 < SZ ≤ 0.95: Pause the current action and switch to the impedance control mode to absorb external forces;

[0070] Third-level response, that is, S > 0.95: Emergency stop and lock the joints.

[0071] Furthermore, the specific operation steps of the energy consumption optimization mechanism in the planning module are as follows:

[0072] The objective function was adjusted to minimize the weighted combination of the end-effector tracking error and the sum of squares of joint torques, with the energy consumption weighting coefficient set to 0.3.

[0073] By optimizing the algorithm, joint movement schemes with lower torque consumption are selected first, while satisfying kinematic and dynamic constraints.

[0074] During the operation of the robotic arm, the current data of each joint motor is collected in real time to calculate the instantaneous power:

[0075] If the power of a certain joint exceeds the safety threshold for 10 consecutive control cycles, the energy consumption weighting coefficient will be automatically increased to 0.6, and the algorithm will be forced to redistribute the joint load.

[0076] Triggering the joint task transfer mechanism: Transferring some motion tasks of high-load joints to low-load joints, and by adjusting the weights of the Jacobian matrix, the end effector motion of the robotic arm is completed by multiple joints working together;

[0077] An elastic buffer is introduced into the joint torque constraint condition: when the torque reaches 90% of the limit value, an early warning signal is generated and the acceleration limit is reduced by 5%, so as to buy time for dynamic adjustment.

[0078] If the torque continues to exceed the limit, an emergency load reduction strategy will be triggered: the current task will be paused and a reverse torque compensation operation will be performed.

[0079] Compared with the prior art, the beneficial effects of the present invention are:

[0080] (1) This invention utilizes a feature fusion perception module to synchronize multi-sensor data using FPGA, combined with technologies such as YOLOv5s, PointNet++, and Doppler frequency shift, to achieve accurate extraction and fusion of 2D / 3D features of obstacles, generating a real-time environmental state matrix. Compared with traditional systems, it can perceive the position, speed, and other information of dynamic obstacles more comprehensively and accurately, providing a more reliable data foundation for subsequent motion prediction and planning. At the same time, the motion chain prediction module, based on historical trajectory clustering and Bayesian prediction, can quantify the probability of obstacle movement, construct a collision risk field, and dynamically assess the collision risk within the next 2 seconds.

[0081] (2) In this invention, an improved A algorithm is used to search for low-risk paths, and a quadratic programming is combined to optimize the joint motion trajectory. Kinematic and dynamic constraints are also embedded. The hierarchical optimization path generation method not only considers the balance between path risk and length, but also fully takes into account the physical limitations of the robotic arm, ensuring that the generated motion trajectory is both safe and feasible. Through the correction mechanism, the feasible range of speed and acceleration of the end effector is dynamically corrected, and an energy consumption optimization mechanism is introduced. This invention can reasonably adjust motion parameters according to the real-time state of the robotic arm and environmental interference, avoiding problems such as motion jitter caused by excessive load or improper parameters, thereby ensuring the stability and safety of the robotic arm's motion during obstacle avoidance.

[0082] (3) This invention introduces an intelligent adaptive mechanism in multiple stages. In motion pattern clustering, it can detect the matching degree between new trajectory data and existing patterns in real time, automatically add or archive motion patterns, and realize adaptive learning and updating of complex motion features. In terms of energy consumption optimization, by collecting joint motor current data in real time, dynamically adjusting the energy consumption weight coefficient and triggering the joint task migration mechanism, it can intelligently optimize the joint load distribution and improve the energy efficiency and reliability of the system. At the same time, the feedback adjustment mechanism uses the force sensor installed on the end effector to obtain contact force information in real time, adopts the impedance control algorithm to adjust the end contact force online, and quickly triggers local replanning when the trajectory tracking error exceeds the threshold, ensuring that the robotic arm can respond to environmental changes and task requirements in a timely and accurate manner, further improving the intelligence and adaptability of the system. Attached Figure Description

[0083] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings;

[0084] Figure 1 This is the overall system block diagram of the present invention. Detailed Implementation

[0085] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0086] It should be understood that the terms “comprising” and “including” used in this disclosure and claims indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0087] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0088] like Figure 1 As shown, the dual-arm adaptive motion neural network optimization system includes a feature fusion perception module, a motion chain prediction module, and a planning module.

[0089] The feature fusion perception module synchronizes multi-sensor data through FPGA, combines YOLOv5s, PointNet++ and Doppler frequency shift to extract 2D / 3D features of obstacles, and uses a spatiotemporal graph neural network (ST-GNN) to fuse position and velocity information to generate a real-time environmental state matrix.

[0090] Using FPGA hardware synchronization signals, a binocular camera (100Hz), a millimeter-wave radar (50Hz), and a robotic arm encoder (1kHz) are connected respectively. Through hardware circuits and synchronization algorithms, a specific timestamp is added to each sensor data. Starting from a fixed time point, time calibration is performed on the sensor data of different frequencies. For example, the data collected by the binocular camera every 10ms, the data collected by the millimeter-wave radar every 20ms, and the data collected by the robotic arm encoder every 1ms are arranged in chronological order to generate a unified time-series data stream, ensuring the consistency of the data of each sensor in the time dimension, which is convenient for subsequent fusion processing.

[0091] Cross-modal feature extraction: The YOLOv5s model is used to detect 2D bounding boxes of obstacles in RGB images. The RGB images acquired by the binocular camera are input into the YOLOv5s model. The YOLOv5s model first performs feature extraction on the image, extracting feature information at different levels in the image through network structures such as convolutional layers and pooling layers. In the detection head, the anchor box mechanism is used to predict the position, category, and confidence of the 2D bounding boxes of obstacles. For multiple obstacles in complex scenes, the model accurately identifies the 2D position of each obstacle, providing 2D spatial information for subsequent fusion.

[0092] PointNet++ is used to extract 3D geometric features of obstacles from point cloud data. Point cloud data acquired from millimeter-wave radar is input into the PointNet++ model. The model extracts local point cloud features at different scales through sampling, grouping, and feature learning operations. First, the original point cloud is sampled to reduce data volume while retaining key feature points. Then, the sampled points are grouped, and feature learning is performed on the point cloud within each group to obtain its 3D geometric features, such as shape and size, providing richer information for the spatial description of obstacles. The radial velocity of obstacles is calculated from millimeter-wave radar data using Doppler frequency shift. Millimeter-wave radar emits continuous wave signals; when the signal is reflected back from an obstacle, a Doppler frequency shift occurs. Based on the Doppler effect formula... Where f d Let v be the Doppler frequency shift, λ be the radial velocity of the obstacle, λ be the radar wavelength, and f0 be the transmitted signal frequency. By performing spectral analysis on the echo signal received by the millimeter-wave radar, the Doppler frequency shift is calculated, and thus the radial velocity of the obstacle is obtained.

[0093] Attention Fusion: A Spatiotemporal Graph Neural Network (ST-GNN) is constructed, using the 2D bounding boxes, 3D geometric features, and radial velocities of each obstacle as node features. The edge weights between nodes are calculated based on the relative distances between obstacles. (x i y i , z i Let x be the coordinates of obstacle i. j y j , z j Let d be the coordinates of obstacle j. ij (The relative distance between two obstacles) and the speed difference Δv ij =|v i -v j |(v i ,v j Let Δv be the velocity of obstacle ij. ij (where the speed difference is between the two obstacles) is expressed by the function (∈ to prevent the denominator from being a minimum value of 0) Dynamically calculate the edge weight w ij By utilizing the message passing mechanism of graph neural networks, node features are fused in the spatiotemporal dimensions, and the final output is a fused obstacle state matrix, which includes position, velocity, size confidence information.

[0094] The motion chain prediction module quantifies the probability of obstacle movement based on historical trajectory clustering and Bayesian prediction, constructs a collision risk field, and dynamically assesses the collision risk within the next 2 seconds.

[0095] Motion pattern clustering: Historical obstacle trajectory data is sliced, with each trajectory segment containing a 0.5-second data fragment including temporal information on position, velocity, and acceleration. The coordinates are converted to a relative coordinate system with the robotic arm base as the origin, and the velocity and acceleration are normalized to limit their values ​​to the range [-1, 1]. For each data fragment, motion state labels are automatically applied, such as indicating whether the trajectory is linear (curvature less than 0.1) or whether there is significant acceleration (average absolute acceleration greater than 0.3 m / s²). 2 Features such as ) are used to construct a deep network model containing an encoder and a decoder: Encoder: A one-dimensional convolutional neural network (1D-CNN) is used to extract spatial features of the trajectory, with a kernel size of 5 and 64 channels; then a Long Short-Term Memory (LSTM) layer with 128 hidden units is connected to capture temporal dependencies. The encoder output is a 32-dimensional latent vector representing the core features of the trajectory; Decoder: The original trajectory data is reconstructed through deconvolutional layers and fully connected layers to ensure that the reconstruction error is minimized; Joint optimization: During training, the trajectory reconstruction error and the clustering separability of the latent space are optimized simultaneously. By introducing a clustering loss function, the latent vector is forced to align with the centroids of 6 preset motion patterns; The matching degree between new trajectory data and existing patterns is detected in real time:

[0096] If the similarity between a new trajectory and all existing patterns is less than 85%, a temporary storage mechanism is triggered. After five consecutive detections of the same type of trajectory, the pattern is automatically added to the feature library (expanded to a maximum of eight patterns). Unconfirmed temporary patterns are stored in a buffer and a decision is made on whether to retain them permanently after manual review. For patterns that have not been used for a long time (e.g., not matched within 72 hours), their confidence weight is automatically reduced by 20% every week. When the weight is below 10%, they are archived as historical patterns and no longer participate in real-time prediction.

[0097] Online Bayesian prediction: Real-time matching of current obstacle motion features with a pattern library, calculating the posterior probability of each pattern; for patterns with a probability >15%, generating multiple predicted trajectories within the next 2 seconds using the corresponding dynamic equation; outputting a set of trajectories with probability weights (e.g., trajectory A with a probability of 42%, trajectory B with a probability of 33%); Collision risk field modeling: converting the predicted trajectories into a probabilistic risk field, with the field strength formula as follows: Where P i Let R be the probability of trajectory i, and ∑ be the safe expansion zone of the robotic arm body; i (x, t) represents the risk field strength of trajectory i at point x and time t; x is a point in space; x i (t): represents the position of trajectory i at time t; T is the transpose symbol.

[0098] The planning module searches for low-risk paths by improving the A* algorithm, optimizes the joint motion trajectory through quadratic programming, and embeds kinematic and dynamic constraints to ensure path feasibility and safety;

[0099] Kinematic constraints: Based on the kinematic principles of the robotic arm, a feasible region of the joint angular velocity-acceleration of the robotic arm is established. Through the D-H parameters of the robotic arm, the Jacobian matrix J is derived. The angular velocity and acceleration limits in the joint space are projected onto the end space through the Jacobian matrix to obtain the feasible ranges of the velocity and acceleration of the end effector, ensuring that the joint motion parameters of the robotic arm are within the safe and operable range during movement; then a correction mechanism is used to correct the feasible ranges of the velocity and acceleration of the end effector. The correction mechanism analyzes by collecting the influencing parameters of the robotic arm, including:

[0100] Joint torque load ratio: The ratio of the actual torque of each joint to the rated torque; Trajectory tracking error: The Euclidean distance between the actual position of the end and the planned position; Instantaneous power consumption: The real-time power calculated based on the motor current; Environmental interference intensity: The amplitude of the external contact force obtained through the end force sensor; Health index: A comprehensive score of joint wear, temperature, and dust concentration based on historical data;

[0101] All parameters are mapped to the interval of 0-1, weights are assigned according to the importance of the influencing parameters, and then the formula: SZ = gl×ψ1 + gj×ψ2 + nh×ψ3 + hg×ψ4 + he×ψ5 is used to obtain the correction score SZ; where gl, gj, nh, hg, he are the mapped values of the joint torque load ratio, trajectory tracking error, instantaneous power consumption, environmental interference intensity, and health index respectively, and ψ1, ψ2, ψ3, ψ4, ψ5 are the preset weight coefficients of the joint torque load ratio, trajectory tracking error, instantaneous power consumption, environmental interference intensity, and health index respectively, and are taken as 0.35, 0.25, 0.15, 0.15, 0.1 respectively;

[0102] Then different risk intervals are divided according to the correction score SZ, which are: Low risk (SZ≤0.3): Allowing the end velocity / acceleration to reach the theoretical maximum value; Medium risk (0.3 < SZ≤0.6): Limiting the motion parameters proportionally: U new = U max ×(1 - SZ), a new = a max ×(1 - SZ); where U new is the limited end velocity, U max is the theoretical maximum end velocity; a new is the limited acceleration; a max is the theoretical maximum acceleration; High risk (SZ>0.6): Triggering a three-level response:

[0103] Level 1 response (0.6 < SZ ≤ 0.8): Reduce speed to 50% and perform local replanning; Level 2 response (0.8 < SZ ≤ 0.95): Pause the current action and switch to impedance control mode to absorb external forces; Level 3 response (S > 0.95): Emergency stop and lock the joints.

[0104] Dynamic constraints: Use the recursive Newton-Euler algorithm to calculate the forces and torques of each joint sequentially starting from the base of the robotic arm. Considering factors such as the motor torque limits of each joint of the robotic arm and the bearing capacity of the transmission mechanism, determine the torque limits of each joint, generate a torque feasible region, and prevent damage to the equipment due to excessive torque during the movement of the robotic arm;

[0105] Hierarchical optimization path generation: Primary planning: Use the improved A* algorithm to search for a path with a low collision probability in the risk field. The cost function is: Cost = α·R total +β·PathLength (α and β are weight coefficients, R total is the total risk value on the path, and PathLength is the path length). Balance the path risk and length by adjusting the weight coefficients. The algorithm starts from the starting point and continuously expands the search nodes according to the heuristic function until the target point is found or it is determined that there is no feasible path, generating the primary planning path; Secondary correction: Discretize the primary planning path into 50 waypoints. For each waypoint, consider it as a quadratic programming problem to solve. The objective function is min||Jq - v d || 2 (q is the joint angle, J is the Jacobian matrix, and v d is the desired end velocity), and the constraint condition is T min ≤ M(q) + C(q, φ) ≤ T max , T min , T max are the upper and lower limits of the joint torque, M(q) is the joint driving torque, and C(q, φ) is the torque generated by Coriolis force and centrifugal force; By solving the quadratic programming problem, optimize the joint angles at each waypoint to make the movement of the robotic arm end smoother and more accurate. And perform joint torque energy consumption optimization through an energy consumption optimization mechanism. The process is as follows:

[0106] On the basis of the original trajectory smoothing target, add a joint torque energy consumption optimization term: The objective function is adjusted to minimize the weighted combination of the end velocity tracking error and the sum of the squares of the joint torques, where the energy consumption weight coefficient is set to 0.3; Through the optimization algorithm, under the premise of satisfying kinematic and dynamic constraints, preferentially select a joint motion plan with lower torque consumption; During the operation of the robotic arm, collect the current data of each joint motor in real time and calculate the instantaneous power: If the power of a certain joint exceeds the safety threshold for 10 consecutive control cycles (such as 10 ms / cycle), automatically increase the energy consumption weight coefficient to 0.6 and force the algorithm to reallocate the joint load;

[0107] Trigger the "Joint Task Migration" mechanism: Transfer some motion tasks of high-load joints to low-load joints, for example, by adjusting the weights of the Jacobian matrix, so that the end effector of the robotic arm is completed by multiple joints in coordination, avoiding overload of a single joint; Introduce an elastic buffer in the joint torque constraint: When the torque approaches 90% of the limit value, generate a warning signal and slightly relax the acceleration limit (e.g., reduce it by 5%) to buy time for dynamic adjustment; If the torque continues to exceed the limit, trigger the emergency deload strategy: Pause the current task and perform reverse torque compensation operation.

[0108] Feedback Adjustment: During the robotic arm's movement, contact force information is acquired in real time via force sensors mounted on the end effector, and the end contact force is adjusted online using an impedance control algorithm. A trajectory tracking error threshold is set; when the error between the actual trajectory and the planned trajectory exceeds the threshold, local replanning is triggered. Utilizing the planning module's dual-constraint modeling and hierarchical optimization path generation method, the local path is replanned within 10ms, ensuring the robotic arm can avoid obstacles or adjust its trajectory in time, guaranteeing the safety and accuracy of the movement.

[0109] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to specific implementations. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A dual-robotic arm adaptive motion neural network optimization system, characterized in that, include: The feature fusion perception module is used to synchronize multi-sensor data through field-programmable gate array, combine target detection model, point cloud data deep learning model and Doppler frequency shift to extract 2D and 3D features of obstacles, and use spatiotemporal graph neural network to fuse position and velocity information to generate real-time environmental state matrix. The motion chain prediction module is used to quantify the probability of obstacle movement based on historical trajectory clustering and Bayesian prediction, construct a collision risk field, and dynamically assess the collision risk within the next 2 seconds. The planning module is used to search for low-risk paths by improving the A algorithm, optimize joint motion trajectories by combining quadratic programming, and embed kinematic and dynamic constraints to ensure path feasibility and safety. The specific operation steps of the planning module are as follows: Kinematic constraints: Based on the kinematic principle of the robotic arm, the feasible region of angular velocity-acceleration of the robotic arm joints is established; through the DH parameters of the robotic arm, the Jacobian matrix J is derived, and the angular velocity and acceleration constraints in the joint space are projected onto the end effector space through the Jacobian matrix to obtain the feasible range of velocity and acceleration of the end effector. Then, the feasible range of velocity and acceleration of the end effector is corrected by the correction mechanism. Dynamic constraints: The recursive Newton-Euler algorithm is used to calculate the forces and torques of each joint sequentially, starting from the base of the robotic arm. Analyze the motor torque limits and transmission capacity factors of each joint of the robotic arm, determine the torque limits of each joint, generate the feasible torque domain, and prevent the robotic arm from being damaged by excessive torque during movement; The hierarchical optimization path generation process is as follows: Primary planning: Use the improved A algorithm to search for low-collision-probability paths in the risk field, with the cost function as follows: ,in These are the weighting coefficients. This represents the total risk value along the path. The path length is determined by adjusting the weight coefficients to balance path risk and length. The algorithm starts from the starting point and expands the search nodes according to the heuristic function until the target point is found or it is determined that no feasible path exists, thus generating a preliminary planned path. Secondary correction: Discretize the primary planning path into 50 waypoints, and treat each waypoint as a quadratic programming problem to be solved; The objective function is Where q is the joint angle and J is the Jacobian matrix. The desired terminal velocity; The constraints are , These represent the upper and lower limits of joint torque. For joint driving torque, The torque generated by the Coriolis force and centrifugal force; By solving a quadratic programming problem, the joint angles at each waypoint are optimized, making the end effector movement of the robotic arm smoother and more accurate. And joint torque energy consumption is optimized through an energy consumption optimization mechanism; Feedback Adjustment: During the movement of the robotic arm, contact force information is acquired in real time by force sensors installed on the end effector, and the end contact force is adjusted online using an impedance control algorithm; Set a trajectory tracking error threshold. When the error between the actual trajectory and the planned trajectory exceeds the threshold, trigger local replanning. By utilizing the dual-constraint modeling and hierarchical optimization path generation method of the planning module, the local path is replanned within 10ms, ensuring that the robotic arm can avoid obstacles or adjust its motion trajectory in time.

2. The dual-arm adaptive motion neural network optimization system according to claim 1, characterized in that, The execution process of the feature fusion perception module is as follows: Using FPGA hardware synchronization signals, binocular cameras, millimeter-wave radar, and robotic arm encoders are connected respectively. By using hardware circuitry and synchronization algorithms, a specific timestamp is added to each sensor data. Using a fixed point in time as the starting reference, time calibration is performed on sensor data of different frequencies; Cross-modal feature extraction: The YOLOv5s model is used to detect 2D bounding boxes of obstacles in the RGB image. The RGB image acquired by the binocular camera is input into the YOLOv5s model. The YOLOv5s model first extracts features from the image by using a network structure of convolutional and pooling layers to extract feature information at different levels in the image. In the detection head, the anchor box mechanism is used to predict the 2D bounding box location, category, and confidence level of obstacles; PointNet++ is used to extract 3D geometric features of obstacles from point cloud data. Point cloud data is obtained from millimeter-wave radar and then input into the PointNet++ model. The model extracts local point cloud features at different scales through sampling, grouping, and feature learning operations. First, the original point cloud is sampled to reduce the amount of data while retaining key feature points. Then, the sampled points are grouped, and feature learning is performed on the point cloud within each group to obtain its 3D geometric features. The radial velocity of an obstacle is calculated from millimeter-wave radar data using Doppler frequency shift. The millimeter-wave radar emits a continuous wave signal, and when the signal is reflected back from an obstacle, a Doppler frequency shift is generated. According to the Doppler effect formula ,in For Doppler frequency shift, The radial velocity of the obstacle. For radar wavelength, Using the transmitted signal frequency, the spectrum of the echo signal received by the millimeter-wave radar is analyzed to calculate the Doppler frequency shift, and then the radial velocity of the obstacle is obtained. Finally, the fused obstacle state matrix is ​​output through attention fusion.

3. The dual-arm adaptive motion neural network optimization system according to claim 2, characterized in that, The specific operation steps of the feature fusion perception module in outputting the fused obstacle state matrix through attention fusion are as follows: A spatiotemporal graph neural network is constructed, using the 2D bounding boxes, 3D geometric features, and radial velocities of each obstacle as node features; For the calculation of edge weights between nodes, the edge weights are dynamically calculated using a function based on the relative distance and speed difference between obstacles. By utilizing the message passing mechanism of graph neural networks, node features are fused in the spatiotemporal dimensions, and the final output is a fused obstacle state matrix containing position, velocity, size confidence information.

4. The dual-arm adaptive motion neural network optimization system according to claim 1, characterized in that, The specific operation steps of the kinetic chain prediction module are as follows: First, motion pattern clustering is performed using an incremental trajectory clustering method based on a spatiotemporal autoencoder; Online Bayesian prediction: Real-time matching of current obstacle motion features with a pattern library to calculate the posterior probability of each pattern; For patterns with a probability >15%, the corresponding dynamic equation is used to generate multiple predicted trajectories within the next 2 seconds; the output is a set of trajectories with probability weights. Collision risk field modeling: The predicted trajectory is transformed into a probabilistic risk field, and the field strength formula is: in Let be the probability of trajectory i. The safe expansion zone for the robotic arm body; This represents the risk field strength of trajectory i at point x and time t; A point in space; : Represents the position of trajectory i at time t; This is the transpose symbol.

5. The dual-arm adaptive motion neural network optimization system according to claim 4, characterized in that, The specific steps for motion pattern clustering in the motion chain prediction module are as follows: The historical obstacle trajectory data is sliced, and each trajectory segment is extracted into a 0.5-second data fragment containing the temporal information of position, velocity, and acceleration. The coordinates are converted to a relative coordinate system with the robotic arm base as the origin, and the velocity and acceleration are normalized to limit the numerical range to [-1, 1]. For each data segment, motion status labels are automatically added. The process of constructing a deep network model containing an encoder and a decoder is as follows: Encoder: A one-dimensional convolutional neural network is used to extract the spatial features of the trajectory, with the kernel size set to 5 and the number of channels to 64; It is connected to the Long Short-Term Memory network layer with 128 hidden units to capture temporal dependencies; The encoder outputs a 32-dimensional latent vector, which represents the core features of the trajectory. Decoder: Reconstructs the original trajectory data through deconvolutional layers and fully connected layers to ensure that the reconstruction error is minimized; Joint optimization: During training, both trajectory reconstruction error and cluster separability of the latent space are optimized simultaneously; by introducing a clustering loss function, the latent vector is forced to align with the centroids of the six preset motion patterns; Real-time detection of the matching degree between new trajectory data and existing patterns: If the similarity between the new trajectory and all existing patterns is lower than 85%, the temporary storage mechanism is triggered. After the same type of trajectory is detected 5 times continuously, a new pattern is automatically added to the feature library; the unconfirmed temporary pattern is stored in the buffer area, and it is decided whether to permanently retain it through manual review; For usage patterns that have not been matched within 72 hours, the confidence weight is automatically reduced by 20% per week. When the weight is lower than 10%, it is archived as a historical pattern and no longer participates in real-time prediction.

6. The dual-arm adaptive motion neural network optimization system according to claim 1, characterized in that, The specific operation steps for the correction mechanism in the planning module to correct the feasible ranges of the speed and acceleration of the end effector are as follows: The correction mechanism analyzes by collecting the influencing parameters of the robotic arm. The influencing parameters include: Joint torque load rate: The ratio of the actual torque of each joint to the rated torque; Trajectory tracking error: The Euclidean distance between the actual position of the end and the planned position; Instantaneous power consumption: The real-time power calculated based on the motor current; Environmental interference intensity: The amplitude of the external contact force obtained through the end force sensor; Health index: The comprehensive score of joint wear degree, temperature, and dust concentration based on historical data; Map all parameters to the 0-1 range, assign weights according to the importance of the parameters, and then use the formula: To obtain the corrected score SZ; where These are the mapping values ​​for joint torque load rate, trajectory tracking error, instantaneous power consumption, environmental interference intensity, and health indicators, respectively. These are the preset weighting coefficients for joint torque load rate, trajectory tracking error, instantaneous power consumption, environmental interference intensity, and health indicators, respectively. Then, different risk intervals are divided according to the correction score SZ, which are: Low risk, i.e., SZ ≤ 0.3: Allow the end speed or acceleration to reach the theoretical maximum value; Medium risk, i.e., 0.3 < SZ ≤ 0.6: Limit the motion parameters proportionally: , where is the limited end velocity, is the theoretical maximum end velocity; the limited acceleration; is the theoretical maximum acceleration; High risk, i.e., SZ > 0.6: Trigger a three-level response, including: First-level response, i.e., 0.6 < SZ ≤ 0.8: Reduce the speed to 50% and perform local replanning; Second-level response, i.e., 0.8 < SZ ≤ 0.95: Pause the current action and switch to the impedance control mode to absorb external forces; Third-level response, i.e., S > 0.95: Emergency stop and lock the joints.

7. The dual-arm adaptive motion neural network optimization system according to claim 1, characterized in that, The specific operation steps for the energy consumption optimization mechanism in the planning module are as follows: The objective function is adjusted to minimize the weighted combination of the end speed tracking error and the sum of the squares of the joint torques, where the energy consumption weight coefficient is set to 0.3; Through the optimization algorithm, on the premise of satisfying the kinematic and dynamic constraints, the joint motion plan with lower torque consumption is preferentially selected; During the operation of the robotic arm, the current data of each joint motor is collected in real time to calculate the instantaneous power: If the power of a certain joint exceeds the safety threshold for 10 consecutive control cycles, the energy consumption weight coefficient is automatically increased to 0.6, forcing the algorithm to reallocate the joint load; Trigger the joint task migration mechanism: Transfer some motion tasks of the high-load joint to the low-load joint, and adjust the weight of the Jacobian matrix to make the end motion of the robotic arm completed by multiple joints in cooperation; Introduce an elastic buffer in the joint torque constraint condition: When the torque reaches 90% of the limit value, a warning signal is generated and the acceleration limit is reduced by 5% to gain time for dynamic adjustment; If the torque continues to exceed the limit, trigger an emergency load reduction strategy: Pause the current task and perform a reverse torque compensation operation.

Images