Dual-arm robot vision modeling and collaborative control method based on long short-term memory network

Abstract

This invention discloses a visual modeling and cooperative control method for a dual-arm robot based on a long short-term memory (LSTM) network. The method includes: Step 1, building a heterogeneous dual-arm robot system platform and establishing the coordinate relationship between the end effector of the robotic arm, the wrist camera, and the target object; Step 2, collecting joint states of the robotic arm and visual observation data from the wrist RGB-D camera, constructing a dataset, and training a camera-robotic arm kinematic model using an LSM network; Step 3, using the trained PI-LSTM model to predict the pose of the target object in the base coordinate system based on real-time visual observations and joint states; and Step 4, introducing the predicted pose information into an optimized inverse kinematics solution framework to generate joint control commands online. This invention, by constructing a data-driven mapping between camera observations and robot kinematics, and combining physical prior constraints such as visual reprojection consistency, can stably estimate the target and end effector pose even in the presence of hand-eye calibration errors and joint perturbations.

Description

Technical Field

[0001] This invention belongs to the field of robot vision servoing and dual-arm cooperative operation control technology, specifically involving a dual-arm robot vision modeling and cooperative control method based on long short-term memory network. Background Technology

[0002] Dual-arm cooperative operation is increasingly used in tasks such as industrial logistics and collaborative assembly due to its ability to improve the stability of grasping and handling through multi-contact and closed-loop coordination. In vision-based closed-loop control, position-based visual servoing (PBVS) and image-based visual servoing (IBVS) are two common frameworks: IBVS directly acts on image features and is more robust to the model; PBVS directly adjusts the pose in the three-dimensional task space, making it easier to integrate redundancy allocation and safety constraints, but its control effect often highly depends on the accuracy of hand-eye relationship and kinematic calibration.

[0003] In PBVS applications, the system requires high accuracy in hand-eye relationship, camera intrinsic and extrinsic parameters, and robot kinematics models. In addition to static geometric offsets caused by assembly deviations and calibration errors, non-geometric errors caused by structural compliance often exhibit characteristics of changing with configuration, drifting over time, and being difficult to parameterize precisely using standard geometric models. These deviations may also propagate through inverse kinematics and control laws, leading to a decrease in end-effector positioning and target alignment performance, thereby affecting the stability and robustness of dual-arm cooperative handling.

[0004] To address the aforementioned issues, existing methods can be broadly categorized into two types: the first is model augmentation and sensor fusion methods, which explicitly model pose uncertainties and utilize filtering or optimization to fuse multi-source measurements, reducing reliance on single static calibration results; the second is data-driven compensation methods, such as online Jacobi adaptation, neural network models, or residual learning, which compensate for unmodeled errors within the control loop. Many existing data-driven methods primarily learn and correct at the level of control variables or velocity / pose increments, still requiring a reliable camera-kinematic calibration foundation. Furthermore, it is difficult to directly and stably output unified 3D pose information suitable for PBVS from raw depth observations and joint measurements. Additionally, bi-arm handling requires constraints such as closed-chain coordination and joint limitations, making it difficult to guarantee the kinematic feasibility of online movements solely through learning strategies. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a visual dual-arm cooperative handling control method based on kinematic modeling using a long short-term memory neural network, oriented towards model mismatch. The method specifically includes the following steps:

[0006] A visual dual-arm cooperative handling control method based on kinematic modeling using a long short-term memory neural network, the method specifically includes the following steps:

[0007] Step 1: Build a dual-arm system platform, derive the kinematic constraints of the robotic arm system with a camera mounted on the wrist, and construct a system environment model including the robotic arm, the wrist camera, and the target object.

[0008] Step 2: Collect data and train the PI-LSTM camera-robotic arm kinematics model;

[0009] Step 3: Predict the pose information of the target object in the robot arm's base coordinate system and the pose information of the robot arm's end effector in the base coordinate system;

[0010] Step 4: Constraint optimization inverse kinematics solution and control execution of each stage of the dual-arm handling.

[0011] Furthermore, the specific steps of step 1 are as follows:

[0012] Step 1.1: Construct a dual-arm robotic arm system platform: This includes two tandem robotic arms, one on the left and one on the right, whose base coordinate systems are fixed to a common base coordinate system. Below; let the coordinate systems of the end effectors of the left and right robotic arms be respectively... and The corresponding joint variable vectors are respectively ;

[0013] Step 1.2: Establish the forward kinematic model of the robotic arm using the exponential product form; for the th One robotic arm ( The pose of its end effector relative to the base coordinate system is expressed as:

[0014] in Indicates the first The joint rotation corresponding to each joint;

[0015] Step 1.3: Install RGB-D cameras on the wrists of both robotic arms, and set the camera coordinate systems as follows: and The extrinsic parameter transformation of the camera relative to the end effector is defined as follows:

[0016] The camera's pose in the base coordinate system can then be expressed as:

[0017] Let the target object's coordinate system be... Its pose in the base coordinate system is represented as: ;

[0018] Step 1.4: Construct a system environment model including a dual-arm robotic arm, a wrist camera, and a target object. Its kinematic state variables can be uniformly represented as: To simulate model mismatch and non-geometric errors caused by factors such as load pressure and structural compliance deformation, a constant perturbation term is introduced into the joint angle of the robotic arm in the environmental model; let the first... The true and measured values ​​of the joint angles of the robotic arm satisfy the following conditions. in This is a constant joint disturbance vector used to characterize non-geometric errors such as configuration-related compliance deviations; in subsequent modeling and control processes, it is used... As input to reflect the effects of model mismatch.

[0019] Furthermore, the specific steps of step 2 are as follows:

[0020] Step 2.1: During the operation of the dual-arm system, visual observation information of the target object is acquired through an RGB-D camera mounted on the wrist, and the joint states of the corresponding robotic arm are acquired simultaneously; let the visual observation of the ARM robotic arm in the k-th sample be represented as... in The pixel coordinates of the target center point in the camera image. For the corresponding depth value, the joint state of the robotic arm is recorded as follows: ;

[0021] Step 2.2: After time-aligning and preprocessing the acquired visual observations and joint states, the input dataset is formed. The corresponding output dataset is defined as follows: ;

[0022] Where N is the total number of training samples. This represents the pose of the target object in the base coordinate system. This corresponds to the pose of the robotic arm's end effector;

[0023] Step 2.3: Construct a camera-robotic arm kinematic modeling training framework based on a Long Short-Term Memory (LSTM) neural network. Use LSTM as the backbone network for temporal feature extraction, and jointly regress the target object pose and the robotic arm end effector pose at its output. Let the PI-LSTM camera-robotic arm kinematic model be... ,in For network parameters;

[0024] Step 2.4: Training the PI-LSTM camera-robotic arm kinematic model In time The visual observations and joint state sequences are used as input, and the input format is as follows: The model output includes the pose of the target object and the corresponding pose of the robotic arm's end effector. in For network parameters;

[0025] By minimizing the composite loss function defined in step 2.3 Gradient descent method is used for and camera parameters Perform end-to-end joint optimization until the loss function converges to obtain the parameters of the trained PI-LSTM camera-robotic arm kinematic model. .

[0026] Furthermore, step 2.3 also includes the following steps:

[0027] To introduce visual geometric consistency constraints, a wrist-eye extrinsic parameter transformation is proposed. The end effector pose remains unchanged during training and execution; predicted by the network. Calculate the corresponding camera pose as follows ;

[0028] Using a learnable pinhole camera model and distortion parameters, the camera intrinsic parameter matrix is ​​denoted as... The distortion parameter is ; predict the location of the target object Transform to the camera coordinate system and project onto the image plane to obtain the predicted pixel coordinates and depth values:

[0029] ;

[0030] Based on the above prediction results, the composite loss function is defined as follows:

[0031] The loss items are as follows:

[0032] Loss of monitoring target location: ;

[0033] End-effector pose supervision loss: ;

[0034] Reprojection consistency loss: ;

[0035] Deep consistency loss: ;

[0036] Noise prediction loss: ;

[0037] in This refers to the configuration-related joint correction amount predicted by the network;

[0038] Camera parameter regularization terms: in and These represent the offsets of the distortion parameters within the camera relative to their initial values.

[0039] Furthermore, the specific steps of step 3 are as follows:

[0040] Step 3.1: During the online operation of the system, at the [number]th [time period]... Real-time acquisition of the joint status of the dual-arm robotic arm at each control moment. and visual observation information from the wrist camera Construct the model input sequence using a time window of length $T$. It is then input into the PI-LSTM camera-robotic arm kinematic model trained in step 2. The predicted poses of the target object and the corresponding end effector of the robotic arm are obtained:

[0041]

[0042] in This represents the pose of the target object in the base coordinate system, predicted by the corresponding wrist camera. This indicates the pose of the corresponding robotic arm end effector in the base coordinate system;

[0043] When both wrist cameras observe the same target object at the same time, the pose predictions of the two target objects are obtained respectively. The target object pose is then fused using an averaging method to obtain the target object pose used for control:

[0044] ;

[0045] Step 3.2: Compare and analyze the target object pose and end effector pose prediction results obtained in Step 3.1 with the pose results calculated based on traditional analytical kinematics and camera calibration methods.

[0046] Furthermore, the specific steps of step 4 are as follows:

[0047] Step 4.1: Based on the target object pose obtained in Step 3 Based on the pose prediction results of the dual-arm end effector, an optimization-based inverse kinematics solution problem is constructed; Let... The inverse kinematics solution for a single-arm robotic arm, representing the end effector pose calculated from forward kinematics, can be expressed as:

[0048]

[0049] in To determine the desired end effector pose, This indicates the end-effector pose error. This is the joint configuration for the previous control cycle. and These represent the upper and lower limits of the joint, respectively;

[0050] Step 4.2: Divide the dual-arm collaborative handling task into the approach phase, grasping phase, lifting phase and placement phase, and execute the corresponding control strategies for each phase.

[0051] Furthermore, step 4.2 specifically includes the following stages:

[0052] (1) Approach phase: Based on the target object pose predicted in step 3 Generate the desired approximate pose for the dual-arm end effector. By solving the single-arm inverse kinematics in step 4.1, the joint control commands are obtained, guiding the ends of both arms to gradually approach the target object.

[0053] (2) Grasping Phase: When the pose error between the end effector and the target object is less than a preset threshold, the end effector pose remains unchanged and the grasping action is performed, and the grasping completion time is recorded. End-point object relative pose ;

[0054] (3) Lifting phase: while maintaining the aforementioned relative pose Under the premise of keeping the target object unchanged, update the lifting pose according to the desired lifting position. By introducing cooperative inverse kinematics solution with closed-chain constraints for both arms, control commands for the joints of both arms are generated to achieve stable lifting of the target object.

[0055] (4) Placement phase: Update the position of the target object based on the pose of the placement location. We continue to use the inverse kinematics solution with closed-chain constraints of the two arms to guide the two arms to move in coordination to the placement position and complete the release of the target object.

[0056] The present invention also relates to a visual dual-arm cooperative handling control system based on kinematic modeling of long short-term memory neural network, the system comprising a computer module that runs the above-mentioned visual dual-arm cooperative handling control method based on kinematic modeling of long short-term memory neural network. Attached Figure Description

[0057] Figure 1 The flowchart shows the visual modeling and cooperative control method for a dual-arm robot based on a long short-term memory network according to the present invention.

[0058] Figure 2 This is a schematic diagram of the heterogeneous dual-arm simulation test platform system architecture for the visual dual-arm cooperative handling control method based on kinematic modeling of long short-term memory neural network according to the present invention.

[0059] Figure 3 A comparison curve showing the variation of modeling error between traditional analytical modeling and PI-LSTM modeling as a function of noise;

[0060] Figure 4 This is a schematic diagram illustrating the convergence of PI-LSTM training loss and prediction accuracy analysis results in this invention.

[0061] Figure 5 This is a schematic diagram illustrating the various stages in which the NN-PBVS of the present invention completes its transport task. Detailed Implementation

[0062] The present invention will be described in detail below with reference to specific embodiments.

[0063] like Figure 1 As shown, the present invention provides a visual dual-arm cooperative handling control method based on kinematic modeling of a long short-term memory neural network, which specifically includes the following steps:

[0064] Step 1: Build a dual-arm system platform, derive the kinematic constraints of the robotic arm system with a camera mounted on the wrist, and construct a system environment model including the robotic arm, the wrist camera, and the target object.

[0065] The specific steps of step 1 are as follows:

[0066] Step 1.1: Construct a dual-arm robotic arm system platform: This includes two tandem robotic arms, one on the left and one on the right, whose base coordinate systems are fixed to a common base coordinate system. Below; let the coordinate systems of the end effectors of the left and right robotic arms be respectively... and The corresponding joint variable vectors are respectively ;

[0067] Step 1.2: Establish the forward kinematic model of the robotic arm using the exponential product form; for the th One robotic arm ( The pose of its end effector relative to the base coordinate system is expressed as:

[0068] in Indicates the first The joint rotation corresponding to each joint;

[0069] Step 1.3: Install RGB-D cameras on the wrists of both robotic arms, and set the camera coordinate systems as follows: and The extrinsic parameter transformation of the camera relative to the end effector is defined as follows:

[0070] The camera's pose in the base coordinate system can then be expressed as:

[0071] Let the target object's coordinate system be... Its pose in the base coordinate system is represented as: ;

[0072] Step 1.4: Construct a system environment model including a dual-arm robotic arm, a wrist camera, and a target object. Its kinematic state variables can be uniformly represented as: To simulate model mismatch and non-geometric errors caused by factors such as load pressure and structural compliance deformation, a constant perturbation term is introduced into the joint angle of the robotic arm in the environmental model; let the first... The true and measured values ​​of the joint angles of the robotic arm satisfy the following conditions. in This is a constant joint disturbance vector used to characterize non-geometric errors such as configuration-related compliance deviations; in subsequent modeling and control processes, it is used... As input to reflect the effects of model mismatch.

[0073] Step 2: Collect data and train the PI-LSTM camera-robotic arm kinematics model;

[0074] The specific steps of step 2 are as follows:

[0075] Step 2.1: During the operation of the dual-arm system, visual observation information of the target object is acquired through an RGB-D camera mounted on the wrist, and the joint states of the corresponding robotic arm are acquired simultaneously; let the visual observation of the ARM robotic arm in the k-th sample be represented as... in The pixel coordinates of the target center point in the camera image. For the corresponding depth value, the joint state of the robotic arm is recorded as follows: ;

[0076] Step 2.2: After time-aligning and preprocessing the acquired visual observations and joint states, the input dataset is formed. The corresponding output dataset is defined as follows: ;

[0077] Where N is the total number of training samples. This represents the pose of the target object in the base coordinate system. This corresponds to the pose of the robotic arm's end effector;

[0078] Step 2.3: Construct a camera-robotic arm kinematic modeling training framework based on a Long Short-Term Memory (LSTM) neural network. Use LSTM as the backbone network for temporal feature extraction, and jointly regress the target object pose and the robotic arm end effector pose at its output. Let the PI-LSTM camera-robotic arm kinematic model be... ,in For network parameters;

[0079] To introduce visual geometric consistency constraints, a wrist-eye extrinsic parameter transformation is proposed. The end effector pose remains unchanged during training and execution; predicted by the network. Calculate the corresponding camera pose as follows ;

[0080] Using a learnable pinhole camera model and distortion parameters, the camera intrinsic parameter matrix is ​​denoted as... The distortion parameter is ; predict the location of the target object Transform to the camera coordinate system and project onto the image plane to obtain the predicted pixel coordinates and depth values:

[0081] ;

[0082] Based on the above prediction results, the composite loss function is defined as follows:

[0083] The loss items are as follows:

[0084] Loss of monitoring target location: ;

[0085] End-effector pose supervision loss: ;

[0086] Reprojection consistency loss: ;

[0087] Deep consistency loss: ;

[0088] Noise prediction loss: ;

[0089] in This refers to the configuration-related joint correction amount predicted by the network;

[0090] Camera parameter regularization terms: in and These represent the offsets of the distortion parameters within the camera relative to their initial values.

[0091] Step 2.4: Training the PI-LSTM camera-robotic arm kinematic model In time The visual observations and joint state sequences are used as input, and the input format is as follows: The model output includes the pose of the target object and the corresponding pose of the robotic arm's end effector. in For network parameters;

[0092] By minimizing the composite loss function defined in step 2.3 Gradient descent method is used for and camera parameters Perform end-to-end joint optimization until the loss function converges to obtain the parameters of the trained PI-LSTM camera-robotic arm kinematic model. .

[0093] Step 3: Predict the pose information of the target object in the robot arm's base coordinate system and the pose information of the robot arm's end effector in the base coordinate system;

[0094] The specific steps of step 3 are as follows:

[0095] Step 3.1: During the online operation of the system, at the [number]th [time period]... Real-time acquisition of the joint status of the dual-arm robotic arm at each control moment. and visual observation information from the wrist camera Construct the model input sequence using a time window of length $T$. It is then input into the PI-LSTM camera-robotic arm kinematic model trained in step 2. The predicted poses of the target object and the corresponding end effector of the robotic arm are obtained:

[0096]

[0097] in This represents the pose of the target object in the base coordinate system, predicted by the corresponding wrist camera. This indicates the pose of the corresponding robotic arm end effector in the base coordinate system;

[0098] When both wrist cameras observe the same target object at the same time, the pose predictions of the two target objects are obtained respectively. The target object pose is then fused using an averaging method to obtain the target object pose used for control:

[0099] ;

[0100] Step 3.2: Compare and analyze the target object pose and end effector pose prediction results obtained in Step 3.1 with the pose results calculated based on traditional analytical kinematics and camera calibration methods.

[0101] Step 4: Constraint optimization inverse kinematics solution and control execution of each stage of the dual-arm handling.

[0102] Step 4.1: Based on the target object pose obtained in Step 3 Based on the pose prediction results of the dual-arm end effector, an optimization-based inverse kinematics solution problem is constructed; Let... The inverse kinematics solution for a single-arm robotic arm, representing the end effector pose calculated from forward kinematics, can be expressed as:

[0103]

[0104] in To determine the desired end effector pose, This indicates the end-effector pose error. This is the joint configuration for the previous control cycle. and These represent the upper and lower limits of the joint, respectively;

[0105] Step 4.2: Divide the dual-arm collaborative handling task into the approach phase, grasping phase, lifting phase and placement phase, and execute the corresponding control strategies for each phase.

[0106] Furthermore, step 4.2 specifically includes the following stages:

[0107] (1) Approach phase: Based on the target object pose predicted in step 3 Generate the desired approximate pose for the dual-arm end effector. By solving the single-arm inverse kinematics in step 4.1, the joint control commands are obtained, guiding the ends of both arms to gradually approach the target object.

[0108] (2) Grasping Phase: When the pose error between the end effector and the target object is less than a preset threshold, the end effector pose remains unchanged and the grasping action is performed, and the grasping completion time is recorded. End-point object relative pose ;

[0109] (3) Lifting phase: while maintaining the aforementioned relative pose Under the premise of keeping the target object unchanged, update the lifting pose according to the desired lifting position. By introducing cooperative inverse kinematics solution with closed-chain constraints for both arms, control commands for the joints of both arms are generated to achieve stable lifting of the target object.

[0110] (4) Placement phase: Update the position of the target object based on the pose of the placement location. We continue to use the inverse kinematics solution with closed-chain constraints of the two arms to guide the two arms to move in coordination to the placement position and complete the release of the target object.

[0111] Example

[0112] This implementation is based on a heterogeneous dual-arm robot system, where the left arm is a Franka FR3 seven-DOF robotic arm and the right arm is a Kinova Gen3 seven-DOF robotic arm. The two robotic arms are symmetrically mounted on a common base, forming a dual-arm collaborative operation platform. Each robotic arm has an RGB-D camera mounted on its wrist for real-time acquisition of visual observation information of the target object.

[0113] In this embodiment, ArUco visual markers are set on the surface of the target object. The ArUco markers are identified and located by a wrist RGB-D camera to obtain the pixel coordinates and depth information of the target object in the camera coordinate system, which serves as the visual input for subsequent camera-robotic arm kinematic modeling and pose prediction.

[0114] The physical experimental platform of this embodiment is as follows: Figure 2 As shown, the simulation verification platform is as follows: Figure 3 As shown. The specific steps of the method are as follows:

[0115] Step 1: Build a heterogeneous dual-arm system (Franka FR3 seven degrees of freedom, Kinova Gen3 seven degrees of freedom), establish a common base coordinate system and coordinate relationships between the coordinate system of the dual-arm end caps and the coordinate system of the wrist camera, so as to provide a unified coordinate description for subsequent simulation modeling and control calculation.

[0116] Step 2: Affix an ArUco code (ID=582) to the surface of the target object as a target detection marker, and affix an ArUco code (ID=120) to the placement location as a placement reference marker. After the wrist-mounted RGB-D camera detects the ArUco center, it obtains the pixel coordinates and depth, and defines the visual observation for each sample as follows:

[0117]

[0118] Step 3: Calculate the camera extrinsic parameters and obtain the initial intrinsic parameters of the uncalibrated RealSense camera.

[0119] Step 4: Build a simulation environment based on the physical platform, set δt=0.002 s, and set the positions of the centers of the two Aruco codes.

[0120] Step 5: In this case, 5000 data points were collected, and the dataset was obtained after time alignment and preprocessing.

[0121] The following steps 6, 7, and 8 are all processes of training the system camera-robotic arm kinematic model using a physical information LSTM network.

[0122] Step 6: Construct a camera-robotic arm kinematic model based on a Physically Constrained Long Short-Term Memory (PI-LSTM) network. Input the dataset obtained in Step 5 into the PI-LSTM network for training. The network receives the robotic arm joint states and visual observation information as input in time-series format. Specifically, at time... The length is Time window:

[0123]

[0124] in The ArUco marker center pixel coordinates and depth information detected by the wrist RGB-D camera. This is the joint angle vector of the robotic arm at the corresponding moment.

[0125] The above time series data is input into an LSTM network, where hidden state representations are obtained through temporal feature extraction, and prediction results are output by a fully connected layer to estimate the pose of the target object and the end effector of the robotic arm.

[0126]

[0127] in, This represents the predicted pose of the target object in the base coordinate system. This indicates the pose of the corresponding robotic arm end effector in the base coordinate system. This represents the camera-robotic arm kinematic mapping relationship characterized by network parameters φ. Simultaneously, learnable camera intrinsic parameter matrices and distortion parameters are introduced into the network. Through visual reprojection constraints and depth consistency constraints, the network simultaneously performs camera parameter correction and non-geometric kinematic error compensation during training, thereby establishing a unified mapping model between visual observation and robot kinematics.

[0128] Step 7: Set the loss function as follows:

[0129] .

[0130] Step 8: Set the network parameters and train the kinematic model. .

[0131] The following steps 9-11 are... Solve for the inverse kinematics.

[0132] Step 9: Based on the trained model, input the current observations and q to predict the target pose and end effector pose.

[0133] Step 10: During simulation, the joint states of both arms and the observations from the two wrist cameras are read in real time and input into the trained PI-LSTM model to predict the pose of the target object in the base coordinate system and the poses of the FR3 and Gen3 arm ends in the base coordinate system. When both wrist cameras simultaneously observe the ArUco code (ID=582), two target pose estimates are obtained respectively. The target pose results are then fused using an averaging method to output the target pose estimate for control. The expected pose of the target object predicted visually is... .

[0134] Step 11: Within each control cycle, generate bi-arm joint position commands by solving a constrained optimization inverse kinematics problem. The optimization problem is represented as:

[0135]

[0136] in, Obtained from step 10 The desired end-effector pose is determined; and To determine the maximum and minimum limits of the joint, the optimized inverse kinematics is solved using IPOPT to obtain the joint position command for the current control cycle.

[0137] Steps 12-15 are the four stages of the handling process.

[0138] Step 12: Approach phase. Based on the pose of the target object with ArUco 582 attached, generate the expected approach pose of the end arms. Call the inverse kinematics solution from step 12 to obtain joint commands so that the end arms reach the grasping pre-pose.

[0139] Step 13: Grasping phase. When the end effector reaches the grasping condition, the grasping action is executed, and the relative relationship at the time of grasping is recorded. Then, the lifting phase begins. Under the premise of maintaining the relative relationship constraints between the arms and the object, the joint commands are generated online by using inverse kinematics with closed-chain constraints to achieve coordinated lifting and stable transport.

[0140] Step 14: Lifting phase. While maintaining the coordination constraint of both arms, update the target pose and repeat step 11 to solve the joint commands online to achieve stable lifting of the target object.

[0141] Step 15: Placement phase. Using the pose corresponding to the placement position ArUco 120 as the placement reference, the desired placement pose of the target object is generated. The joint commands of the two arms are then solved online using optimized inverse kinematics with closed-chain constraints to guide the target to the placement position. After placement, the target object is released, thus completing the collaborative handling and placement task.

[0142] In the lifting and placement phases described in steps 14 and 15, to ensure the stability of the target object under force during the dual-arm collaborative handling process and to maintain the relative pose relationship between the end effectors of the two arms and the target object, a dual-arm closed-chain constraint is used for control. Let the time of grasping completion be... The pose of the target object at that moment is Then define the relative transformation of the end-object after the grabbing is established as a constant: in To indicate the first The robotic arm has a joint angle of... The pose transformation of the end effector in the base coordinate system.

[0143] During the promotion and placement process, the closed-chain consistency constraint is satisfied: The closed-chain constraint is introduced as an equality constraint in the optimized inverse kinematics described in step 11, thereby ensuring that the cooperative relationship between the two arms remains consistent with the relative pose of the target object when solving joint commands online.

[0144] Simulation Experiment

[0145] To verify the effectiveness of the method of the present invention, the method described in the above specific embodiments was tested on a heterogeneous dual-arm simulation platform. The simulation system consists of two 7-DOF serial robotic arms, corresponding to Franka FR3 and Kinova Gen3 respectively. Each robotic arm is equipped with an RGB-D camera at its wrist to acquire visual observation information of the target object, which is a regular box with ArUco codes. The system structure is as follows. Figure 3 As shown. To simulate model mismatch and non-geometric errors in actual working conditions, a perturbation is injected into the joint angle measurement during the simulation. The perturbation takes the form of... ,in This is used to describe the quasi-static deviation term caused by the load. The zero-mean Gaussian white noise term is used to simulate high-frequency random disturbances in the environment. The target point for the grasping and placement phase is set 0.03 m outside the edge of the corresponding ArUco marker, and the physical size of the marker is 0.1 m.

[0146] The simulation first verifies the accuracy of the PI-LSTM camera-robotic arm kinematic modeling of this invention, analyzing the convergence of training loss and the prediction errors of target pose and end-effector pose under different joint noise settings. Then, the method of this invention is compared with traditional modeling processes based on camera calibration and analytical kinematics, as well as related baseline methods. The results show that this invention can maintain small target position and end-effector pose errors without explicit camera calibration, and averaging the two estimates when both wrist cameras simultaneously observe the same target can further improve the stability of pose estimation. In the control performance verification, the dual-arm handling task is divided into four stages: approach, grasping, lifting, and placement. In the approach stage, the wrist camera uses ArUco codes to obtain the target pose and guides the end-effector to approach; grasping is performed after the error meets the threshold. In the lifting and placement stages, under the premise of maintaining dual-arm cooperative constraints, constraint optimization inverse kinematics with closed-chain constraints is used to solve control commands online to complete lifting and placement. Simulation comparison results show that this invention still has a more stable cooperative handling effect under conditions of inaccurate camera parameters and joint disturbance injection.

[0147] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A visual dual-arm cooperative handling control method based on kinematic modeling using long short-term memory neural networks, characterized in that, The method specifically includes the following steps: Step 1: Build a dual-arm system platform, derive the kinematic constraints of the robotic arm system with a camera mounted on the wrist, and construct a system environment model including the robotic arm, the wrist camera, and the target object. Step 2: Collect data and train the PI-LSTM camera-robotic arm kinematics model; Step 3: Predict the pose information of the target object in the robot arm's base coordinate system and the pose information of the robot arm's end effector in the base coordinate system; Step 4: Constraint optimization inverse kinematics solution and control execution of each stage of the dual-arm handling.

2. The method for visual modeling and cooperative control of a dual-arm robot based on a long short-term memory network according to claim 1, characterized in that, The specific steps of step 1 are as follows: Step 1.1: Construct a dual-arm robotic arm system platform: This includes two tandem robotic arms, one on the left and one on the right, whose base coordinate systems are fixed to a common base coordinate system. Below; let the coordinate systems of the end effectors of the left and right robotic arms be respectively... and The corresponding joint variable vectors are respectively ; ； Step 1.2: Establish the forward kinematic model of the robotic arm using the exponential product form; for the th One robotic arm ( The pose of its end effector relative to the base coordinate system is expressed as: in Indicates the first The joint rotation corresponding to each joint; Step 1.3: Install RGB-D cameras on the wrists of both robotic arms, and set the camera coordinate systems as follows: and The extrinsic parameter transformation of the camera relative to the end effector is defined as follows: The camera's pose in the base coordinate system can then be expressed as: ; Let the target object's coordinate system be... Its pose in the base coordinate system is represented as: ; Step 1.4: Construct a system environment model including a dual-arm robotic arm, a wrist camera, and a target object. Its kinematic state variables can be uniformly represented as: To simulate model mismatch and non-geometric errors caused by factors such as load pressure and structural compliance deformation, a constant perturbation term is introduced into the joint angle of the robotic arm in the environmental model; let the first... The true and measured values ​​of the joint angles of the robotic arm satisfy the following conditions. in This is a constant joint disturbance vector used to characterize non-geometric errors such as configuration-related compliance deviations; in subsequent modeling and control processes, it is used... As input to reflect the effects of model mismatch.

3. The visual dual-arm cooperative handling control method based on kinematic modeling of long short-term memory neural network according to claim 1, characterized in that, The specific steps of step 2 are as follows: Step 2.1: During the operation of the dual-arm system, visual observation information of the target object is acquired through an RGB-D camera mounted on the wrist, and the joint states of the corresponding robotic arm are acquired simultaneously; let the visual observation of the ARM robotic arm in the k-th sample be represented as... in The pixel coordinates of the target center point in the camera image. For the corresponding depth value, the joint state of the robotic arm is recorded as follows: ; Step 2.2: After time alignment and preprocessing of the acquired visual observations and joint states, the input dataset is formed. The corresponding output dataset is defined as follows: ; Where N is the total number of training samples. This represents the pose of the target object in the base coordinate system. This corresponds to the pose of the robotic arm's end effector; Step 2.3: Construct a camera-robotic arm kinematic modeling training framework based on a Long Short-Term Memory (LSTM) neural network. Use LSTM as the backbone network for temporal feature extraction, and jointly regress the target object pose and the robotic arm end effector pose at its output. Let the PI-LSTM camera-robotic arm kinematic model be... ,in For network parameters; Step 2.4: Training the PI-LSTM camera-robotic arm kinematic model In time The visual observations and joint state sequences are used as input, and the input format is as follows: The model output includes the pose of the target object and the corresponding pose of the robotic arm's end effector. in For network parameters; By minimizing the composite loss function defined in step 2.3 Gradient descent method is used for and camera parameters Perform end-to-end joint optimization until the loss function converges to obtain the parameters of the trained PI-LSTM camera-robotic arm kinematic model. .

4. The visual dual-arm cooperative handling control method based on kinematic modeling of long short-term memory neural network according to claim 3, characterized in that, Step 2.3 also includes the following steps: To introduce visual geometric consistency constraints, a wrist-eye extrinsic parameter transformation is proposed. The end effector pose remains unchanged during training and execution; predicted by the network. Calculate the corresponding camera pose as follows ; Using a learnable pinhole camera model and distortion parameters, the camera intrinsic parameter matrix is ​​denoted as... The distortion parameters are ; predict the location of the target object Transform to the camera coordinate system and project onto the image plane to obtain the predicted pixel coordinates and depth values: ； Based on the above prediction results, the composite loss function is defined as follows: ; The loss items are as follows: Loss of monitoring target location: ; End-effector pose supervision loss: ; Reprojection consistency loss: ; Deep consistency loss: ; Noise prediction loss: ; in This refers to the configuration-related joint correction amount predicted by the network; Camera parameter regularization terms: in and These represent the offsets of the distortion parameters within the camera relative to their initial values.

5. The visual dual-arm cooperative handling control method based on long short-term memory neural network kinematic modeling according to claim 1, characterized in that, The specific steps of step 3 are as follows: Step 3.1: During the online operation of the system, at the [number]th [time period]... Real-time acquisition of the joint status of the dual-arm robotic arm at each control moment. and visual observation information from the wrist camera Construct the model input sequence using a time window of length $T$. It is then input into the PI-LSTM camera-robotic arm kinematic model trained in step 2. The predicted poses of the target object and the corresponding end effector of the robotic arm are obtained: ； in This represents the pose of the target object in the base coordinate system, predicted by the corresponding wrist camera. This indicates the pose of the corresponding robotic arm end effector in the base coordinate system; When both wrist cameras observe the same target object at the same time, the pose predictions of the two target objects are obtained respectively. The target object pose is then fused using an averaging method to obtain the target object pose used for control: ； Step 3.2: Compare and analyze the target object pose and end effector pose prediction results obtained in Step 3.1 with the pose results calculated based on traditional analytical kinematics and camera calibration methods.

6. The visual dual-arm cooperative handling control method based on long short-term memory neural network kinematic modeling according to claim 1, characterized in that, The specific steps of step 4 are as follows: Step 4.1: Based on the target object pose obtained in Step 3 Based on the pose prediction results of the dual-arm end effector, an optimization-based inverse kinematics solution problem is constructed; Let The inverse kinematics solution for a single-arm robotic arm, representing the end effector pose calculated from forward kinematics, can be expressed as: ； in To determine the desired end effector pose, This indicates the end-effector pose error. This is the joint configuration for the previous control cycle. and These represent the upper and lower limits of the joint, respectively; Step 4.2: Divide the dual-arm collaborative handling task into the approach phase, grasping phase, lifting phase and placement phase, and execute the corresponding control strategies for each phase.

7. The visual dual-arm cooperative handling control method based on kinematic modeling of long short-term memory neural network according to claim 6, characterized in that, Step 4.2 specifically includes the following stages: (1) Approach phase: Based on the target object pose predicted in step 3 Generate the desired approximate pose for the dual-arm end effector. By solving the single-arm inverse kinematics in step 4.1, the joint control commands are obtained, guiding the ends of both arms to gradually approach the target object. (2) Grasping Phase: When the pose error between the end effector and the target object is less than a preset threshold, the end effector pose remains unchanged and the grasping action is performed, and the grasping completion time is recorded. End-point object relative pose ; (3) Lifting phase: while maintaining the relative pose Under the premise of keeping the target object unchanged, update the lifting pose according to the desired lifting position. By introducing cooperative inverse kinematics solution with closed-chain constraints for both arms, control commands for the joints of both arms are generated to achieve stable lifting of the target object. (4) Placement phase: Update the position of the target object based on the pose of the placement location. We continue to use the inverse kinematics solution with closed-chain constraints of the two arms to guide the two arms to move in coordination to the placement position and complete the release of the target object.

8. A visual dual-arm cooperative handling control system based on kinematic modeling using long short-term memory neural networks, characterized in that, The system includes a computer module that runs the visual modeling and cooperative control method for dual-arm robots based on long short-term memory networks as described in any one of claims 1 to 7.

Images