A differential game-based dual-arm robot cooperative control method and device

Abstract

The application provides a dual-arm robot cooperative control method and device based on differential game, relates to the technical field of robot control, and comprises the following steps: collecting multi-modal data of a dual-arm robot, inputting the multi-modal data into a graph neural network to extract observation feature vectors; obtaining a state vector of the dual-arm robot, inputting the observation feature vectors and the state vector into an extended Kalman filter for recursive estimation to obtain a state estimation value of the dual-arm robot; constructing a left-arm control benefit function and a right-arm control benefit function of the dual-arm robot; based on the state estimation value, iteratively calculating a Nash equilibrium point of the left-arm control benefit function and the right-arm control benefit function through an argmin function to obtain left-arm optimal impedance parameters and right-arm optimal impedance parameters corresponding to the Nash equilibrium point; and driving the dual-arm robot to complete a cooperative operation task according to the left-arm optimal impedance parameters and the right-arm optimal impedance parameters.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a collaborative control method and device for a dual-arm robot based on differential game theory. Background Technology

[0002] With the rapid development of industrial automation, intelligent manufacturing and other fields, dual-arm robots are widely used in precision assembly, flexible grasping and complex workpiece processing due to their human-like collaborative operation capabilities.

[0003] Existing collaborative control technologies for dual-arm robots fail to effectively correct the robot's state during the state estimation stage, resulting in inaccurate state estimation. Furthermore, the design of the control model lacks dynamic representation of the interaction and coupling between the two arms, causing its operational coordination, anti-interference ability, and operational stability to fail to meet the requirements of complex working environments. Summary of the Invention

[0004] To address the aforementioned problems, in a first aspect, the present invention provides a cooperative control method for a dual-arm robot based on differential game theory, comprising:

[0005] Multimodal data of the dual-arm robot are collected through a multi-source sensor system;

[0006] A graph neural network and an extended Kalman filter are constructed. Multimodal data is input into the graph neural network to extract observation features and obtain observation feature vectors. The state vector of the dual-arm robot is obtained. The observation feature vector and the state vector are input into the extended Kalman filter for recursive estimation to obtain the state estimate of the dual-arm robot.

[0007] A differential game model for a dual-arm robot is constructed, which includes the control payoff function of the left arm and the control payoff function of the right arm. Based on the state estimate, the Nash equilibrium point that makes the control payoff functions of the left arm and the right arm reach local minima is calculated iteratively by using the argmin function. The optimal impedance parameters of the left arm and the right arm corresponding to the Nash equilibrium point are then obtained.

[0008] The optimal impedance parameters of the left arm are input into the left arm impedance controller of the dual-arm robot to generate the left arm joint torque command; the optimal impedance parameters of the right arm are input into the right arm impedance controller of the dual-arm robot to generate the right arm joint torque command; the dual-arm robot is driven to complete the collaborative operation task according to the left arm joint torque command and the right arm joint torque command.

[0009] Optionally, the acquisition of multimodal data of the dual-arm robot through a multi-source sensor system includes:

[0010] The system collects visual image data of the working environment, tactile sensing data of the robotic arm end effector, and motion posture data of the inertial measurement unit through a multi-source sensor system.

[0011] Based on the sampling frequency of motion posture data, time stamp interpolation is performed on visual image data and tactile sensing data to synchronize them, thereby obtaining synchronized visual image data and synchronized tactile sensing data.

[0012] Multimodal data consists of synchronized visual image data, synchronized tactile sensing data, and motion posture data.

[0013] Optionally, the graph neural network includes a feature node extraction layer, a hidden layer, a global average pooling layer, and a fully connected layer. The step of inputting multimodal data into the graph neural network for observation feature extraction to obtain an observation feature vector includes:

[0014] The feature node extraction layer converts synchronized visual image data, synchronized tactile sensing data, and motion posture data into visual feature nodes, tactile feature nodes, and inertial navigation feature nodes.

[0015] Obtain the historical association database, and establish connections between visual feature nodes, tactile feature nodes, and inertial navigation feature nodes based on the historical association database to construct an initial feature map;

[0016] The initial feature map is input into the hidden layer and subjected to multi-layer graph convolution operations to obtain the final feature map;

[0017] The final feature map is transformed into a spatial feature vector through a global average pooling layer. The spatial feature vector is then perceptually quantized to obtain a binary hash code. Finally, the binary hash code is mapped into an observed feature vector through a fully connected layer.

[0018] Optionally, the step of inputting the observed feature vector and state vector into an extended Kalman filter for recursive estimation to obtain the state estimate of the dual-arm robot includes:

[0019] S11: Obtain the state vector of the dual-arm robot at the previous moment and the prior state vector at the current moment;

[0020] S12: Input the state vector and observation feature vector from the previous time step into the extended Kalman filter to calculate the state correction.

[0021] S13: Add the prior state vector at the current moment to the state correction amount to obtain the posterior state vector at the current moment;

[0022] S14: Calculate the visual confidence level based on the state correction amount; when the visual confidence level is greater than or equal to the first preset threshold, adjust the parameters of the extended Kalman filter according to the visual confidence level and return to step S11; when the visual confidence level is less than the first preset threshold, use the posterior state vector at the current time as the state estimate.

[0023] Optionally, the state vector includes left arm end-effector position tracking error, right arm end-effector position tracking error, left arm velocity error, right arm velocity error, left arm end-effector contact force, and right arm end-effector contact force.

[0024] Optionally, the construction process of the left-arm control payoff function and the right-arm control payoff function includes:

[0025] Set the weight coefficients for the first left arm, the second left arm, the third left arm, the first right arm, the second right arm, and the third right arm;

[0026] Obtain the current impedance parameters of the left arm and the right arm of the dual-arm robot;

[0027] The left arm task performance term is constructed based on the first left arm weight coefficient and the state vector; the left arm energy consumption constraint term is constructed based on the current impedance parameter of the left arm and the second left arm weight coefficient; and the left arm game coupling term is constructed based on the state vector, the current impedance parameter of the left arm, the current impedance parameter of the right arm, and the third left arm weight coefficient.

[0028] Based on the left arm task performance term, the left arm energy consumption constraint term, and the left arm game coupling term, a left arm control benefit function is constructed.

[0029] The right arm task performance term is constructed based on the first right arm weight coefficient and the state vector; the right arm energy consumption constraint term is constructed based on the current impedance parameter of the right arm and the second right arm weight coefficient; and the right arm game coupling term is constructed based on the state vector, the current impedance parameter of the left arm, the current impedance parameter of the right arm, and the third right arm weight coefficient.

[0030] Based on the right arm task performance term, right arm energy consumption constraint term, and right arm game coupling term, a right arm control benefit function is constructed.

[0031] Optionally, the step of iteratively calculating the Nash equilibrium point that simultaneously minimizes the left-arm and right-arm control payoff functions based on the state estimate using the argmin function, and obtaining the optimal impedance parameters of the left and right arms corresponding to the Nash equilibrium point, includes:

[0032] S21: Based on the pre-trained policy network, obtain the initial values ​​of the current impedance parameters of the left arm and the right arm at the current time.

[0033] S22: Fix the current impedance parameters of the right arm, input the state estimate into the control benefit function of the left arm, and optimize the current impedance parameters of the left arm by using the argmin function with the goal of minimizing the control benefit function of the left arm.

[0034] S23: Fix the current impedance parameters of the left arm, input the state estimate into the control benefit function of the right arm, and use the argmin function to solve for the current impedance parameters of the right arm with the goal of minimizing the control benefit function of the right arm.

[0035] S24: Calculate the iterative change of impedance parameters based on the current impedance parameters of the left arm, the optimized current impedance parameters of the left arm, the current impedance parameters of the right arm, and the optimized current impedance parameters of the right arm.

[0036] S25: When the iterative change of the impedance parameter is greater than or equal to the second preset threshold, adjust the parameters of the argmin function according to the iterative change of the impedance parameter and return to step S22; when the iterative change of the impedance parameter is less than the second preset threshold, optimize the current impedance parameter of the left arm and optimize the current impedance parameter of the right arm as the Nash equilibrium point, optimize the current impedance parameter of the left arm as the optimal impedance parameter of the left arm, and optimize the current impedance parameter of the right arm as the optimal impedance parameter of the right arm.

[0037] Secondly, the present invention provides a cooperative control device for a dual-arm robot based on differential game theory, used to implement the aforementioned cooperative control method for a dual-arm robot based on differential game theory, the device comprising:

[0038] The multimodal data acquisition module is used to collect multimodal data of the dual-arm robot through a multi-source sensor system;

[0039] The state estimation module is used to construct a graph neural network and an extended Kalman filter. It inputs multimodal data into the graph neural network to extract observation features and obtains observation feature vectors. It also obtains the state vector of the dual-arm robot and inputs the observation feature vector and state vector into the extended Kalman filter for recursive estimation to obtain the state estimation value of the dual-arm robot.

[0040] The optimal impedance parameter acquisition module is used to construct a differential game model for the dual-arm robot. The differential game model includes the control payoff function of the left arm and the control payoff function of the right arm. Based on the state estimate, the Nash equilibrium point that makes the control payoff functions of the left arm and the right arm reach local minima simultaneously is calculated through the argmin function. The optimal impedance parameters of the left arm and the right arm corresponding to the Nash equilibrium point are then obtained.

[0041] The collaborative operation module is used to input the optimal impedance parameters of the left arm into the left arm impedance controller of the dual-arm robot to generate left arm joint torque commands; input the optimal impedance parameters of the right arm into the right arm impedance controller of the dual-arm robot to generate right arm joint torque commands; and drive the dual-arm robot to complete collaborative operation tasks according to the left arm joint torque commands and the right arm joint torque commands.

[0042] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aforementioned collaborative control method for a dual-arm robot based on differential game theory.

[0043] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned collaborative control method for a dual-arm robot based on differential game theory.

[0044] The present invention has the following beneficial effects:

[0045] 1. A graph neural network is used to extract cross-modal observation features from multimodal data, resulting in a robust observation feature vector. An extended Kalman filter is used to recursively estimate and correct the state vector using the observation feature vector, ensuring that state perception is not affected by single-modal noise and accurately capturing the state estimate of the dual-arm robot. A differential game model is used to dynamically represent the interaction coupling between the two arms, greatly improving the coordination of the dual-arm robot when performing collaborative tasks. By iteratively solving the differential game model using the argmin function, the Nash equilibrium point is found, allowing the impedance parameters of both arms to reach local optima simultaneously, significantly improving the anti-interference ability and operational stability of the dual-arm robot in complex working environments.

[0046] 2. By extracting the observation feature vectors of multimodal data through graph neural networks, graph neural networks can deeply integrate cross-modal correlation features of multimodal data and enhance feature reliability by combining historical correlation databases. The output observation feature vectors are robust and avoid the limitations of single-modal data. The extended Kalman filter corrects the state vector by recursively estimating and observing the feature vectors, which cancels the interference of modal noise. Combined with visual confidence feedback, the extended Kalman filter can be dynamically tuned, which improves the accuracy of the state estimation.

[0047] 3. The differential game model includes a left-arm control payoff function and a right-arm control payoff function. This model can accurately represent the interaction between the two arms and is suitable for various collaborative task scenarios such as rigid assembly and flexible grasping. Both the left-arm and right-arm control payoff functions are composed of task performance, energy consumption constraints, and game coupling terms, taking into account task accuracy, energy consumption control, and the coordination between the two arms, avoiding uncoordinated actions or energy waste caused by single-objective optimization. By iteratively solving the Nash equilibrium point of the control payoff function using the argmin function, the impedance parameters of both arms are simultaneously optimized locally, significantly improving the anti-interference capability and operational stability of the two-arm collaboration. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0050] Figure 2 This is a structural diagram of the device according to an embodiment of the present invention. Detailed Implementation

[0051] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0052] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0053] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0054] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0055] Reference Figure 1 This invention provides a cooperative control method for a dual-arm robot based on differential game theory, comprising:

[0056] The S100 collects multimodal data of the dual-arm robot through a multi-source sensor system.

[0057] In some embodiments, the acquisition of multimodal data of the dual-arm robot through a multi-source sensor system includes:

[0058] The system collects visual image data of the working environment, tactile sensing data of the robotic arm end effector, and motion posture data of the inertial measurement unit through a multi-source sensor system.

[0059] Based on the sampling frequency of motion posture data, time stamp interpolation is performed on visual image data and tactile sensing data to synchronize them, thereby obtaining synchronized visual image data and synchronized tactile sensing data.

[0060] Multimodal data consists of synchronized visual image data, synchronized tactile sensing data, and motion posture data.

[0061] In some embodiments, an Intel RealSense D455 RGB-D camera is used to acquire 3D coordinates and attitude of the working environment; an ATI Mini 45 six-dimensional force sensor is used to acquire the end-effector contact force and torque; and a BNO055 nine-axis inertial navigation system is used to record joint angular velocities and attitude angles. Data is transmitted in parallel via USB 3.2, EtherCAT, and I2C interfaces to avoid transmission bottlenecks.

[0062] Using the 200Hz sampling frequency of the inertial navigation system as a reference, a synchronization signal is generated through the NI9401 hardware trigger board to trigger all sensors to sample simultaneously; linear interpolation is used to fill the time gaps in visual image data and tactile sensing data to ensure that the timestamp error is <±1ms; a circular queue is used to buffer the data, and when a missing frame is detected, it is automatically filled with interpolation from adjacent frames to ensure data continuity.

[0063] The visual image data is filtered by a high-pass filter to remove low-frequency highlight noise. The expression for the filter function is:

[0064]

[0065] in, These are the pixel coordinates. This is the frequency domain filtered response. The cutoff frequency, denoted as , where is the distance from the pixel to the center of the frequency domain, and n is the filter order.

[0066] The image is then reconstructed using a reflection suppression network after filtering. The expression for the loss function L of the reflection suppression network is:

[0067]

[0068] in, Based on the loss weight, For structural similarity loss weights, To output the image, is the truth image, || ||1 is the L1 norm notation, and SSIM() is the structural similarity function.

[0069] The tactile sensing data was denoised using Gaussian filtering, and the motion posture data was combined with an improved EPNP algorithm to complete the vision-robot coordinate calibration, constructing a coordinate system transformation model. The expression for the error function Q of the coordinate system transformation model is as follows:

[0070]

[0071]

[0072] Where i is the number of the feature point. For feature point weights, For noise level, The projection point in the image coordinate system. Let K be the gradient response at the projection point, and K be the camera intrinsic parameter. Let be the transformation matrix of the camera's end coordinate system. It is the inverse matrix of the world coordinate system. For feature points in the world coordinate system, These are feature points in the reference coordinate system.

[0073] The Levenberg-Marquardt algorithm is used to iteratively solve for the error function Q, finding the motion posture data that minimizes Q. Finally, the visual image data is saved as a PCL format point cloud, the tactile sensor data as a CSV format force value, and the motion posture data as a JSON format pose, for use by subsequent modules.

[0074] The S200 constructs a graph neural network and an extended Kalman filter. Multimodal data is input into the graph neural network for observation feature extraction to obtain the observation feature vector. The state vector of the dual-arm robot is obtained, and the observation feature vector and the state vector are input into the extended Kalman filter for recursive estimation to obtain the state estimate of the dual-arm robot.

[0075] In some embodiments, the graph neural network includes a feature node extraction layer, a hidden layer, a global average pooling layer, and a fully connected layer. The step of inputting multimodal data into the graph neural network for observation feature extraction to obtain an observation feature vector includes:

[0076] The feature node extraction layer converts synchronized visual image data, synchronized tactile sensing data, and motion posture data into visual feature nodes, tactile feature nodes, and inertial navigation feature nodes.

[0077] Obtain the historical association database, and establish connections between visual feature nodes, tactile feature nodes, and inertial navigation feature nodes based on the historical association database to construct an initial feature map;

[0078] The initial feature map is input into the hidden layer and subjected to multi-layer graph convolution operations to obtain the final feature map;

[0079] The final feature map is transformed into a spatial feature vector through a global average pooling layer. The spatial feature vector is then perceptually quantized to obtain a binary hash code. Finally, the binary hash code is mapped into an observed feature vector through a fully connected layer.

[0080] In some embodiments, for synchronized visual image data, multi-scale features are first extracted using FAST corner detection. Feature points with gradient response values ​​greater than 0.15 times the maximum grayscale value of the image are selected. Then, the 3D coordinates (x, y, z), grayscale gradient (dx, dy), and normal vector (nx, ny, nz) of each feature point are extracted to form a 10-dimensional visual feature vector, ultimately generating a set of visual feature nodes. For synchronized tactile data, after Gaussian filtering for noise reduction, the peak value, mean value, torque change rate, and force vector direction angle of the contact force are extracted to construct an 8-dimensional tactile feature vector, generating one tactile feature node at each sampling time. Based on synchronized motion posture data, joint angular velocity, attitude angle, and angular acceleration are extracted. Combined with the coordinate transformation parameters calibrated by the EPnP algorithm, a 9-dimensional inertial navigation feature vector is constructed, and inertial navigation feature nodes are generated according to the sampling frequency of the inertial measurement unit.

[0081] The historical association database stores over 100,000 multimodal collaborative samples, recording the correlations between visual, tactile, and inertial navigation features under different operational scenarios. For example, in the bolt assembly scenario, it records the corresponding thresholds for visual positioning deviation and excessive tactile force values. The cosine similarity between the current feature node and similar nodes in the historical database is calculated; a connection is established when the similarity is ≥95%. Connections between visual and tactile nodes are based on the position-force feedback correlation weights, for example, 0.6 for rigid tasks and 0.4 for flexible tasks. Connections between visual and inertial navigation nodes are established based on the calibration error of the coordinate system transformation matrix. An adjacency matrix is ​​constructed with feature nodes as rows or columns and connection weights as elements, with weight values ​​ranging from [0,1]. Elements that do not meet the connection conditions are set to 0, forming an initial feature map. The dimension of the initial feature map = the number of visual nodes + the number of tactile nodes + the number of inertial navigation nodes.

[0082] A three-layer graph convolutional network (GCN) is used as the hidden layer, with 64 neurons in each layer. ReLU is chosen as the activation function, and L2 regularization is introduced to prevent overfitting. The first convolutional layer fuses spatial features from the initial feature map, outputting a 64-dimensional feature map. The second layer updates node association weights through the adjacency matrix, strengthening cross-modal feature associations. The third layer performs a non-linear mapping on the fused features, outputting a final 128-dimensional feature map. .

[0083] The global average pooling layer uses a 2×2 pooling window to pool the final 128-dimensional feature map, calculates the mean of each feature channel, and generates a 128-dimensional spatial feature vector. Eliminate the impact of node number fluctuations on feature representation. () represents the global average pooling layer. A quantization function maps the spatial feature vector to the interval [-1, 1], and the sgn sign function converts it into a 64-bit binary hash code. , () is the quantization function. () represents the sign function; the hash value is 1 when the spatial feature is ≥0, and 0 otherwise. A single fully connected layer maps the 64-bit binary hash code to a 64-dimensional observation feature vector. , The weights of the fully connected layer, For the bias term of the fully connected layer, () is the Sigmoid activation function. The weights of the fully connected layer are initialized with a uniform distribution of Xavier, and the output vector is normalized to the interval [0,1] to ensure the robustness of the feature vector and its compatibility with the subsequent extended Kalman filter.

[0084] In some embodiments, the step of inputting the observed feature vector and the state vector into an extended Kalman filter for recursive estimation to obtain the state estimate of the dual-arm robot includes:

[0085] S11: Obtain the state vector of the dual-arm robot at the previous moment and the prior state vector at the current moment;

[0086] S12: Input the state vector and observation feature vector from the previous time step into the extended Kalman filter to calculate the state correction.

[0087] S13: Add the prior state vector at the current moment to the state correction amount to obtain the posterior state vector at the current moment;

[0088] S14: Calculate the visual confidence level based on the state correction amount; when the visual confidence level is greater than or equal to the first preset threshold, adjust the parameters of the extended Kalman filter according to the visual confidence level and return to step S11; when the visual confidence level is less than the first preset threshold, use the posterior state vector at the current time as the state estimate.

[0089] In some embodiments, reading from a circular buffer The state vector stored at each time step The prior state vector at time t is predicted based on the state-space equation. Where F is the state transition matrix and B is the associated impedance parameter input matrix. for The impedance parameter at time t.

[0090] The state vector and observed feature vectors Input the extended Kalman filter and calculate the state correction. The calculation formula is:

[0091]

[0092] in, For Carl Fischer gain, , The prior covariance matrix, H is the weight matrix for observing noise covariance.

[0093] Prior state vector With state correction amount By adding each element one by one, we obtain the posterior state vector at time t. According to the state correction amount Computational visual confidence The first preset threshold is set to 0.05. If err This increases the process noise covariance of the extended Kalman filter. Simultaneously, the observation noise covariance is optimized based on the visual occlusion rate. Return to S11 to recalculate; if err ,but As the state estimate at time t.

[0094] In some embodiments, the state vector includes left arm end-effector position tracking error, right arm end-effector position tracking error, left arm velocity error, right arm velocity error, left arm end-effector contact force, and right arm end-effector contact force.

[0095] In some embodiments, the state vector of the dual-arm robot at time t is represented as: The definition is as follows:

[0096]

[0097] in, For the left arm end-position tracking error, For the tracking error of the right arm end position, For the left arm speed error, For the right arm speed error, The contact force at the end of the left arm. The contact force is at the end of the right arm.

[0098] The S300 constructs a differential game model for the dual-arm robot, which includes the control payoff function for the left arm and the control payoff function for the right arm. Based on the state estimate, the Nash equilibrium point that makes the control payoff functions for the left and right arms simultaneously reach local minima is calculated through the argmin function iterative calculation. The optimal impedance parameters of the left and right arms corresponding to the Nash equilibrium point are then obtained.

[0099] In some embodiments, the construction process of the left-arm control payoff function and the right-arm control payoff function includes:

[0100] Set the weight coefficients for the first left arm, the second left arm, the third left arm, the first right arm, the second right arm, and the third right arm;

[0101] Obtain the current impedance parameters of the left arm and the right arm of the dual-arm robot;

[0102] The left arm task performance term is constructed based on the first left arm weight coefficient and the state vector; the left arm energy consumption constraint term is constructed based on the current impedance parameter of the left arm and the second left arm weight coefficient; and the left arm game coupling term is constructed based on the state vector, the current impedance parameter of the left arm, the current impedance parameter of the right arm, and the third left arm weight coefficient.

[0103] Based on the left arm task performance term, the left arm energy consumption constraint term, and the left arm game coupling term, a left arm control benefit function is constructed.

[0104] The right arm task performance term is constructed based on the first right arm weight coefficient and the state vector; the right arm energy consumption constraint term is constructed based on the current impedance parameter of the right arm and the second right arm weight coefficient; and the right arm game coupling term is constructed based on the state vector, the current impedance parameter of the left arm, the current impedance parameter of the right arm, and the third right arm weight coefficient.

[0105] Based on the right arm task performance term, right arm energy consumption constraint term, and right arm game coupling term, a right arm control benefit function is constructed.

[0106] In some embodiments, the left arm controls the payoff function The expression is:

[0107]

[0108] Where t is the current time; Let be the state vector at time t; Let be the current impedance parameter of the left arm at time t; Let be the current impedance parameter of the right arm at time t; The first left arm weighting coefficient; This is the weighting coefficient for the second left arm; This is the weighting coefficient for the third left arm; This is a performance parameter for the left arm task, used to penalize positional errors and excessive contact forces; This is an energy consumption constraint term for the left arm to prevent sudden changes in the control quantity; For the left-hand game coupling terms, where For game coupling terms, the game coupling terms describe the internal force conflict between the two arms. When the pushing force of the left arm is too large, causing the right arm to be forced to displace, the value of the game coupling terms will increase significantly, thereby forcing the left arm to adjust its strategy. It is an L2 norm; The square of the weighted L2 norm, =a T Ma.

[0109] Right arm control benefit function The expression is:

[0110]

[0111] in, The first right arm weighting coefficient; This is the weighting coefficient for the second right arm; This is the weighting coefficient for the third right arm; For right arm task performance items, For the energy consumption constraint term of the right arm; This is the coupling term of the right-hand game.

[0112] In some embodiments, the step of iteratively calculating the Nash equilibrium point that simultaneously minimizes the left-arm and right-arm control payoff functions based on the state estimate using the argmin function, and obtaining the optimal impedance parameters of the left and right arms corresponding to the Nash equilibrium point, includes:

[0113] S21: Based on the pre-trained policy network, obtain the initial values ​​of the current impedance parameters of the left arm and the right arm at the current time.

[0114] S22: Fix the current impedance parameters of the right arm, input the state estimate into the control benefit function of the left arm, and optimize the current impedance parameters of the left arm by using the argmin function with the goal of minimizing the control benefit function of the left arm.

[0115] S23: Fix the current impedance parameters of the left arm, input the state estimate into the control benefit function of the right arm, and use the argmin function to solve for the current impedance parameters of the right arm with the goal of minimizing the control benefit function of the right arm.

[0116] S24: Calculate the iterative change of impedance parameters based on the current impedance parameters of the left arm, the optimized current impedance parameters of the left arm, the current impedance parameters of the right arm, and the optimized current impedance parameters of the right arm.

[0117] S25: When the iterative change of the impedance parameter is greater than or equal to the second preset threshold, adjust the parameters of the argmin function according to the iterative change of the impedance parameter and return to step S22; when the iterative change of the impedance parameter is less than the second preset threshold, optimize the current impedance parameter of the left arm and optimize the current impedance parameter of the right arm as the Nash equilibrium point, optimize the current impedance parameter of the left arm as the optimal impedance parameter of the left arm, and optimize the current impedance parameter of the right arm as the optimal impedance parameter of the right arm.

[0118] In some embodiments, a 3-layer fully connected deep neural network (DNN) is used as the policy network. The inputs are the current job scenario type, initial posture error and environmental stiffness level, and the outputs are the impedance parameters of the left and right arms. The hidden layer has 128 neurons and the activation function is ReLU.

[0119] We collected over 100,000 historical data points on collaborative operations of dual-arm robots, covering state vectors and optimal impedance parameter samples under different scenarios. Using the mean square error between the predicted impedance parameters and the actual optimal parameters as the loss function, we employed the Adam optimizer for iterative training. The initial learning rate was set to 0.001, and the algorithm converged after 500 iterations.

[0120] The current task scenario identifier and initial pose detection data are input into the trained policy network, and the initial value of the left arm's current impedance parameter at time t is directly output through forward inference. And the initial value of the current impedance parameter of the right arm. Set the number of iterations k to 1 to .

[0121] Start the k-th iteration and set the state estimate. Input the left arm control revenue function, and fix the current impedance parameter of the right arm. Optimize the current impedance parameters of the left arm The expression is:

[0122]

[0123] Here, argmin is the argmin function, which is used to find the optimal current impedance parameter of the left arm that minimizes the control benefit function of the left arm.

[0124] Fixed optimization of current impedance parameters of the left arm Optimize the current impedance parameters of the right arm The expression is:

[0125]

[0126] The iterative change U of the impedance parameter is expressed as: When U < second preset threshold At that time, and As the Nash equilibrium point, As the optimal impedance parameter for the left arm ,Will As the optimal impedance parameter of the right arm .

[0127] The S400 inputs the optimal impedance parameters of the left arm into the left arm impedance controller of the dual-arm robot to generate left arm joint torque commands; it inputs the optimal impedance parameters of the right arm into the right arm impedance controller of the dual-arm robot to generate right arm joint torque commands; and drives the dual-arm robot to complete collaborative operation tasks according to the left arm joint torque commands and the right arm joint torque commands.

[0128] In some embodiments, the optimal impedance parameter of the left arm Includes the stiffness matrix of the left arm and left arm damping matrix Optimal impedance parameters of the right arm Includes right arm stiffness matrix and right arm damping matrix The controller first analyzes the diagonal and off-diagonal elements of each parameter matrix and maps them to the controller channels according to the joint number. For rigid tasks, impedance control is used to prioritize position tracking accuracy; for flexible tasks, admittance control is switched to focus on suppressing contact force overshoot.

[0129] Calculation of foundation torque based on impedance control formula ,in, Let D be the inertia matrix, D be the damping matrix, and K be the stiffness matrix. For end-position tracking error, For speed error, This represents the acceleration of the end-position tracking error.

[0130] Real-time acquisition of end contact force using the ATIMini45 sensor ,when When the threshold is approached, a force feedback term is introduced. , For force control gain, we set it to 0.3; For reference contact force; The end contact force is used; the basic torque is corrected through force feedback to avoid damage from excessive force. An upper limit is set according to the rated torque of the robot joint. If it is exceeded, linear clipping is performed to prevent joint overload.

[0131] The torque is converted into a torque command through an impedance controller, and the torque command is transmitted to the joint actuator via the EtherCAT protocol, with a communication cycle of [missing information]. To ensure real-time command performance, a CRC check is performed on the command before transmission to avoid data loss or errors. After receiving the command, the driver outputs the corresponding joint torque through a closed-loop current loop control to drive the servo motor, while simultaneously transmitting the actual joint torque and position feedback values ​​back to the control system in real time.

[0132] Synchronize the timing of torque command issuance to both arms to ensure coordinated action, such as the timing difference between the left arm fixing and the right arm tightening during bolt assembly. The system monitors assembly clearance and force control deviations in real time. If deviations exceed the limits, the performance optimization module fine-tunes the torque command. When positional deviations are excessive, the stiffness coefficient is increased; when force deviations exceed limits, the damping coefficient is decreased, dynamically maintaining operational accuracy. When the anomaly detection unit detects a sudden increase in contact force or a change in posture, it immediately triggers emergency deceleration, reducing the speed to the original speed. At the same time, adjust the torque command to the safety threshold.

[0133] Reference Figure 2 A collaborative control device 20 for a dual-arm robot based on differential game theory is provided for implementing a collaborative control method for a dual-arm robot based on differential game theory. The device includes:

[0134] The multimodal data acquisition module 21 is used to acquire multimodal data of the dual-arm robot through a multi-source sensor system;

[0135] The state estimation module 22 is used to construct a graph neural network and an extended Kalman filter. It inputs multimodal data into the graph neural network to extract observation features and obtains observation feature vectors. It also obtains the state vector of the dual-arm robot and inputs the observation feature vector and state vector into the extended Kalman filter for recursive estimation to obtain the state estimation value of the dual-arm robot.

[0136] The optimal impedance parameter acquisition module 23 is used to construct a differential game model of the dual-arm robot. The differential game model includes the control payoff function of the left arm and the control payoff function of the right arm. Based on the state estimate, the Nash equilibrium point that makes the control payoff function of the left arm and the control payoff function of the right arm reach local minima simultaneously is calculated through the argmin function. The optimal impedance parameters of the left arm and the optimal impedance parameters of the right arm corresponding to the Nash equilibrium point are then obtained.

[0137] The collaborative operation module 24 is used to input the optimal impedance parameters of the left arm into the left arm impedance controller of the dual-arm robot to generate the left arm joint torque command; input the optimal impedance parameters of the right arm into the right arm impedance controller of the dual-arm robot to generate the right arm joint torque command; and drive the dual-arm robot to complete the collaborative operation task according to the left arm joint torque command and the right arm joint torque command.

[0138] This application provides an electronic device, including a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements any of the above-described schemes of a collaborative control method for a dual-arm robot based on differential game theory.

[0139] Specifically, the processor may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0140] Memory can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and also random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0141] This application also provides a computer-readable medium storing a computer program that, when executed by a processor, implements the differential game-based cooperative control method for a dual-arm robot as described above. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into that device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the method as described in the embodiments of this application.

[0142] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0143] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments, but should be defined not only by the appended claims, but also by their equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A cooperative control method for a dual-arm robot based on differential game theory, characterized in that, include: Multimodal data of the dual-arm robot are collected through a multi-source sensor system; A graph neural network and an extended Kalman filter are constructed. Multimodal data is input into the graph neural network to extract observation features and obtain observation feature vectors. The state vector of the dual-arm robot is obtained. The observation feature vector and the state vector are input into the extended Kalman filter for recursive estimation to obtain the state estimate of the dual-arm robot. A differential game model for a dual-arm robot is constructed, which includes the control payoff function of the left arm and the control payoff function of the right arm. Based on the state estimate, the Nash equilibrium point that makes the control payoff functions of the left arm and the right arm reach local minima is calculated iteratively by using the argmin function. The optimal impedance parameters of the left arm and the right arm corresponding to the Nash equilibrium point are then obtained. Input the optimal impedance parameters of the left arm into the impedance controller of the left arm of the dual-arm robot to generate the torque command of the left arm joint; input the optimal impedance parameters of the right arm into the impedance controller of the right arm of the dual-arm robot to generate the torque command of the right arm joint. The dual-arm robot is driven to complete collaborative operation tasks based on the torque commands of the left and right arm joints.

2. The collaborative control method for a dual-arm robot based on differential game theory according to claim 1, characterized in that, The acquisition of multimodal data of the dual-arm robot through a multi-source sensor system includes: The system collects visual image data of the working environment, tactile sensing data of the robotic arm end effector, and motion posture data of the inertial measurement unit through a multi-source sensor system. Based on the sampling frequency of motion posture data, time stamp interpolation is performed on visual image data and tactile sensing data to synchronize them, thereby obtaining synchronized visual image data and synchronized tactile sensing data. Multimodal data consists of synchronized visual image data, synchronized tactile sensing data, and motion posture data.

3. The collaborative control method for a dual-arm robot based on differential game theory according to claim 2, characterized in that, The graph neural network includes a feature node extraction layer, a hidden layer, a global average pooling layer, and a fully connected layer. The step of inputting multimodal data into the graph neural network for observation feature extraction to obtain an observation feature vector includes: The feature node extraction layer converts synchronized visual image data, synchronized tactile sensing data, and motion posture data into visual feature nodes, tactile feature nodes, and inertial navigation feature nodes. Obtain the historical association database, and establish connections between visual feature nodes, tactile feature nodes, and inertial navigation feature nodes based on the historical association database to construct an initial feature map; The initial feature map is input into the hidden layer and subjected to multi-layer graph convolution operations to obtain the final feature map; The final feature map is transformed into a spatial feature vector through a global average pooling layer. The spatial feature vector is then perceptually quantized to obtain a binary hash code. Finally, the binary hash code is mapped into an observed feature vector through a fully connected layer.

4. The collaborative control method for a dual-arm robot based on differential game theory according to claim 1, characterized in that, The step of inputting the observed feature vector and state vector into an extended Kalman filter for recursive estimation to obtain the state estimate of the dual-arm robot includes: S11: Obtain the state vector of the dual-arm robot at the previous moment and the prior state vector at the current moment; S12: Input the state vector and observation feature vector from the previous time step into the extended Kalman filter to calculate the state correction. S13: Add the prior state vector at the current moment to the state correction amount to obtain the posterior state vector at the current moment; S14: Calculate the visual confidence level based on the state correction amount; when the visual confidence level is greater than or equal to the first preset threshold, adjust the parameters of the extended Kalman filter according to the visual confidence level and return to step S11; when the visual confidence level is less than the first preset threshold, use the posterior state vector at the current time as the state estimate.

5. The collaborative control method for a dual-arm robot based on differential game theory according to claim 1, characterized in that, The state vector includes the left arm end-effector position tracking error, the right arm end-effector position tracking error, the left arm velocity error, the right arm velocity error, the left arm end-effector contact force, and the right arm end-effector contact force.

6. The collaborative control method for a dual-arm robot based on differential game theory according to claim 1, characterized in that, The construction process of the left-arm control payoff function and the right-arm control payoff function includes: Set the weight coefficients for the first left arm, the second left arm, the third left arm, the first right arm, the second right arm, and the third right arm; Obtain the current impedance parameters of the left arm and the right arm of the dual-arm robot; The left arm task performance term is constructed based on the first left arm weight coefficient and the state vector; the left arm energy consumption constraint term is constructed based on the current impedance parameter of the left arm and the second left arm weight coefficient; and the left arm game coupling term is constructed based on the state vector, the current impedance parameter of the left arm, the current impedance parameter of the right arm, and the third left arm weight coefficient. Based on the left arm task performance term, the left arm energy consumption constraint term, and the left arm game coupling term, a left arm control benefit function is constructed. The right arm task performance term is constructed based on the first right arm weight coefficient and the state vector; the right arm energy consumption constraint term is constructed based on the current impedance parameter of the right arm and the second right arm weight coefficient; and the right arm game coupling term is constructed based on the state vector, the current impedance parameter of the left arm, the current impedance parameter of the right arm, and the third right arm weight coefficient. Based on the right arm task performance term, right arm energy consumption constraint term, and right arm game coupling term, a right arm control benefit function is constructed.

7. The cooperative control method for a dual-arm robot based on differential game theory according to claim 6, characterized in that, The process involves iteratively calculating, based on state estimates, the Nash equilibrium point where both the left-arm and right-arm control payoff functions simultaneously reach a local minimum using the argmin function. The optimal impedance parameters for the left and right arms corresponding to this Nash equilibrium point are then obtained, including: S21: Based on the pre-trained policy network, obtain the initial values ​​of the current impedance parameters of the left arm and the right arm at the current time. S22: Fix the current impedance parameters of the right arm, input the state estimate into the control benefit function of the left arm, and optimize the current impedance parameters of the left arm by using the argmin function with the goal of minimizing the control benefit function of the left arm. S23: Fix the current impedance parameters of the left arm, input the state estimate into the control benefit function of the right arm, and use the argmin function to solve for the current impedance parameters of the right arm with the goal of minimizing the control benefit function of the right arm. S24: Calculate the iterative change of impedance parameters based on the current impedance parameters of the left arm, the optimized current impedance parameters of the left arm, the current impedance parameters of the right arm, and the optimized current impedance parameters of the right arm. S25: When the iterative change of the impedance parameter is greater than or equal to the second preset threshold, adjust the parameters of the argmin function according to the iterative change of the impedance parameter and return to step S22; when the iterative change of the impedance parameter is less than the second preset threshold, optimize the current impedance parameter of the left arm and optimize the current impedance parameter of the right arm as the Nash equilibrium point, optimize the current impedance parameter of the left arm as the optimal impedance parameter of the left arm, and optimize the current impedance parameter of the right arm as the optimal impedance parameter of the right arm.

8. A collaborative control device for a dual-arm robot based on differential game theory, used to implement the collaborative control method for a dual-arm robot based on differential game theory as described in any one of claims 1 to 7, characterized in that, The device includes: The multimodal data acquisition module is used to collect multimodal data of the dual-arm robot through a multi-source sensor system; The state estimation module is used to construct a graph neural network and an extended Kalman filter. It inputs multimodal data into the graph neural network to extract observation features and obtains observation feature vectors. It also obtains the state vector of the dual-arm robot and inputs the observation feature vector and state vector into the extended Kalman filter for recursive estimation to obtain the state estimation value of the dual-arm robot. The optimal impedance parameter acquisition module is used to construct a differential game model for the dual-arm robot. The differential game model includes the control payoff function of the left arm and the control payoff function of the right arm. Based on the state estimate, the Nash equilibrium point that makes the control payoff functions of the left arm and the right arm reach local minima simultaneously is calculated through the argmin function. The optimal impedance parameters of the left arm and the right arm corresponding to the Nash equilibrium point are then obtained. The collaborative operation module is used to input the optimal impedance parameters of the left arm into the left arm impedance controller of the dual-arm robot to generate left arm joint torque commands; input the optimal impedance parameters of the right arm into the right arm impedance controller of the dual-arm robot to generate right arm joint torque commands; and drive the dual-arm robot to complete collaborative operation tasks according to the left arm joint torque commands and the right arm joint torque commands.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the collaborative control method for a dual-arm robot based on differential game theory as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the collaborative control method for a dual-arm robot based on differential game theory as described in any one of claims 1 to 7.

Images