A three-arm cooperative control method and system based on visual servoing

Abstract

This invention discloses a heterogeneous three-arm cooperative control method and system based on visual servoing, comprising: determining the joint joint variable matrix of the three-arm system in the global coordinate system based on the hand-eye transformation matrix, the base coordinate systems of the three robotic arms, and the camera coordinate system; constructing a heterogeneous joint Jacobian matrix based on the motion velocity relationship of the three-arm system and the joint joint variable matrix; extracting feature points to determine feature errors and establishing the main task control law of visual servoing; introducing an obstacle avoidance cost function based on gradient descent, and obtaining the optimal joint speed control law based on null projection, the heterogeneous joint Jacobian matrix, and the main task control law of visual servoing. By embedding the obstacle avoidance cost function into the underlying control law through null projection technology, the optimal anti-collision joint speed can be calculated at a frequency of 150Hz, ensuring absolute mechanical safety of the three arms when working in a compact space, and fundamentally solving the problem of high-precision coordination and safety of heterogeneous three-arm systems in dynamic operations.

Description

Technical Field

[0001] This invention belongs to the field of intelligent manufacturing technology, and more specifically, relates to a three-arm collaborative control method and system based on visual servoing. Background Technology

[0002] As intelligent manufacturing systems evolve towards higher precision, flexibility, and multi-task parallelism, traditional single-arm robotic systems are increasingly unable to meet the demands of complex spatial operations. Compared to traditional single-arm or dual-arm systems, three-arm collaborative control systems demonstrate greater flexibility and load distribution capabilities in tasks such as handling heavy objects under statically indeterminate constraints, obstacle avoidance and grasping in complex spaces, and precision flexible assembly. Especially in applications such as the assembly of large flexible components, precision machining of irregular curved surfaces, stable clamping of heavy-duty parts, and high-precision collaborative operations, multi-arm collaborative systems are gradually becoming a core technological trend.

[0003] In current industrial scenarios, dual-arm systems have achieved a certain degree of collaborative control, but there are still obvious limitations in the following task scenarios: (1) two robotic arms are needed to stably clamp large flexible workpieces; (2) a third execution unit is needed to perform precision operations (grinding, drilling, insertion); (3) multi-arm cross operations are needed in a narrow space.

[0004] Compared with the two-arm system, the three-arm collaborative system has higher degree of freedom redundancy and task allocation flexibility, and can achieve: (1) the two arms form a stable clamping closed chain structure; (2) the third arm performs high-precision dynamic operations; (3) the overall envelope coordinated movement and local fine adjustment are carried out in parallel.

[0005] However, while the degrees of freedom increase, the system dimensions grow exponentially, leading to a sharp increase in control complexity and extremely complex kinematic coupling relationships in high-dimensional state spaces.

[0006] Existing multi-arm cooperative control systems face the following core challenges in practical applications: 1. Difficulty in modeling and unifying the mapping of heterogeneous systems: There are significant differences in mechanical configuration, link parameters and underlying dynamic response of robotic arms with different degrees of freedom, making it difficult to establish a unified cooperative kinematic mapping model.

[0007] 2. Dynamic error accumulation in unstructured environments: Traditional open-loop or semi-closed-loop control based on pure kinematics lacks the ability to perceive and compensate for errors in the workspace in real time when facing assembly tolerances or dynamic deformation of the target object.

[0008] 3. Self-collision and singularity problems in high-dimensional space: Three-arm (≥18 degrees of freedom) robots are prone to interference between each other in a compact workspace. Traditional obstacle avoidance algorithms have exponentially increasing computational complexity when dealing with heterogeneous three-arm robots, making it difficult to meet real-time control requirements.

[0009] To address the aforementioned issues, this invention proposes a heterogeneous three-arm collaborative control method and system based on visual servoing, which solves the problems of high-precision collaboration and safety in dynamic operations of heterogeneous three-arm systems. Summary of the Invention

[0010] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a heterogeneous three-arm cooperative control method and system based on visual servoing. Within a zero-space projection task priority framework, visual servoing tracking is designated as the first priority task, while self-collision avoidance and joint limiting are designated as the second priority tasks. Under the premise of ensuring accurate end-effector pose tracking as the primary task, secondary tasks (obstacle avoidance and singularity avoidance) are achieved by utilizing the redundant degrees of freedom of the three-arm system. This fundamentally solves the problems of high-precision coordination and safety of heterogeneous three-arm systems in dynamic operations.

[0011] To achieve the above objectives, according to one aspect of the present invention, a heterogeneous three-arm cooperative control method based on visual servoing is provided, comprising the following steps: The joint joint variable matrix of the three-arm system in the global coordinate system is determined based on the hand-eye transformation matrix, the base coordinate system of the three robotic arms, and the camera coordinate system. Heterogeneous joint Jacobian matrix is ​​constructed based on the motion velocity relationship of the three-arm system and the joint joint variable matrix; Feature point extraction is performed to determine feature errors, and the main control law for visual servoing is established. An obstacle avoidance cost function based on gradient descent is introduced, and the optimal joint velocity control law is obtained based on null space projection, heterogeneous joint Jacobian matrix and visual servoing master task control law.

[0012] Furthermore, in an optional embodiment of the present invention, determining the joint joint variable matrix of the three-arm system in the global coordinate system based on the hand-eye transformation matrix, the base coordinate system, and the camera coordinate system includes: Establish a global coordinate system, calibrate and determine the transformation relationship between the base coordinate system and the global coordinate system of the three robotic arms, and transform the base coordinate system of the three robotic arms to the global coordinate system; The target pose in the camera coordinate system is transformed to the global coordinate system based on the hand-eye transformation matrix, and the joint joint variable matrix of the three robotic arms in the global coordinate system is solved inversely.

[0013] Furthermore, in an optional embodiment of the present invention, the construction of a heterogeneous joint Jacobian matrix from the motion velocity relationship of the three-arm system and the joint joint variable matrix includes: In the global coordinate system, the overall motion of the envelope formed by the three robotic arms is taken as the absolute motion, and the motion of the manipulator relative to the other two gripping arms is taken as the relative motion. The relationship between the absolute motion speed, the relative motion speed and the task space speed of the three-arm system is obtained. Based on the relationship between absolute motion velocity, relative motion velocity and the task space velocity of the three-arm system, and the joint joint variable matrix, the heterogeneous joint Jacobian matrix is ​​obtained.

[0014] Furthermore, in an optional embodiment of the present invention, the relationship between the absolute motion speed, the relative motion speed, and the task space speed of the three-arm system is as follows: , in, For the three-arm system's mission space velocity, The absolute velocity of the center of motion of the three-arm coordination. Let i be the relative velocity of the manipulator arm with respect to the gripper arm, where i is selected from 1 to 2, and T is the matrix transpose.

[0015] Furthermore, in an optional embodiment of the invention, the heterogeneous joint Jacobian matrix... for: in, For the three-arm system's mission space velocity, This is the joint joint variable matrix.

[0016] Furthermore, in an optional embodiment of the present invention, the step of extracting feature points to determine feature errors and establishing the main task control law for visual servoing includes: Feature point extraction is performed to determine the feature error, resulting in a feature error vector: , in, For the extracted feature points, The desired characteristic state; Construct an exponentially decaying servo control law to obtain the desired task space command speed: , in It is a positive definite gain matrix. , It is the pseudo-inverse of the visual interaction matrix.

[0017] Furthermore, in an optional embodiment of the present invention, the introduction of an obstacle avoidance cost function based on gradient descent, and the obtaining of the optimal joint velocity control law based on null space projection, heterogeneous joint Jacobian matrix, and visual servoing master task control law, includes: An obstacle avoidance cost function based on gradient descent is introduced into the task control law for self-collision avoidance and joint restraint. In the task priority framework of null-space projection, visual servo tracking is taken as the first priority task, and self-collision avoidance and joint restraint are taken as the second priority tasks, thus obtaining the optimal joint velocity control law: , in, For Moore-Penrose pseudo-inverse, Let I be the desired task space instruction velocity, and let I be the identity mapping matrix of the joint joint space. For heterogeneous joint Jacobian matrices, The gradient vector of the obstacle avoidance cost function H with respect to the joint joint variable matrix. This is the joint joint variable matrix.

[0018] According to a second aspect of the present invention, a heterogeneous three-arm cooperative control system based on visual servoing is provided, wherein the heterogeneous three-arm cooperative control method is executed, comprising: The visual perception and pose estimation module is used to control the camera to collect point cloud data in the workspace, and to calculate the pose of the target workpiece in the camera coordinate system in real time through point cloud registration and feature extraction algorithms. The multi-arm collaborative control module is used to establish a global coordinate system, combine the end effector states of the three robotic arms into a generalized workspace state vector, and decompose the task objective into absolute motion and relative motion according to task requirements. The closed-loop visual servo control module is used to generate joint space speed commands based on the visual error between the current pose and the desired pose and the heterogeneous joint Jacobian matrix, and then send them to the underlying driver of the three robotic arms to achieve high-frequency closed-loop servo control.

[0019] Furthermore, in the global coordinate system, the overall motion of the envelope formed by the three robotic arms is taken as the absolute motion, and the motion of the manipulator relative to the other two gripping arms is taken as the relative motion.

[0020] Furthermore, the control law is the optimal joint speed control law.

[0021] According to a third aspect of the present invention, an electronic device is provided, comprising: at least one central processing unit (CPU); and at least one memory communicatively connected to the CPU, wherein: the memory stores program instructions executable by the CPU, and the CPU invokes the program instructions to execute the method described herein.

[0022] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing computer instructions that cause a computer to perform the method described herein.

[0023] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: 1. The method of this invention utilizes high-precision image data and depth information provided by a depth camera to construct a closed-loop visual feedback, establishing a position-based visual servoing model. It also introduces null-space projection theory to achieve secondary tasks (obstacle avoidance and singularity avoidance) using the redundant degrees of freedom of the three-arm system, while ensuring accurate end-effector pose tracking as the primary task. The 20-DOF redundant state space is analyzed using null-space projection technology, embedding the obstacle avoidance cost function into the underlying control law. The algorithm can calculate the optimal anti-collision joint speed at a frequency of 150Hz, ensuring absolute mechanical safety of the three arms during overlapping operations in a compact space, fundamentally solving the problem of high-precision coordination and safety in dynamic operations of heterogeneous three-arm systems.

[0024] 2. The method of this invention determines the joint joint variable matrix of the three-arm system in the global coordinate system based on the hand-eye transformation matrix, the base coordinate system of the three robotic arms, and the camera coordinate system; based on the motion velocity relationship of the three-arm system and the joint joint variable matrix, a heterogeneous joint Jacobian matrix is ​​constructed to decouple and remap the manipulator and gripper from the underlying kinematic dimension, thereby constructing a strictly corresponding joint Jacobian matrix, eliminating the collaborative trajectory tearing and chattering phenomenon caused by inconsistent dynamic parameters between heterogeneous robotic arms, and realizing the mathematical unification and smooth collaboration of heterogeneous platforms.

[0025] 3. The method of the present invention extracts feature points to determine feature errors, establishes the main task control law of visual servoing, and constructs a closed-loop error function based on the depth vision information provided by the depth camera. The system can adaptively compensate for workpiece deformation, displacement and base installation tolerance in real time, which greatly improves the system's unstructured operation accuracy and high-precision operation capability against dynamic interference. Attached Figure Description

[0026] Figure 1 This is a flowchart of the heterogeneous three-arm cooperative control method based on visual servoing in an embodiment of the present invention. Figure 2 This is a diagram of the architecture of a heterogeneous three-arm cooperative control system based on visual servoing in an embodiment of the present invention (the macro-architecture is divided into a perception layer, a cooperative decision-making layer, and a bottom control layer). Figure 3 This is a flowchart of the precise position control method for a robotic arm in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0028] This invention provides a heterogeneous three-arm cooperative control method based on vision servoing, which can be used for high-precision cooperative control and high-precision operation control against dynamic interference in any three-arm system, such as... Figure 1 As shown, the specific steps include: S100 determines the joint joint variable matrix of the three-arm system in the global coordinate system based on the hand-eye transformation matrix, the base coordinate system of the three robotic arms, and the camera coordinate system; S200 constructs a heterogeneous joint Jacobian matrix based on the motion velocity relationship of the three-arm system and the joint joint variable matrix; S300 performs feature point extraction to determine feature error and establishes the main task control law for visual servoing. The S400 introduces an obstacle avoidance cost function based on gradient descent, and obtains the optimal joint velocity control law based on null space projection, heterogeneous joint Jacobian matrix and visual servoing master task control law.

[0029] The method of this invention utilizes high-precision image data and depth information provided by a depth camera to construct a closed-loop visual feedback, establishing a position-based visual servoing (PBVS) model. It also introduces null-space projection theory to achieve secondary tasks (obstacle avoidance and singularity avoidance) using the redundant degrees of freedom of the three-arm system, while ensuring accurate end-effector pose tracking as the primary task. The 20-DOF redundant state space is analyzed using null-space projection technology, embedding the obstacle avoidance cost function into the underlying control law. The algorithm can calculate the optimal anti-collision joint velocity at a frequency of 150Hz, ensuring absolute mechanical safety of the three arms during overlapping operations in a compact space, fundamentally solving the problem of high-precision coordination and safety in dynamic operations of heterogeneous three-arm systems.

[0030] In an embodiment of the present invention, S100 specifically includes the following steps: Establish a global coordinate system, calibrate and determine the transformation relationship between the base coordinate system and the global coordinate system of the three robotic arms, and transform the base coordinate system of the three robotic arms to the global coordinate system; The target pose in the camera coordinate system is transformed to the global coordinate system based on the hand-eye transformation matrix, and the joint joint variable matrix of the three robotic arms in the global coordinate system is solved inversely.

[0031] To implement the method of this invention, it is first necessary to establish a unified kinematic model of the heterogeneous three-arm system, assuming the global coordinate system is... The L515 camera coordinate system is The base coordinate systems of the two RM75-6F units are respectively The base coordinate system of the Z1 tree is... .

[0032] Matrix calibration by hand and eye This transforms the target pose in the camera coordinate system to the global coordinate system.

[0033] The joint variable matrix of the three-arm system is: Each subvector of the joint joint variable matrix satisfies its own differential kinematic equation. Here, i is selected from R1, R2, Z1. Let be the first derivative of the velocity vector of robotic arm i. It represents 20 degrees of freedom.

[0034] In an embodiment of the present invention, S200 specifically includes the following steps: In the global coordinate system, the overall motion of the envelope formed by the three robotic arms is taken as the absolute motion, and the motion of the manipulator relative to the other two gripping arms is taken as the relative motion. The relationship between the absolute motion speed, the relative motion speed and the task space speed of the three-arm system is obtained. Based on the relationship between absolute motion velocity, relative motion velocity and the task space velocity of the three-arm system, and the joint joint variable matrix, the heterogeneous joint Jacobian matrix is ​​obtained.

[0035] Specifically, to achieve close coordination among the three robotic arms, the overall motion of the envelope formed by the three robotic arms is taken as the absolute motion, and the motion of the manipulator relative to the other two gripping arms is taken as the relative motion. The generalized velocity in the task space is defined as follows: , in, For the three-arm system's mission space velocity, The absolute velocity of the center of motion of the three-arm coordination. Let i be the relative velocity of the manipulator arm with respect to the gripper arm, where i is selected from 1 to 2, and T is the matrix transpose.

[0036] Furthermore, the heterogeneous joint Jacobian matrix can be obtained based on the generalized velocity of the task space and the joint joint variable matrix. for: in, For the three-arm system's mission space velocity, For the joint joint variable matrix, , This indicates the dimension of the null projection.

[0037] The method of this invention constructs a strictly corresponding joint Jacobian matrix by decoupling and remapping the three arms, eliminating the collaborative trajectory tearing and chattering phenomena caused by inconsistent dynamic parameters between heterogeneous robotic arms, and realizing the mathematical unification and smooth collaboration of heterogeneous platforms.

[0038] In an embodiment of the present invention, S300 specifically includes the following steps: Feature point extraction is performed to determine the feature error, resulting in a feature error vector. Construct an exponentially decaying servo control law to obtain the desired task space speed.

[0039] Specifically, the feature error vector is: , in, The extracted feature points were obtained from the depth camera. Let t be the desired characteristic state, and t be the current time. Construct an exponentially decaying servo control law to obtain the desired task space velocity (i.e., the generalized velocity of the task space): , in It is a positive definite gain matrix. , This is the pseudo-inverse of the visual interaction matrix, and its solution is a conventional method in this field, which will not be elaborated here.

[0040] The method of this invention relies on the depth vision information provided by the depth camera to construct a closed-loop error function. The system can adaptively compensate for the deformation, displacement and base installation tolerance of the workpiece in real time, thereby improving the accuracy of the unstructured operation of the system.

[0041] In an embodiment of the present invention, S400 specifically includes: An obstacle avoidance cost function based on gradient descent is introduced into the task control law for self-collision avoidance and joint restraint. In the zero-space projection task priority framework, visual servo tracking is taken as the first priority task, and self-collision avoidance and joint restraint are taken as the second priority tasks, thus obtaining the optimal joint velocity control law.

[0042] The optimal joint velocity control law is: , in, For Moore-Penrose pseudo-inverse, Let I be the desired task space command velocity, and let I be the identity mapping matrix of the joint joint space. In the three-arm system of this invention, its dimension is consistent with the total degrees of freedom. , For heterogeneous joint Jacobian matrices, The gradient vector of the obstacle avoidance cost function H with respect to the joint joint variable matrix. This is the joint joint variable matrix.

[0043] It should be noted that solving the Moore-Penrose pseudoinverse is a standard method in this field and will not be elaborated upon here.

[0044] Specifically, in order to solve the self-collision problem of the three arms, an obstacle avoidance cost function based on gradient descent is introduced. When the distance between the links of the robotic arm is less than the safety threshold, the H value increases sharply, thereby achieving joint limiting.

[0045] The method of this invention prioritizes visual servo tracking as the first priority task, and collision avoidance and joint restraint as the second priority tasks. By projecting the gradient vector of the secondary task into the null space of the main task, the tracking accuracy of the end-effector visual servo is never interfered with during obstacle avoidance and configuration reconstruction, thus ensuring the intrinsic safety of the high-dimensional space.

[0046] The above-described implementation of this invention is achieved through programmed processing using a device with a central processing unit (CPU). Therefore, in practical engineering, the technical solutions and functions of the various embodiments of this invention can be encapsulated into various modules. Based on this reality, and building upon the above embodiments, this invention provides a visual servoing-based heterogeneous three-arm cooperative control system. This device is used to execute the visual servoing-based heterogeneous three-arm cooperative control method in the above method embodiments. It includes: The visual perception and pose estimation module is used to control the camera to collect point cloud data in the workspace, and to calculate the pose of the target workpiece in the camera coordinate system in real time through point cloud registration and feature extraction algorithms. The multi-arm collaborative control module is used to establish a global coordinate system, combine the end effector states of the three robotic arms into a generalized workspace state vector, and decompose the task objective into absolute motion and relative motion according to task requirements. The closed-loop visual servo control module is used to generate joint space speed commands based on the visual error between the current pose and the desired pose and the heterogeneous joint Jacobian matrix, and then send them to the underlying driver of the three robotic arms to achieve high-frequency closed-loop servo control.

[0047] Example: In an embodiment of the present invention, the system uses two RM75-6F robotic arms (2×7 dof) and one Yushu Z1 robotic arm (6 dof). It utilizes high-precision RGB image data and depth information provided by the Intel RealSense L515 depth camera to construct closed-loop visual feedback and establish a position-based visual servoing (PBVS) model. Furthermore, it introduces the theory of null-space projection to achieve secondary tasks by utilizing the redundant degrees of freedom of the three-arm system while ensuring the primary task of accurate end-effector pose tracking.

[0048] Specifically, such as Figure 2 As shown, the macroscopic architecture of this system is divided into three core modules: perception layer, collaborative decision layer and bottom control layer. (1) Perception layer (L515 visual perception and pose estimation): Intel RealSense L515 depth camera is used to collect RGB-D point cloud data of the workspace in eye-in-hand mode. The three-dimensional pose matrix of the target workpiece in the camera coordinate system is calculated in real time through point cloud registration and feature extraction algorithms. (2) Collaborative decision layer (heterogeneous multi-arm collaborative mapping): The system establishes a global base coordinate system. Since the configurations of RM75-6F and Z1 are different, the end effector states of the three are combined into a generalized workspace state vector. The decision layer decomposes "absolute motion" (i.e., the overall motion of the envelope formed by the three arms) and "relative motion" (i.e., the working motion of Z1 relative to the two RM75-6Fs) according to the task requirements, and the precise position control method of the robotic arm. (3) Control layer (closed-loop visual servo calculation): The precise position control method of the robotic arm is as follows. Figure 3 As shown, the control layer calculates the visual error between the current pose and the desired pose, combines the heterogeneous joint Jacobian matrix, uses the optimal joint velocity control law to generate velocity commands in the joint space, and sends them to the underlying drivers of the RM75-6F and Z1 to achieve high-frequency closed-loop servoing.

[0049] The system of this invention decouples and remaps the Relmann RM75-6F and the Yushu Z1 from the underlying kinematic dimension, constructing a strictly corresponding joint Jacobian matrix, eliminating the collaborative trajectory tearing and chattering phenomenon caused by inconsistent dynamic parameters between heterogeneous robotic arms; the algorithm can calculate the optimal anti-collision joint speed at a frequency of 150Hz, ensuring absolute mechanical safety when the three arms work together in a compact space.

[0050] It should be noted that the apparatus in the device embodiments provided by the present invention can be used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The only difference is that corresponding functional modules are set. The principle is basically the same as that of the above device embodiments provided by the present invention. As long as those skilled in the art can improve the apparatus in the above device embodiments by referring to the specific technical solutions in other method embodiments and combining technical features to obtain corresponding technical means and technical solutions composed of these technical means, on the basis of the above device embodiments, under the premise of ensuring the practicality of the technical solutions, so as to obtain corresponding device-type embodiments for implementing the methods in other method-type embodiments.

[0051] The method in this embodiment of the invention is implemented using an electronic device; therefore, it is necessary to introduce the relevant electronic device. For this purpose, this embodiment of the invention provides an electronic device, such as... Figure 3 As shown, the electronic device includes: at least one central processor, a communications interface, at least one memory, and a communications bus, wherein the at least one central processor, the communications interface, and the at least one memory communicate with each other via the communications bus. The at least one central processor can invoke logical instructions in the at least one memory to execute all or part of the steps of the methods provided in the foregoing method embodiments.

[0052] Furthermore, when the logical instructions in at least one of the aforementioned memories can be implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various method embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0053] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0054] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0055] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Based on this understanding, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0056] In this patent, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0057] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A heterogeneous three-arm cooperative control method based on visual servoing, characterized in that, Includes the following steps: The joint joint variable matrix of the three-arm system in the global coordinate system is determined based on the hand-eye transformation matrix, the base coordinate system of the three robotic arms, and the camera coordinate system. Heterogeneous joint Jacobian matrix is ​​constructed based on the motion velocity relationship of the three-arm system and the joint joint variable matrix; Feature point extraction is performed to determine feature errors, and the main control law for visual servoing is established. An obstacle avoidance cost function based on gradient descent is introduced, and the optimal joint velocity control law is obtained based on null space projection, heterogeneous joint Jacobian matrix and visual servoing master task control law.

2. The heterogeneous three-arm cooperative control method based on visual servoing according to claim 1, wherein, The determination of the joint joint variable matrix of the three-arm system in the global coordinate system based on the hand-eye transformation matrix, the base coordinate system, and the camera coordinate system includes: Establish a global coordinate system, calibrate and determine the transformation relationship between the base coordinate system and the global coordinate system of the three robotic arms, and transform the base coordinate system of the three robotic arms to the global coordinate system; The target pose in the camera coordinate system is transformed to the global coordinate system based on the hand-eye transformation matrix, and the joint joint variable matrix of the three robotic arms in the global coordinate system is solved inversely.

3. The heterogeneous three-arm cooperative control method based on visual servoing according to claim 1, wherein, The heterogeneous joint Jacobian matrix is ​​constructed from the motion velocity relationships of the three-arm system and the joint joint variable matrix, including: In the global coordinate system, the overall motion of the envelope formed by the three robotic arms is taken as the absolute motion, and the motion of the manipulator relative to the other two gripping arms is taken as the relative motion. The relationship between the absolute motion speed, the relative motion speed and the task space speed of the three-arm system is obtained. Based on the relationship between absolute motion velocity, relative motion velocity and the task space velocity of the three-arm system, and the joint joint variable matrix, the heterogeneous joint Jacobian matrix is ​​obtained.

4. The heterogeneous three-arm cooperative control method based on visual servoing according to claim 1, wherein, The relationship between the absolute motion velocity, relative motion velocity, and the mission space velocity of the three-arm system is as follows: ， wherein, is the three-arm system task space velocity, is the absolute motion velocity of the three-arm synergic center, is the relative velocity of the operating arm with respect to the gripping arm, i is selected from 1, 2, and T is the matrix transpose.

5. The heterogeneous three-arm cooperative control method based on visual servoing according to claim 4, characterized in that, The heterogeneous joint Jacobian matrix for: in, For the three-arm system's mission space velocity, This is the joint joint variable matrix.

6. The heterogeneous three-arm cooperative control method based on visual servoing according to claim 1, characterized in that, The process of extracting feature points to determine feature errors and establishing the main control law for visual servoing includes: Feature point extraction is performed to determine the feature error, resulting in a feature error vector: ， in, For the extracted feature points, Let t be the desired characteristic state, and t be the current time. Construct an exponentially decaying servo control law to obtain the desired task space command speed: ， in It is a positive definite gain matrix. , It is the pseudo-inverse of the visual interaction matrix.

7. The heterogeneous three-arm cooperative control method based on visual servoing according to claim 6, characterized in that, The introduction of an obstacle avoidance cost function based on gradient descent, and the obtaining of the optimal joint velocity control law based on null space projection, heterogeneous joint Jacobian matrix, and visual servoing master task control law, includes: An obstacle avoidance cost function based on gradient descent is introduced into the task control law for self-collision avoidance and joint restraint. In the task priority framework of null-space projection, visual servo tracking is taken as the first priority task, and self-collision avoidance and joint restraint are taken as the second priority tasks, thus obtaining the optimal joint velocity control law: ， in, For Moore-Penrose pseudo-inverse, Let I be the desired task space instruction velocity, and let I be the identity mapping matrix of the joint joint space. For heterogeneous joint Jacobian matrices, The gradient vector of the obstacle avoidance cost function H with respect to the joint joint variable matrix. This is the joint joint variable matrix.

8. A heterogeneous three-arm cooperative control system based on visual servoing, executing the heterogeneous three-arm cooperative control method as described in claim 1, characterized in that, include: The visual perception and pose estimation module is used to control the camera to collect point cloud data in the workspace, and to calculate the pose of the target workpiece in the camera coordinate system in real time through point cloud registration and feature extraction algorithms. The multi-arm collaborative control module is used to establish a global coordinate system, combine the end effector states of the three robotic arms into a generalized workspace state vector, and decompose the task objective into absolute motion and relative motion according to task requirements. The closed-loop visual servo control module is used to generate joint space speed commands based on the visual error between the current pose and the desired pose and the heterogeneous joint Jacobian matrix, and then send them to the underlying driver of the three robotic arms to achieve high-frequency closed-loop servo control.

9. An electronic device, characterized in that, include: At least one central processing unit; And at least one memory communicatively connected to a central processing unit, wherein: the memory stores program instructions executable by the central processing unit, and the central processing unit invokes the program instructions to perform the method of any one of claims 1-7.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method of any one of claims 1-7.

Images