A method and device for calibrating a robot arm, an electronic device and a storage medium

Abstract

The present disclosure provides a kind of calibration method, device, electronic equipment and storage medium of mechanical arm, applied in multi-mechanical arm system, multi-mechanical arm system at least includes first mechanical arm and second mechanical arm, camera is installed on the first mechanical arm, method includes: using the camera on the first mechanical arm to photograph target device, obtain the image of target device, and determine the first pixel coordinate of target device;First pose information when the first mechanical arm is photographed to target device is acquired;Using the camera on the first mechanical arm, when the first calibration is carried out to the second mechanical arm, the second mechanical arm is photographed;Second pose information when the first mechanical arm is photographed to the second mechanical arm is acquired;Based on first pose information, first pixel coordinate and second pose information, determine the second pixel coordinate of target device when the first calibration is carried out to the second mechanical arm;Based on second pixel coordinate, determine the world coordinate of target device under the base coordinate system of the second mechanical arm.

Description

Technical Field

[0001] This disclosure relates to the field of robot hand-eye calibration technology, and in particular to a calibration method, apparatus, electronic device and storage medium for a robotic arm. Background Technology

[0002] In multi-arm collaborative systems, to enable multiple arms to conduct joint tests, the physical world coordinates of the target point need to be input to each arm. This requires all arms to be in the same spatial coordinate system; the process of obtaining this coordinate system is called world coordinate system calibration. Traditional world coordinate system calibration schemes can employ a direct method, utilizing known geometric relationships to determine the relative positions and orientations of other arms with respect to the selected world coordinate system through measurement or calculation; or a physical contact method, where calibration is achieved by having the end effectors of different arms contact the same physical point. However, in related technologies, large-field-of-view vision sensors are predominantly used for world coordinate system calibration, which is insufficient for high-precision scenarios. Summary of the Invention

[0003] This disclosure provides a calibration method, apparatus, electronic device, and storage medium for a robotic arm, to at least solve the above-mentioned technical problems existing in the prior art.

[0004] According to a first aspect of this disclosure, a calibration method for a robotic arm is provided, applied to a multi-robotic arm system, the multi-robotic arm system including at least a first robotic arm and a second robotic arm, wherein a camera is mounted on the first robotic arm, wherein the method includes: The target device is photographed using a camera on the first robotic arm to obtain an image of the target device; Based on the image of the target device, determine the first pixel coordinates of the target device; Obtain the first pose information of the first robotic arm when taking a picture of the target device; Using the camera on the first robotic arm, take a picture of the second robotic arm when the second robotic arm is calibrated for the first time; Obtain the second pose information of the first robotic arm when it takes a picture of the second robotic arm; Based on the first pose information, the first pixel coordinates, and the second pose information, the second pixel coordinates of the target device are determined when the second robotic arm performs the first calibration. Based on the second pixel coordinates, the world coordinates of the target device in the base coordinate system of the second robotic arm are determined.

[0005] In one possible implementation, before taking a picture of the target device using a camera on the first robotic arm, the method further includes: Perform a second calibration on the first robotic arm and obtain the first hand-eye matrix of the first robotic arm during the second calibration. The step of determining the second pixel coordinates of the target device during the first calibration of the second robotic arm based on the first pose information, the first pixel coordinates, and the second pose information includes: Based on the first pose information and the first hand-eye matrix, a first association relationship is determined between the first pixel coordinates and the first world coordinates of the target device in the base coordinate system of the first robotic arm; Based on the second pose information and the first hand-eye matrix, a second association relationship is determined between the second pixel coordinates and the second world coordinates of the target device in the base coordinate system of the first robotic arm; Based on the mapping relationship between the first world coordinates and the second world coordinates, as well as the first association relationship and the second association relationship, the second pixel coordinates of the target device are determined when the second robotic arm performs the first calibration.

[0006] In one possible implementation, when performing a first calibration on the second robotic arm, the method further includes: Obtain the third pose information of the second robotic arm during the first calibration; Determining the world coordinates of the target device in the base coordinate system of the second robotic arm based on the second pixel coordinates includes: Based on the third pose information and the second pixel coordinates, the world coordinates of the target device in the base coordinate system of the second robotic arm are determined.

[0007] In one possible implementation, the first pose information includes a first rotation matrix and a first translation matrix; the first hand-eye matrix includes a second rotation matrix and a second translation matrix; The step of determining the first association relationship between the first pixel coordinates and the first world coordinates of the target device in the base coordinate system of the first robotic arm, based on the first pose information and the first hand-eye matrix, includes: Based on the second rotation matrix and the second translation matrix, a first sub-association relationship between the first pixel coordinates and the first coordinate is determined, wherein the first coordinate is the coordinate of the first pixel coordinates transformed from the camera coordinate system to the end coordinate system of the first robotic arm; Based on the first rotation matrix, the first translation matrix, and the first sub-association relationship, a first association relationship between the first pixel coordinates and the first world coordinates is determined.

[0008] In one possible implementation, the second pose information includes a third rotation matrix and a third translation matrix; The step of determining the second association relationship between the second pixel coordinates and the second world coordinates of the target device in the base coordinate system of the first robotic arm, based on the second pose information and the first hand-eye matrix, includes: Based on the second rotation matrix and the second translation matrix, a second sub-association relationship between the second pixel coordinates and the second coordinate is determined, wherein the second coordinate is the coordinate of the second pixel coordinate transformed from the camera coordinate system to the end coordinate system of the first robotic arm; Based on the third rotation matrix, the third translation matrix, and the second sub-association relationship, a second association relationship between the second pixel coordinates and the second world coordinates is determined.

[0009] In one possible implementation, determining the second pixel coordinates of the target device during the first calibration of the second robotic arm, based on the mapping relationship between the first world coordinates and the second world coordinates, and the first association relationship and the second association relationship, includes: The mapping relationship between the first world coordinates and the second world coordinates includes the first world coordinates and the second world coordinates being the same; Based on the fact that the first world coordinates and the second world coordinates are the same, it is determined that the first association relationship and the second association relationship are the same; Based on the fact that the first association relationship and the second association relationship are the same, the second pixel coordinates in the second association relationship are determined.

[0010] In one possible implementation, the third pose information includes a fourth rotation matrix and a fourth translation matrix; Determining the world coordinates of the target device in the base coordinate system of the second robotic arm based on the third pose information and the second pixel coordinates includes: Based on the fourth rotation matrix, the fourth translation matrix, and the second pixel coordinates, the world coordinates of the second pixel coordinates are determined from the camera coordinate system to the base coordinate system of the second robotic arm.

[0011] According to a second aspect of this disclosure, a calibration device for a robotic arm is provided, wherein the device comprises: A multi-robotic arm system, the multi-robotic arm system including at least a first robotic arm and a second robotic arm, wherein a camera is mounted on the first robotic arm; The first acquisition unit is used to take a picture of the target device using a camera on the first robotic arm, and acquire an image of the target device; The first determining unit is configured to determine the first pixel coordinates of the target device based on the image of the target device; The second acquisition unit is used to acquire the first pose information of the first robotic arm when taking a picture of the target device; The photographing unit is used to take a picture of the second robotic arm using a camera on the first robotic arm when the second robotic arm is calibrated for the first time. The third acquisition unit is used to acquire the second pose information of the first robotic arm when it takes a picture of the second robotic arm; The second determining unit is used to determine the second pixel coordinates of the target device when the second robotic arm performs the first calibration, based on the first pose information, the first pixel coordinates and the second pose information; The third determining unit is used to determine the world coordinates of the target device in the base coordinate system of the second robotic arm based on the second pixel coordinates.

[0012] According to a third aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and a memory communicatively connected to said at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described in this disclosure.

[0013] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in this disclosure.

[0014] The present disclosure discloses a calibration method, apparatus, electronic device, and storage medium for a robotic arm. First, using a camera on a first robotic arm, the first pixel coordinates of the target device and the first pose information of the first robotic arm when taking a picture of the target device are determined. Then, during the first calibration of the second robotic arm, the second pose information of the first robotic arm when taking a picture of the second robotic arm is obtained. Based on the first pose information, the first pixel coordinates, and the second pose information, the second pixel coordinates of the target device during the first calibration of the second robotic arm are determined. These second pixel coordinates are not the pixel coordinates of the actual image position obtained by the camera, but rather virtual pixel coordinates of a virtual image position set during the first calibration of the second robotic arm. These second pixel coordinates can be transferred to the base coordinate system of each robotic arm to achieve the same effect as world coordinates. Because the first calibration is performed on the second robotic arm, the camera can use a small field-of-view high-precision visual sensor, resulting in high data accuracy and simple operation, making it suitable for small field-of-view sensor scenarios.

[0015] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0016] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0017] Figure 1 A flowchart of a calibration method for a robotic arm provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of the structure of the multi-robotic arm system in an embodiment of this disclosure; Figure 3 A schematic diagram of the structure of the calibration device for the robotic arm provided in the embodiments of this disclosure; Figure 4 A schematic diagram of the composition structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0018] To make the objectives, features, and advantages of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0019] This disclosure provides a calibration method for a robotic arm, applicable to a multi-robotic arm system, which includes at least a first robotic arm and a second robotic arm, with a camera mounted on the first robotic arm. Figure 1 This is a flowchart of a calibration method for a robotic arm provided in an embodiment of this disclosure. Figure 2 This is a schematic diagram of the structure of the multi-robotic arm system in an embodiment of this disclosure, as shown below. Figure 1 and Figure 2 As shown, the method includes: Step 101: Use the camera 20 on the first robotic arm 11 to take a picture of the target device 30 and obtain an image of the target device 30.

[0020] Before taking a picture of the target device 30 using the camera 20 on the first robotic arm 11, the method further includes: The first robotic arm 11 is calibrated a second time, and the first hand-eye matrix of the first robotic arm 11 during the second calibration is obtained. The second calibration is an eye-in-hand calibration.

[0021] Specifically, the first robotic arm 11 is first positioned above the target device 30 in a fixed photographing posture, and the target device is then positioned on the imaging surface of the camera 20.

[0022] Then, the camera on the first robotic arm 11 is used to perform a second calibration of the first robotic arm 11, obtaining the first hand-eye matrix of the first robotic arm 11 during calibration. The first hand-eye matrix includes the second rotation matrix. Second translation matrix .

[0023] Then, the camera 20 on the first robotic arm 11 is used to take pictures of the target device 30 to obtain an image of the target device 30.

[0024] Camera 20 can employ a small field-of-view, high-precision vision sensor, resulting in high data accuracy. The field of view of camera 20 is 40mm. 30mm.

[0025] Next, step 102 is executed to determine the first pixel coordinates of the target device 30 based on the image of the target device 30; step 103 is executed to obtain the first pose information of the first robotic arm 11 when taking a picture of the target device 30.

[0026] Specifically, the first robotic arm 11 is controlled to move to the position of the target device 30, and a picture of the target device 30 is taken to obtain an image of the target device 30. The first pixel coordinates of the target device 30 are then obtained from the image. Because there are many points on the target device 30, the first pixel coordinates of the target device 30 are determined... In this case, the coordinates of the midpoint of one edge of the target device 30 can be used as the coordinates of the entire device. First pixel coordinates These are coordinates in the camera coordinate system, which can be the camera's built-in coordinate system. Next, the first pose information of the first robotic arm 11 when taking a picture of the target device 30 is obtained. This first pose information includes the first rotation matrix. and the first translation matrix .

[0027] Next, step 104 is executed, using the camera 20 on the first robotic arm 11 to take a picture of the second robotic arm 12 during the first calibration; step 105 is executed, to obtain the second pose information of the first robotic arm 11 when taking a picture of the second robotic arm 12.

[0028] The first calibration is the eye-to-hand calibration.

[0029] In one embodiment, when calibrating the second robotic arm 12 for the first time, the method further includes: obtaining the third pose information of the second robotic arm 12 during the calibration.

[0030] Specifically, the first robotic arm 11 is first fixed above the target device 30 in a fixed photographing posture. At this time, the second robotic arm 12 can be calibrated relative to the camera 20. The camera 20 is used to take a picture of the second robotic arm 12 to obtain an image of the second robotic arm 12. Based on the image, the third pose information of the second robotic arm during calibration can be obtained. The third pose information includes a fourth rotation matrix. and the fourth translation matrix Simultaneously, the second pose information of the first robotic arm 11 when taking a picture of the second robotic arm 12 is acquired. The second pose information includes a third rotation matrix. and the third translation matrix The second pose information is also the kinematics of the first robotic arm 11 at this time.

[0031] It should be explained that the order of steps 101, 102, and 103 can be reversed with that of steps 104 and 105. That is, steps 104 and 105 can be executed first, followed by steps 101, 102, and 103.

[0032] Next, step 106 is executed, based on the first pose information, the first pixel coordinates and the second pose information, to determine the second pixel coordinates of the target device 30 when the second robotic arm 12 performs the first calibration.

[0033] In one embodiment, determining the second pixel coordinates of the target device 30 during the first calibration of the second robotic arm 12, based on the first pose information, the first pixel coordinates, and the second pose information, includes: Based on the first pose information and the first hand-eye matrix, the first association relationship between the first pixel coordinates and the first world coordinates of the target device 30 in the base coordinate system of the first robotic arm 11 is determined; Based on the second pose information and the first hand-eye matrix, a second association relationship is determined between the second pixel coordinates and the second world coordinates of the target device 30 in the base coordinate system of the first robotic arm 11; Based on the mapping relationship between the first world coordinates and the second world coordinates, as well as the first association relationship and the second association relationship, the second pixel coordinates of the target device 30 are determined when the second robotic arm 12 performs the first calibration.

[0034] In one embodiment, based on the first pose information and the first hand-eye matrix, determining the first association relationship between the first pixel coordinates and the first world coordinates of the target device 30 in the base coordinate system of the first robotic arm 11 includes: Based on the second rotation matrix and the second translation matrix, the first sub-association relationship between the first pixel coordinate and the first coordinate is determined. The first coordinate is the coordinate of the first pixel transformed from the camera coordinate system to the end coordinate system of the first robotic arm 11. Based on the first rotation matrix, the first translation matrix, and the first sub-association relationship, the first association relationship between the first pixel coordinates and the first world coordinates is determined.

[0035] Specifically, based on the following formula, the first correlation relationship between the first pixel coordinates and the first world coordinates of the target device 30 in the base coordinate system of the first robotic arm 11 is determined.

[0036] in, Using first-world coordinates, Let be the first rotation matrix. This is the first translation matrix. This is the second rotation matrix. This is the second translation matrix. The coordinates of the first pixel.

[0037] The above formula represents the first correlation between the first pixel coordinates and the first world coordinates. The above formula means that the first pixel coordinates are first transformed from the camera coordinate system to the end coordinate system of the first robotic arm 11, and then transformed from the end coordinate system of the first robotic arm 11 to the base coordinate system of the first robotic arm 11.

[0038] As shown in the above formula, This formula represents the first sub-association relationship between the first pixel coordinate and the first coordinate, where the formula yields the first coordinate.

[0039] In one embodiment, based on the second pose information and the first hand-eye matrix, determining a second association between the second pixel coordinates and the second world coordinates of the target device 30 in the base coordinate system of the first robotic arm 11 includes: Based on the second rotation matrix and the second translation matrix, the second sub-association relationship between the second pixel coordinates and the second coordinates is determined. The second coordinates are the coordinates of the second pixel coordinates transformed from the camera coordinate system to the end coordinate system of the first robotic arm 11. Based on the third rotation matrix, the third translation matrix, and the second sub-association relationship, the second association relationship between the second pixel coordinates and the second world coordinates is determined.

[0040] Specifically, based on the following formula, the second association relationship between the second pixel coordinates and the second world coordinates is determined.

[0041] in, Second world coordinates, This is the third rotation matrix. This is the third translation matrix. The coordinates are the second pixel coordinates.

[0042] The above formula represents the second relationship between the second pixel coordinates and the second world coordinates. The formula means that the second pixel coordinates are first transformed from the camera coordinate system to the end coordinate system of the first robotic arm, and then transformed from the end coordinate system of the first robotic arm to the base coordinate system of the first robotic arm.

[0043] As shown in the above formula, This formula represents the second sub-association relationship between the second pixel coordinate and the second coordinate, where the formula yields the second coordinate.

[0044] The second pixel coordinates here are not the pixel coordinates of the actual shooting position obtained by taking a picture with the camera, but virtual pixel coordinates of a virtual shooting position set when the second robotic arm is calibrated for the first time. The second pixel coordinates can be transferred to the base coordinate system of each robotic arm to achieve the same effect as the world coordinates.

[0045] In one embodiment, based on the mapping relationship between the first world coordinates and the second world coordinates, as well as the first association relationship and the second association relationship, the second pixel coordinates of the target device 30 are determined when the second robotic arm 12 performs the first calibration, including: The mapping relationship between first-world coordinates and second-world coordinates includes the first-world coordinates and second-world coordinates being the same; Based on the fact that the first world coordinates and the second world coordinates are the same, it is determined that the first association relationship and the second association relationship are the same; Based on the fact that the first association relationship and the second association relationship are the same, the coordinates of the second pixel in the second association relationship are determined.

[0046] The first world coordinates and the second world coordinates obtained above are essentially different mathematical expressions of the same point in space. That is, for the same position, the first world coordinates obtained by the transformation relationship between the image of the target device acquired by the camera and the base coordinate system should be completely the same as the second world coordinates obtained by the kinematics of the first robotic arm. In other words, in this disclosure, the mapping relationship is that the first world coordinates are equal to the second world coordinates.

[0047] In one embodiment, determining the second pixel coordinates in the second association relationship based on the first association relationship being equal to the second association relationship includes: The coordinates of the second pixel are determined based on the following formula.

[0048] In the above formula, the only unknown quantity is the coordinate of the second pixel. Therefore, expanding the above formula, we can obtain,

[0049] For ease of calculation, let

[0050]

[0051] Therefore, we can obtain , Solving .

[0052] Next, step 107 is executed to determine the world coordinates of the target device 30 in the base coordinate system of the second robotic arm 12 based on the second pixel coordinates.

[0053] In one embodiment, determining the world coordinates of the target device 30 in the base coordinate system of the second robotic arm 12 based on the second pixel coordinates includes: Based on the third pose information and the second pixel coordinates, the world coordinates of the target device 30 in the base coordinate system of the second robotic arm 12 are determined.

[0054] In one embodiment, determining the world coordinates of the target device 30 in the base coordinate system of the second robotic arm 12 based on the third pose information and the second pixel coordinates includes: Based on the fourth rotation matrix, the fourth translation matrix, and the second pixel coordinates, the world coordinates of the second pixel coordinates are determined from the camera coordinate system to the base coordinate system of the second robotic arm 12.

[0055] Specifically, the world coordinates of the target device 30 in the base coordinate system of the second robotic arm 12 are determined based on the following formula.

[0056] in, Let be the world coordinates of the target device in the base coordinate system of the second robotic arm. This is the fourth rotation matrix. This is the fourth translation matrix. The coordinates are the second pixel coordinates.

[0057] In this way, the transformation from pixel coordinates to the base coordinate system of the cameraless robotic arm was completed. Without performing traditional world coordinate system calibration, the same result as world coordinate system calibration was obtained by using camera data input.

[0058] When testing multiple robotic arms, a unified coordinate system is essential for effective testing and subsequent data calculation; this requires world coordinate system calibration. The visual sensors used in related technologies are large-field-of-view sensors, capable of easily acquiring global information about the entire scene, allowing for the extraction of necessary data for world coordinate system calibration. However, small-field-of-view sensors (cameras) significantly impact information acquisition, making world coordinate system calibration impossible using the aforementioned method. For small-field-of-view sensor applications, a first calibration scheme using a second robotic arm (without a camera) is adopted as an alternative to world coordinate system calibration. In hand-eye calibration with the eye outside the hand, the visual sensor must be fixed, which contradicts the requirement in this disclosure for the robotic arm to move while carrying the visual sensor for image capture. Therefore, this disclosure proposes using a virtual image capture position to transfer the coordinate information obtained from each image capture in subsequent tasks to the position information under the hand-eye calibration posture. Then, the third pose information from the first calibration is used to transfer the coordinates to the base coordinate system of the second robotic arm.

[0059] This disclosure also provides a calibration device for a robotic arm. Figure 3 A schematic diagram of the structure of the calibration device for the robotic arm provided in the embodiments of this disclosure is shown below. Figure 2 and Figure 3 As shown, the device includes: A multi-arm robotic system, which includes at least a first robotic arm 11 and a second robotic arm, wherein a camera 20 is mounted on the first robotic arm 11; The first acquisition unit 301 is used to take pictures of the target device 30 using the camera 20 on the first robotic arm 11, and acquire an image of the target device 30; The first determining unit 302 is used to determine the first pixel coordinates of the target device 30 based on the image of the target device 30. The second acquisition unit 303 is used to acquire the first pose information of the first robotic arm 11 when taking a picture of the target device 30; The photographing unit 304 is used to take a picture of the second robotic arm 12 when the second robotic arm 12 is calibrated for the first time using the camera 20 on the first robotic arm 11. The third acquisition unit 305 is used to acquire the second pose information of the first robotic arm 11 when taking a picture of the second robotic arm 12. The second determining unit 306 is used to determine the second pixel coordinates of the target device 30 when the second robotic arm 12 performs the first calibration based on the first pose information, the first pixel coordinates and the second pose information. The third determining unit 307 is used to determine the world coordinates of the target device 30 in the base coordinate system of the second robotic arm 12 based on the second pixel coordinates.

[0060] The specific details of each part of the above-mentioned device have been described in detail in the method section of the implementation plan. For any undisclosed details, please refer to the implementation plan of the method section, and therefore will not be repeated here.

[0061] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0062] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0063] like Figure 4 As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 408 into random access memory (RAM) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0064] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0065] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as the calibration method for a robotic arm. For example, in some embodiments, the calibration method for a robotic arm may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the calibration method for a robotic arm described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the calibration method for a robotic arm by any other suitable means (e.g., by means of firmware).

[0066] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0067] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0068] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0069] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0070] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0071] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0072] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0073] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0074] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A calibration method for a robotic arm, applied to a multi-robotic arm system, wherein the multi-robotic arm system includes at least a first robotic arm and a second robotic arm, the first robotic arm being equipped with a camera, characterized in that, The method includes: The target device is photographed using a camera on the first robotic arm to obtain an image of the target device; Based on the image of the target device, determine the first pixel coordinates of the target device; Obtain the first pose information of the first robotic arm when taking a picture of the target device; Using the camera on the first robotic arm, take a picture of the second robotic arm when the second robotic arm is calibrated for the first time; Obtain the second pose information of the first robotic arm when it takes a picture of the second robotic arm; Based on the first pose information, the first pixel coordinates, and the second pose information, the second pixel coordinates of the target device are determined when the second robotic arm performs the first calibration. Based on the second pixel coordinates, the world coordinates of the target device in the base coordinate system of the second robotic arm are determined.

2. The calibration method for a robotic arm according to claim 1, characterized in that, Before taking a picture of the target device using the camera on the first robotic arm, the method further includes: Perform a second calibration on the first robotic arm and obtain the first hand-eye matrix of the first robotic arm during the second calibration. The step of determining the second pixel coordinates of the target device during the first calibration of the second robotic arm based on the first pose information, the first pixel coordinates, and the second pose information includes: Based on the first pose information and the first hand-eye matrix, a first association relationship is determined between the first pixel coordinates and the first world coordinates of the target device in the base coordinate system of the first robotic arm; Based on the second pose information and the first hand-eye matrix, a second association relationship is determined between the second pixel coordinates and the second world coordinates of the target device in the base coordinate system of the first robotic arm; Based on the mapping relationship between the first world coordinates and the second world coordinates, as well as the first association relationship and the second association relationship, the second pixel coordinates of the target device are determined when the second robotic arm performs the first calibration.

3. The calibration method for a robotic arm according to claim 1, characterized in that, When performing a first calibration on the second robotic arm, the method further includes: Obtain the third pose information of the second robotic arm during the first calibration; Determining the world coordinates of the target device in the base coordinate system of the second robotic arm based on the second pixel coordinates includes: Based on the third pose information and the second pixel coordinates, the world coordinates of the target device in the base coordinate system of the second robotic arm are determined.

4. The calibration method for a robotic arm according to claim 2, characterized in that, The first pose information includes a first rotation matrix and a first translation matrix; the first hand-eye matrix includes a second rotation matrix and a second translation matrix; The step of determining the first association relationship between the first pixel coordinates and the first world coordinates of the target device in the base coordinate system of the first robotic arm, based on the first pose information and the first hand-eye matrix, includes: Based on the second rotation matrix and the second translation matrix, a first sub-association relationship between the first pixel coordinates and the first coordinate is determined, wherein the first coordinate is the coordinate of the first pixel coordinates transformed from the camera coordinate system to the end coordinate system of the first robotic arm; Based on the first rotation matrix, the first translation matrix, and the first sub-association relationship, a first association relationship between the first pixel coordinates and the first world coordinates is determined.

5. The calibration method for a robotic arm according to claim 4, characterized in that, The second pose information includes a third rotation matrix and a third translation matrix; The step of determining the second association relationship between the second pixel coordinates and the second world coordinates of the target device in the base coordinate system of the first robotic arm, based on the second pose information and the first hand-eye matrix, includes: Based on the second rotation matrix and the second translation matrix, a second sub-association relationship between the second pixel coordinates and the second coordinates is determined, wherein the second coordinates are the coordinates of the second pixel coordinates transformed from the camera coordinate system to the end coordinate system of the first robotic arm; Based on the third rotation matrix, the third translation matrix, and the second sub-association relationship, a second association relationship between the second pixel coordinates and the second world coordinates is determined.

6. The calibration method for a robotic arm according to claim 2, characterized in that, The determination of the second pixel coordinates of the target device during the first calibration of the second robotic arm, based on the mapping relationship between the first world coordinates and the second world coordinates, and the first association relationship and the second association relationship, includes: The mapping relationship between the first world coordinates and the second world coordinates includes the first world coordinates and the second world coordinates being the same; Based on the fact that the first world coordinates and the second world coordinates are the same, it is determined that the first association relationship and the second association relationship are the same; Based on the fact that the first association relationship and the second association relationship are the same, the second pixel coordinates in the second association relationship are determined.

7. The calibration method for a robotic arm according to claim 3, characterized in that, The third pose information includes a fourth rotation matrix and a fourth translation matrix; Determining the world coordinates of the target device in the base coordinate system of the second robotic arm based on the third pose information and the second pixel coordinates includes: Based on the fourth rotation matrix, the fourth translation matrix, and the second pixel coordinates, the world coordinates of the second pixel coordinates are determined from the camera coordinate system to the base coordinate system of the second robotic arm.

8. A calibration device for a robotic arm, characterized in that, The device includes: A multi-robotic arm system, the multi-robotic arm system including at least a first robotic arm and a second robotic arm, wherein a camera is mounted on the first robotic arm; The first acquisition unit is used to take a picture of the target device using a camera on the first robotic arm, and acquire an image of the target device; The first determining unit is configured to determine the first pixel coordinates of the target device based on the image of the target device; The second acquisition unit is used to acquire the first pose information of the first robotic arm when taking a picture of the target device; The photographing unit is used to take a picture of the second robotic arm using a camera on the first robotic arm when the second robotic arm is calibrated for the first time. The third acquisition unit is used to acquire the second pose information of the first robotic arm when it takes a picture of the second robotic arm; The second determining unit is used to determine the second pixel coordinates of the target device when the second robotic arm performs the first calibration, based on the first pose information, the first pixel coordinates and the second pose information; The third determining unit is used to determine the world coordinates of the target device in the base coordinate system of the second robotic arm based on the second pixel coordinates.

9. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images