An environment information-based hand-eye calibration method, device and electronic equipment

Abstract

The present disclosure provides a kind of based on environmental information hand-eye calibration method, device and electronic equipment, method includes: by calibration camera obtains the first environment image when mechanical arm is in first pose and the second environment image when in second pose;Determine the first pixel coordinate and the second pixel coordinate corresponding to target feature point in first environment image and second environment image;First pixel coordinate is converted into first world coordinate, and second pixel coordinate is converted into second world coordinate;Determine the imaging projection error of hand-eye calibration based on the first world coordinate and the second world coordinate of all target feature points;Imaging projection error is handled, and the hand-eye calibration matrix of hand-eye calibration is obtained, and hand-eye calibration is carried out according to hand-eye calibration matrix.The application of this method, with the target feature point in environment image instead of calibration object completes calibration, without specific calibration object, calibration process and mechanical arm operation process do not need to separate, can guarantee the consistency of calibration environment and operation environment.

Description

Technical Field

[0001] This disclosure relates to the field of robot vision, and in particular to a hand-eye calibration method, apparatus, and electronic device based on environmental information. Background Technology

[0002] Vision-guided robotic arm systems can perform numerous automated tasks, and visual calibration (i.e., hand-eye calibration) is the foundation and key to vision-guided robotic arm operations. Currently, hand-eye calibration typically uses a specific calibration object. Through manual operation, the coordinates of this specific calibration object in both the visual coordinate system and the coordinate system to be calibrated are obtained. The hand-eye calibration matrix is ​​then obtained through coordinate transformation, and the robotic arm movement is guided according to the hand-eye calibration matrix to complete the calibration. However, when using a specific calibration object for hand-eye calibration, it is necessary to obtain the coordinates of this specific calibration object in both the visual coordinate system and the coordinate system to be calibrated. The coordinates in the coordinate system to be calibrated usually need to be obtained manually, which is difficult to obtain directly. The accuracy of manual operation affects the accuracy of hand-eye calibration. Furthermore, during calibration, the calibration process is separated from the robotic arm operation process. Calibration requires interrupting the work environment to place the calibration object, etc., making it impossible to guarantee that the calibration environment is completely consistent with the robotic arm's working environment. Summary of the Invention

[0003] This disclosure provides a hand-eye calibration method, apparatus, and electronic device based on environmental information, to at least solve the above-mentioned technical problems existing in the prior art.

[0004] According to a first aspect of this disclosure, a hand-eye calibration method based on environmental information is provided. The method includes: acquiring a first environmental image of the robotic arm in a first pose and a second environmental image in a second pose using a calibration camera in a robotic arm system; determining first pixel coordinates and second pixel coordinates corresponding to target feature points in the first environmental image and the second environmental image; converting the first pixel coordinates into first world coordinates and the second pixel coordinates into second world coordinates; determining the imaging projection error of hand-eye calibration based on the first world coordinates and second world coordinates of all target feature points; processing the imaging projection error of hand-eye calibration according to the method of Lie groups and Lie algebras to obtain a hand-eye calibration matrix, and performing hand-eye calibration according to the hand-eye calibration matrix.

[0005] In one possible implementation, determining the first pixel coordinates and second pixel coordinates corresponding to the target feature points in the first environmental image and the second environmental image includes: extracting initial feature points from the first environmental image and the second environmental image respectively; matching the initial feature points to determine the same initial feature points in the first environmental image and the second environmental image as target feature points; determining the first pixel coordinates corresponding to each target feature point in the first environmental image; and determining the second pixel coordinates corresponding to each target feature point in the second environmental image.

[0006] In one possible implementation, the step of converting the first pixel coordinates into first world coordinates and the second pixel coordinates into second world coordinates includes: converting the first pixel coordinates into first world coordinates based on the first pixel coordinates, the intrinsic parameters of the calibration camera, and the kinematics of the robotic arm; and converting the second pixel coordinates into second world coordinates based on the second pixel coordinates, the intrinsic parameters of the calibration camera, and the kinematics of the robotic arm. The calibration camera in the robotic arm system has two mounting methods: eye-on-hand and eye-out-of-hand. When the calibration camera is mounted with the eye on the hand, the robotic arm kinematics are forward kinematics; when the calibration camera is mounted with the eye-out-of-hand, the robotic arm kinematics are inverse kinematics.

[0007] In one possible implementation, determining the imaging projection error of hand-eye calibration based on the first world coordinates and the second world coordinates of all target feature points includes: determining the distance between the first world coordinates and the second world coordinates of each target feature point as the imaging projection error of the target feature point; and summing the imaging projection errors of all target feature points to determine the imaging projection error of hand-eye calibration.

[0008] In one embodiment, processing the imaging projection error of the hand-eye calibration using the method of Lie groups and Lie algebras to obtain the hand-eye calibration matrix includes: converting the imaging projection error of the hand-eye calibration into Lie algebra form according to the exponential mapping transformation of Lie groups and Lie algebras; determining the minimum value of the imaging projection error converted into Lie algebra form; determining the optimal value of the Lie algebra of the hand-eye calibration matrix based on the minimum value; and determining the hand-eye calibration matrix based on the optimal value and the logarithmic mapping transformation of Lie groups and Lie algebras.

[0009] According to a second aspect of this disclosure, a hand-eye calibration device based on environmental information is provided. The device includes: an acquisition module, configured to acquire a first environmental image of the robotic arm in a first pose and a second environmental image of the robotic arm in a second pose via a calibration camera in a robotic arm system; a first determination module, configured to determine the first pixel coordinates and the second pixel coordinates corresponding to target feature points in the first environmental image and the second environmental image; a conversion module, configured to convert the first pixel coordinates into first world coordinates and the second pixel coordinates into second world coordinates; a second determination module, configured to determine the imaging projection error of hand-eye calibration based on the first world coordinates and the second world coordinates of all target feature points; and a processing module, configured to process the imaging projection error of hand-eye calibration according to the methods of Lie groups and Lie algebras to obtain a hand-eye calibration matrix, and perform hand-eye calibration according to the hand-eye calibration matrix.

[0010] In one embodiment, the first determining module includes: an extraction submodule, configured to extract initial feature points from the first environmental image and the second environmental image respectively; a matching submodule, configured to match the initial feature points and determine the same initial feature points in the first environmental image and the second environmental image as target feature points; a first determining submodule, configured to determine the first pixel coordinates corresponding to each target feature point in the first environmental image; and the first determining submodule is further configured to determine the second pixel coordinates corresponding to each target feature point in the second environmental image.

[0011] In one possible implementation, the conversion module is specifically used to convert the first pixel coordinates into first world coordinates based on the first pixel coordinates, the intrinsic parameters of the calibration camera, and the robotic arm kinematics; and to convert the second pixel coordinates into second world coordinates based on the second pixel coordinates, the intrinsic parameters of the calibration camera, and the robotic arm kinematics; wherein, the calibration camera in the robotic arm system has two mounting methods: eye-on-hand and eye-outside-hand. When the calibration camera adopts the eye-on-hand mounting method, the robotic arm kinematics are forward robotic arm kinematics; when the calibration camera adopts the eye-outside-hand mounting method, the robotic arm kinematics are inverse robotic arm kinematics.

[0012] In one possible implementation, the second determining module is specifically used to determine the distance between the first world coordinate and the second world coordinate of each target feature point as the imaging projection error of the target feature point; and to add up the imaging projection errors of all target feature points to determine the imaging projection error of the hand-eye calibration.

[0013] In one embodiment, the processing module includes: a transformation submodule, configured to transform the imaging projection error of the hand-eye calibration into Lie algebra form according to the exponential mapping transformation of Lie groups and Lie algebras; a second determination submodule, configured to determine the minimum value of the imaging projection error transformed into Lie algebra form; the second determination submodule is further configured to determine the optimal value of the Lie algebra of the hand-eye calibration matrix according to the minimum value; the second determination submodule is further configured to determine the hand-eye calibration matrix of the hand-eye calibration according to the optimal value based on the logarithmic mapping transformation of Lie groups and Lie algebras.

[0014] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described in this disclosure.

[0018] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in this disclosure.

[0019] This disclosure discloses a hand-eye calibration method, apparatus, electronic device, and storage medium based on environmental information. Using a calibration camera in a robotic arm system, it acquires a first environmental image when the robotic arm is in a first pose and a second environmental image when it is in a second pose. It determines the first pixel coordinates and second pixel coordinates corresponding to target feature points in the first and second environmental images, respectively, and converts them into first world coordinates and second world coordinates. Based on the first and second world coordinates of the target feature points, it determines the imaging projection error of hand-eye calibration. The imaging projection error of hand-eye calibration is processed using Lie groups and Lie algebras to obtain the hand-eye calibration matrix, and hand-eye calibration is performed based on the calibration matrix. This method uses environmental information instead of calibration objects, completing calibration through target feature points in the environmental images. It eliminates the need for specific calibration objects, and the calibration process does not need to be separated from the robotic arm operation process, ensuring consistency between the calibration environment and the operating environment for the vision system. By determining the imaging projection error and processing it using Lie groups and Lie algebras to solve for the hand-eye calibration matrix, it improves the calibration accuracy of the vision system and reduces the computational complexity.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0022] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0023] Figure 1 This illustration shows the implementation flow of a hand-eye calibration method based on environmental information according to an embodiment of the present disclosure. Figure 1 ;

[0024] Figure 2 This illustration shows the implementation flow of a hand-eye calibration method based on environmental information according to an embodiment of the present disclosure. Figure 2 ;

[0025] Figure 3 A schematic diagram of a hand-eye calibration device based on environmental information according to an embodiment of the present disclosure is shown.

[0026] Figure 4 A schematic diagram of the composition structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0027] To make the objectives, features, and advantages of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0028] Figure 1 This illustration shows the implementation flow of a hand-eye calibration method based on environmental information according to an embodiment of the present disclosure. Figure 1 ,include:

[0029] Step 101: Using the calibration camera in the robotic arm system, acquire the first environmental image when the robotic arm is in the first pose and the second environmental image when it is in the second pose.

[0030] Vision-guided robotic arm systems can perform numerous automated tasks, and hand-eye calibration is the foundation and key to vision-guided robotic arm operations. The purpose of hand-eye calibration is to determine the relative positional relationship between the robotic arm and the calibration camera, so that the robotic arm can accurately locate and manipulate target objects. In this application, the robotic arm first moves to a first pose, and the calibration camera takes a picture, using the environmental image of the robotic arm in the first pose as the first environmental image; then the robotic arm moves to a second pose, which is different from the first pose, and the calibration camera continues to take pictures, using the environmental image of the robotic arm in the second pose as the second environmental image.

[0031] Step 102: Determine the first pixel coordinates and the second pixel coordinates corresponding to the target feature points in the first environmental image and the second environmental image.

[0032] The first and second environmental images each include multiple feature points. Feature points refer to special locations in the image, such as corners, edges, or regions. Identical feature points in both the first and second environmental images are identified as target feature points. The first pixel coordinates of the target feature points in the first environmental image are determined based on the first environmental image, and the second pixel coordinates of the target feature points in the second environmental image are determined based on the second environmental image.

[0033] Step 103: Convert the first pixel coordinates to first world coordinates and the second pixel coordinates to second world coordinates.

[0034] The first pixel coordinates and the second pixel coordinates are the coordinates of the target feature point in the image. Now, the first pixel coordinates and the second pixel coordinates are transformed into first world coordinates and second world coordinates, which are the coordinates of the target feature point in world space.

[0035] Step 104: Determine the imaging projection error of hand-eye calibration based on the first world coordinates and second world coordinates of all target feature points.

[0036] The world coordinates of a target feature point are unique, with one target feature point corresponding to one world coordinate. However, in the actual acquisition process, there may be errors between the world coordinates of the target feature point determined by different methods. Therefore, in this application, the imaging projection error of hand-eye calibration is determined based on the first world coordinate and the second world coordinate of the target feature point.

[0037] Step 105: Process the imaging projection error of hand-eye calibration according to the method of Lie group and Lie algebra to obtain the hand-eye calibration matrix, and perform hand-eye calibration according to the hand-eye calibration matrix.

[0038] By using Lie groups and Lie algebraic mapping transformations to process the imaging projection error of hand-eye calibration, the hand-eye calibration matrix can be obtained, which can reduce the computational complexity of solving the problem.

[0039] This disclosure provides a hand-eye calibration method based on environmental information. Using a calibration camera in a robotic arm system, it acquires a first environmental image when the robotic arm is in a first pose and a second environmental image when it is in a second pose. The first and second pixel coordinates corresponding to target feature points in the first and second environmental images are determined. These coordinates are then converted into first and second world coordinates. Based on the first and second world coordinates of all target feature points, the imaging projection error for hand-eye calibration is determined. The imaging projection error is processed using Lie groups and Lie algebras to obtain a hand-eye calibration matrix. Hand-eye calibration is then performed based on this matrix. This method uses environmental information instead of calibration objects, completing calibration through target feature points in the environmental images. It eliminates the need for specific calibration objects, and the calibration process does not need to be separated from the robotic arm operation process. This ensures the consistency of the calibration environment and the operating environment for the vision system, improving the calibration accuracy of the vision system.

[0040] In one possible implementation, such as Figure 2 As shown, determining the first pixel coordinates and second pixel coordinates corresponding to the target feature points in the first environment image and the second environment image includes:

[0041] Step 201: Extract initial feature points from the first environmental image and the second environmental image respectively;

[0042] Step 202: Match the initial feature points and determine the same initial feature points in the first environment image and the second environment image as target feature points;

[0043] Step 203: Determine the first pixel coordinates corresponding to each target feature point in the first environment image;

[0044] Step 204: Determine the second pixel coordinates corresponding to each target feature point in the second environment image.

[0045] Initial feature points are extracted from the first and second environmental images using a visual algorithm. A visual matching algorithm then matches these initial feature points, identifying the common initial feature points as target feature points. For each target feature point, its pixel coordinates in the first environmental image are determined as the first pixel coordinates, and its pixel coordinates in the second environmental image are determined as the second pixel coordinates. It is understood that when determining the first and second pixel coordinates of the target feature points, the pixel coordinate system should be constructed in both the first and second environmental images in the same way. For example, one possible implementation is to use the top-left pixel of both images as the origin, and the axes as rays pointing from the top-left to the top-right and bottom-left corners. By using the pixel coordinates of the target feature points in the environmental images to replace the coordinates of the calibration object in the reference coordinate system, the calibration environment is made identical to the working environment, resulting in strong adaptability.

[0046] In one possible implementation, converting the first pixel coordinates into first world coordinates and the second pixel coordinates into second world coordinates includes:

[0047] The first pixel coordinates are transformed into first world coordinates based on the first pixel coordinates, the intrinsic parameters of the calibrated camera, and the kinematics of the robotic arm.

[0048] The second pixel coordinates are transformed into second world coordinates based on the second pixel coordinates, the intrinsic parameters of the calibrated camera, and the kinematics of the robotic arm.

[0049] The calibration camera in the robotic arm system has two mounting methods: eye-on-hand and eye-out-of-hand. When the calibration camera is mounted with the eye on the hand, the robotic arm kinematics is the forward kinematics. When the calibration camera is mounted with the eye-out-of-hand, the robotic arm kinematics is the inverse kinematics.

[0050] Based on the calibrated camera model, the pixel imaging model is determined, and the pixel coordinates of the target feature point in the environmental image are transformed into world coordinates in the base coordinate system. That is, the first pixel coordinate of the target feature point in the first environmental image is transformed into the first world coordinate in the base coordinate system, and the second pixel coordinate of the target feature point in the second environmental image is transformed into the second world coordinate in the base coordinate system.

[0051] Specifically, the first pixel coordinates are transformed into first world coordinates using the following formula model: Where i represents the i-th target feature point; This represents the kinematics of the robotic arm when it is in its first pose. When the calibration camera is mounted with the eye on the hand, the kinematics of the robotic arm here is the forward kinematics of the robotic arm. When the calibration camera is mounted with the eye outside the hand, the kinematics of the robotic arm here is the inverse kinematics of the robotic arm. For hand-eye calibration, K is the intrinsic parameter of the calibration camera; p 1,i Let be the first pixel coordinate of the i-th target feature point in the first environment image; Let be the first-world coordinates of the i-th target feature point. Similarly, the second pixel coordinates are transformed into second-world coordinates using the following formula: Similarly, i represents the i-th target feature point; This represents the kinematics of the robotic arm when it is in the second pose. When the calibration camera is mounted with the eye on the hand, the kinematics of the robotic arm here is the forward kinematics of the robotic arm. When the calibration camera is mounted with the eye outside the hand, the kinematics of the robotic arm here is the inverse kinematics of the robotic arm. For hand-eye calibration, K is the intrinsic parameter of the calibration camera; p 2,i The second pixel coordinate of the target feature point in the second environment image; Let be the second world coordinates of the i-th target feature point.

[0052] In the two formulas above, the kinematics of the robotic arm and the intrinsic parameters of the calibration camera are constant known quantities during the calibration process, and the hand-eye calibration matrix... The process of transforming the first pixel coordinates and the second pixel coordinates into first world coordinates and second world coordinates, which are the unknown quantities that need to be determined in this application, is equivalent to the process of representing the world coordinates of the target feature points with the help of the kinematics of the robotic arm, camera intrinsics, pixel coordinates of the target feature points, and the hand-eye calibration matrix to be determined.

[0053] In one possible implementation, the imaging projection error of hand-eye calibration is determined based on the first world coordinates and second world coordinates of all target feature points, including:

[0054] The distance between the first world coordinates and the second world coordinates of each target feature point is defined as the imaging projection error of the target feature point;

[0055] The imaging projection errors of all target feature points are summed to determine the imaging projection error of hand-eye calibration.

[0056] In principle, only one world coordinate exists for the same target feature point. Therefore, theoretically, the first world coordinate and the second world coordinate of the same target feature point are exactly the same. However, in actual measurement, considering different measurement methods, there are measurement errors. The world coordinates of the same target feature point obtained by different methods may deviate. In this application, the distance between the first world coordinate and the second world coordinate of each target feature point is determined as the imaging projection error of the target feature point. Specifically, the imaging projection error of the target feature point is determined by the following formula: e i This represents the imaging projection error of target feature point i.

[0057] The imaging projection errors of all target feature points are summed to determine the hand-eye calibration imaging projection error. For ease of subsequent calculations, the square of the distance between the first-world coordinates and the second-world coordinates can be used as the imaging projection error of the target feature points. The hand-eye calibration imaging projection error can be determined using the following formula: Q represents the imaging projection error determined by hand-eye calibration, and n represents the total number of target feature points. For e i The transpose matrix is ​​obtained. By using pixel coordinate reprojection error as an optimization, the solution process is simplified and the calibration accuracy of the vision system is improved.

[0058] In one possible implementation, the imaging projection error of hand-eye calibration is processed using methods based on Lie groups and Lie algebras to obtain the hand-eye calibration matrix, including:

[0059] Based on the exponential mapping transformation of Lie groups and Lie algebras, the imaging projection error of hand-eye calibration is transformed into Lie algebra form;

[0060] Determine the minimum value of the imaging projection error converted to Lie algebraic form;

[0061] The optimal value of the Lie algebra of the hand-eye calibration matrix is ​​determined based on the minimum value;

[0062] Based on the optimal value, the hand-eye calibration matrix for hand-eye calibration is determined according to the logarithmic mapping transformation of Lie groups and Lie algebras.

[0063] Because the imaging projection error of hand-eye calibration represents the sum of the errors in the first-world and second-world coordinates of all target feature points, to make the calibration more accurate, the imaging projection error of hand-eye calibration needs to be minimized, which means minimizing the aforementioned Q. To simplify calculations, define Hand-eye calibration matrix Lie algebras, where, By employing the exponential mapping transformation of Lie groups and Lie algebras, the aforementioned hand-eye calibration imaging projection error is transformed into...

[0064] Based on the properties of Lie algebras, the above formula for the imaging projection error of hand-eye calibration can be transformed into an unconstrained optimization problem. The minimum value of the imaging projection error Q of hand-eye calibration is at the extreme point of the derivative. The derivative of the Lie algebra can be calculated using the following perturbation model BCH (Baker-Campbell-Hausdorff) formula: The derivative is calculated as follows: Based on the aforementioned derivatives, partial derivatives are determined. Since the partial derivatives are zero, the minimum value of the imaging projection error in hand-eye calibration can be determined. Based on this minimum value, the optimal value of the Lie algebra of the hand-eye calibration matrix can be determined. When the Lie algebra of the hand-eye calibration matrix is ​​optimal, the imaging projection error in hand-eye calibration is minimized. Then, based on the logarithmic mapping relationship between Lie groups and Lie algebras, and according to the optimal value of the Lie algebra of the hand-eye calibration matrix, the rotation matrix and translation vector of the hand-eye calibration matrix can be determined, specifically: through... Ra = a determines the rotation matrix of the hand-eye calibration matrix, and t = Jρ determines the translation vector of the hand-eye calibration matrix. Finally, hand-eye calibration is performed based on this rotation matrix and translation vector. Representing the hand-eye calibration matrix using Lie groups and Lie algebras reduces the computational complexity of the solution.

[0065] Figure 3 A schematic diagram of a hand-eye calibration device based on environmental information according to an embodiment of the present disclosure is shown.

[0066] See Figure 3 According to a second aspect of the present disclosure, a hand-eye calibration device based on environmental information is provided. The device includes: an acquisition module 301, configured to acquire a first environmental image of the robotic arm in a first pose and a second environmental image of the robotic arm in a second pose via a calibration camera in the robotic arm system; a first determination module 302, configured to determine the first pixel coordinates and the second pixel coordinates corresponding to target feature points in the first environmental image and the second environmental image; a conversion module 303, configured to convert the first pixel coordinates into first world coordinates and the second pixel coordinates into second world coordinates; a second determination module 304, configured to determine the imaging projection error of hand-eye calibration based on the first world coordinates and the second world coordinates of all target feature points; and a processing module 305, configured to process the imaging projection error of hand-eye calibration according to the methods of Lie groups and Lie algebras to obtain a hand-eye calibration matrix, and perform hand-eye calibration according to the hand-eye calibration matrix.

[0067] In one embodiment, the first determining module 302 includes: an extraction submodule 3021, used to extract initial feature points from a first environmental image and a second environmental image respectively; a matching submodule 3022, used to match the initial feature points and determine the same initial feature points in the first environmental image and the second environmental image as target feature points; a first determining submodule 3023, used to determine the first pixel coordinates corresponding to each target feature point in the first environmental image; and the first determining submodule 3023 is further used to determine the second pixel coordinates corresponding to each target feature point in the second environmental image.

[0068] In one embodiment, the conversion module 303 is specifically used to convert the first pixel coordinates into first world coordinates based on the first pixel coordinates, the intrinsic parameters of the calibration camera, and the kinematics of the robotic arm; and to convert the second pixel coordinates into second world coordinates based on the second pixel coordinates, the intrinsic parameters of the calibration camera, and the kinematics of the robotic arm. The calibration camera in the robotic arm system has two mounting methods: eye-on-hand and eye-out-of-hand. When the calibration camera adopts the eye-on-hand mounting method, the robotic arm kinematics is the forward kinematics of the robotic arm; when the calibration camera adopts the eye-out-of-hand mounting method, the robotic arm kinematics is the inverse kinematics of the robotic arm.

[0069] In one embodiment, the second determining module 304 is specifically used to determine the distance between the first world coordinate and the second world coordinate of each target feature point as the imaging projection error of the target feature point; and to add up the imaging projection errors of all target feature points to determine the imaging projection error of hand-eye calibration.

[0070] In one embodiment, the processing module 305 includes: a transformation submodule 3051, used to transform the imaging projection error of hand-eye calibration into Lie algebra form according to the exponential mapping transformation of Lie groups and Lie algebras; a second determination submodule 3052, used to determine the minimum value of the imaging projection error transformed into Lie algebra form; the second determination submodule 3052 is also used to determine the optimal value of the Lie algebra of the hand-eye calibration matrix according to the minimum value; the second determination submodule 3052 is also used to determine the hand-eye calibration matrix of hand-eye calibration according to the optimal value based on the logarithmic mapping transformation of Lie groups and Lie algebras.

[0071] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0072] Figure 4A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0073] like Figure 4 As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 408 into random access memory (RAM) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0074] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0075] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as a hand-eye calibration method based on environmental information. For example, in some embodiments, a hand-eye calibration method based on environmental information can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of a hand-eye calibration method based on environmental information described above can be performed. Alternatively, in other embodiments, the computing unit 401 may be configured by any other suitable means (e.g., by means of firmware) to perform a hand-eye calibration method based on environmental information.

[0076] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0077] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0078] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0079] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0080] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0081] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0082] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0083] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0084] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A hand-eye calibration method based on environmental information, characterized in that, The method includes: The calibration camera in the robotic arm system is used to acquire the first environmental image when the robotic arm is in the first pose and the second environmental image when it is in the second pose. Determine the first pixel coordinates and the second pixel coordinates corresponding to the target feature points in the first environmental image and the second environmental image; The first pixel coordinates are converted into first world coordinates, and the second pixel coordinates are converted into second world coordinates; The imaging projection error of hand-eye calibration is determined based on the first-world and second-world coordinates of all target feature points; The imaging projection error of the hand-eye calibration is processed according to the method of Lie group and Lie algebra to obtain the hand-eye calibration matrix, and hand-eye calibration is performed according to the hand-eye calibration matrix. Determining the first pixel coordinates and second pixel coordinates corresponding to the target feature points in the first environmental image and the second environmental image includes: extracting initial feature points from the first environmental image and the second environmental image respectively; matching the initial feature points to determine the same initial feature points in the first environmental image and the second environmental image as target feature points; determining the first pixel coordinates corresponding to each target feature point in the first environmental image; and determining the second pixel coordinates corresponding to each target feature point in the second environmental image.

2. The method according to claim 1, characterized in that, The step of converting the first pixel coordinates into first world coordinates and the second pixel coordinates into second world coordinates includes: The first pixel coordinates are converted into first world coordinates based on the first pixel coordinates, the intrinsic parameters of the calibration camera, and the kinematics of the robotic arm. The second pixel coordinates are converted into second world coordinates based on the second pixel coordinates, the intrinsic parameters of the calibration camera, and the kinematics of the robotic arm; The calibration camera in the robotic arm system has two mounting methods: eye-on-hand and eye-out-of-hand. When the calibration camera is mounted with the eye on the hand, the kinematics of the robotic arm is the forward kinematics. When the calibration camera is mounted with the eye-out-of-hand, the kinematics of the robotic arm is the inverse kinematics.

3. The method according to claim 1, characterized in that, The determination of the imaging projection error for hand-eye calibration based on the first and second world coordinates of all target feature points includes: The distance between the first world coordinates and the second world coordinates of each target feature point is determined as the imaging projection error of the target feature point; The imaging projection errors of all target feature points are summed to determine the imaging projection error of the hand-eye calibration.

4. The method according to claim 1, characterized in that, The method of processing the imaging projection error of the hand-eye calibration based on Lie groups and Lie algebras to obtain the hand-eye calibration matrix includes: Based on the exponential mapping transformation of Lie groups and Lie algebras, the imaging projection error of hand-eye calibration is transformed into Lie algebra form; Determine the minimum value of the imaging projection error converted to Lie algebraic form; The optimal value of the Lie algebra of the hand-eye calibration matrix is ​​determined based on the minimum value. Based on the optimal value, the hand-eye calibration matrix for hand-eye calibration is determined according to the logarithmic mapping transformation of Lie groups and Lie algebras.

5. A hand-eye calibration device based on environmental information, characterized in that, The device includes: The acquisition module is used to acquire, through the calibration camera in the robotic arm system, a first environmental image when the robotic arm is in the first pose and a second environmental image when it is in the second pose. The first determining module is used to determine the first pixel coordinates and the second pixel coordinates corresponding to the target feature points in the first environmental image and the second environmental image; The conversion module is used to convert the first pixel coordinates into first world coordinates and the second pixel coordinates into second world coordinates; The second determining module is used to determine the imaging projection error of hand-eye calibration based on the first world coordinates and the second world coordinates of all target feature points. The processing module is used to process the imaging projection error of the hand-eye calibration according to the Lie group and Lie algebra method, to obtain the hand-eye calibration matrix, and to perform hand-eye calibration according to the hand-eye calibration matrix; The first determining module includes: an extraction submodule, configured to extract initial feature points from the first environmental image and the second environmental image respectively; a matching submodule, configured to match the initial feature points and determine the same initial feature points in the first environmental image and the second environmental image as target feature points; a first determining submodule, configured to determine the first pixel coordinates corresponding to each target feature point in the first environmental image; and the first determining submodule is further configured to determine the second pixel coordinates corresponding to each target feature point in the second environmental image.

6. The apparatus according to claim 5, characterized in that, The conversion module is specifically used for, The first pixel coordinates are converted into first world coordinates based on the first pixel coordinates, the intrinsic parameters of the calibration camera, and the kinematics of the robotic arm. The second pixel coordinates are converted into second world coordinates based on the second pixel coordinates, the intrinsic parameters of the calibration camera, and the kinematics of the robotic arm; The calibration camera in the robotic arm system has two mounting methods: eye-on-hand and eye-out-of-hand. When the calibration camera is mounted with the eye on the hand, the kinematics of the robotic arm is the forward kinematics. When the calibration camera is mounted with the eye-out-of-hand, the kinematics of the robotic arm is the inverse kinematics.

7. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4.

Images