A method, device and control device for calibrating camera parameters

Abstract

The present application provides a method, apparatus and control device for calibrating camera parameters. The method includes: determining a plurality of initial expected poses of the robotic arm based on the initial camera parameters of the camera; for each initial expected pose, controlling the robotic arm to move the calibration board based on the initial expected pose, and collecting a first calibration image of the calibration board through the camera; determining candidate camera parameters of the camera based on the first calibration images corresponding to the plurality of initial expected poses; determining a plurality of target expected poses of the robotic arm based on the candidate camera parameters; wherein the plurality of target expected poses are used to determine target camera parameters and calibrate the target camera parameters. Through the technical solution of the present application, automatic calibration of camera parameters can be achieved.

Description

Technical Field

[0001] The present application relates to the technical field of machine vision, and in particular to a method and device for calibrating camera parameters and a control device. Background Art

[0002] Camera parameters may include camera internal parameters (i.e., intrinsic parameters), distortion coefficients, and camera external parameters (i.e., extrinsic parameters). The camera internal parameters are composed of (cx, cy, fx, fy), where (cx, cy) are the principal point coordinates, and (fx, fy) are the camera focal lengths. The camera internal parameters are parameters related to the characteristics of the camera itself. The camera external parameters are composed of (ω, δ, θ, Tx, Ty, Tz), where (ω, δ, θ) are the rotation parameters of three axes, and (Tx, Ty, Tz) are the translation parameters of three axes. The camera external parameters are used to represent the conversion relationship between the world coordinate system and the camera coordinate system. (ω, δ, θ) can also be referred to as (roll, pitch, yaw), where roll represents the roll angle, pitch represents the pitch angle, and yaw represents the yaw angle. Tx represents the horizontal parameter, i.e., the translation parameter value of the x-axis, Ty represents the vertical parameter, i.e., the vertical parameter value of the y-axis, and Tz represents the vertical parameter, i.e., the vertical parameter value of the z-axis.

[0003] Before the camera leaves the factory, it is necessary to calibrate the camera parameters for each camera. However, there is no reasonable calibration method in the related art for how to calibrate the camera parameters for each camera, such as calibrating the camera internal parameters, distortion coefficients, and camera external parameters, resulting in problems such as cumbersome calibration operations, long calibration time, and high calibration labor costs. Summary of the Invention

[0004] The present application provides a method for calibrating camera parameters. A calibration board is fixedly connected to the robotic arm of a robot, and the calibration board moves synchronously with the robotic arm. The method includes:

[0005] Determining a plurality of initial desired poses of the robotic arm based on the initial camera parameters of the camera;

[0006] For each initial desired pose, controlling the robotic arm to move the calibration board based on the initial desired pose, and collecting a first calibration image of the calibration board through the camera;

[0007] Determining candidate camera parameters of the camera based on the first calibration images corresponding to the plurality of initial desired poses;

[0008] Determining a plurality of target desired poses of the robotic arm based on the candidate camera parameters; wherein, the plurality of target desired poses are used to determine target camera parameters and calibrate the target camera parameters.

[0009] This application provides a calibration device for camera parameters. A calibration board is fixedly connected to the robotic arm of a robot, and the calibration board moves synchronously with the robotic arm. The device includes: an initial expected pose determination module, configured to determine a plurality of initial expected poses of the robotic arm based on the initial camera parameters of a camera; a first control module, configured to, for each initial expected pose, control the robotic arm to move the calibration board based on the initial expected pose, and collect a first calibration image of the calibration board through the camera; a target expected pose determination module, configured to determine candidate camera parameters of the camera based on the first calibration images corresponding to the plurality of initial expected poses; and determine a plurality of target expected poses of the robotic arm based on the candidate camera parameters; wherein the plurality of target expected poses are used to determine target camera parameters and calibrate the target camera parameters.

[0010] This application provides a control device, including: a transmitter, a receiver, and a processor; wherein:

[0011] The processor is configured to obtain the initial camera parameters of a camera, and determine a plurality of initial expected poses of the robotic arm of the robot based on the initial camera parameters of the camera; wherein a calibration board is fixedly connected to the robotic arm of the robot, and the calibration board moves synchronously with the robotic arm;

[0012] The transmitter is configured to, for each initial expected pose, send a first control instruction to the robot and send a second control instruction to the camera; the first control instruction is used to cause the robot to move the robotic arm based on the initial expected pose, and the calibration board moves synchronously based on the initial expected pose; the second control instruction is used to cause the camera to collect a first calibration image of the calibration board;

[0013] The receiver is configured to receive the first calibration image collected by the camera;

[0014] The processor is configured to determine candidate camera parameters of the camera based on the first calibration images corresponding to the plurality of initial expected poses; and determine a plurality of target expected poses of the robotic arm based on the candidate camera parameters; wherein the plurality of target expected poses are used to determine target camera parameters and calibrate the target camera parameters.

[0015] This application provides a control device, including: a processor and a machine-readable storage medium, the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is configured to execute the machine-executable instructions to implement the above-mentioned camera parameter calibration method.

[0016] This application provides a computer program product, the computer program product includes a computer program, and when the computer program is executed by a processor, it implements the above-mentioned camera parameter calibration method.

[0017] The present application provides a machine-readable storage medium storing machine-executable instructions executable by a processor. Among them, the processor is configured to execute the machine-executable instructions, and when executing the machine-executable instructions, implement the above-mentioned camera parameter calibration method.

[0018] As can be seen from the above technical solutions, in the embodiments of the present application, automatic calibration of camera parameters can be achieved. The calibration operation is simple, the calibration time is short, and the calibration labor cost is low. It can automatically generate the pose of each desired point of the robotic arm (i.e., multiple initial desired poses), without the need for manual testing and calibration of the desired point poses, reducing the testing and calibration of the desired point poses, improving automation, and reducing the interference of human factors. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flowchart of a method for calibrating camera parameters in an embodiment of the present application;

[0020] Figure 2 is a flowchart of a process for obtaining a target desired pose in an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of optical distortion and TV distortion in an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a camera coordinate system and a robotic arm coordinate system in an embodiment of the present application;

[0023] Figure 5 is a flowchart of a process for calibrating camera parameters in an embodiment of the present application;

[0024] Figure 6 is a schematic structural diagram of a device for calibrating camera parameters in an embodiment of the present application;

[0025] Figure 7 is a hardware structure diagram of a control device in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] In the embodiments of the present application, a method for calibrating camera parameters is proposed. This method can be applied to a control device, such as a personal computer, a laptop computer, a smart phone, etc. There is no limitation on the type of this control device, as long as it can control the robotic arm of the robot to move and control the camera to collect images. The calibration board is placed on the robotic arm of the robot and is fixedly connected to the robotic arm of the robot, that is, the calibration board moves synchronously with the robotic arm. Refer to Figure 1 As shown, for the flowchart of this method, this method may include:

[0027] Step 101: Determine multiple initial expected poses of the robotic arm based on the initial camera parameters of the camera.

[0028] Step 102: For each initial expected pose, control the robotic arm to move the calibration plate based on this initial expected pose, and collect the first calibration image of the calibration plate through the camera.

[0029] Step 103: Determine the candidate camera parameters of the camera based on the first calibration images corresponding to the multiple initial expected poses (i.e., the multiple first calibration images corresponding to the multiple initial expected poses).

[0030] Step 104: Determine multiple target expected poses of the robotic arm based on the candidate camera parameters; among them, the multiple target expected poses are used to determine and calibrate the target camera parameters.

[0031] Exemplarily, before determining the multiple initial expected poses of the robotic arm based on the initial camera parameters of the camera, the acquisition process of the initial camera parameters of the camera may include, but is not limited to: determining the initial distortion parameters of the camera based on the horizontal field of view angle, vertical field of view angle, optical distortion coefficient, and TV distortion coefficient of the camera; determining the initial focal length parameters of the camera based on the focal length of the camera lens and the pixel size of the camera sensor; determining the initial principal point coordinate parameters of the camera based on the imaging resolution of the camera; obtaining the initial camera parameters; the initial camera parameters include the initial distortion parameters, the initial focal length parameters, and the initial principal point coordinate parameters.

[0032] Exemplarily, determining the multiple initial expected poses of the robotic arm based on the initial camera parameters of the camera may include, but is not limited to: obtaining multiple configured expected data, and each expected data may include an expected distance and an expected pixel coordinate; where the expected distance may represent the distance between the calibration plate and the camera, and the expected pixel coordinate may represent the pixel coordinate of a specific position of the calibration plate on the image.

[0033] For each expected data, perform a distortion removal operation on the expected pixel coordinate in the expected data based on the initial camera parameters to obtain the undistorted pixel coordinate; determine the first expected position of the calibration plate in the camera coordinate system based on the expected distance, the undistorted pixel coordinate, and the initial camera parameters in the expected data; determine the initial expected pose corresponding to the expected data based on the first expected position.

[0034] Exemplarily, determining the initial desired pose corresponding to the desired data based on the first desired position may include, but is not limited to: determining the second desired position of the calibration board in the robotic arm coordinate system based on the first desired position, the initial rotation matrix, and the initial translation matrix between the camera coordinate system and the robotic arm coordinate system; determining the second pose based on the first pose of the robotic arm in the robotic arm coordinate system and the second desired position, where the first pose is the pose of the robotic arm when the calibration board is in the initial state; adjusting the orientation in the second pose based on a randomly obtained rotation angle to obtain the initial desired pose of the robotic arm, and the second pose includes a position and an orientation.

[0035] Exemplarily, determining multiple target desired poses of the robotic arm based on candidate camera parameters may include, but is not limited to: determining the pixel distance between the desired pixel coordinates in the desired data and the actual pixel coordinates of a specific position on the first calibration image based on the initial desired pose corresponding to each desired data; determining a position correction amount based on the candidate camera parameters and the pixel distance; correcting the initial desired pose based on the position correction amount to obtain the target desired pose corresponding to the desired data. Alternatively, for each desired data, performing a distortion removal operation on the desired pixel coordinates in the desired data based on the candidate camera parameters to obtain undistorted pixel coordinates; determining the desired position of the calibration board in the camera coordinate system based on the desired distance, the undistorted pixel coordinates, and the candidate camera parameters in the desired data, and determining the target desired pose corresponding to the desired data based on the desired position. Alternatively, based on the initial desired pose corresponding to each desired data, correcting the initial desired pose with the obtained position correction amount to obtain the first desired pose; determining the second desired pose based on the obtained desired position; projecting the calibration board in the first desired pose onto the image of the camera to obtain the first actual pixel coordinates of the specific position on the image, and projecting the calibration board in the second desired pose onto the image of the camera to obtain the second actual pixel coordinates of the specific position on the image. If the first pixel distance between the first actual pixel coordinates and the desired pixel coordinates in the desired data is less than the second pixel distance between the second actual pixel coordinates and the desired pixel coordinates, the first desired pose may be determined as the target desired pose corresponding to the desired data; if the second pixel distance is less than the first pixel distance, the second desired pose may be determined as the target desired pose corresponding to the desired data.

[0036] Exemplarily, determining a position correction amount based on the candidate camera parameters and the pixel distance may include, but is not limited to: converting the pixel distance to a physical distance based on the candidate camera parameters and the desired distance in the desired data; converting the physical distance to a position correction amount in the robotic arm coordinate system based on the target rotation matrix between the camera coordinate system and the robotic arm coordinate system.

[0037] Exemplarily, determining the target desired pose corresponding to the desired data based on the desired position may include, but is not limited to: determining the desired position of the calibration board in the robotic arm coordinate system based on the desired position, the target rotation matrix, and the target translation matrix between the camera coordinate system and the robotic arm coordinate system, and determining the target desired pose based on the desired position of the calibration board in the robotic arm coordinate system.

[0038] Exemplarily, projecting the calibration board in the first desired pose onto the image of the camera to obtain the first actual pixel coordinates of a specific position on the image may include, but is not limited to: controlling the robotic arm to move the calibration board based on the first desired pose to determine the first physical coordinates of the specific position of the calibration board in the robotic arm coordinate system; converting the first physical coordinates into the second physical coordinates of the specific position in the camera coordinate system based on the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system; and determining the first actual pixel coordinates of the specific position on the image based on the second physical coordinates.

[0039] Exemplarily, projecting the calibration board in the second desired pose onto the image of the camera to obtain the second actual pixel coordinates of a specific position on the image may include, but is not limited to: controlling the robotic arm to move the calibration board based on the second desired pose to determine the third physical coordinates of the specific position of the calibration board in the robotic arm coordinate system; converting the third physical coordinates into the fourth physical coordinates of the specific position in the camera coordinate system based on the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system, and determining the second actual pixel coordinates of the specific position on the image based on the fourth physical coordinates.

[0040] Exemplarily, the process of obtaining the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system may include, but is not limited to: determining the third transformation matrix between the camera coordinate system and the robotic arm coordinate system based on the first transformation matrix between the robotic arm coordinate system and the end - effector coordinate system and the second transformation matrix between the calibration board coordinate system and the camera coordinate system, where the third transformation matrix may include the target rotation matrix and the target translation matrix. For example, when obtaining the initial desired pose of the robotic arm, determining the first transformation matrix based on the initial desired pose; and when obtaining the first calibration image of the calibration board, determining the second transformation matrix based on the physical coordinates in the calibration board coordinate system and the pixel coordinates in the first calibration image.

[0041] Exemplarily, the camera in the above - mentioned embodiment may be a reference camera, and multiple target desired poses are used to determine the target camera parameters of the reference camera. Alternatively, multiple target desired poses may be used to determine the target camera parameters of multiple cameras to be calibrated, and the models of the multiple cameras to be calibrated are the same as the model of the reference camera.

[0042] On this basis, when determining the target camera parameters of the reference camera or the camera to be calibrated, for each target desired pose, the robotic arm is controlled to move the calibration board based on the target desired pose, and the second calibration image of the calibration board is collected through the reference camera or the camera to be calibrated; the target camera parameters of the reference camera or the camera to be calibrated are determined based on the second calibration images corresponding to multiple target desired poses.

[0043] As can be seen from the above technical solutions, in the embodiments of the present application, automatic calibration of camera parameters can be achieved, the calibration operation is simple, the calibration time is short, and the calibration labor cost is low. It can automatically generate the desired pose of each point of the robotic arm (i.e., multiple initial desired poses), without manual testing and calibration of the desired pose of the point, reducing the testing and calibration of the desired pose of the point, improving automation, and reducing the interference of human factors.

[0044] The above technical solutions of the embodiments of the present application will be described below in conjunction with specific application scenarios.

[0045] The embodiments of the present application propose a method for calibrating camera parameters, which is used to calibrate the camera parameters of the camera of the device to be calibrated, such as calibrating the camera internal parameters, distortion coefficients, and camera external parameters. The camera internal parameters are composed of (cx, cy, fx, fy), where (cx, cy) are the principal point coordinates, (fx, fy) are the camera focal lengths, the camera external parameters are composed of (roll, pitch, yaw, Tx, Ty, Tz), (roll, pitch, yaw) are the rotation parameters of the three axes, (Tx, Ty, Tz) are the translation parameters of the three axes, roll represents the roll angle, pitch represents the pitch angle, and yaw represents the yaw angle. Tx represents the horizontal parameter, Ty represents the vertical parameter, and Tz represents the vertical parameter.

[0046] The device to be calibrated is a device with at least one camera, and the type of this device to be calibrated is not limited, as long as it has at least one camera. For example, the device to be calibrated may include an RGBD camera, or an RGB camera, or a depth camera. The RGB camera can be one camera. The depth camera may include a left-eye camera and a right-eye camera. The RGBD camera may include an RGB camera and a depth camera, and the depth camera may be a left-eye camera and a right-eye camera, or the depth camera may be one camera. For the convenience of description, taking the device to be calibrated including an RGBD camera as an example, that is, using the camera parameter calibration method of this embodiment to calibrate the camera parameters of the RGB camera in the RGBD camera and calibrate the camera parameters of the depth camera in the RGBD camera.

[0047] In this embodiment, it involves the process of obtaining the target desired pose and the process of calibrating the camera parameters. In the process of obtaining the target desired pose, multiple target desired poses corresponding to the reference camera can be obtained. In the process of calibrating the camera parameters, the camera parameters of each camera to be calibrated can be calibrated based on the target desired pose.

[0048] For example, for multiple cameras to be calibrated of the same model, one camera to be calibrated can be selected from the multiple cameras to be calibrated as the reference camera, and multiple target desired poses corresponding to the reference camera are obtained and stored. Based on the multiple target desired poses, the camera parameters of each camera to be calibrated (including the reference camera and other cameras to be calibrated of the same model) can be calibrated.

[0049] For example, when calibrating the RGB camera among multiple RGBD cameras, multiple RGB cameras can be regarded as multiple cameras to be calibrated, and one RGB camera is selected from the multiple RGB cameras as the reference camera. When calibrating the depth camera among multiple RGBD cameras, multiple depth cameras can be regarded as multiple cameras to be calibrated, and one depth camera is selected from the multiple depth cameras as the reference camera.

[0050] In this embodiment, in the process of obtaining the target desired pose, the reference camera can be placed at a specified position, that is, the reference camera is fixed at this specified position, and images within the field of view are collected through the reference camera. For example, a certain RGBD camera can be placed at the specified position. In this way, the RGB camera in this RGBD camera serves as the reference camera, and / or the depth camera in this RGBD camera serves as the reference camera.

[0051] In the process of calibrating the camera parameters, multiple cameras to be calibrated can be placed at the specified position in sequence, that is, the cameras to be calibrated are fixed at this specified position, and images within the field of view are collected through the cameras to be calibrated. For example, RGBD Camera 1 can be placed at the specified position. The RGB camera in RGBD Camera 1 serves as the camera to be calibrated, and / or the depth camera in RGBD Camera 1 serves as the camera to be calibrated. After the parameter calibration of RGBD Camera 1 is completed, RGBD Camera 2 can be placed at the specified position, and so on.

[0052] In this embodiment, in the process of obtaining the target desired pose and the process of calibrating the camera parameters, the calibration board can be placed on the robotic arm of the robot, and the calibration board is fixedly connected to the robotic arm of the robot (it can be considered that the calibration board and the robotic arm are in the same coordinate system), that is, the calibration board moves synchronously with the robotic arm.

[0053] Regarding the calibration board, the calibration board can be a calibration board with a checkerboard pattern, or a calibration board with a target pattern. There is no limitation on the calibration pattern of this calibration board, and it can be a calibration board with any pattern.

[0054] Exemplarily, for the process of obtaining the target desired pose, one of the cameras to be calibrated can be selected as the reference camera from multiple cameras to be calibrated, and multiple target desired poses corresponding to the reference camera can be obtained. Refer to Figure 2 As shown, it is a schematic flowchart of the process of obtaining the target desired pose, and this process may include:

[0055] Step 201, obtain the initial camera parameters of the reference camera.

[0056] Exemplarily, the initial camera parameters can be estimated camera parameters, and there may be certain errors. There is no limitation on the process of obtaining such initial camera parameters, as long as the initial camera parameters can be obtained. For example, the initial camera parameters can include initial distortion parameters, initial focal length parameters, and initial principal point coordinate parameters.

[0057] In a possible implementation manner, the following steps can be adopted to obtain the initial camera parameters.

[0058] Step S11, based on the horizontal field of view angle, vertical field of view angle, optical distortion coefficient, and TV (Transverse Vertical) distortion coefficient of the reference camera, determine the initial distortion parameters of the reference camera.

[0059] For example, the horizontal field of view angle, vertical field of view angle, optical distortion coefficient, and TV distortion coefficient are all inherent parameters of the reference camera and are all known parameters. On this basis, the initial distortion parameters can be determined based on the horizontal field of view angle, vertical field of view angle, optical distortion coefficient, and TV distortion coefficient of the reference camera.

[0060] Exemplarily, the initial distortion parameters can include radial distortion coefficients and tangential distortion coefficients . If the reference camera provides the ideal values and distorted values (hereinafter simply referred to as the distortion table) of some pixel points within the field of view, the initial distortion parameters of the reference camera can be solved based on the distortion model.

[0061] For example, the distortion model can be as shown in formula (1), and the distortion table includes the corresponding relationship between the distorted value and the ideal value . represents the coordinates after distortion of the ideal value , that is to say, due to the existence of the initial distortion parameters, the ideal value is distorted into .

[0062] Formula (1)

[0063] In formula (1), , obviously, after substituting multiple groups of data into formula (1), each group of data may include distortion values and ideal values , and if multiple groups of data can be obtained from the distortion table, the radial distortion coefficient and the tangential distortion coefficient can be solved.

[0064] Exemplarily, if the distortion table of some pixel points within the field of view is not provided by the reference camera, the initial distortion parameters of the reference camera can be determined based on the horizontal field of view angle, vertical field of view angle, optical distortion coefficient, and TV distortion coefficient of the reference camera, which can also be referred to as fitting and estimating the initial distortion parameters of the reference camera.

[0065] See Figure 3 shown, which is a schematic diagram of optical distortion and TV distortion. In Figure 3 , Y 0 / Y represents the optical distortion coefficient, and the optical distortion coefficient can be denoted as , h 0 / h represents the TV distortion coefficient, and the TV distortion coefficient can be denoted as . In addition, the horizontal field of view angle can be denoted as , the vertical field of view angle can be denoted as , and let , . On this basis, is the coordinate of the ideal lower right corner edge point of the image, that is, the lower right corner edge point without distortion, and is . is the coordinate of the ideal center point of the lower edge of the image, that is, the coordinate of the ideal center point of the lower edge without distortion, and is . Due to the existence of the initial distortion parameters, the coordinate of the lower right corner edge point is distorted into the edge point coordinate , and is , and the coordinate of the center point of the lower edge is distorted into the edge point coordinate , and is .

[0066] Take the coordinate as the ideal value , take the coordinate as the distortion value and substitute it into formula (1), take the coordinate as the ideal value , take coordinates as the distortion value Substitute into formula (1). Based on Figure 3 the geometric relationship shown, the coordinates and the coordinates and the optical distortion coefficient (Y 0 / Y) and the TV distortion coefficient (h 0 / h), update the coordinates and the coordinates to the optical distortion coefficient and the TV distortion coefficient , and substitute the above relationship into formula (1) as well. Assume that the radial distortion coefficient and the tangential distortion coefficient are 0, and substitute this relationship into formula (1) as well. Thus, the following formula (2) can be obtained.

[0067] Formula (2)

[0068] In formula (2), the radial distortion coefficient is an unknown parameter, the optical distortion coefficient and the TV distortion coefficient are known parameters, the horizontal field of view angle and the vertical field of view angle are known parameters. Therefore, and are also known parameters. Based on this, the radial distortion coefficient can be approximately solved. In addition, the radial distortion coefficient and the tangential distortion coefficient are 0. Therefore, the initial distortion parameters can be obtained, and the initial distortion parameters include the radial distortion coefficient and the tangential distortion coefficient .

[0069] Step S12: Based on the lens focal length of the reference camera and the pixel size of the sensor of the reference camera, determine the initial focal length parameters of the reference camera. For example, the initial focal length parameters can include the camera focal lengths (fx, fy).

[0070] For example, both the focal length of the reference camera's lens and the pixel size of the reference camera's sensor (i.e., the sensor size) are inherent parameters of the reference camera and are known parameters. Based on this, the initial focal length parameters can be determined based on the focal length of the lens and the pixel size of the sensor (such as the pixel size in the x direction and the pixel size in the y direction). For example, fx is the focal length of the lens divided by the pixel size in the x direction of the sensor, and fy is the focal length of the lens divided by the pixel size in the y direction of the sensor.

[0071] Step S13: Determine the initial principal point coordinate parameters of the reference camera based on the imaging resolution of the reference camera (i.e., the sensor resolution). For example, the initial principal point coordinate parameters can include the principal point coordinates (cx, cy).

[0072] For example, the imaging resolution of the reference camera is an inherent parameter of the reference camera and is a known parameter. Based on this, the initial principal point coordinate parameters can be determined based on the imaging resolution. For example, cx is the image width resolution in the imaging resolution divided by 2, and cy is the image height resolution in the imaging resolution divided by 2.

[0073] Step S14: Obtain the initial camera parameters of the reference camera, and the initial camera parameters can include initial distortion parameters , initial focal length parameters (fx, fy), and initial principal point coordinate parameters (cx, cy).

[0074] Thus, step 201 is completed, and the initial camera parameters of the reference camera are obtained.

[0075] Step 202: Determine multiple initial desired poses of the robotic arm based on the initial camera parameters of the reference camera.

[0076] Exemplarily, the initial desired poses of the robotic arm can be determined based on the initial camera parameters of the reference camera and the desired data. The initial desired poses can be the point desired poses of the robotic arm. For example, multiple desired data can be obtained, and each desired data can correspond to an initial desired pose, so as to obtain multiple initial desired poses of the robotic arm. Subsequently, taking the acquisition of one initial desired pose as an example for illustration.

[0077] In a possible implementation manner, the following steps can be adopted to obtain multiple initial desired poses.

[0078] Step S21: Obtain multiple configured expected data. Each expected data can include an expected distance and expected pixel coordinates. Each expected data is configured by the user according to their own needs, and there is no limitation in this regard. For example, an expected distance that is relatively far, an expected distance that is moderate, or an expected distance that is relatively close can be configured. For example, the expected pixel coordinates can be the pixel coordinates at the center of the image, the expected pixel coordinates can be the pixel coordinates at the upper left corner of the image, the expected pixel coordinates can be the pixel coordinates at the upper right corner of the image, etc.

[0079] Exemplarily, each expected data can correspond to an initial expected pose. Subsequently, take the processing process for one expected data as an example. This expected data can include an expected distance, and the expected distance can represent the distance between the calibration board and the reference camera. The expected distance can be denoted as d. This expected data can include expected pixel coordinates, and the expected pixel coordinates can represent the pixel coordinates of a specific position of the calibration board on the image. The specific position of the calibration board can be any position of the calibration board, such as the center position of the calibration board, the upper left corner position of the calibration board, the upper right corner position of the calibration board, the lower left corner position of the calibration board, the lower right corner position of the calibration board, etc. There is no limitation on this specific position, and the expected pixel coordinates can be denoted as 。

[0080] Step S22: For each expected data, perform a distortion removal operation on the expected pixel coordinates in the expected data based on the initial camera parameters of the reference camera to obtain undistorted pixel coordinates.

[0081] Exemplarily, the expected pixel coordinates in the expected data can be ,and the undistorted pixel coordinates after the distortion removal operation are denoted as ,The initial camera parameters include initial distortion parameters 、initial focal length parameters (fx, fy) and initial principal point coordinate parameters (cx, cy). On this basis, formula (3) can be obtained based on the camera imaging model. In formula (3), z can be set to 1.

[0082] Formula (3)

[0083] ( )represents the expected pixel coordinates. The expected pixel coordinates ( )correspond to the distortion value ,that is, based on the distortion value determine the expected pixel coordinates ( ).( )represents the undistorted pixel coordinates, which are the pixel coordinates after removing the distortion. The undistorted pixel coordinates ( )correspond to the ideal value ,that is, based on the ideal value determine the undistorted pixel coordinates ( ).

[0084] In addition, the functional relationship between the distortion value and the ideal value can be determined based on the distortion model, and the distortion model can be seen in Formula (1). To sum up, by combining Formula (1) and Formula (3) and through iterative optimization, the distortion-free pixel coordinates ( ) can be calculated.

[0085] Step S23: Determine the first expected position of the calibration board in the camera coordinate system based on the expected distance d in the expected data, the distortion-free pixel coordinates ( ) and the initial camera parameters.

[0086] Exemplarily, the expected distance in the expected data can be d, the distortion-free pixel coordinates are ( ), and the initial camera parameters can include the initial focal length parameters (fx, fy) and the initial principal point coordinate parameters (cx, cy). On this basis, Formula (4) can be obtained based on the camera imaging model.

[0087] Formula (4)

[0088] In Formula (4), is the expected distance d, ( ), (fx, fy) and (cx, cy) are all known values. Therefore, the first expected position ( ) of the calibration board in the camera coordinate system can be obtained.

[0089] Step S24: Determine the second expected position of the calibration board in the robotic arm coordinate system based on the first expected position, the initial rotation matrix and the initial translation matrix between the camera coordinate system and the robotic arm coordinate system.

[0090] For example, the first expected position ( ) is the position of the calibration board in the camera coordinate system, and the initial rotation matrix and the initial translation matrix are the conversion relationships between the camera coordinate system and the robotic arm coordinate system. Therefore, the position in the camera coordinate system can be converted into the position in the robotic arm coordinate system based on the initial rotation matrix and the initial translation matrix, that is, the second expected position ( ) of the calibration board in the robotic arm coordinate system.

[0091] For example, the first expected position ( ) of the calibration board in the camera coordinate system can be converted into the second expected position ( ) of the calibration board in the robotic arm coordinate system through the following Formula (5).

[0092] Formula (5)

[0093] In formula (5), represents the initial rotation matrix between the camera coordinate system and the robotic arm coordinate system, and represents the initial translation matrix between the camera coordinate system and the robotic arm coordinate system. Regarding the initial rotation matrix and the initial translation matrix between the camera coordinate system and the robotic arm coordinate system, they can be obtained through measurement or other means, and there is no restriction on this. The initial rotation matrix and the initial translation matrix can be rough values, and in the subsequent process, the initial rotation matrix and the initial translation matrix still need to be adjusted and optimized.

[0094] Step S25: Determine the second pose based on the first pose of the robotic arm in the robotic arm coordinate system and the second desired position of the calibration board in the robotic arm coordinate system. The first pose is the pose of the robotic arm when the calibration board is in the initial state, and the second pose is the intermediate pose to be optimized. Subsequently, the second pose needs to be optimized.

[0095] For example, when the calibration board is in the initial state, that is, when the calibration board has not been moved, the pose of the robotic arm in the current situation can be obtained, and this pose of the robotic arm is used as the first pose in the robotic arm coordinate system , the first pose can be denoted as . In the initial state, the first pose in the robotic arm coordinate system can be directly obtained .

[0096] Based on the first pose and the second desired position of the calibration board in the robotic arm coordinate system ( ), the desired second pose is obtained, that is , that is, replacing the position ( ) in the first pose with the second desired position ( ).

[0097] Step S26: The second pose can include a position ( ) and an attitude ( , , ). Based on the randomly obtained rotation angle, the attitude ( , , ) in the second pose is adjusted to obtain the initial desired pose of the robotic arm, and the initial desired pose can be denoted as .

[0098] For example, during the camera calibration process, it is required that the calibration board planes in each pose are not parallel. Therefore, for the attitude ( ) in the second pose , , ), increase the random rotation angle within a certain range ( ), to obtain the initial expected pose of the robotic arm . . ( ) is randomly obtained and can be within the preset range, and there is no restriction on this.

[0099] Based on each expected data, the above steps can be used to obtain the initial expected pose corresponding to the expected data, and then multiple initial expected poses of the robotic arm can be obtained, completing step 202.

[0100] Step 203: For each initial expected pose, based on the initial expected pose, control the robotic arm to move the calibration plate, and collect the first calibration image of the calibration plate through the reference camera.

[0101] For example, based on the initial expected pose 1, control the robotic arm to move the calibration plate, that is, the robotic arm moves to the initial expected pose 1, and the calibration plate moves synchronously with the robotic arm, and the positional relationship between the calibration plate and the robotic arm remains unchanged. After the robotic arm moves to the initial expected pose 1, collect the first calibration image a1 of the calibration plate through the reference camera. Similarly, after the robotic arm moves to the initial expected pose 2, collect the first calibration image a2 of the calibration plate through the reference camera, and so on, multiple first calibration images can be obtained. To sum up, by controlling the movement of the robotic arm, multiple first calibration images can be collected through the reference camera.

[0102] Step 204: Based on the first calibration images corresponding to multiple initial expected poses (multiple first calibration images), determine the candidate camera parameters of the reference camera, that is, calibrate the candidate camera parameters for the reference camera.

[0103] For example, for the key points on the calibration plate (such as one or more key points), the key point can be the black and white intersection point (for the calibration plate with a checkerboard pattern), the key point can be the center point of the target (for the calibration plate with a target pattern), and the key point can also be other position points, and there is no restriction on this.

[0104] For each key point, the physical coordinates of the key point in the world coordinate system can be determined, and the pixel coordinates corresponding to the key point in the first calibration image can be determined, so as to form a coordinate point pair for the key point, and the coordinate point pair can include the physical coordinates of the key point and the pixel coordinates of the key point.

[0105] For example, when the robotic arm moves to the initial desired pose 1, the physical coordinates of the key points on the calibration board in the world coordinate system can be determined, and the pixel coordinates corresponding to the key points in the first calibration image 1 can be determined, thereby forming a coordinate point pair corresponding to the key points. When the robotic arm moves to the initial desired pose 2, the physical coordinates of the key points on the calibration board in the world coordinate system can be determined, and the pixel coordinates corresponding to the key points in the first calibration image 2 can be determined, thereby forming a coordinate point pair corresponding to the key points, and so on.

[0106] In summary, multiple coordinate point pairs under multiple initial desired poses can be obtained. Each coordinate point pair includes physical coordinates and pixel coordinates, and then candidate camera parameters are determined based on the multiple coordinate point pairs.

[0107] For example, the relationship between the world coordinate system and the image coordinate system can be represented by the following function: Y = f(X) , f is related to camera parameters (such as camera internal parameters, distortion coefficients, and camera external parameters). Therefore, the physical coordinates in the coordinate point pair can be substituted into the above function as Y, and the pixel coordinates in the coordinate point pair can be substituted into the above function as X, thereby obtaining an equation related to camera parameters. After performing the above operations on multiple coordinate point pairs, multiple equations can be obtained. By solving these equations, candidate camera parameters can be obtained, and the candidate camera parameters can include camera internal parameters, distortion coefficients, and camera external parameters.

[0108] Regarding the calibration process of the candidate camera parameters, no limitation is made in this embodiment. The calibration of the candidate camera parameters can be completed through multiple coordinate point pairs. During the calibration process of the candidate camera parameters, if there are multiple reference cameras, each reference camera can be calibrated one by one, and then one reference camera is fixed, and the remaining reference cameras are jointly calibrated with this reference camera (i.e., calibrating the camera external parameters).

[0109] Step 205: Obtain a third transformation matrix between the camera coordinate system and the robotic arm coordinate system. The third transformation matrix includes a target rotation matrix and a target translation matrix between the camera coordinate system and the robotic arm coordinate system.

[0110] In step 202, the initial desired pose is determined using the initial rotation matrix and the initial translation matrix between the camera coordinate system and the robotic arm coordinate system. However, the initial rotation matrix and the initial translation matrix are rough values with certain errors and need to be corrected. Based on this, in step 205, the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system can be obtained, and the target rotation matrix and the target translation matrix are accurate values that can represent the accurate pose relationship between the camera coordinate system and the robotic arm coordinate system.

[0111] Exemplarily, refer to Figure 4As shown in the figure, it is a schematic diagram of the camera coordinate system and the robotic arm coordinate system. "base" can represent the robotic arm coordinate system (which can also be called the base coordinate system of the robot), "gripper" can represent the end coordinate system (i.e., the coordinate system at the end of the robotic arm, which is the coordinate system of the above-mentioned initial desired pose), "camera" can represent the camera coordinate system, and "target" can represent the calibration board coordinate system. The reference camera is fixedly installed in the world and remains stationary, while the calibration board is fixedly installed at the end of the robotic arm and moves with the movement of the robotic arm.

[0112] See Figure 4 as shown in represents the transformation matrix (such as the rotation matrix and the translation matrix) from the robotic arm coordinate system to the end coordinate system. represents the transformation matrix (such as the rotation matrix and the translation matrix) from the calibration board coordinate system to the end coordinate system. represents the transformation matrix (such as the rotation matrix and the translation matrix) from the calibration board coordinate system to the camera coordinate system. represents the transformation matrix (such as the rotation matrix and the translation matrix) from the camera coordinate system to the robotic arm coordinate system, that is is the third transformation matrix that finally needs to be calibrated.

[0113] Based on the relationships between the transformation matrices, when the robotic arm is at any position, the following relationship can be satisfied: . Since the calibration board is fixedly installed at the end of the robotic arm and moves with the movement of the robotic arm, therefore, no matter which position the robotic arm is in, the transformation matrix from the calibration board coordinate system to the end coordinate system remains unchanged, that is, , (1) represents the transformation matrices when the robotic arm is at position 1, and (2) represents the transformation matrices when the robotic arm is at position 2.

[0114] By transforming the above equation, the following equation can be obtained: , that is to say, by performing left multiplication and right multiplication simplification on the above equation respectively, the form of AX = XB is obtained. In the above equation, let , let , let .

[0115] To sum up, the matrix X can be calculated based on matrix A and matrix B, and the matrix X is the transformation matrix from the camera coordinate system to the robotic arm coordinate system. In addition, matrix A is the transformation matrix from the robotic arm coordinate system to the end coordinate system, and matrix B is the transformation matrix from the calibration board coordinate system to the camera coordinate system.

[0116] In summary, it can be seen that based on the first transformation matrix (i.e., matrix A) between the robotic arm coordinate system and the end - effector coordinate system, and the second transformation matrix (i.e., matrix B) between the calibration plate coordinate system and the camera coordinate system, the third transformation matrix (matrix X) between the camera coordinate system and the robotic arm coordinate system can be determined, and the third transformation matrix can include the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system.

[0117] Regarding the first transformation matrix between the robotic arm coordinate system and the end - effector coordinate system, when obtaining the initial desired pose of the robotic arm, since the initial desired pose represents the final pose of the robotic arm, that is, the robotic arm is in this initial desired pose, therefore, the first transformation matrix can be determined based on the initial desired pose.

[0118] For example, (1) in matrix A represents the first transformation matrix determined based on the initial desired pose 1, and (2) in matrix A represents the first transformation matrix determined based on the initial desired pose 2.

[0119] Regarding the second transformation matrix between the calibration plate coordinate system and the camera coordinate system, when obtaining the first calibration image of the calibration plate, the second transformation matrix is determined based on the physical coordinates in the calibration plate coordinate system and the pixel coordinates in the first calibration image. For example, the coordinate point pairs include the physical coordinates of the key points on the calibration plate in the calibration plate coordinate system and the pixel coordinates of the key points in the first calibration image. The second transformation matrix between the calibration plate coordinate system and the camera coordinate system can be calculated based on multiple coordinate point pairs, and this process is not restricted.

[0120] For example, the first calibration image a1 corresponds to the initial desired pose 1, and (1) in matrix B represents the second transformation matrix determined based on the first calibration image a1. The first calibration image a2 corresponds to the initial desired pose 2, and (2) in matrix B represents the second transformation matrix determined based on the first calibration image a2.

[0121] In summary, it can be seen that based on at least two sets of transformation matrices (such as the first transformation matrix and the second transformation matrix), the third transformation matrix between the camera coordinate system and the robotic arm coordinate system can be obtained. When there is more data of the transformation matrix, a more accurate third transformation matrix can be obtained. In this embodiment, two sets of transformation matrices are taken as an example.

[0122] Step 206: Determine multiple target desired poses of the robotic arm based on the candidate camera parameters of the reference camera.

[0123] Exemplarily, for multiple initial desired poses of the robotic arm, these initial desired poses are not accurate. The initial desired poses (i.e., the desired poses of each point) can also be adjusted to obtain the target desired poses corresponding to each initial desired pose. For example, based on the first calibration image under each initial desired pose, the candidate camera parameters of the reference camera, and the transformation matrix between the camera coordinate system and the robotic arm coordinate system, multiple initial desired poses can be adjusted to obtain multiple target desired poses (i.e., the final poses) of the robotic arm.

[0124] In one possible implementation, the following steps can be used to determine multiple target desired poses:

[0125] Step S31: For each desired data (for the convenience of description, one desired data is taken as an example hereinafter), determine the pixel distance between the desired pixel coordinates in the desired data and the actual pixel coordinates of a specific position of the calibration board on the first calibration image, that is, the pixel distance between the two pixel coordinates.

[0126] For example, the desired pixel coordinates represent the pixel coordinates of a specific position of the calibration board on the image and are the expected pixel coordinates of the specific position on the image. After obtaining the first calibration image (i.e., the first calibration image under the initial desired pose corresponding to this desired data), the actual pixel coordinates of the specific position of the calibration board on the first calibration image can be determined . Then, the pixel distance between the desired pixel coordinates and the actual pixel coordinates can be calculated . For example, the pixel distance can be determined by the following formula : .

[0127] Step S32: Based on the candidate camera parameters and the desired distance in this desired data, convert the pixel distance into a physical distance. For example, the candidate camera parameters can include candidate focal length parameters (fx, fy), and the desired distance in this desired data can be d. Combining the desired distance d and the candidate focal length parameters (fx, fy), the pixel distance can be converted into a physical distance , and the physical distance can represent the distance in the world coordinate system. For example, the pixel distance can be converted into a physical distance by the following formula: .

[0128] Step S33: Based on the target rotation matrix between the camera coordinate system and the robotic arm coordinate system, convert the physical distance into a position correction amount in the robotic arm coordinate system。

[0129] For example, the third transformation matrix between the camera coordinate system and the robotic arm coordinate system includes a target rotation matrix and a target translation matrix, and the physical distance can be the distance in the camera coordinate system, and the physical distance in the camera coordinate system can be converted into a position correction amount in the robotic arm coordinate system based on the target rotation matrix 。For example, the following formula is used to convert the physical distance 。 into a position correction amount : 。In the above formula, represents the target rotation matrix between the camera coordinate system and the robotic arm coordinate system.

[0130] Step S34: Correct the initial expected pose (i.e., the initial expected pose corresponding to the expected data) based on the position correction amount to obtain the target expected pose corresponding to the expected data.

[0131] For example, based on the position correction amount, the following formula can be used to determine the target expected pose: 。In the above formula, represents the target expected pose, represents the initial expected pose, represents the position correction amount.

[0132] For example, steps S31 - S33 can be used to determine the position correction amount 1 corresponding to the expected data 1, and based on the position correction amount 1, correct the initial expected pose corresponding to the expected data 1 to obtain the target expected pose corresponding to the expected data 1. Steps S31 - S33 can be used to determine the position correction amount 2 corresponding to the expected data 2, and based on the position correction amount 2, correct the initial expected pose corresponding to the expected data 2 to obtain the target expected pose corresponding to the expected data 2, and so on, so that the target expected pose corresponding to each expected data can be obtained, that is, multiple target expected poses of the robotic arm are obtained.

[0133] In a possible implementation manner, the following steps can be used to determine multiple target expected poses:

[0134] Step S41: For each expected data (subsequently taking one expected data as an example), perform a distortion removal operation on the expected pixel coordinates in the expected data based on the candidate camera parameters to obtain undistorted pixel coordinates.

[0135] Step S42: Based on the expected distance d in the expected data, the undistorted pixel coordinates, and the candidate camera parameters, determine the expected position of the calibration board in the camera coordinate system (denoted as expected position A).

[0136] Step S43: Based on the desired position A, the target rotation matrix, and the target translation matrix between the camera coordinate system and the robotic arm coordinate system, determine the desired position of the calibration board in the robotic arm coordinate system (denoted as the desired position B).

[0137] Step S44: Based on the first pose of the robotic arm in the robotic arm coordinate system and the desired position B of the calibration board in the robotic arm coordinate system, determine the third pose. The first pose is the pose of the robotic arm when the calibration board is in its initial state, and the third pose is the intermediate pose to be optimized. Subsequently, the third pose needs to be optimized.

[0138] Step S45: Adjust the pose in the third pose based on the randomly obtained rotation angle to obtain the target desired pose of the robotic arm, that is, obtain the target desired pose corresponding to this desired data.

[0139] Steps S41 - S45 can refer to Steps S22 - S26. The differences are as follows: Replace the initial camera parameters with the candidate camera parameters, replace the initial rotation matrix with the target rotation matrix, replace the initial translation matrix with the target translation matrix, and the other processing procedures are similar and will not be repeated here.

[0140] Exemplarily, for each desired data, Steps S41 - S45 can be used to obtain the target desired pose corresponding to this desired data, so that multiple target desired poses of the robotic arm can be obtained.

[0141] In a possible implementation manner, the following steps can be used to determine multiple target desired poses:

[0142] Step S51: For each desired data, determine the pixel distance between the desired pixel coordinates in this desired data and the actual pixel coordinates of a specific position of the calibration board on the first calibration image.

[0143] Step S52: Based on the candidate camera parameters and the desired distance in this desired data, convert this pixel distance (i.e., the distance between two pixel coordinates) into a physical distance.

[0144] Step S53: Based on the target rotation matrix between the camera coordinate system and the robotic arm coordinate system, convert this physical distance into a position correction amount in the robotic arm coordinate system.

[0145] Step S54: Based on this position correction amount, correct the initial desired pose corresponding to this desired data to obtain the first desired pose corresponding to this desired data.

[0146] Step S55: Perform a distortion removal operation on the desired pixel coordinates in this desired data based on the candidate camera parameters to obtain the undistorted pixel coordinates corresponding to this desired pixel coordinate.

[0147] Step S56: Determine the expected position A of the calibration board in the camera coordinate system based on the expected distance d in the expected data, the undistorted pixel coordinates, and the candidate camera parameters.

[0148] Step S57: Determine the expected position B of the calibration board in the robotic arm coordinate system based on the expected position A, the target rotation matrix, and the target translation matrix between the camera coordinate system and the robotic arm coordinate system.

[0149] Step S58: Determine the third pose of the robotic arm based on the first pose of the robotic arm in the robotic arm coordinate system and the expected position B of the calibration board in the robotic arm coordinate system.

[0150] Step S59: Adjust the pose in the third pose based on the randomly obtained rotation angle to obtain the second expected pose of the robotic arm, that is, obtain the second expected pose corresponding to the expected data.

[0151] In summary, the first expected pose and the second expected pose corresponding to each expected data can be obtained.

[0152] Step S60: Project the calibration board in the first expected pose onto the image of the reference camera (i.e., the first calibration image) to obtain the first actual pixel coordinates of the specific position of the calibration board on the image.

[0153] For example, control the robotic arm to move the calibration board based on the first expected pose, that is, control the robotic arm to move to the first expected pose. After the robotic arm moves to the first expected pose, determine the first physical coordinates of the specific position of the calibration board (such as the origin of the calibration board) in the robotic arm coordinate system .

[0154] Then, based on the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system, convert the first physical coordinates to the second physical coordinates of the specific position of the calibration board in the camera coordinate system . For example, the following formula can be used to convert the first physical coordinates to the second physical coordinates : .

[0155] Based on the second physical coordinates determine the first actual pixel coordinates of the specific position of the calibration board on the image , that is, the first actual pixel coordinates in the image coordinate system . For example, the following formula can be used to determine the first actual pixel coordinates : .(fx, fy) is the focal length parameter in the candidate camera parameters, and (cx, cy) is the principal point coordinate parameter in the candidate camera parameters.

[0156] Step S61: Project the calibration board in the second desired pose onto the image of the reference camera (i.e., the first calibration image) to obtain the second actual pixel coordinates of the specific position of the calibration board on the image.

[0157] For example, control the robotic arm to move the calibration board based on the second desired pose, that is, control the robotic arm to move to the second desired pose. After the robotic arm moves to the second desired pose, determine the third physical coordinates of the specific position (such as the origin) of the calibration board in the robotic arm coordinate system. Based on the target rotation matrix and the target translation matrix , convert the third physical coordinates to the fourth physical coordinates of the specific position of the calibration board in the camera coordinate system. Determine the second actual pixel coordinates of the specific position of the calibration board on the image based on the fourth physical coordinates, that is, the second actual pixel coordinates in the image coordinate system.

[0158] Step S62: Calculate the first pixel distance between the first actual pixel coordinates and the desired pixel coordinates in the desired data, and calculate the second pixel distance between the second actual pixel coordinates and the desired pixel coordinates.

[0159] Step S63: If the first pixel distance is less than the second pixel distance, determine the first desired pose as the target desired pose corresponding to the desired data. If the second pixel distance is less than the first pixel distance, determine the second desired pose as the target desired pose corresponding to the desired data. If the first pixel distance is equal to the second pixel distance, determine either the first desired pose or the second desired pose as the target desired pose corresponding to the desired data.

[0160] Exemplarily, for each desired data, steps S51 - S63 can be used to obtain the target desired pose corresponding to the desired data, so that multiple target desired poses of the robotic arm can be obtained.

[0161] So far, step 206 is completed, and multiple target desired poses of the robotic arm can be obtained. Obviously, after step 206, the process of obtaining the target desired pose is completed, and then the camera parameter calibration process is performed based on the multiple target desired poses. During the camera parameter calibration process, multiple cameras to be calibrated can be placed at the specified positions in sequence. For the currently placed camera to be calibrated, the camera parameters of the camera to be calibrated can be calibrated based on the multiple target desired poses, so that the camera parameters of multiple cameras to be calibrated can be calibrated.

[0162] Exemplarily, for the camera parameter calibration process, the camera parameters of each camera to be calibrated can be calibrated in sequence. Subsequently, the camera parameter calibration process of one camera to be calibrated will be described. Refer to Figure 5 As shown in

[0163] Step 501: For each target desired pose, control the robotic arm to move the calibration board based on the target desired pose, and collect the second calibration image of the calibration board through the camera to be calibrated.

[0164] For example, control the robotic arm to move the calibration board based on the target desired pose 1, that is, the robotic arm moves to the target desired pose 1, and the calibration board moves synchronously with the robotic arm, and the positional relationship between the calibration board and the robotic arm remains unchanged. After the robotic arm moves to the target desired pose 1, collect the second calibration image b1 of the calibration board through the camera to be calibrated. After the robotic arm moves to the target desired pose 2, collect the second calibration image b2 of the calibration board through the camera to be calibrated, and so on, multiple second calibration images can be obtained.

[0165] Step 502: Determine the target camera parameters of the camera to be calibrated based on the second calibration images corresponding to multiple target desired poses (multiple second calibration images), that is, calibrate the target camera parameters for the camera to be calibrated.

[0166] For example, for the key points on the calibration board, the key point can be the black and white intersection point, the key point can be the center point of the target, and the key point can also be other position points, which are not restricted here. For each key point, determine the physical coordinates of the key point in the world coordinate system, and determine the pixel coordinates corresponding to the key point in the second calibration image, so as to form a coordinate point pair for the key point. The coordinate point pair can include the physical coordinates of the key point and the pixel coordinates of the key point. In this way, multiple coordinate point pairs under multiple target desired poses can be obtained, and then the target camera parameters can be determined based on the multiple coordinate point pairs.

[0167] For example, the relationship between the world coordinate system and the image coordinate system can be expressed by the following function: Y = f(X) , f which is related to the camera parameters (such as camera internal parameters, distortion coefficients, and camera external parameters). By substituting the physical coordinates and pixel coordinates in the coordinate point pair into the above function, the target camera parameters can be solved, and the target camera parameters can include camera internal parameters, distortion coefficients, and camera external parameters. The calibration process is not restricted here.

[0168] As can be seen from the above technical solutions, in the embodiments of the present application, automatic calibration of camera parameters (such as the camera parameters of an RGBD camera) can be achieved. The calibration operation is simple, the calibration time is short, and the calibration labor cost is low. It can automatically generate the pose of each desired point of the robotic arm (i.e., multiple initial desired poses), without the need for manual experimentation and calibration of the pose of the desired point, reducing the experimentation and calibration of the pose of the desired point, improving automation, and automatically generating and verifying the pose of the robotic arm according to the expectation, reducing the interference of human factors. Based on the known camera lens parameters, desired data, and the position relationship between the camera and the robotic arm, the pose of each desired point can be accurately determined, and then the calibration of the camera can be completed, significantly improving convenience, accuracy, and consistency.

[0169] Based on the same inventive concept as the above method, an embodiment of the present application provides a calibration device for camera parameters. A calibration board is fixedly connected to the robotic arm of the robot, and the calibration board moves synchronously with the robotic arm. Refer to Figure 6 As shown in the figure, which is a schematic structural diagram of the device. The device includes: an initial desired pose determination module 61, configured to determine multiple initial desired poses of the robotic arm based on the initial camera parameters of the camera; a first control module 62, configured to, for each initial desired pose, control the robotic arm to move the calibration board based on the initial desired pose, and collect a first calibration image of the calibration board through the camera; a target desired pose determination module 63, configured to determine candidate camera parameters of the camera based on the first calibration images corresponding to the multiple initial desired poses; and determine multiple target desired poses of the robotic arm based on the candidate camera parameters; wherein the multiple target desired poses are used to determine and calibrate the target camera parameters.

[0170] Exemplarily, the initial desired pose determination module 61 is further configured to, before determining multiple initial desired poses of the robotic arm based on the initial camera parameters, determine initial distortion parameters of the camera based on the horizontal field of view angle, vertical field of view angle, optical distortion coefficient, and TV distortion coefficient of the camera; determine initial focal length parameters of the camera based on the focal length of the camera lens and the pixel size of the sensor of the camera; determine initial principal point coordinate parameters of the camera based on the imaging resolution of the camera; and obtain the initial camera parameters of the camera; wherein the initial camera parameters include the initial distortion parameters, the initial focal length parameters, and the initial principal point coordinate parameters.

[0171] Exemplarily, when the initial desired pose determination module 61 determines multiple initial desired poses of the robotic arm based on the initial camera parameters of the camera, it is specifically configured to: obtain multiple desired data, each desired data including a desired distance and a desired pixel coordinate; the desired distance represents the distance between the calibration board and the camera, and the desired pixel coordinate represents the pixel coordinate of a specific position of the calibration board on the image; for each desired data, perform a distortion removal operation on the desired pixel coordinate in the desired data based on the initial camera parameters to obtain an undistorted pixel coordinate; determine a first desired position of the calibration board in the camera coordinate system based on the desired distance, the undistorted pixel coordinate, and the initial camera parameters in the desired data; and determine the initial desired pose corresponding to the desired data based on the first desired position.

[0172] Exemplarily, when the initial desired pose determination module 61 determines the initial desired pose corresponding to the desired data based on the first desired position, it is specifically configured to: determine a second desired position of the calibration board in the robotic arm coordinate system based on the first desired position, the initial rotation matrix and the initial translation matrix between the camera coordinate system and the robotic arm coordinate system; determine a second pose based on the first pose of the robotic arm in the robotic arm coordinate system and the second desired position, where the first pose is the robotic arm pose when the calibration board is in the initial situation; adjust the pose in the second pose based on a randomly obtained rotation angle to obtain the initial desired pose of the robotic arm; the second pose includes a position and a pose.

[0173] Exemplarily, when determining multiple target desired poses of the robotic arm based on the candidate camera parameters, the target desired pose determination module 63 is specifically configured to: determine the pixel distance between the desired pixel coordinates in the desired data and the actual pixel coordinates of the specific position on the first calibration image based on the initial desired pose corresponding to each desired data; determine a position correction amount based on the candidate camera parameters and the pixel distance; correct the initial desired pose based on the position correction amount to obtain the target desired pose corresponding to the desired data; or, for each desired data, perform a distortion removal operation on the desired pixel coordinates in the desired data based on the candidate camera parameters to obtain undistorted pixel coordinates; determine the desired position of the calibration board in the camera coordinate system based on the desired distance, the undistorted pixel coordinates, and the candidate camera parameters in the desired data, and determine the target desired pose corresponding to the desired data based on the desired position; or, correct the initial desired pose based on the initially obtained position correction amount to obtain a first desired pose based on the initial desired pose corresponding to each desired data; determine a second desired pose based on the initially obtained desired position; project the calibration board in the first desired pose onto the image of the camera to obtain the first actual pixel coordinates of the specific position on the image, and project the calibration board in the second desired pose onto the image of the camera to obtain the second actual pixel coordinates of the specific position on the image; if the first pixel distance between the first actual pixel coordinates and the desired pixel coordinates in the desired data is less than the second pixel distance between the second actual pixel coordinates and the desired pixel coordinates, determine the first desired pose as the target desired pose corresponding to the desired data; if the second pixel distance is less than the first pixel distance, determine the second desired pose as the target desired pose corresponding to the desired data.

[0174] When determining the position correction amount based on the candidate camera parameters and the pixel distance, the target desired pose determination module 63 is specifically configured to: convert the pixel distance into a physical distance based on the candidate camera parameters and the desired distance in the desired data; convert the physical distance into the position correction amount in the robotic arm coordinate system based on the target rotation matrix between the camera coordinate system and the robotic arm coordinate system; when determining the target desired pose corresponding to the desired data based on the desired position, the target desired pose determination module 63 is specifically configured to: determine the desired position of the calibration board in the robotic arm coordinate system based on the desired position, the target rotation matrix, and the target translation matrix between the camera coordinate system and the robotic arm coordinate system, and determine the target desired pose based on the desired position of the calibration board in the robotic arm coordinate system.

[0175] Exemplarily, when the target desired pose determination module 63 projects the calibration board in the first desired pose onto the image of the camera to obtain the first actual pixel coordinates of the specific position on the image, it specifically is used for: controlling the robotic arm to move the calibration board based on the first desired pose, and determining the first physical coordinates of the specific position of the calibration board in the robotic arm coordinate system; based on the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system, converting the first physical coordinates into the second physical coordinates of the specific position in the camera coordinate system; and determining the first actual pixel coordinates of the specific position on the image based on the second physical coordinates. When the target desired pose determination module 63 projects the calibration board in the second desired pose onto the image of the camera to obtain the second actual pixel coordinates of the specific position on the image, it specifically is used for: controlling the robotic arm to move the calibration board based on the second desired pose, and determining the third physical coordinates of the specific position in the robotic arm coordinate system; based on the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system, converting the third physical coordinates into the fourth physical coordinates of the specific position in the camera coordinate system, and determining the second actual pixel coordinates of the specific position on the image based on the fourth physical coordinates.

[0176] Exemplarily, the device further includes: an acquisition module, configured to acquire the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system. Wherein, when the acquisition module acquires the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system, it specifically is used for: determining a third transformation matrix between the camera coordinate system and the robotic arm coordinate system based on a first transformation matrix between the robotic arm coordinate system and the end effector coordinate system and a second transformation matrix between the calibration board coordinate system and the camera coordinate system, and the third transformation matrix includes a target rotation matrix and a target translation matrix;

[0177] Wherein, when obtaining the initial desired pose of the robotic arm, determining the first transformation matrix based on the initial desired pose; when obtaining the first calibration image of the calibration board, determining the second transformation matrix based on the physical coordinates in the calibration board coordinate system and the pixel coordinates in the first calibration image.

[0178] Exemplarily, the camera is a reference camera, and the multiple target desired poses are used to determine the target camera parameters of the reference camera; or, the multiple target desired poses are used to determine the target camera parameters of multiple cameras to be calibrated, and the models of the multiple cameras to be calibrated are the same as the model of the reference camera;

[0179] When determining the target camera parameters of the reference camera or the camera to be calibrated, the device further includes: a second control module, configured to, for each target desired pose, control the robotic arm to move the calibration board based on the target desired pose, and collect a second calibration image of the calibration board through the reference camera or the camera to be calibrated; a camera parameter calibration module, configured to determine the target camera parameters of the reference camera or the camera to be calibrated based on the second calibration images corresponding to multiple target desired poses.

[0180] Based on the same inventive concept as the above method, an embodiment of the present application provides a control device, including: a transmitter, a receiver, and a processor; wherein:

[0181] The processor is configured to obtain the initial camera parameters of the camera, and determine multiple initial desired poses of the robotic arm of the robot based on the initial camera parameters of the camera; wherein, the calibration board is fixedly connected to the robotic arm of the robot, and the calibration board moves synchronously with the robotic arm.

[0182] The transmitter is configured to, for each initial desired pose, send a first control instruction to the robot and send a second control instruction to the camera; the first control instruction is used to make the robot move the robotic arm based on the initial desired pose, and the calibration board moves synchronously based on the initial desired pose; the second control instruction is used to make the camera collect a first calibration image of the calibration board.

[0183] The receiver is configured to receive the first calibration image collected by the camera.

[0184] The processor is configured to determine candidate camera parameters of the camera based on the first calibration images corresponding to multiple initial desired poses; determine multiple target desired poses of the robotic arm based on the candidate camera parameters; wherein, the multiple target desired poses are used to determine the target camera parameters and calibrate the target camera parameters.

[0185] Based on the same inventive concept as the above method, an embodiment of the present application provides a control device, as shown in Figure 7 shown, the control device includes: a processor 71 and a machine-readable storage medium 72, the machine-readable storage medium 72 stores machine-executable instructions that can be executed by the processor 71; the processor 71 is configured to execute the machine-executable instructions to implement the camera parameter calibration method disclosed in the above examples of the present application.

[0186] Based on the same inventive concept as the above method, an embodiment of the present application further provides a machine-readable storage medium, on which several computer instructions are stored, and when the computer instructions are executed by a processor, the camera parameter calibration method disclosed in the above examples of the present application can be implemented.

[0187] Among them, the above-mentioned machine-readable storage medium can be any electronic, magnetic, optical or other physical storage device that can contain or store information, such as executable instructions, data, and so on. For example, the machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or a combination thereof.

[0188] Based on the same application concept as the above method, an embodiment of the present application further provides a computer program product. The computer program product may include a computer program, and when the computer program is executed by a processor, it can implement the calibration method of the camera parameters disclosed in the above examples.

[0189] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0190] The above description is only for the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A camera parameter calibration method, characterized in that: The calibration plate is fixedly connected to the mechanical arm of the robot, and the calibration plate and the mechanical arm move synchronously. The method includes: Determining a plurality of initial desired poses of the robotic arm based on initial camera parameters of the camera; For each initial desired posture, controlling the mechanical arm to move the calibration plate based on the initial desired posture, and acquiring a first calibration image for the calibration plate through the camera; Determining candidate camera parameters of the camera based on first calibration images corresponding to a plurality of initial desired poses; Determine a plurality of target desired poses of the robotic arm based on the candidate camera parameters; wherein the plurality of target desired poses are used to determine target camera parameters and calibrate the target camera parameters; The multiple initial expected postures correspond one-to-one to the multiple expected data that have been configured, and each expected data includes an expected distance and an expected pixel coordinate; the expected distance represents the distance between the calibration plate and the camera, and the expected pixel coordinate represents the pixel coordinate of a specific position of the calibration plate on the image; The step of determining a plurality of target desired poses of the robotic arm based on the candidate camera parameters comprises: Based on the initial expected posture corresponding to each expected data, determine the pixel distance between the expected pixel coordinates in the expected data and the actual pixel coordinates of the specific position on the first calibration image; determine the position correction amount based on the candidate camera parameters and the pixel distance; and correct the initial expected posture based on the position correction amount to obtain the target expected posture corresponding to the expected data; Alternatively, for each expected data, a dedistortion operation is performed on the expected pixel coordinates in the expected data based on the candidate camera parameters to obtain undistorted pixel coordinates; based on the expected distance in the expected data, the undistorted pixel coordinates and the candidate camera parameters, an expected position of the calibration plate in the camera coordinate system is determined, and based on the expected position, an expected target pose corresponding to the expected data is determined; Alternatively, based on the initial expected posture corresponding to each expected data, the initial expected posture is corrected by the acquired position correction amount to obtain a first expected posture; the second expected posture is determined by the acquired expected position; the calibration plate under the first expected posture is projected onto the image of the camera to obtain the first actual pixel coordinates of the specific position on the image, and the calibration plate under the second expected posture is projected onto the image of the camera to obtain the second actual pixel coordinates of the specific position on the image; If the first pixel distance between the first actual pixel coordinate and the expected pixel coordinate in the expected data is less than the second pixel distance between the second actual pixel coordinate and the expected pixel coordinate, the first expected posture is determined as the target expected posture corresponding to the expected data; if the second pixel distance is less than the first pixel distance, the second expected posture is determined as the target expected posture corresponding to the expected data.

2. The method according to claim 1, characterized in that Before determining a plurality of initial desired positions of the robotic arm based on the initial camera parameters of the camera, the process of acquiring the initial camera parameters includes: Determine the initial distortion parameters of the camera based on the horizontal field of view angle, vertical field of view angle, optical distortion coefficient and TV distortion coefficient of the camera; determine the initial focal length parameters of the camera based on the focal length of the camera lens and the pixel size of the camera sensor; determine the initial principal point coordinate parameters of the camera based on the imaging resolution of the camera; Acquire initial camera parameters of the camera; wherein the initial camera parameters include the initial distortion parameters, the initial focal length parameters and the initial principal point coordinate parameters.

3. The method according to claim 1, characterized in that The determining of a plurality of initial desired positions of the robotic arm based on initial camera parameters of the camera comprises: Acquire a plurality of configured expected data; for each expected data, perform a dedistortion operation on the expected pixel coordinates in the expected data based on the initial camera parameters to obtain undistorted pixel coordinates; Determine a first expected position of the calibration plate in a camera coordinate system based on the expected distance in the expected data, the distortion-free pixel coordinates, and the initial camera parameters; An initial expected posture corresponding to the expected data is determined based on the first expected position.

4. The method according to claim 3, characterized in that The determining, based on the first expected position, an initial expected posture corresponding to the expected data includes: Determine a second desired position of the calibration plate in the robotic arm coordinate system based on the first desired position, an initial rotation matrix and an initial translation matrix between the camera coordinate system and the robotic arm coordinate system; Determine a second posture based on a first posture of the robotic arm in the robotic arm coordinate system and the second expected position, wherein the first posture is a posture of the robotic arm when the calibration plate is in an initial state; The posture in the second posture is adjusted based on the randomly acquired rotation angle to obtain an initial desired posture of the robotic arm; wherein the second posture includes a position and a posture.

5. The method according to claim 1, characterized in that The determining of the position correction amount based on the candidate camera parameters and the pixel distance comprises: Based on the candidate camera parameters and the expected distance in the expected data, the pixel distance is converted into a physical distance; based on the target rotation matrix between the camera coordinate system and the robot coordinate system, the physical distance is converted into the position correction value in the robot coordinate system; The step of determining the target expected posture corresponding to the expected data based on the expected position includes: Based on the expected position, the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system, the expected position of the calibration plate in the robotic arm coordinate system is determined, and the target expected posture is determined based on the expected position of the calibration plate in the robotic arm coordinate system.

6. The method according to claim 1, characterized in that The step of projecting the calibration plate in the first desired posture onto the image of the camera to obtain the first actual pixel coordinates of the specific position on the image includes: controlling the robotic arm to move the calibration plate based on the first desired posture to determine the first physical coordinates of the specific position of the calibration plate in the robotic arm coordinate system; converting the first physical coordinates into the second physical coordinates of the specific position in the camera coordinate system based on the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system; and determining the first actual pixel coordinates of the specific position on the image based on the second physical coordinates; The method of projecting the calibration plate in the second desired posture onto the image of the camera to obtain the second actual pixel coordinates of the specific position on the image includes: controlling the robotic arm to move the calibration plate based on the second desired posture to determine the third physical coordinates of the specific position in the robotic arm coordinate system; based on the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system, converting the third physical coordinates into the fourth physical coordinates of the specific position in the camera coordinate system, and determining the second actual pixel coordinates of the specific position on the image based on the fourth physical coordinates.

7. The method according to claim 5 or 6, characterized in that: The process of obtaining the target rotation matrix and the target translation matrix between the camera coordinate system and the robot arm coordinate system includes: Based on the first transformation matrix between the manipulator coordinate system and the end coordinate system and the second transformation matrix between the calibration plate coordinate system and the camera coordinate system, determine a third transformation matrix between the camera coordinate system and the manipulator coordinate system, the third transformation matrix including a target rotation matrix and a target translation matrix; Among them, when the initial expected posture of the robotic arm is obtained, the first transformation matrix is ​​determined based on the initial expected posture; when the first calibration image of the calibration plate is obtained, the second transformation matrix is ​​determined based on the physical coordinates in the calibration plate coordinate system and the pixel coordinates in the first calibration image.

8. The method according to any one of claims 1 to 6, characterized in that: The camera is a reference camera, and the multiple target expected postures are used to determine the target camera parameters of the reference camera; or, the multiple target expected postures are used to determine the target camera parameters of multiple cameras to be calibrated, and the models of the multiple cameras to be calibrated are the same as the model of the reference camera; When determining the target camera parameters of the reference camera or the camera to be calibrated, the method further includes: For each target expected posture, the mechanical arm is controlled to move the calibration plate based on the target expected posture, and a second calibration image for the calibration plate is acquired through the reference camera or the camera to be calibrated; The target camera parameters of the reference camera or the camera to be calibrated are determined based on the second calibration images corresponding to the multiple target expected postures.

9. A camera parameter calibration device, characterized in that: The calibration plate is fixedly connected to the mechanical arm of the robot, and the calibration plate moves synchronously with the mechanical arm. The device comprises: An initial desired posture determination module, used to determine a plurality of initial desired postures of the robotic arm based on initial camera parameters of the camera; A first control module is used for controlling the mechanical arm to move the calibration plate based on each initial expected posture, and collecting a first calibration image for the calibration plate through the camera; A target desired posture determination module, used to determine candidate camera parameters of the camera based on a plurality of first calibration images corresponding to initial desired postures; and determine a plurality of target desired postures of the robotic arm based on the candidate camera parameters; wherein the plurality of target desired postures are used to determine target camera parameters and calibrate the target camera parameters; The multiple initial expected postures correspond one-to-one to the multiple expected data that have been configured, and each expected data includes an expected distance and an expected pixel coordinate; the expected distance represents the distance between the calibration plate and the camera, and the expected pixel coordinate represents the pixel coordinate of a specific position of the calibration plate on the image; The target expected posture determination module is specifically used to determine multiple target expected postures of the robot arm based on the candidate camera parameters: based on the initial expected posture corresponding to each expected data, determine the pixel distance between the expected pixel coordinates in the expected data and the actual pixel coordinates of the specific position on the first calibration image; determine the position correction amount based on the candidate camera parameters and the pixel distance; correct the initial expected posture based on the position correction amount to obtain the target expected posture corresponding to the expected data; or, for each expected data, perform a dedistortion operation on the expected pixel coordinates in the expected data based on the candidate camera parameters to obtain undistorted pixel coordinates; determine the expected position of the calibration plate in the camera coordinate system based on the expected distance in the expected data, the undistorted pixel coordinates and the candidate camera parameters, and determine the target expected posture corresponding to the expected data based on the expected position; Alternatively, based on the initial expected posture corresponding to each expected data, the initial expected posture is corrected by the acquired position correction amount to obtain a first expected posture; the second expected posture is determined by the acquired expected position; the calibration plate under the first expected posture is projected onto the image of the camera to obtain the first actual pixel coordinates of the specific position on the image, and the calibration plate under the second expected posture is projected onto the image of the camera to obtain the second actual pixel coordinates of the specific position on the image; If the first pixel distance between the first actual pixel coordinate and the expected pixel coordinate in the expected data is less than the second pixel distance between the second actual pixel coordinate and the expected pixel coordinate, the first expected posture is determined as the target expected posture corresponding to the expected data; if the second pixel distance is less than the first pixel distance, the second expected posture is determined as the target expected posture corresponding to the expected data.

10. A control device, characterized in that: include: A transmitter, a receiver, and a processor; wherein: The processor is used to obtain initial camera parameters of the camera, and determine multiple initial desired positions of the robot's mechanical arm based on the initial camera parameters of the camera; wherein the calibration plate is fixedly connected to the robot's mechanical arm, and the calibration plate moves synchronously with the mechanical arm; The transmitter is used to send a first control instruction to the robot and a second control instruction to the camera for each initial expected posture; the first control instruction is used to enable the robot to move the mechanical arm based on the initial expected posture, and the calibration plate to move synchronously based on the initial expected posture; the second control instruction is used to enable the camera to collect a first calibration image for the calibration plate; The receiver is used to receive a first calibration image captured by the camera; The processor is used to determine candidate camera parameters of the camera based on a first calibration image corresponding to a plurality of initial desired poses; and determine a plurality of target desired poses of the robotic arm based on the candidate camera parameters; wherein the plurality of target desired poses are used to determine target camera parameters and calibrate the target camera parameters; The multiple initial expected postures correspond one-to-one to the multiple expected data that have been configured, and each expected data includes an expected distance and an expected pixel coordinate; the expected distance represents the distance between the calibration plate and the camera, and the expected pixel coordinate represents the pixel coordinate of a specific position of the calibration plate on the image; Wherein, when the processor determines the multiple target expected postures of the robot arm based on the candidate camera parameters, it is specifically used to: determine the pixel distance between the expected pixel coordinates in the expected data and the actual pixel coordinates of the specific position on the first calibration image based on the initial expected posture corresponding to each expected data; determine the position correction amount based on the candidate camera parameters and the pixel distance; correct the initial expected posture based on the position correction amount to obtain the target expected posture corresponding to the expected data; Alternatively, for each expected data, a dedistortion operation is performed on the expected pixel coordinates in the expected data based on the candidate camera parameters to obtain undistorted pixel coordinates; based on the expected distance in the expected data, the undistorted pixel coordinates and the candidate camera parameters, an expected position of the calibration plate in the camera coordinate system is determined, and based on the expected position, an expected target pose corresponding to the expected data is determined; Alternatively, based on the initial expected posture corresponding to each expected data, the initial expected posture is corrected by the acquired position correction amount to obtain a first expected posture; the second expected posture is determined by the acquired expected position; the calibration plate under the first expected posture is projected onto the image of the camera to obtain the first actual pixel coordinates of the specific position on the image, and the calibration plate under the second expected posture is projected onto the image of the camera to obtain the second actual pixel coordinates of the specific position on the image; If the first pixel distance between the first actual pixel coordinate and the expected pixel coordinate in the expected data is less than the second pixel distance between the second actual pixel coordinate and the expected pixel coordinate, the first expected posture is determined as the target expected posture corresponding to the expected data; if the second pixel distance is less than the first pixel distance, the second expected posture is determined as the target expected posture corresponding to the expected data.

Images