Camera parameter calibration method and device and control equipment

Abstract

The invention provides a camera parameter calibration method and device and control equipment. The method comprises the steps that multiple initial expected poses of a mechanical arm are determined based on initial camera parameters of a camera; aiming at each initial expected pose, controlling the mechanical arm to move the calibration plate based on the initial expected pose, and collecting a first calibration image aiming at the calibration plate through the camera; determining candidate camera parameters of the camera based on the first calibration images corresponding to the plurality of initial expected poses; determining a plurality of target expected poses of the mechanical arm based on the candidate camera parameters; wherein the multiple target expected poses are used for determining target camera parameters and calibrating the target camera parameters. According to the technical scheme, automatic calibration of the camera parameters can be realized.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a camera parameter calibration method, device and control equipment. Background Art

[0002] Camera parameters can include camera intrinsic parameters (i.e. internal parameters), distortion coefficients and camera extrinsic parameters (i.e. external parameters). Camera intrinsic parameters are composed of (cx, cy, fx, fy), (cx, cy) are the coordinates of the principal point, (fx, fy) are the focal length of the camera, and camera intrinsic parameters are parameters related to the characteristics of the camera itself. Camera extrinsic parameters are composed of (ω, δ, θ, Tx, Ty, Tz), (ω, δ, θ) are the rotation parameters of the three axes, (Tx, Ty, Tz) are the translation parameters of the three axes, and camera extrinsic parameters are used to represent the conversion relationship between the world coordinate system and the camera coordinate system. (ω, δ, θ) can also be called (roll, pitch, yaw), roll represents the roll angle, pitch represents the pitch angle, and yaw represents the yaw angle. Tx represents the horizontal parameter, that is, the translation parameter value of the x-axis, Ty represents the vertical parameter, that is, the vertical parameter value of the y-axis, and Tz represents the vertical parameter, that is, the vertical parameter value of the z-axis.

[0003] Before a camera leaves the factory, it is necessary to calibrate the camera parameters for each camera. However, there is no reasonable calibration method in the relevant technology for calibrating the camera parameters, such as camera intrinsic parameters, distortion coefficients, and camera extrinsic parameters. There are problems such as cumbersome calibration operation, long calibration time, and high calibration labor cost. Summary of the invention

[0004] The present application provides a camera parameter calibration method, wherein a calibration plate is fixedly connected to a robot arm, and the calibration plate and the robot arm move synchronously, and the method comprises: Determining a plurality of initial desired poses of the robotic arm based on initial camera parameters of the camera; For each initial desired posture, controlling the mechanical arm to move the calibration plate based on the initial desired posture, and acquiring a first calibration image for the calibration plate through the camera; Determining candidate camera parameters of the camera based on first calibration images corresponding to a plurality of initial desired poses; Based on the candidate camera parameters, a plurality of target desired postures of the robotic arm are determined; wherein the plurality of target desired postures are used to determine target camera parameters and calibrate the target camera parameters.

[0005] The present application provides a camera parameter calibration device, wherein a calibration plate is fixedly connected to a robot arm, and the calibration plate moves synchronously with the robot arm. The device comprises: an initial expected posture determination module, used to determine a plurality of initial expected postures of the robot arm based on the initial camera parameters of the camera; a first control module, used to control the robot arm to move the calibration plate based on each initial expected posture, and to collect a first calibration image of the calibration plate through the camera; a target expected posture determination module, used to determine candidate camera parameters of the camera based on the first calibration images corresponding to the plurality of initial expected postures; and determine a plurality of target expected postures of the robot arm based on the candidate camera parameters; wherein the plurality of target expected postures are used to determine target camera parameters and calibrate the target camera parameters.

[0006] The present application provides a control device, comprising: a transmitter, a receiver and a processor; wherein: The processor is used to obtain initial camera parameters of the camera, and determine multiple initial desired positions of the robot's mechanical arm based on the initial camera parameters of the camera; wherein the calibration plate is fixedly connected to the robot's mechanical arm, and the calibration plate moves synchronously with the mechanical arm; The transmitter is used to send a first control instruction to the robot and a second control instruction to the camera for each initial expected posture; the first control instruction is used to enable the robot to move the mechanical arm based on the initial expected posture, and the calibration plate to move synchronously based on the initial expected posture; the second control instruction is used to enable the camera to collect a first calibration image for the calibration plate; The receiver is used to receive a first calibration image captured by the camera; The processor is used to determine candidate camera parameters of the camera based on a first calibration image corresponding to a plurality of initial desired poses; and determine a plurality of target desired poses of the robotic arm based on the candidate camera parameters; wherein the plurality of target desired poses are used to determine target camera parameters and calibrate the target camera parameters.

[0007] The present application provides a control device, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-mentioned camera parameter calibration method.

[0008] The present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned camera parameter calibration method is implemented.

[0009] The present application provides a machine-readable storage medium, which stores machine-executable instructions that can be executed by a processor; wherein the processor is used to execute the machine-executable instructions, and implements the above-mentioned camera parameter calibration method when executing the machine-executable instructions.

[0010] It can be seen from the above technical solutions that in the embodiments of the present application, the camera parameters can be automatically calibrated, and the calibration operation is simple, the calibration time is short, and the calibration labor cost is low. The desired point positions (i.e., multiple initial desired positions) of the robot arm can be automatically generated without manual testing and calibration of the desired point positions, thereby reducing the testing and calibration of the desired point positions, improving automation, and reducing the interference of human factors. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a flowchart of a camera parameter calibration method in one embodiment of the present application; Figure 2 It is a flowchart of a target desired posture acquisition process in one embodiment of the present application; Figure 3 is a schematic diagram of optical distortion and TV distortion in one embodiment of the present application; Figure 4 is a schematic diagram of a camera coordinate system and a robotic arm coordinate system in one embodiment of the present application; Figure 5 is a flowchart of a camera parameter calibration process in one embodiment of the present application; Figure 6 is a schematic structural diagram of a camera parameter calibration device in one embodiment of the present application; Figure 7 It is a hardware structure diagram of a control device in one embodiment of the present application. DETAILED DESCRIPTION

[0012] In the embodiment of the present application, a camera parameter calibration method is proposed. The method can be applied to a control device, such as a personal computer, a laptop computer, a smart phone, etc. The type of the control device is not limited. It can control the robot to move the mechanical arm and control the camera to capture images. The calibration plate is placed on the robot's mechanical arm, and the calibration plate is fixedly connected to the robot's mechanical arm, that is, the calibration plate moves synchronously with the mechanical arm, see Figure 1 FIG. 4 is a flow chart of the method, which may include: Step 101: Determine a plurality of initial desired positions of the robot arm based on initial camera parameters of the camera.

[0013] Step 102: for each initial desired posture, control the robot arm to move the calibration plate based on the initial desired posture, and collect a first calibration image for the calibration plate through a camera.

[0014] Step 103 : determining candidate camera parameters of the camera based on the first calibration images corresponding to the multiple initial desired postures (ie, the multiple first calibration images corresponding to the multiple initial desired postures).

[0015] Step 104: determine multiple target desired postures of the robotic arm based on the candidate camera parameters; wherein the multiple target desired postures are used to determine the target camera parameters and calibrate the target camera parameters.

[0016] Exemplarily, before determining multiple initial desired postures of the robotic arm based on the initial camera parameters of the camera, the process of acquiring the initial camera parameters of the camera may include but is not limited to: determining the initial distortion parameters of the camera based on the horizontal field of view angle, vertical field of view angle, optical distortion coefficient and TV distortion coefficient of the camera; determining the initial focal length parameters of the camera based on the focal length of the camera lens and the pixel size of the camera sensor; determining the initial principal point coordinate parameters of the camera based on the imaging resolution of the camera; acquiring the initial camera parameters of the camera; the initial camera parameters include the initial distortion parameters, the initial focal length parameters and the initial principal point coordinate parameters.

[0017] Exemplarily, determining multiple initial expected postures of the robotic arm based on the initial camera parameters of the camera may include, but is not limited to: obtaining multiple configured expected data, each expected data may include an expected distance and an expected pixel coordinate; wherein the expected distance may represent the distance between the calibration plate and the camera, and the expected pixel coordinate may represent the pixel coordinate of a specific position of the calibration plate on the image.

[0018] For each expected data, a dedistortion operation is performed on the expected pixel coordinates in the expected data based on the initial camera parameters to obtain undistorted pixel coordinates; a first expected position of the calibration plate in the camera coordinate system is determined based on the expected distance in the expected data, the undistorted pixel coordinates and the initial camera parameters; and an initial expected posture corresponding to the expected data is determined based on the first expected position.

[0019] Exemplarily, determining the initial expected posture corresponding to the expected data based on the first expected position may include but is not limited to: determining the second expected position of the calibration plate in the robotic arm coordinate system based on the first expected position, the initial rotation matrix and the initial translation matrix between the camera coordinate system and the robotic arm coordinate system; determining the second posture of the robotic arm based on the first posture and the second expected position of the robotic arm in the robotic arm coordinate system, the first posture being the posture of the robotic arm in the initial state of the calibration plate; adjusting the posture in the second posture based on the randomly acquired rotation angle to obtain the initial expected posture of the robotic arm, the second posture including the position and posture.

[0020] Exemplarily, determining multiple target expected postures of the manipulator based on the candidate camera parameters may include, but is not limited to: determining the pixel distance between the expected pixel coordinates in the expected data and the actual pixel coordinates of the specific position on the first calibration image based on the initial expected posture corresponding to each expected data; determining the position correction amount based on the candidate camera parameters and the pixel distance; correcting the initial expected posture based on the position correction amount to obtain the target expected posture corresponding to the expected data. Alternatively, for each expected data, performing a dedistortion operation on the expected pixel coordinates in the expected data based on the candidate camera parameters to obtain undistorted pixel coordinates; determining the expected position of the calibration plate in the camera coordinate system based on the expected distance, undistorted pixel coordinates and candidate camera parameters in the expected data, and determining the target expected posture corresponding to the expected data based on the expected position. Alternatively, based on the initial expected posture corresponding to each expected data, the initial expected posture is corrected by the acquired position correction amount to obtain the first expected posture; the second expected posture is determined by the acquired expected position; the calibration plate under the first expected posture is projected onto the image of the camera to obtain the first actual pixel coordinates of the specific position on the image, and the calibration plate under the second expected posture is projected onto the image of the camera to obtain the second actual pixel coordinates of the specific position on the image. If the first pixel distance between the first actual pixel coordinate and the expected pixel coordinate in the expected data is less than the second pixel distance between the second actual pixel coordinate and the expected pixel coordinate, the first expected posture can be determined as the target expected posture corresponding to the expected data; if the second pixel distance is less than the first pixel distance, the second expected posture can be determined as the target expected posture corresponding to the expected data.

[0021] Exemplarily, determining the position correction amount based on the candidate camera parameters and the pixel distance may include, but is not limited to: based on the candidate camera parameters and the expected distance in the expected data, the pixel distance may be converted into a physical distance; based on the target rotation matrix between the camera coordinate system and the robotic arm coordinate system, the physical distance may be converted into a position correction amount in the robotic arm coordinate system.

[0022] Exemplarily, determining the expected target pose corresponding to the expected data based on the expected position may include but is not limited to: determining the expected position of the calibration plate in the robotic arm coordinate system based on the expected position, the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system, and determining the expected target pose based on the expected position of the calibration plate in the robotic arm coordinate system.

[0023] Exemplarily, a calibration plate in a first desired posture is projected onto an image of a camera to obtain a first actual pixel coordinate of a specific position on the image, which may include but is not limited to: controlling a robotic arm to move the calibration plate based on the first desired posture to determine a first physical coordinate of a specific position of the calibration plate in the robotic arm coordinate system; based on a target rotation matrix and a target translation matrix between the camera coordinate system and the robotic arm coordinate system, converting the first physical coordinate into a second physical coordinate of the specific position in the camera coordinate system; and determining the first actual pixel coordinate of the specific position on the image based on the second physical coordinate.

[0024] Exemplarily, the calibration plate in the second desired posture is projected onto the image of the camera to obtain the second actual pixel coordinates of the specific position on the image, which may include but is not limited to: controlling the robotic arm to move the calibration plate based on the second desired posture to determine the third physical coordinates of the specific position of the calibration plate in the robotic arm coordinate system; based on the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system, converting the third physical coordinates into the fourth physical coordinates of the specific position in the camera coordinate system, and determining the second actual pixel coordinates of the specific position on the image based on the fourth physical coordinates.

[0025] Exemplarily, the process of acquiring the target rotation matrix and the target translation matrix between the camera coordinate system and the manipulator coordinate system may include, but is not limited to: determining the third transformation matrix between the camera coordinate system and the manipulator coordinate system based on the first transformation matrix between the manipulator coordinate system and the end coordinate system and the second transformation matrix between the calibration plate coordinate system and the camera coordinate system, and the third transformation matrix may include the target rotation matrix and the target translation matrix. For example, when the initial expected posture of the manipulator is obtained, the first transformation matrix is ​​determined based on the initial expected posture; when the first calibration image of the calibration plate is obtained, the second transformation matrix is ​​determined based on the physical coordinates in the calibration plate coordinate system and the pixel coordinates in the first calibration image.

[0026] Exemplarily, the camera in the above embodiment may be a reference camera, and the multiple target desired poses are used to determine the target camera parameters of the reference camera. Alternatively, the multiple target desired poses may be used to determine the target camera parameters of multiple cameras to be calibrated, and the models of the multiple cameras to be calibrated are the same as the model of the reference camera.

[0027] On this basis, when determining the target camera parameters of the reference camera or the camera to be calibrated, for each target expected posture, the robotic arm is controlled to move the calibration plate based on the target expected posture, and a second calibration image for the calibration plate is collected through the reference camera or the camera to be calibrated; the target camera parameters of the reference camera or the camera to be calibrated are determined based on the second calibration images corresponding to multiple target expected postures.

[0028] It can be seen from the above technical solutions that in the embodiments of the present application, the camera parameters can be automatically calibrated, and the calibration operation is simple, the calibration time is short, and the calibration labor cost is low. The desired point positions (i.e., multiple initial desired positions) of the robot arm can be automatically generated without manual testing and calibration of the desired point positions, thereby reducing the testing and calibration of the desired point positions, improving automation, and reducing the interference of human factors.

[0029] The above technical solutions of the embodiments of the present application are described below in combination with specific application scenarios.

[0030] The embodiment of the present application proposes a camera parameter calibration method for calibrating the camera parameters of the camera of the calibrated device, such as calibrating the camera intrinsic parameters, distortion coefficients and camera extrinsic parameters. The camera intrinsic parameters are composed of (cx, cy, fx, fy), (cx, cy) are the main point coordinates, (fx, fy) are the focal length of the camera, and the camera extrinsic parameters are composed of (roll, pitch, yaw, Tx, Ty, Tz), (roll, pitch, yaw) are the rotation parameters of the three axes, (Tx, Ty, Tz) are the translation parameters of the three axes, roll represents the roll angle, pitch represents the pitch angle, and yaw represents the yaw angle. Tx represents the horizontal parameter, Ty represents the vertical parameter, and Tz represents the vertical parameter.

[0031] The device to be calibrated is a device having at least one camera. There is no restriction on the type of the device to be calibrated, as long as it has at least one camera. For example, the device to be calibrated may include an RGBD camera, an RGB camera, or a depth camera. The RGB camera may be a single camera. The depth camera may include a left-eye camera and a right-eye camera. The RGBD camera may include an RGB camera and a depth camera, and the depth camera may be a left-eye camera and a right-eye camera, and the depth camera may also be a single camera. For the convenience of description, it is taken as an example that the device to be calibrated includes an RGBD camera, that is, the camera parameter calibration method of the present embodiment is adopted to calibrate the camera parameters of the RGB camera in the RGBD camera, and to calibrate the camera parameters of the depth camera in the RGBD camera.

[0032] In this embodiment, a target expected posture acquisition process and a camera parameter calibration process are involved. In the target expected posture acquisition process, multiple target expected postures corresponding to the reference camera can be acquired. In the camera parameter calibration process, camera parameters of each camera to be calibrated can be calibrated based on the target expected posture.

[0033] For example, for multiple cameras of the same model to be calibrated, one camera to be calibrated can be selected as a reference camera, and multiple target expected poses corresponding to the reference camera can be obtained and stored. Based on the multiple target expected poses, the camera parameters of each camera to be calibrated (including the reference camera and other cameras of the same model to be calibrated) can be calibrated.

[0034] For example, when calibrating RGB cameras among multiple RGBD cameras, multiple RGB cameras can be used as multiple cameras to be calibrated, and one RGB camera can be selected from the multiple RGB cameras as a reference camera. When calibrating depth cameras among multiple RGBD cameras, multiple depth cameras can be used as multiple cameras to be calibrated, and one depth camera can be selected from the multiple depth cameras as a reference camera.

[0035] In this embodiment, during the process of acquiring the desired position of the target, the reference camera can be placed at a specified position, that is, the reference camera is fixed at the specified position, and the image within the field of view is collected by the reference camera. For example, a certain RGBD camera can be placed at a specified position, so that the RGB camera in the RGBD camera is used as a reference camera, and / or the depth camera in the RGBD camera is used as a reference camera.

[0036] During the camera parameter calibration process, multiple cameras to be calibrated can be placed in the specified position in sequence, that is, the camera to be calibrated is fixed at the specified position, and the image within the field of view is collected by the camera to be calibrated. For example, RGBD camera 1 can be placed at the specified position, the RGB camera in RGBD camera 1 is used as the camera to be calibrated, and / or the depth camera in RGBD camera 1 is used as the camera to be calibrated. After completing the parameter calibration of RGBD camera 1, RGBD camera 2 can be placed at the specified position, and so on.

[0037] In this embodiment, during the process of acquiring the target desired posture and calibrating the camera parameters, the calibration plate can be placed on the robot's robotic arm, and the calibration plate is fixedly connected to the robot's robotic arm (the calibration plate and the robotic arm can be considered to be in the same coordinate system), that is, the calibration plate and the robotic arm move synchronously.

[0038] For the calibration plate, the calibration plate may be a calibration plate with a checkerboard pattern or a calibration plate with a target pattern. There is no limitation on the calibration pattern of the calibration plate, and the calibration plate may be a calibration plate with any pattern.

[0039] For example, in the process of obtaining the target desired pose, a camera to be calibrated may be selected from multiple cameras to be calibrated as a reference camera, and multiple target desired poses corresponding to the reference camera may be obtained. Figure 2 FIG. 1 is a flow chart of a process for obtaining the target desired posture, which may include: Step 201: Obtain initial camera parameters of a reference camera.

[0040] Exemplarily, the initial camera parameters may be estimated camera parameters, which may have certain errors. There is no restriction on the process of obtaining the initial camera parameters, as long as the initial camera parameters can be obtained. For example, the initial camera parameters may include initial distortion parameters, initial focal length parameters, and initial principal point coordinate parameters.

[0041] In a possible implementation, the following steps may be used to obtain initial camera parameters.

[0042] Step S11: determining initial distortion parameters of the reference camera based on the horizontal field of view angle, vertical field of view angle, optical distortion coefficient, and TV (Transverse Vertical) distortion coefficient of the reference camera.

[0043] For example, the horizontal field of view angle, vertical field of view angle, optical distortion coefficient and TV distortion coefficient are all inherent parameters of the reference camera and are all known parameters. On this basis, the initial distortion parameters can be determined based on the horizontal field of view angle, vertical field of view angle, optical distortion coefficient and TV distortion coefficient of the reference camera.

[0044] For example, the initial distortion parameters may include radial distortion coefficients and tangential distortion coefficient ,If the reference camera provides the ideal values ​​and distortion values ​​of some pixels within the field of view (hereinafter referred to as the distortion table), the initial distortion parameters of the reference camera can be solved based on the distortion model.

[0045] For example, the distortion model can be shown in formula (1), and the distortion table includes the distortion value and ideal value The corresponding relationship, Indicates ideal value The distorted coordinates, that is, due to the existence of the initial distortion parameters, lead to the ideal value Distorted into .

[0046] Formula (1) In formula (1), Obviously, after substituting multiple sets of data into formula (1), each set of data can include the distortion value and ideal value , and multiple sets of data can be obtained from the distortion table to solve the radial distortion coefficient and tangential distortion coefficient .

[0047] Exemplarily, if the reference camera does not provide a distortion table for some pixels within the field of view, the initial distortion parameters of the reference camera can be determined based on the horizontal field of view angle, vertical field of view angle, optical distortion coefficient, and TV distortion coefficient of the reference camera, which can also be called fitting to estimate the initial distortion parameters of the reference camera.

[0048] See also Figure 3 The figure below is a schematic diagram of optical distortion and TV distortion. Figure 3 In the equation, Y0 / Y represents the optical distortion coefficient, which can be expressed as , h0 / h represents the TV distortion coefficient, which can be expressed as In addition, the horizontal field of view can be expressed as , and the vertical field angle is recorded as , and order , On this basis, is the ideal coordinate of the lower right corner edge point of the image, that is, the lower right corner edge point without distortion, and for . is the ideal coordinate of the center point of the lower edge of the image, that is, the coordinate of the center point of the lower edge without distortion, and for Due to the existence of the initial distortion parameters, the coordinates of the lower right edge point are Distorted into edge point coordinates ,and for , and results in the coordinates of the center point of the lower edge Distorted into edge point coordinates ,and for .

[0049] Will Coordinates As an ideal value ,Will Coordinates As the distortion value Substituting into formula (1), Coordinates As an ideal value ,Will Coordinates As the distortion value Substitute into formula (1). Based on Figure 3 The geometric relationship shown can be used to obtain the coordinates and coordinates and optical distortion coefficient (Y0 / Y) and TV distortion coefficient (h0 / h), the coordinates and coordinates Updated to optical distortion coefficients and TV distortion coefficient , substitute the above relationship into formula (1). Assuming the radial distortion coefficient and tangential distortion coefficient is 0, and this relationship is also substituted into formula (1). So far, the following formula (2) can be obtained.

[0050] Formula (2) In formula (2), the radial distortion coefficient is an unknown parameter, the optical distortion coefficient and TV distortion coefficient is a known parameter, horizontal field of view angle and vertical field of view are known parameters, so and is also a known parameter. On this basis, the radial distortion coefficient can be approximately solved In addition, the radial distortion coefficient and tangential distortion coefficient is 0, so the initial distortion parameters can be obtained, and the initial distortion parameters include the radial distortion coefficient and tangential distortion coefficient .

[0051] Step S12: determining initial focal length parameters of the reference camera based on the focal length of the lens of the reference camera and the pixel size of the sensor of the reference camera. For example, the initial focal length parameters may include the focal length of the camera (fx, fy).

[0052] For example, the focal length of the reference camera lens and the pixel size of the reference camera sensor (i.e., sensor size) are both inherent parameters of the reference camera and are known parameters. On this basis, the initial focal length parameters can be determined based on the focal length of the lens and the pixel size of the sensor (such as the pixel size in the x direction and the pixel size in the y direction). For example, fx is the focal length of the lens divided by the pixel size in the x direction of the sensor, and fy is the focal length of the lens divided by the pixel size in the y direction of the sensor.

[0053] Step S13: determining the initial principal point coordinate parameters of the reference camera based on the imaging resolution (ie, sensor resolution) of the reference camera. For example, the initial principal point coordinate parameters may include the principal point coordinates (cx, cy).

[0054] For example, the imaging resolution of the reference camera is an intrinsic parameter of the reference camera and is a known parameter. On this basis, the initial principal point coordinate parameters can be determined based on the imaging resolution. For example, cx is the image width resolution in the imaging resolution divided by 2, and cy is the image height resolution in the imaging resolution divided by 2.

[0055] Step S14: Obtain initial camera parameters of the reference camera, which may include initial distortion parameters. , initial focal length parameters (fx, fy) and initial principal point coordinate parameters (cx, cy).

[0056] At this point, step 201 is completed, and the initial camera parameters of the reference camera are obtained.

[0057] Step 202: Determine a plurality of initial desired positions of the robot arm based on initial camera parameters of a reference camera.

[0058] Exemplarily, the initial expected posture of the robot arm can be determined based on the initial camera parameters and expected data of the reference camera, and the initial expected posture can be the point expected posture of the robot arm. For example, multiple expected data can be obtained, each expected data can correspond to an initial expected posture, thereby obtaining multiple initial expected postures of the robot arm, and the following is explained by taking obtaining an initial expected posture as an example.

[0059] In a possible implementation, the following steps may be used to obtain multiple initial desired postures.

[0060] Step S21, obtaining multiple configured expected data, each of which may include an expected distance and an expected pixel coordinate, each of which is configured by the user according to his own needs, and there is no restriction on this. For example, a farther expected distance, a moderate expected distance, and a closer expected distance may be configured. For example, the expected pixel coordinate may be the pixel coordinate of the center of the image, the expected pixel coordinate may be the pixel coordinate of the upper left corner of the image, the expected pixel coordinate may be the pixel coordinate of the upper right corner of the image, etc.

[0061] Exemplarily, each expected data may correspond to an initial expected posture, and the processing process for one expected data is taken as an example. The expected data may include an expected distance, and the expected distance may represent the distance between the calibration plate and the reference camera, and the expected distance may be denoted as d. The expected data may include expected pixel coordinates, and the expected pixel coordinates may represent the pixel coordinates of a specific position of the calibration plate on the image. The specific position of the calibration plate may be any position of the calibration plate, such as the center position of the calibration plate, the upper left corner position of the calibration plate, the upper right corner position of the calibration plate, the lower left corner position of the calibration plate, the lower right corner position of the calibration plate, etc. There is no limitation on this specific position, and the expected pixel coordinates may be denoted as .

[0062] Step S22: for each expected data, dedistortion operation is performed on the expected pixel coordinates in the expected data based on the initial camera parameters of the reference camera to obtain undistorted pixel coordinates.

[0063] For example, the expected pixel coordinates in the expected data may be , the coordinates of the undistorted pixels after the dedistortion operation are denoted as , the initial camera parameters include the initial distortion parameters , initial focal length parameters (fx, fy) and initial principal point coordinate parameters (cx, cy). On this basis, based on the camera imaging model, formula (3) can be obtained. In formula (3), z can be set to 1.

[0064] Formula (3) ( ) represents the expected pixel coordinates, the expected pixel coordinates ( ) corresponds to the distortion value , that is, based on the distortion value Determine the desired pixel coordinates ( ). ( ) represents the undistorted pixel coordinates, which are the pixel coordinates after the distortion is removed. The undistorted pixel coordinates ( ) corresponds to the ideal value , that is, based on the ideal value Determine the undistorted pixel coordinates ( ).

[0065] In addition, the distortion value With ideal value The functional relationship between can be determined based on the distortion model. The distortion model can be shown in formula (1). In summary, by combining formula (1) and formula (3), the undistorted pixel coordinates can be calculated through iterative optimization ( ).

[0066] Step S23: based on the expected distance d in the expected data and the undistorted pixel coordinates ( ) and the initial camera parameters determine the first expected position of the calibration plate in the camera coordinate system.

[0067] For example, the expected distance in the expected data may be d, and the undistorted pixel coordinates may be ( ), the initial camera parameters may include the initial focal length parameters (fx, fy) and the initial principal point coordinate parameters (cx, cy). On this basis, the formula (4) can be obtained based on the camera imaging model.

[0068] Formula (4) In formula (4), is the expected distance d, ( ), (fx, fy) and (cx, cy) are all known values, so the first expected position of the calibration plate in the camera coordinate system can be obtained ( ).

[0069] Step S24: determining a second expected position of the calibration plate in the robotic arm coordinate system based on the first expected position, an initial rotation matrix and an initial translation matrix between the camera coordinate system and the robotic arm coordinate system.

[0070] For example, the first expected position ( ) is the position of the calibration plate in the camera coordinate system, and the initial rotation matrix and the initial translation matrix are the conversion relationship between the camera coordinate system and the robotic arm coordinate system. Therefore, the position in the camera coordinate system can be converted to the position in the robotic arm coordinate system based on the initial rotation matrix and the initial translation matrix, that is, the second expected position of the calibration plate in the robotic arm coordinate system ( ).

[0071] For example, the first expected position of the calibration plate in the camera coordinate system ( ) is converted to the second desired position of the calibration plate in the robot coordinate system ( ).

[0072] Formula (5) In formula (5), represents the initial rotation matrix between the camera coordinate system and the robot arm coordinate system, Represents the initial translation matrix between the camera coordinate system and the robot coordinate system. The initial rotation matrix and initial translation matrix between the camera coordinate system and the robot coordinate system can be obtained by measurement or other methods, and there is no restriction on this. The initial rotation matrix and initial translation matrix can be rough values. In the subsequent process, the initial rotation matrix and initial translation matrix need to be adjusted and optimized.

[0073] Step S25, determine the second posture based on the first posture of the robot arm in the robot arm coordinate system and the second expected position of the calibration plate in the robot arm coordinate system, the first posture is the robot arm posture when the calibration plate is in the initial state, and the second posture is the intermediate posture to be optimized, and the second posture needs to be optimized later.

[0074] For example, when the calibration plate is in the initial state, that is, when the calibration plate is not moved, the current position of the robot arm can be obtained, and this position of the robot arm is used as the first position of the robot arm in the robot arm coordinate system. , the first position can be Recorded as In the initial state, the first position in the robot coordinate system can be directly obtained. .

[0075] Based on the first position and the second desired position of the calibration plate in the robot coordinate system ( ), and obtain the desired second pose ,Right now , will be the first position The position in ) is replaced by the second desired position ( ).

[0076] Step S26: The second posture may include a position ( ) and posture ( , , ), based on the randomly obtained rotation angle, the posture in the second pose ( , , ) to obtain the initial desired posture of the robot arm, which can be recorded as .

[0077] For example, in the camera calibration process, the calibration plate planes at each pose need not be parallel, so for the second pose The posture in , , ) Increase the random rotation angle within a certain range ( ), get the initial desired position of the robot arm . . ( ) are obtained randomly and are within the preset range without any restrictions.

[0078] Based on each expected data, the above steps can be used to obtain the initial expected posture corresponding to the expected data, and then multiple initial expected postures of the robot arm are obtained to complete step 202.

[0079] Step 203: for each initial desired posture, control the robot arm to move the calibration plate based on the initial desired posture, and acquire a first calibration image for the calibration plate through a reference camera.

[0080] For example, based on the initial expected posture 1, the robotic arm is controlled to move the calibration plate, that is, the robotic arm moves to the initial expected posture 1, and the calibration plate moves synchronously with the robotic arm, and the positional relationship between the calibration plate and the robotic arm remains unchanged. After the robotic arm moves to the initial expected posture 1, the first calibration image a1 of the calibration plate is captured by the reference camera. Similarly, after the robotic arm moves to the initial expected posture 2, the first calibration image a2 of the calibration plate is captured by the reference camera, and so on, multiple first calibration images can be obtained. In summary, by controlling the movement of the robotic arm, multiple first calibration images can be captured by the reference camera.

[0081] Step 204 : determining candidate camera parameters of the reference camera based on the first calibration images (multiple first calibration images) corresponding to the multiple initial desired poses, that is, calibrating the candidate camera parameters for the reference camera.

[0082] For example, for a key point on a calibration plate (such as one or more key points), the key point may be a black and white intersection point (for a calibration plate with a checkerboard pattern), the key point may be a target center point (for a calibration plate with a target pattern), or other position points, without limitation.

[0083] For each key point, the physical coordinates of the key point in the world coordinate system can be determined, and the pixel coordinates corresponding to the key point in the first calibration image can be determined, thereby forming a coordinate point pair for the key point, which can include the physical coordinates of the key point and the pixel coordinates of the key point.

[0084] For example, when the robot moves to the initial desired posture 1, the physical coordinates of the key point on the calibration plate in the world coordinate system can be determined, and the pixel coordinates corresponding to the key point in the first calibration image 1 can be determined, thereby forming a coordinate point pair corresponding to the key point. When the robot moves to the initial desired posture 2, the physical coordinates of the key point on the calibration plate in the world coordinate system can be determined, and the pixel coordinates corresponding to the key point in the first calibration image 2 can be determined, thereby forming a coordinate point pair corresponding to the key point, and so on.

[0085] In summary, multiple coordinate point pairs under multiple initial desired poses can be obtained, each coordinate point pair includes physical coordinates and pixel coordinates, and then candidate camera parameters are determined based on the multiple coordinate point pairs.

[0086] For example, the relationship between the world coordinate system and the image coordinate system can be expressed by the following function: Y=f(X) , fIt is related to camera parameters (such as camera intrinsic parameters, distortion coefficients and camera extrinsic parameters). Therefore, the physical coordinates in the coordinate point pair can be substituted into the above function as Y, and the pixel coordinates in the coordinate point pair can be substituted into the above function as X, so as to obtain equations related to camera parameters. After performing the above operations on multiple coordinate point pairs, multiple equations can be obtained. By solving these equations, candidate camera parameters can be obtained, and the candidate camera parameters can include camera intrinsic parameters, distortion coefficients and camera extrinsic parameters.

[0087] Regarding the calibration process of the candidate camera parameters, there is no limitation in this embodiment, and the calibration of the candidate camera parameters can be completed through multiple coordinate point pairs. In the calibration process of the candidate camera parameters, if there are multiple reference cameras, each reference camera can be calibrated one by one, and then a reference camera is fixed, and the remaining reference cameras are jointly calibrated with the reference camera (i.e., the camera extrinsic parameters are calibrated).

[0088] Step 205 : Obtain a third transformation matrix between the camera coordinate system and the robotic arm coordinate system, where the third transformation matrix includes a target rotation matrix and a target translation matrix between the camera coordinate system and the robotic arm coordinate system.

[0089] In step 202, the initial rotation matrix and the initial translation matrix between the camera coordinate system and the manipulator coordinate system are used to determine the initial desired posture, and the initial rotation matrix and the initial translation matrix are rough values, and there are certain errors, which need to be corrected. Based on this, in step 205, the target rotation matrix and the target translation matrix between the camera coordinate system and the manipulator coordinate system can be obtained, and the target rotation matrix and the target translation matrix are accurate values, which can represent the accurate posture relationship between the camera coordinate system and the manipulator coordinate system.

[0090] For example, see Figure 4 The figure shows the schematic diagram of the camera coordinate system and the robot coordinate system. base can represent the robot coordinate system (also called the base coordinate system of the robot), gripper can represent the end coordinate system (that is, the coordinate system of the end of the robot, which is the coordinate system of the initial desired pose mentioned above), camera can represent the camera coordinate system, and target can represent the calibration plate coordinate system. The reference camera is fixed in the world and remains stationary, while the calibration plate is fixed at the end of the robot and moves with the movement of the robot.

[0091] See also Figure 4 As shown, Represents the transformation matrix between the robot coordinate system and the end coordinate system (such as rotation matrix and translation matrix), Represents the transformation matrix between the calibration plate coordinate system and the terminal coordinate system (such as rotation matrix and translation matrix), Represents the transformation matrix between the calibration plate coordinate system and the camera coordinate system (such as rotation matrix and translation matrix), Represents the transformation matrix between the camera coordinate system and the robot arm coordinate system (such as rotation matrix and translation matrix), that is It is the third transformation matrix that needs to be calibrated in the end.

[0092] Based on the relationship between each transformation matrix, when the robot arm is in any position, the following relationship can be satisfied: Since the calibration plate is fixed at the end of the robot arm and moves with the movement of the robot arm, no matter where the robot arm is, the conversion matrix between the calibration plate coordinate system and the end coordinate system is remain unchanged, that is, , (1) represents the transformation matrices when the robot is in position 1, and (2) represents the transformation matrices when the robot is in position 2.

[0093] By transforming the above equation, we can get the following equation: In other words, the above equation is simplified by multiplying the left and right sides respectively, and the form of AX=XB is obtained. , you can also , you can also .

[0094] In summary, the matrix X can be calculated based on the matrix A and the matrix B, and the matrix X is the transformation matrix between the camera coordinate system and the robot coordinate system. In addition, the matrix A is the transformation matrix between the robot coordinate system and the end coordinate system, and the matrix B is the transformation matrix between the calibration plate coordinate system and the camera coordinate system.

[0095] In summary, it can be seen that the third transformation matrix (matrix X) between the camera coordinate system and the manipulator coordinate system can be determined based on the first transformation matrix (i.e., matrix A) between the manipulator coordinate system and the end coordinate system and the second transformation matrix (i.e., matrix B) between the calibration plate coordinate system and the camera coordinate system, and the third transformation matrix can include the target rotation matrix and target translation matrix between the camera coordinate system and the manipulator coordinate system.

[0096] For the first transformation matrix between the robot coordinate system and the end coordinate system, when the initial expected posture of the robot is obtained, since the initial expected posture represents the final posture of the robot, that is, the robot is in the initial expected posture, the first transformation matrix can be determined based on the initial expected posture.

[0097] For example, (1) in matrix A represents the first transformation matrix determined based on the initial desired posture 1, and (2) in matrix A represents the first transformation matrix determined based on the initial desired posture 2.

[0098] For the second transformation matrix between the calibration plate coordinate system and the camera coordinate system, when the first calibration image of the calibration plate is obtained, the second transformation matrix is ​​determined based on the physical coordinates in the calibration plate coordinate system and the pixel coordinates in the first calibration image. For example, the coordinate point pairs include the physical coordinates of the key points on the calibration plate in the calibration plate coordinate system and the pixel coordinates of the key points in the first calibration image. The second transformation matrix between the calibration plate coordinate system and the camera coordinate system can be calculated based on multiple coordinate point pairs, and there is no restriction on this process.

[0099] For example, the first calibration image a1 corresponds to the initial desired pose 1, and (1) in the matrix B represents the second transformation matrix determined based on the first calibration image a1. The first calibration image a2 corresponds to the initial desired pose 2, and (2) in the matrix B represents the second transformation matrix determined based on the first calibration image a2.

[0100] In summary, it can be seen that based on at least two sets of transformation matrices (such as the first transformation matrix and the second transformation matrix), the third transformation matrix between the camera coordinate system and the robot arm coordinate system can be obtained. When the data of the transformation matrix is ​​more, a more accurate third transformation matrix can be obtained. This embodiment takes two sets of transformation matrices as an example.

[0101] Step 206: Determine multiple target desired poses of the robotic arm based on the candidate camera parameters of the reference camera.

[0102] For example, for multiple initial expected postures of the robot arm, these initial expected postures are not accurate, and each initial expected posture (i.e., the expected posture of each point position) can be adjusted to obtain the target expected posture corresponding to each initial expected posture. For example, based on the first calibration image under each initial expected posture, the candidate camera parameters of the reference camera, and the transformation matrix between the camera coordinate system and the robot arm coordinate system, multiple initial expected postures can be adjusted to obtain multiple target expected postures (i.e., final postures) of the robot arm.

[0103] In a possible implementation, the following steps may be used to determine the desired positions of multiple targets: Step S31, for each expected data (for the convenience of description, one expected data is taken as an example below), determine the pixel distance between the expected pixel coordinates in the expected data and the actual pixel coordinates of the specific position of the calibration plate on the first calibration image, that is, the pixel distance between the two pixel coordinates.

[0104] For example, the expected pixel coordinates are Indicates the pixel coordinates of the specific position of the calibration plate on the image, which are the expected pixel coordinates of the specific position on the image. After obtaining the first calibration image (i.e., the first calibration image under the initial expected posture corresponding to the expected data), the actual pixel coordinates of the specific position of the calibration plate on the first calibration image can be determined. Then, the desired pixel coordinates can be calculated The actual pixel coordinates The pixel distance between For example, the pixel distance can be determined by the following formula : .

[0105] Step S32: based on the candidate camera parameters and the expected distance in the expected data, convert the pixel distance into a physical distance. For example, the candidate camera parameters may include candidate focal length parameters (fx, fy), the expected distance in the expected data may be d, and the pixel distance may be converted into a physical distance by combining the expected distance d and the candidate focal length parameters (fx, fy). Convert to physical distance , physical distance It can represent the distance in the world coordinate system. For example, the pixel distance can be expressed as Convert to physical distance : .

[0106] Step S33: Based on the target rotation matrix between the camera coordinate system and the robot arm coordinate system, the physical distance Converted to the position correction value in the robot coordinate system .

[0107] For example, the third transformation matrix between the camera coordinate system and the robot coordinate system includes the target rotation matrix and the target translation matrix, and the physical distance It can be the distance in the camera coordinate system. The physical distance in the camera coordinate system can be converted based on the target rotation matrix. Converted to the position correction value in the robot coordinate system For example, the physical distance is calculated using the following formula Convert to position correction : In the above formula, Represents the target rotation matrix between the camera coordinate system and the robot arm coordinate system.

[0108] Step S34: correcting the initial expected posture (i.e., the initial expected posture corresponding to the expected data) based on the position correction amount to obtain the target expected posture corresponding to the expected data.

[0109] For example, based on the position correction, the target expected posture can be determined using the following formula: In the above formula, represents the target desired position, represents the initial desired pose, Indicates the position correction amount.

[0110] For example, steps S31 to S33 may be used to determine position correction amount 1 corresponding to expected data 1, and the initial expected posture corresponding to expected data 1 may be corrected based on position correction amount 1 to obtain the target expected posture corresponding to expected data 1. Steps S31 to S33 may be used to determine position correction amount 2 corresponding to expected data 2, and the initial expected posture corresponding to expected data 2 may be corrected based on position correction amount 2 to obtain the target expected posture corresponding to expected data 2, and so on, so that the target expected posture corresponding to each expected data may be obtained, that is, multiple target expected postures of the robot arm may be obtained.

[0111] In a possible implementation, the following steps may be used to determine the desired positions of multiple targets: Step S41 : for each expected data (hereinafter, one expected data is taken as an example), a dedistortion operation is performed on the expected pixel coordinates in the expected data based on the candidate camera parameters to obtain undistorted pixel coordinates.

[0112] Step S42: determining an expected position (denoted as expected position A) of the calibration plate in the camera coordinate system based on the expected distance d in the expected data, the distortion-free pixel coordinates and the candidate camera parameters.

[0113] Step S43, based on the expected position A, the target rotation matrix and the target translation matrix between the camera coordinate system and the robot coordinate system, determine the expected position of the calibration plate in the robot coordinate system (referred to as the expected position B).

[0114] Step S44, determine the third posture based on the first posture of the robot arm in the robot arm coordinate system and the expected position B of the calibration plate in the robot arm coordinate system. The first posture is the posture of the robot arm when the calibration plate is in the initial state, and the third posture is the intermediate posture to be optimized. The third posture needs to be optimized later.

[0115] Step S45, adjusting the posture in the third posture based on the randomly acquired rotation angle to obtain the target expected posture of the robot arm, that is, obtaining the target expected posture corresponding to the expected data.

[0116] Step S41-step S45 can refer to step S22-step S26, the difference is: the initial camera parameters are replaced by candidate camera parameters, the initial rotation matrix is ​​replaced by the target rotation matrix, and the initial translation matrix is ​​replaced by the target translation matrix. The other processing processes are similar and will not be repeated here.

[0117] Exemplarily, for each expected data, steps S41 to S45 may be used to obtain the target expected posture corresponding to the expected data, thereby obtaining multiple target expected postures of the robotic arm.

[0118] In a possible implementation, the following steps may be used to determine the desired positions of multiple targets: Step S51: for each expected data, determine the pixel distance between the expected pixel coordinates in the expected data and the actual pixel coordinates of the specific position of the calibration plate on the first calibration image.

[0119] Step S52: based on the candidate camera parameters and the expected distance in the expected data, convert the pixel distance (ie, the distance between two pixel coordinates) into a physical distance.

[0120] Step S53: based on the target rotation matrix between the camera coordinate system and the robot coordinate system, convert the physical distance into a position correction value in the robot coordinate system.

[0121] Step S54: correct the initial expected posture corresponding to the expected data based on the position correction amount to obtain a first expected posture corresponding to the expected data.

[0122] Step S55: performing a dedistortion operation on the expected pixel coordinates in the expected data based on the candidate camera parameters to obtain undistorted pixel coordinates corresponding to the expected pixel coordinates.

[0123] Step S56: determining an expected position A of the calibration plate in the camera coordinate system based on the expected distance d in the expected data, the distortion-free pixel coordinates and the candidate camera parameters.

[0124] Step S57, determining the expected position B of the calibration plate in the robotic arm coordinate system based on the expected position A, the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system.

[0125] Step S58: Determine the third posture of the robotic arm based on the first posture of the robotic arm in the robotic arm coordinate system and the expected position B of the calibration plate in the robotic arm coordinate system.

[0126] Step S59: adjusting the posture in the third posture based on the randomly acquired rotation angle to obtain a second expected posture of the robot arm, that is, obtaining a second expected posture corresponding to the expected data.

[0127] In summary, the first expected posture and the second expected posture corresponding to each expected data can be obtained.

[0128] Step S60: project the calibration plate in the first desired posture onto the image of the reference camera (ie, the first calibration image) to obtain the first actual pixel coordinates of the specific position of the calibration plate on the image.

[0129] For example, based on the first desired posture, the robot arm is controlled to move the calibration plate, that is, the robot arm is controlled to move to the first desired posture. After the robot arm moves to the first desired posture, the first physical coordinate of the specific position of the calibration plate (such as the origin of the calibration plate) in the robot arm coordinate system is determined. .

[0130] Then, based on the target rotation matrix between the camera coordinate system and the robot arm coordinate system and the target translation matrix , the first physical coordinate Converted into the second physical coordinate of the specific position of the calibration plate in the camera coordinate system For example, the first physical coordinate can be converted into Convert to the second physical coordinate : .

[0131] Based on the second physical coordinate Determine the first actual pixel coordinate of the specific position of the calibration plate on the image , that is, the first actual pixel coordinate in the image coordinate system For example, the first actual pixel coordinates can be determined using the following formula: : . (fx, fy) is the focal length parameter in the candidate camera parameters, and (cx, cy) is the principal point coordinate parameter in the candidate camera parameters.

[0132] Step S61: project the calibration plate in the second desired posture onto the image of the reference camera (ie, the first calibration image) to obtain a second actual pixel coordinate of a specific position of the calibration plate on the image.

[0133] For example, based on the second desired posture, the robot arm is controlled to move the calibration plate, that is, the robot arm is controlled to move to the second desired posture. After the robot arm moves to the second desired posture, the third physical coordinate of the specific position (such as the origin) of the calibration plate in the robot arm coordinate system is determined. Based on the target rotation matrix between the camera coordinate system and the robot arm coordinate system and the target translation matrix , converting the third physical coordinate into a fourth physical coordinate of the specific position of the calibration plate in the camera coordinate system, and determining the second actual pixel coordinate of the specific position of the calibration plate on the image based on the fourth physical coordinate, that is, the second actual pixel coordinate in the image coordinate system.

[0134] Step S62: Calculate a first pixel distance between the first actual pixel coordinate and the expected pixel coordinate in the expected data, and calculate a second pixel distance between the second actual pixel coordinate and the expected pixel coordinate.

[0135] Step S63: If the first pixel distance is less than the second pixel distance, the first expected posture is determined as the target expected posture corresponding to the expected data. If the second pixel distance is less than the first pixel distance, the second expected posture is determined as the target expected posture corresponding to the expected data. If the first pixel distance is equal to the second pixel distance, the first expected posture or the second expected posture is determined as the target expected posture corresponding to the expected data.

[0136] Exemplarily, for each expected data, steps S51 to S63 may be used to obtain the target expected posture corresponding to the expected data, thereby obtaining multiple target expected postures of the robotic arm.

[0137] At this point, step 206 is completed, and multiple target expected postures of the robot arm can be obtained. Obviously, after step 206, the target expected posture acquisition process is completed, and then the camera parameter calibration process is performed based on the multiple target expected postures. In the camera parameter calibration process, multiple cameras to be calibrated can be placed in the specified position in sequence, and for the currently placed camera to be calibrated, the camera parameters of the camera to be calibrated can be calibrated based on the multiple target expected postures, so that the camera parameters of multiple cameras to be calibrated can be calibrated.

[0138] For example, for the camera parameter calibration process, the camera parameters of each camera to be calibrated can be calibrated in turn, and the camera parameter calibration process of a camera to be calibrated is described below. Figure 5 FIG. 1 is a flow chart of a camera parameter calibration process, which may include: Step 501: for each target expected posture, control the robot arm to move the calibration plate based on the target expected posture, and collect a second calibration image for the calibration plate through the camera to be calibrated.

[0139] For example, based on the target desired posture 1, the robot arm is controlled to move the calibration plate, that is, the robot arm moves to the target desired posture 1, and the calibration plate moves synchronously with the robot arm, and the positional relationship between the calibration plate and the robot arm remains unchanged. After the robot arm moves to the target desired posture 1, the camera to be calibrated collects the second calibration image b1 for the calibration plate. After the robot arm moves to the target desired posture 2, the camera to be calibrated collects the second calibration image b2 for the calibration plate, and so on, multiple second calibration images can be obtained.

[0140] Step 502: determine target camera parameters of the camera to be calibrated based on the second calibration images (multiple second calibration images) corresponding to the multiple target expected postures, that is, calibrate the target camera parameters for the camera to be calibrated.

[0141] For example, for the key point on the calibration plate, the key point can be the intersection of black and white, the key point can be the center point of the target, or other position points, without limitation. For each key point, determine the physical coordinates of the key point in the world coordinate system, and determine the pixel coordinates corresponding to the key point in the second calibration image, thereby forming a coordinate point pair for the key point, which can include the physical coordinates of the key point and the pixel coordinates of the key point. In this way, multiple coordinate point pairs under multiple target expected postures can be obtained, and then the target camera parameters can be determined based on the multiple coordinate point pairs.

[0142] For example, the relationship between the world coordinate system and the image coordinate system can be expressed by the following function: Y=f(X) , f It is related to camera parameters (such as camera intrinsic parameters, distortion coefficients and camera extrinsic parameters). By substituting the physical coordinates and pixel coordinates in the coordinate point pair into the above function, the target camera parameters can be solved. The target camera parameters can include camera intrinsic parameters, distortion coefficients and camera extrinsic parameters. There is no restriction on this calibration process.

[0143] It can be seen from the above technical solutions that in the embodiments of the present application, it is possible to realize automatic calibration of camera parameters (such as camera parameters of RGBD cameras), with simple calibration operation, short calibration time and low calibration labor cost. It is possible to automatically generate the desired point positions and postures of the robotic arm (i.e., multiple initial desired postures) without the need for manual testing and calibration of the desired point positions and postures, thereby reducing the testing and calibration of the desired point positions and postures, improving automation, and automatically generating and verifying the robotic arm postures as desired, thereby reducing interference from human factors. Based on the known camera lens parameters, expected data, and the positional relationship between the camera and the robotic arm, it is possible to accurately determine the desired point positions and postures, thereby completing the calibration of the camera, which can significantly improve convenience, accuracy, and consistency.

[0144] Based on the same application concept as the above method, the embodiment of the present application proposes a camera parameter calibration device, wherein the calibration plate is fixedly connected to the robot arm, and the calibration plate moves synchronously with the robot arm, see Figure 6 As shown, it is a schematic diagram of the structure of the device, and the device includes: an initial expected posture determination module 61, which is used to determine multiple initial expected postures of the robot arm based on the initial camera parameters of the camera; a first control module 62, which is used to control the robot arm to move the calibration plate based on each initial expected posture, and collect a first calibration image for the calibration plate through the camera; a target expected posture determination module 63, which is used to determine the candidate camera parameters of the camera based on the first calibration images corresponding to the multiple initial expected postures; determine multiple target expected postures of the robot arm based on the candidate camera parameters; wherein, the multiple target expected postures are used to determine the target camera parameters and calibrate the target camera parameters.

[0145] Exemplarily, the initial expected posture determination module 61 is also used to determine the initial distortion parameters of the camera based on the horizontal field of view angle, vertical field of view angle, optical distortion coefficient and TV distortion coefficient of the camera before determining multiple initial expected postures of the robotic arm based on the initial camera parameters; determine the initial focal length parameters of the camera based on the focal length of the camera lens and the pixel size of the camera sensor; determine the initial principal point coordinate parameters of the camera based on the imaging resolution of the camera; and obtain the initial camera parameters of the camera; wherein the initial camera parameters include the initial distortion parameters, the initial focal length parameters and the initial principal point coordinate parameters.

[0146] Exemplarily, when the initial expected posture determination module 61 determines multiple initial expected postures of the robotic arm based on the initial camera parameters of the camera, it is specifically used to: obtain multiple expected data, each expected data includes an expected distance and an expected pixel coordinate; the expected distance represents the distance between the calibration plate and the camera, and the expected pixel coordinate represents the pixel coordinate of a specific position of the calibration plate on the image; for each expected data, perform a dedistortion operation on the expected pixel coordinates in the expected data based on the initial camera parameters to obtain distortion-free pixel coordinates; determine a first expected position of the calibration plate in the camera coordinate system based on the expected distance, the distortion-free pixel coordinates and the initial camera parameters in the expected data; and determine the initial expected posture corresponding to the expected data based on the first expected position.

[0147] Exemplarily, when the initial expected posture determination module 61 determines the initial expected posture corresponding to the expected data based on the first expected position, it is specifically used to: determine the second expected position of the calibration plate in the robotic arm coordinate system based on the first expected position, the initial rotation matrix and the initial translation matrix between the camera coordinate system and the robotic arm coordinate system; determine the second posture based on the first posture of the robotic arm in the robotic arm coordinate system and the second expected position, the first posture being the robotic arm posture in the initial state of the calibration plate; adjust the posture in the second posture based on the randomly acquired rotation angle to obtain the initial expected posture of the robotic arm; the second posture includes position and posture.

[0148] Exemplarily, when the target expected posture determination module 63 determines multiple target expected postures of the robot arm based on the candidate camera parameters, it is specifically used to: determine the pixel distance between the expected pixel coordinates in the expected data and the actual pixel coordinates of the specific position on the first calibration image based on the initial expected posture corresponding to each expected data; determine the position correction amount based on the candidate camera parameters and the pixel distance; correct the initial expected posture based on the position correction amount to obtain the target expected posture corresponding to the expected data; or, for each expected data, perform a dedistortion operation on the expected pixel coordinates in the expected data based on the candidate camera parameters to obtain undistorted pixel coordinates; determine the expected position of the calibration plate in the camera coordinate system based on the expected distance in the expected data, the undistorted pixel coordinates and the candidate camera parameters, and determine the corresponding position of the expected data based on the expected position. The target expected posture; or, based on the initial expected posture corresponding to each expected data, the initial expected posture is corrected by the acquired position correction amount to obtain a first expected posture; the second expected posture is determined by the acquired expected position; the calibration plate under the first expected posture is projected onto the image of the camera to obtain the first actual pixel coordinates of the specific position on the image, and the calibration plate under the second expected posture is projected onto the image of the camera to obtain the second actual pixel coordinates of the specific position on the image; if the first pixel distance between the first actual pixel coordinate and the expected pixel coordinate in the expected data is less than the second pixel distance between the second actual pixel coordinate and the expected pixel coordinate, the first expected posture is determined as the target expected posture corresponding to the expected data; if the second pixel distance is less than the first pixel distance, the second expected posture is determined as the target expected posture corresponding to the expected data.

[0149] When the target expected posture determination module 63 determines the position correction amount based on the candidate camera parameters and the pixel distance, it is specifically used to: convert the pixel distance into a physical distance based on the candidate camera parameters and the expected distance in the expected data; convert the physical distance into the position correction amount in the robotic arm coordinate system based on the target rotation matrix between the camera coordinate system and the robotic arm coordinate system; when the target expected posture determination module 63 determines the target expected posture corresponding to the expected data based on the expected position, it is specifically used to: determine the expected position of the calibration plate in the robotic arm coordinate system based on the expected position, the target rotation matrix between the camera coordinate system and the robotic arm coordinate system, and the target translation matrix; and determine the target expected posture based on the expected position of the calibration plate in the robotic arm coordinate system.

[0150] Exemplarily, the target expected posture determination module 63 projects the calibration plate in the first expected posture to the image of the camera, and obtains the first actual pixel coordinates of the specific position on the image, which is specifically used to: control the robotic arm to move the calibration plate based on the first expected posture, and determine the first physical coordinates of the specific position of the calibration plate in the robotic arm coordinate system; based on the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system, convert the first physical coordinates into the second physical coordinates of the specific position in the camera coordinate system; and determine the first actual pixel coordinates of the specific position on the image based on the second physical coordinates. The target expected posture determination module 63 projects the calibration plate under the second expected posture to the image of the camera, and obtains the second actual pixel coordinates of the specific position on the image, which is specifically used for: controlling the robotic arm to move the calibration plate based on the second expected posture, and determining the third physical coordinates of the specific position in the robotic arm coordinate system; based on the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system, converting the third physical coordinates into the fourth physical coordinates of the specific position in the camera coordinate system, and determining the second actual pixel coordinates of the specific position on the image based on the fourth physical coordinates.

[0151] Exemplarily, the device further includes: an acquisition module, used to acquire a target rotation matrix and a target translation matrix between the camera coordinate system and the manipulator coordinate system. Wherein, when the acquisition module acquires the target rotation matrix and the target translation matrix between the camera coordinate system and the manipulator coordinate system, it is specifically used to: determine a third transformation matrix between the camera coordinate system and the manipulator coordinate system based on a first transformation matrix between the manipulator coordinate system and the end coordinate system and a second transformation matrix between the calibration plate coordinate system and the camera coordinate system, wherein the third transformation matrix includes a target rotation matrix and a target translation matrix; Among them, when the initial expected posture of the robotic arm is obtained, the first transformation matrix is ​​determined based on the initial expected posture; when the first calibration image of the calibration plate is obtained, the second transformation matrix is ​​determined based on the physical coordinates in the calibration plate coordinate system and the pixel coordinates in the first calibration image.

[0152] Exemplarily, the camera is a reference camera, and the multiple target expected postures are used to determine the target camera parameters of the reference camera; or, the multiple target expected postures are used to determine the target camera parameters of multiple cameras to be calibrated, and the models of the multiple cameras to be calibrated are the same as the model of the reference camera; When determining the target camera parameters of the reference camera or the camera to be calibrated, the device also includes: a second control module, used to control the mechanical arm to move the calibration plate based on each target expected posture, and collect a second calibration image for the calibration plate through the reference camera or the camera to be calibrated; a camera parameter calibration module, used to determine the target camera parameters of the reference camera or the camera to be calibrated based on the second calibration images corresponding to multiple target expected postures.

[0153] Based on the same application concept as the above method, a control device is proposed in an embodiment of the present application, including: a transmitter, a receiver and a processor; wherein: The processor is used to obtain initial camera parameters of the camera, and determine multiple initial desired positions of the robot's mechanical arm based on the initial camera parameters of the camera; wherein the calibration plate is fixedly connected to the robot's mechanical arm, and the calibration plate moves synchronously with the mechanical arm; The transmitter is used to send a first control instruction to the robot and a second control instruction to the camera for each initial expected posture; the first control instruction is used to enable the robot to move the mechanical arm based on the initial expected posture, and the calibration plate to move synchronously based on the initial expected posture; the second control instruction is used to enable the camera to collect a first calibration image for the calibration plate; The receiver is used to receive a first calibration image captured by the camera; The processor is used to determine candidate camera parameters of the camera based on a first calibration image corresponding to a plurality of initial desired poses; and determine a plurality of target desired poses of the robotic arm based on the candidate camera parameters; wherein the plurality of target desired poses are used to determine target camera parameters and calibrate the target camera parameters.

[0154] Based on the same application concept as the above method, a control device is proposed in the embodiment of the present application, see Figure 7 As shown, the control device includes: a processor 71 and a machine-readable storage medium 72, the machine-readable storage medium 72 stores machine-executable instructions that can be executed by the processor 71; the processor 71 is used to execute the machine-executable instructions to implement the camera parameter calibration method disclosed in the above example of this application.

[0155] Based on the same application concept as the above method, an embodiment of the present application further provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the camera parameter calibration method disclosed in the above example of the present application can be implemented.

[0156] The above-mentioned machine-readable storage medium may be any electronic, magnetic, optical or other physical storage device, which may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof.

[0157] Based on the same application concept as the above method, an embodiment of the present application further provides a computer program product, which may include a computer program. When the computer program is executed by a processor, the camera parameter calibration method disclosed in the above example can be implemented.

[0158] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0159] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A camera parameter calibration method, characterized in that: The calibration plate is fixedly connected to the mechanical arm of the robot, and the calibration plate and the mechanical arm move synchronously. The method includes: Determining a plurality of initial desired positions of the robotic arm based on initial camera parameters of the camera; For each initial desired posture, controlling the mechanical arm to move the calibration plate based on the initial desired posture, and acquiring a first calibration image for the calibration plate through the camera; Determining candidate camera parameters of the camera based on first calibration images corresponding to a plurality of initial desired poses; Based on the candidate camera parameters, a plurality of target desired postures of the robotic arm are determined; wherein the plurality of target desired postures are used to determine target camera parameters and calibrate the target camera parameters.

2. The method according to claim 1, characterized in that Before determining a plurality of initial desired positions of the robotic arm based on the initial camera parameters of the camera, the process of acquiring the initial camera parameters includes: Determine the initial distortion parameters of the camera based on the horizontal field of view angle, vertical field of view angle, optical distortion coefficient and TV distortion coefficient of the camera; determine the initial focal length parameters of the camera based on the focal length of the camera lens and the pixel size of the camera sensor; determine the initial principal point coordinate parameters of the camera based on the imaging resolution of the camera; Acquire initial camera parameters of the camera; wherein the initial camera parameters include the initial distortion parameters, the initial focal length parameters and the initial principal point coordinate parameters.

3. The method according to claim 1, characterized in that The determining of a plurality of initial desired positions of the robotic arm based on initial camera parameters of the camera comprises: Acquire a plurality of configured expected data, each of which includes an expected distance and an expected pixel coordinate; wherein the expected distance represents the distance between the calibration plate and the camera, and the expected pixel coordinate represents the pixel coordinate of a specific position of the calibration plate on the image; For each expected data, performing a dedistortion operation on the expected pixel coordinates in the expected data based on the initial camera parameters to obtain undistorted pixel coordinates; Determine a first expected position of the calibration plate in a camera coordinate system based on the expected distance in the expected data, the distortion-free pixel coordinates, and the initial camera parameters; An initial expected posture corresponding to the expected data is determined based on the first expected position.

4. The method according to claim 3, characterized in that The determining, based on the first expected position, an initial expected posture corresponding to the expected data includes: Determine a second desired position of the calibration plate in the robotic arm coordinate system based on the first desired position, an initial rotation matrix and an initial translation matrix between the camera coordinate system and the robotic arm coordinate system; Determine a second posture based on a first posture of the robotic arm in the robotic arm coordinate system and the second expected position, wherein the first posture is a posture of the robotic arm when the calibration plate is in an initial state; The posture in the second posture is adjusted based on the randomly acquired rotation angle to obtain an initial desired posture of the robotic arm; wherein the second posture includes a position and a posture.

5. The method according to claim 3, characterized in that: The step of determining a plurality of target desired poses of the robotic arm based on the candidate camera parameters comprises: Based on the initial expected posture corresponding to each expected data, determine the pixel distance between the expected pixel coordinates in the expected data and the actual pixel coordinates of the specific position on the first calibration image; determine the position correction amount based on the candidate camera parameters and the pixel distance; and correct the initial expected posture based on the position correction amount to obtain the target expected posture corresponding to the expected data; Alternatively, for each expected data, a dedistortion operation is performed on the expected pixel coordinates in the expected data based on the candidate camera parameters to obtain undistorted pixel coordinates; based on the expected distance in the expected data, the undistorted pixel coordinates and the candidate camera parameters, an expected position of the calibration plate in the camera coordinate system is determined, and based on the expected position, an expected target pose corresponding to the expected data is determined; Alternatively, based on the initial expected posture corresponding to each expected data, the initial expected posture is corrected by the acquired position correction amount to obtain a first expected posture; the second expected posture is determined by the acquired expected position; the calibration plate under the first expected posture is projected onto the image of the camera to obtain the first actual pixel coordinates of the specific position on the image, and the calibration plate under the second expected posture is projected onto the image of the camera to obtain the second actual pixel coordinates of the specific position on the image; If the first pixel distance between the first actual pixel coordinate and the expected pixel coordinate in the expected data is less than the second pixel distance between the second actual pixel coordinate and the expected pixel coordinate, the first expected posture is determined as the target expected posture corresponding to the expected data; if the second pixel distance is less than the first pixel distance, the second expected posture is determined as the target expected posture corresponding to the expected data.

6. The method according to claim 5, characterized in that The determining of the position correction amount based on the candidate camera parameters and the pixel distance comprises: Based on the candidate camera parameters and the expected distance in the expected data, the pixel distance is converted into a physical distance; based on the target rotation matrix between the camera coordinate system and the robot coordinate system, the physical distance is converted into the position correction value in the robot coordinate system; The step of determining the target expected posture corresponding to the expected data based on the expected position includes: Based on the expected position, the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system, the expected position of the calibration plate in the robotic arm coordinate system is determined, and the target expected posture is determined based on the expected position of the calibration plate in the robotic arm coordinate system.

7. The method according to claim 5, characterized in that The step of projecting the calibration plate in the first desired posture onto the image of the camera to obtain the first actual pixel coordinates of the specific position on the image includes: controlling the robotic arm to move the calibration plate based on the first desired posture to determine the first physical coordinates of the specific position of the calibration plate in the robotic arm coordinate system; converting the first physical coordinates into the second physical coordinates of the specific position in the camera coordinate system based on the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system; and determining the first actual pixel coordinates of the specific position on the image based on the second physical coordinates; The method of projecting the calibration plate in the second desired posture onto the image of the camera to obtain the second actual pixel coordinates of the specific position on the image includes: controlling the robotic arm to move the calibration plate based on the second desired posture to determine the third physical coordinates of the specific position in the robotic arm coordinate system; based on the target rotation matrix and the target translation matrix between the camera coordinate system and the robotic arm coordinate system, converting the third physical coordinates into the fourth physical coordinates of the specific position in the camera coordinate system, and determining the second actual pixel coordinates of the specific position on the image based on the fourth physical coordinates.

8. The method according to claim 6 or 7, characterized in that: The process of obtaining the target rotation matrix and the target translation matrix between the camera coordinate system and the robot arm coordinate system includes: Based on the first transformation matrix between the manipulator coordinate system and the end coordinate system and the second transformation matrix between the calibration plate coordinate system and the camera coordinate system, determine a third transformation matrix between the camera coordinate system and the manipulator coordinate system, the third transformation matrix including a target rotation matrix and a target translation matrix; Among them, when the initial expected posture of the robotic arm is obtained, the first transformation matrix is ​​determined based on the initial expected posture; when the first calibration image of the calibration plate is obtained, the second transformation matrix is ​​determined based on the physical coordinates in the calibration plate coordinate system and the pixel coordinates in the first calibration image.

9. The method according to any one of claims 1 to 7, characterized in that: The camera is a reference camera, and the multiple target expected postures are used to determine the target camera parameters of the reference camera; or, the multiple target expected postures are used to determine the target camera parameters of multiple cameras to be calibrated, and the models of the multiple cameras to be calibrated are the same as the model of the reference camera; When determining the target camera parameters of the reference camera or the camera to be calibrated, the method further includes: For each target expected posture, the mechanical arm is controlled to move the calibration plate based on the target expected posture, and a second calibration image for the calibration plate is acquired through the reference camera or the camera to be calibrated; The target camera parameters of the reference camera or the camera to be calibrated are determined based on the second calibration images corresponding to the multiple target expected postures.

10. A camera parameter calibration device, characterized in that: The calibration plate is fixedly connected to the mechanical arm of the robot, and the calibration plate moves synchronously with the mechanical arm. The device comprises: An initial desired posture determination module, used to determine a plurality of initial desired postures of the robotic arm based on initial camera parameters of the camera; A first control module is used for controlling the mechanical arm to move the calibration plate based on each initial expected posture, and collecting a first calibration image for the calibration plate through the camera; A target expected posture determination module is used to determine candidate camera parameters of the camera based on a first calibration image corresponding to a plurality of initial expected postures; and determine a plurality of target expected postures of the robotic arm based on the candidate camera parameters; wherein the plurality of target expected postures are used to determine target camera parameters and calibrate the target camera parameters.

11. A control device, characterized in that: include: A transmitter, a receiver, and a processor; wherein: The processor is used to obtain initial camera parameters of the camera, and determine multiple initial desired positions of the robot's mechanical arm based on the initial camera parameters of the camera; wherein the calibration plate is fixedly connected to the robot's mechanical arm, and the calibration plate moves synchronously with the mechanical arm; The transmitter is used to send a first control instruction to the robot and a second control instruction to the camera for each initial expected posture; the first control instruction is used to enable the robot to move the mechanical arm based on the initial expected posture, and the calibration plate to move synchronously based on the initial expected posture; the second control instruction is used to enable the camera to collect a first calibration image for the calibration plate; The receiver is used to receive a first calibration image captured by the camera; The processor is used to determine candidate camera parameters of the camera based on a first calibration image corresponding to a plurality of initial desired poses; and determine a plurality of target desired poses of the robotic arm based on the candidate camera parameters; wherein the plurality of target desired poses are used to determine target camera parameters and calibrate the target camera parameters.

Images