A camera calibration method, system, apparatus and electronic device

Abstract

This application provides a camera calibration method, system, device, and electronic device. The method includes: acquiring calibration board images captured by the camera to be calibrated in multiple poses at the end effector of a robotic arm; calculating initial intrinsic parameters of the camera to be calibrated based on the pixel coordinates and calibration coordinates of feature points on the calibration board; determining a first transformation relationship based on the pixel coordinates, calibration coordinates, and initial intrinsic parameters of the feature points on the calibration board, and determining a second transformation relationship based on the poses of the end effector of the robotic arm; determining a third and fourth transformation relationship based on the first and second transformation relationships; constructing a first optimization function with the initial intrinsic parameters, the third and fourth transformation relationships as first optimization terms, and the difference between the pixel coordinates and the first projected coordinates as the first optimization objective; solving the first optimization function to obtain the target intrinsic parameters. Applying the method provided in this application can improve the efficiency and accuracy of camera calibration.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and in particular to a camera calibration method, system, apparatus and electronic device. Background Technology

[0002] With the continuous development of industrialization and informatization, robotic arms are increasingly becoming an indispensable part of the production process for automated grasping, sorting, and assembly. Camera calibration establishes the correspondence between image pixel coordinates and spatial position coordinates. When a robotic arm performs operations on a target object, it first acquires an image of the target object using a camera. Then, based on the pixel coordinates of the target object in the image and the camera calibration results, it determines the spatial position coordinates of the target object and performs related operations on it. Therefore, the accuracy of camera calibration affects the positioning accuracy of the target object, and consequently, the operational accuracy of the robotic arm.

[0003] In related technologies, camera calibration is usually performed by manually placing calibration plates in different poses. This calibration process is cumbersome and has low efficiency and accuracy. Summary of the Invention

[0004] The purpose of this application is to provide a camera calibration method, system, apparatus, and electronic device to improve the accuracy of camera calibration. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide a camera calibration method, the method comprising:

[0006] The calibration plate image is obtained by taking pictures of the preset calibration plate when the camera to be calibrated is in multiple poses at the end of the target robotic arm; wherein the relative position of the camera to be calibrated and the preset calibration plate changes with the pose of the end of the robotic arm.

[0007] Based on the pixel coordinates of the calibration board feature points of the preset calibration board in each calibration board image, and the calibration coordinates of the calibration board feature points in the calibration board coordinate system of the preset calibration board, the initial intrinsic parameters of the camera to be calibrated are calculated.

[0008] For each calibration board image, based on the pixel coordinates of the feature points of the calibration board, the calibration coordinates, and the initial intrinsic parameters, a first transformation relationship between the calibration board coordinate system and the camera coordinate system of the camera to be calibrated is determined when the calibration board image is captured. Based on the pose of the end effector of the robotic arm when the calibration board image is captured, a second transformation relationship between the end effector coordinate system of the target robotic arm and the base coordinate system of the target robotic arm is determined when the calibration board image is captured.

[0009] Based on the calculated first and second transformation relationships, a third transformation relationship between the camera coordinate system and the first coordinate system, and a fourth transformation relationship between the calibration plate coordinate system and the second coordinate system are determined; wherein, the first and second coordinate systems are determined based on the relative positions of the camera to be calibrated, the preset calibration plate, and the end effector of the robotic arm, the first coordinate system is the base coordinate system and the second coordinate system is the end effector coordinate system, or, the first coordinate system is the end effector coordinate system and the second coordinate system is the base coordinate system;

[0010] Using the initial intrinsic parameters, the third transformation relationship, and the fourth transformation relationship as the first optimization terms, and the difference between the pixel coordinates of the calibration board feature points in the first image and the first projection coordinates of the first image as the first optimization objective, a first optimization function is constructed and solved to obtain the target intrinsic parameters of the camera to be calibrated that satisfy the first preset requirement; wherein, the first image is any one of the calibration board images, and the first projection coordinates are the pixel coordinates of the calibration coordinates projected onto the first image by the fourth transformation relationship, the second transformation relationship when the first image was captured, the third transformation relationship, and the initial intrinsic parameters.

[0011] Optionally, in one specific implementation, the number of cameras to be calibrated is multiple, and the method further includes:

[0012] A reference camera is determined among the cameras to be calibrated, and the initial extrinsic parameters of the non-reference camera are calculated based on the third transformation relationship of the reference camera and the third transformation relationship of each non-reference camera.

[0013] For each non-reference camera, the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera are used as the second optimization terms. The difference between the pixel coordinates of the calibration board feature points in the second image and the second projection coordinates of the second image is used as the second optimization objective. A second optimization function is constructed and solved to obtain the target extrinsic parameters of the non-reference camera that satisfy the second preset requirement. The second image is any one of the calibration board images captured by the non-reference camera, and the second projection coordinates are: the pixel coordinates of the calibration coordinates projected onto the second image by the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the second image, the third transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera.

[0014] Optionally, in one specific implementation, solving the first optimization function to obtain the target intrinsic parameters of the camera to be calibrated that make the first optimization objective satisfy the first preset requirement includes:

[0015] Solve the first optimization function to obtain the target intrinsic parameters of the camera to be calibrated that make the first optimization objective meet the first preset requirements, and the target transformation relationship between the camera coordinate system and the first coordinate system;

[0016] The step of calculating the initial extrinsic parameters of a non-reference camera based on the third transformation relationship of the reference camera and the third transformation relationship of each non-reference camera includes:

[0017] Based on the target transformation relationship of the reference camera and the target transformation relationship of each non-reference camera, calculate the initial extrinsic parameters of the non-reference camera;

[0018] The second optimization function is constructed using the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera as the second optimization terms, and the difference between the pixel coordinates of the calibration board feature points in the second image and the second projection coordinates of the second image as the second optimization objective. The second optimization function includes:

[0019] Using the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the target transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera as the second optimization terms, and the difference between the pixel coordinates of the calibration plate feature points in the second image and the second projection coordinates of the second image as the second optimization objective, a second optimization function is constructed; wherein, the second projection coordinates are: the pixel coordinates of the calibration coordinates projected onto the second image by the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the second image, the target transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera.

[0020] Optionally, in one specific implementation, the number of cameras to be calibrated is multiple, and the method further includes:

[0021] A reference camera is determined among the cameras to be calibrated, and for each non-reference camera, the initial extrinsic parameters of the non-reference camera are calculated according to the third transformation relationship of the reference camera and the third transformation relationship of the non-reference camera, so as to obtain the initial extrinsic parameters of each non-reference camera.

[0022] Using the initial extrinsic parameters of each non-reference camera, the target intrinsic parameters of each non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera as the third optimization terms, and the sum of the target differences corresponding to each non-reference camera as the third optimization objective, a third optimization function is constructed and solved to obtain the target extrinsic parameters of each non-reference camera, the target intrinsic parameters of each non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera that satisfy the third preset requirement of the third optimization objective;

[0023] The target difference value corresponding to each non-reference camera is: the difference between the pixel coordinates of the calibration board feature points in the third image and the third projection coordinates of the third image. The third image is any one of the calibration board images captured by the non-reference camera. The third projection coordinates are: the pixel coordinates of the calibration coordinates projected onto the third image by the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the third image, the third transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera.

[0024] Optionally, in one specific implementation, the first optimization function is a nonlinear least squares equation.

[0025] Optionally, in one specific implementation, if the camera to be calibrated is fixed at the end of the robotic arm and the preset calibration plate is fixed at a designated position outside the end of the robotic arm, then the first coordinate system is the end coordinate system and the second coordinate system is the base coordinate system;

[0026] If the preset calibration plate is fixed at the end of the robotic arm and the camera to be calibrated is fixed at a designated position outside the end of the robotic arm, then the first coordinate system is the base coordinate system and the second coordinate system is the end coordinate system.

[0027] Secondly, embodiments of this application provide a camera calibration system, the system comprising: a camera to be calibrated, a target robotic arm, a preset calibration plate, and a processing device;

[0028] The camera to be calibrated is used to acquire images of the preset calibration plate;

[0029] The target robotic arm includes a robotic arm base and a robotic arm end effector, and the robotic arm end effector can change into various poses. The relative position between the camera to be calibrated and the preset calibration plate can change with the pose of the robotic arm end effector.

[0030] The preset calibration plate is within the shooting range of the camera to be calibrated;

[0031] The processing device is used to implement any of the camera calibration methods described above.

[0032] Thirdly, embodiments of this application provide a camera calibration device, the device comprising:

[0033] The image acquisition module is used to acquire images of the calibration plate taken by the camera to be calibrated when the end of the target robotic arm is in multiple poses; wherein the relative position of the camera to be calibrated and the preset calibration plate changes with the pose of the end of the robotic arm.

[0034] The intrinsic parameter calculation module is used to calculate the initial intrinsic parameters of the camera to be calibrated based on the pixel coordinates of the calibration board feature points of the preset calibration board in each calibration board image, and the calibration coordinates of the calibration board feature points in the calibration board coordinate system of the preset calibration board.

[0035] The first determining module is used to determine, for each calibration board image, a first transformation relationship between the calibration board coordinate system and the camera coordinate system of the camera to be calibrated when the calibration board image is captured, based on the pixel coordinates of the feature points of the calibration board, the calibration coordinates, and the initial intrinsic parameters; and to determine, based on the pose of the end effector of the robotic arm when the calibration board image is captured, a second transformation relationship between the end effector coordinate system and the base coordinate system of the target robotic arm when the calibration board image is captured.

[0036] The second determining module is used to determine, based on a plurality of calculated first transformation relationships and a plurality of second transformation relationships, a third transformation relationship between the camera coordinate system and the first coordinate system, and a fourth transformation relationship between the calibration plate coordinate system and the second coordinate system; wherein, the first coordinate system and the second coordinate system are determined based on the relative positions of the camera to be calibrated, the preset calibration plate, and the end effector of the robotic arm, wherein the first coordinate system is the base coordinate system and the second coordinate system is the end effector coordinate system, or, the first coordinate system is the end effector coordinate system and the second coordinate system is the base coordinate system;

[0037] An intrinsic parameter optimization module is used to construct a first optimization function with the initial intrinsic parameters, the third transformation relationship, and the fourth transformation relationship as the first optimization terms, and the difference between the pixel coordinates of the calibration board feature points in the first image and the first projection coordinates of the first image as the first optimization objective. The module then solves the first optimization function to obtain the target intrinsic parameters of the camera to be calibrated that satisfy the first preset requirement. The first image is any one of the calibration board images, and the first projection coordinates are the pixel coordinates of the calibration coordinates projected onto the first image using the fourth transformation relationship, the second transformation relationship when the first image was captured, the third transformation relationship, and the initial intrinsic parameters.

[0038] Optionally, in one specific implementation, the number of cameras to be calibrated is multiple, and the device further includes:

[0039] The extrinsic parameter calculation module is used to determine the reference camera among the cameras to be calibrated, and to calculate the initial extrinsic parameters of the non-reference camera according to the third transformation relationship of the reference camera and the third transformation relationship of each non-reference camera;

[0040] An extrinsic parameter optimization module is used to, for each non-reference camera, take the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera as the second optimization terms, and take the difference between the pixel coordinates of the calibration board feature points in the second image and the second projection coordinates of the second image as the second optimization objective, construct a second optimization function, and solve the second optimization function to obtain the target extrinsic parameters of the non-reference camera that make the second optimization objective meet the second preset requirements; wherein, the second image is any image among the calibration board images captured by the non-reference camera, and the second projection coordinates are: the pixel coordinates of the calibration coordinates projected into the second image by the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the second image, the third transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera;

[0041] Optionally, in one specific implementation, the intrinsic parameter optimization module is specifically used for:

[0042] Solve the first optimization function to obtain the target intrinsic parameters of the camera to be calibrated that make the first optimization objective meet the first preset requirements, and the target transformation relationship between the camera coordinate system and the first coordinate system;

[0043] The external parameter calculation module is specifically used for:

[0044] Based on the target transformation relationship of the reference camera and the target transformation relationship of each non-reference camera, calculate the initial extrinsic parameters of the non-reference camera;

[0045] The extrinsic parameter optimization module is specifically used for:

[0046] Using the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the target transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera as the second optimization terms, and the difference between the pixel coordinates of the calibration plate feature points in the second image and the second projection coordinates of the second image as the second optimization objective, a second optimization function is constructed; wherein, the second projection coordinates are: the pixel coordinates of the calibration coordinates projected onto the second image through the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the second image, the target transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera;

[0047] Optionally, in one specific implementation, the number of cameras to be calibrated is multiple, and the device further includes:

[0048] A camera determination module is used to determine a reference camera among the cameras to be calibrated, and for each non-reference camera, calculate the initial extrinsic parameters of the non-reference camera according to the third transformation relationship of the reference camera and the third transformation relationship of the non-reference camera, so as to obtain the initial extrinsic parameters of each non-reference camera.

[0049] The function construction module is used to construct a third optimization function with the initial extrinsic parameters of each non-reference camera, the target intrinsic parameters of each non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera as the third optimization terms, and the sum of the target differences corresponding to each non-reference camera as the third optimization objective. The module then solves the third optimization function to obtain the target extrinsic parameters of each non-reference camera, the target intrinsic parameters of each non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera that satisfy the third preset requirement for the third optimization objective.

[0050] The target difference value corresponding to each non-reference camera is: the difference between the pixel coordinates of the calibration board feature points in the third image and the third projection coordinates of the third image. The third image is any one of the calibration board images captured by the non-reference camera. The third projection coordinates are: the pixel coordinates of the calibration coordinates projected onto the third image by the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the third image, the third transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera.

[0051] Optionally, in one specific implementation, the first optimization function is a nonlinear least squares equation.

[0052] Optionally, in one specific implementation, if the camera to be calibrated is fixed at the end of the robotic arm and the preset calibration plate is fixed at a designated position outside the end of the robotic arm, then the first coordinate system is the end coordinate system and the second coordinate system is the base coordinate system;

[0053] If the preset calibration plate is fixed at the end of the robotic arm and the camera to be calibrated is fixed at a designated position outside the end of the robotic arm, then the first coordinate system is the base coordinate system and the second coordinate system is the end coordinate system;

[0054] Fourthly, embodiments of this application provide an electronic device, including:

[0055] Memory, used to store computer programs;

[0056] The processor, when executing a program stored in memory, implements any of the camera calibration methods described above.

[0057] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the camera calibration methods described above.

[0058] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the camera calibration methods described above.

[0059] Beneficial effects of the embodiments in this application:

[0060] As can be seen above, the camera to be calibrated can capture images of a preset calibration plate, and the relative position between the camera and the preset calibration plate can change with the pose of the end effector of the target robotic arm. When calibrating the camera, the end effector of the robotic arm can be controlled to be in multiple different poses, and the camera to be calibrated can capture images of the preset calibration plate at each pose, thereby obtaining images of the preset calibration plate captured by the camera to be calibrated at various poses of the end effector of the target robotic arm.

[0061] Furthermore, based on the pixel coordinates of the calibration board feature points in each calibration board image, and the calibration coordinates of the calibration board feature points in the calibration board coordinate system of the preset calibration board, the initial intrinsic parameters of the camera to be calibrated can be calculated. Then, based on the pixel coordinates of the calibration board feature points, the calibration coordinates, and the initial intrinsic parameters of the camera to be calibrated, the following transformation relationships can be determined: a first transformation relationship between the calibration board coordinate system and the camera coordinate system of the camera to be calibrated; a second transformation relationship between the end effector coordinate system of the target robotic arm and the base coordinate system of the target robotic arm; a third transformation relationship between the camera coordinate system and the first coordinate system; and a fourth transformation relationship between the calibration board coordinate system and the second coordinate system. Wherein, the first coordinate system is the base coordinate system and the second coordinate system is the end effector coordinate system, or the first coordinate system is the end effector coordinate system and the second coordinate system is the base coordinate system.

[0062] After determining the initial intrinsic parameters and various transformation relationships, one image can be selected from the calibration board images as the first image. The pixel coordinates obtained by projecting the calibration coordinates onto the first image using the fourth transformation relationship, the second transformation relationship when capturing the first image, the third transformation relationship, and the initial intrinsic parameters can be used as the first projected coordinates of the calibration board feature points relative to the first image. Using the initial intrinsic parameters, the third transformation relationship, and the fourth transformation relationship as the first optimization terms, and the difference between the pixel coordinates of the calibration board feature points in the first image and the first projected coordinates relative to the first image as the first optimization objective, a first optimization function can be constructed. By solving the first optimization function, the target intrinsic parameters of the camera to be calibrated that satisfy the first preset requirement for the first optimization objective can be obtained.

[0063] Based on this, by applying the solution provided in the embodiments of this application, the relative position between the camera to be calibrated and the preset calibration plate can be changed by altering the pose of the robotic arm's end effector, thereby enabling the camera to acquire images of the calibration plate from different angles. Therefore, in the camera calibration process of the embodiments of this application, it is not necessary to manually place calibration plates in different poses, thus automating camera calibration, reducing the complexity of camera calibration work, and improving the efficiency of camera calibration.

[0064] Furthermore, since the pose accuracy of the robotic arm's end effector is high, the accuracy of the second transformation relationship between the target robotic arm's end effector coordinate system and the base coordinate system determined based on the end effector pose is also high. Consequently, the target intrinsic parameters obtained by optimizing the initial intrinsic parameters based on the second transformation relationship can also have high accuracy. Therefore, applying the solution provided in the embodiments of this application can also improve the calibration accuracy of camera calibration.

[0065] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0067] Figure 1 A schematic diagram of the world coordinate system, camera coordinate system, pixel coordinate system, and image coordinate system provided in the embodiments of this application;

[0068] Figure 2 A schematic diagram of the camera calibration system provided in the embodiments of this application;

[0069] Figure 3 A schematic flowchart of a camera calibration method provided in an embodiment of this application;

[0070] Figure 4(a) is a schematic diagram of a checkerboard calibration board provided in an embodiment of this application;

[0071] Figure 4(b) is a schematic diagram of a dot calibration plate provided in an embodiment of this application;

[0072] Figure 5(a) is a schematic diagram of the positional relationship between a target robotic arm, a camera to be calibrated, and a preset chessboard grid provided in an embodiment of this application;

[0073] Figure 5(b) is a schematic diagram of the transformation relationship between various coordinate systems provided in an embodiment of this application;

[0074] Figure 6(a) is a schematic diagram of the positional relationship between another target robotic arm, a camera to be calibrated, and a preset chessboard grid provided in an embodiment of this application;

[0075] Figure 6(b) is a schematic diagram of another transformation relationship between coordinate systems provided in an embodiment of this application;

[0076] Figure 7 Another schematic flowchart of the camera calibration method provided in the embodiments of this application;

[0077] Figure 8 This is a schematic diagram of the structure of a camera calibration device provided in an embodiment of this application;

[0078] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0079] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0080] In related technologies, camera calibration is usually performed by manually placing calibration plates in different poses. This calibration process is cumbersome and has low efficiency and accuracy.

[0081] To address the aforementioned issues, this application provides a camera calibration method.

[0082] This method is applicable to various camera calibration scenarios. For example, it can be used to calibrate a camera mounted at the end of the robotic arm when the positional relationship between the robotic arm and the camera is "eye on the hand"; it can also be used to calibrate a camera fixedly mounted at a designated position when the positional relationship is "eye outside the hand"; it can be used to calibrate a monocular camera; it can be used to calibrate a multi-view camera, and so on. The application scenarios of this application are not specifically limited.

[0083] Furthermore, the executing entity of this method can be any electronic device capable of acquiring and processing data such as images. This electronic device can be a camera with data processing capabilities, or any device capable of processing data that has a communication connection with both the camera and the robotic arm, such as a mobile phone, laptop, or desktop computer. Moreover, this electronic device can be a standalone device or a cluster of multiple electronic devices. This application does not specifically limit this aspect; the device will be referred to as an electronic device.

[0084] The camera used in this method can be any device with image acquisition capabilities; this application does not specifically limit its use. Furthermore, when the camera has data processing capabilities, it can also serve as the executing entity of this method.

[0085] This application provides a camera calibration method that may include the following steps:

[0086] The calibration plate image is obtained by taking pictures of the preset calibration plate when the camera to be calibrated is in multiple poses at the end of the target robotic arm; wherein the relative position of the camera to be calibrated and the preset calibration plate changes with the pose of the end of the robotic arm.

[0087] Based on the pixel coordinates of the calibration board feature points of the preset calibration board in each calibration board image, and the calibration coordinates of the calibration board feature points in the calibration board coordinate system of the preset calibration board, the initial intrinsic parameters of the camera to be calibrated are calculated.

[0088] For each calibration board image, based on the pixel coordinates of the feature points of the calibration board, the calibration coordinates, and the initial intrinsic parameters, a first transformation relationship between the calibration board coordinate system and the camera coordinate system of the camera to be calibrated is determined when the calibration board image is captured. Based on the pose of the end effector of the robotic arm when the calibration board image is captured, a second transformation relationship between the end effector coordinate system of the target robotic arm and the base coordinate system of the target robotic arm is determined when the calibration board image is captured.

[0089] Based on the calculated first and second transformation relationships, a third transformation relationship between the camera coordinate system and the first coordinate system, and a fourth transformation relationship between the calibration plate coordinate system and the second coordinate system are determined; wherein, the first and second coordinate systems are determined based on the relative positions of the camera to be calibrated, the preset calibration plate, and the end effector of the robotic arm, the first coordinate system is the base coordinate system and the second coordinate system is the end effector coordinate system, or, the first coordinate system is the end effector coordinate system and the second coordinate system is the base coordinate system;

[0090] Using the initial intrinsic parameters, the third transformation relationship, and the fourth transformation relationship as the first optimization terms, and the difference between the pixel coordinates of the calibration board feature points in the first image and the first projection coordinates of the first image as the first optimization objective, a first optimization function is constructed and solved to obtain the target intrinsic parameters of the camera to be calibrated that satisfy the first preset requirement; wherein, the first image is any one of the calibration board images, and the first projection coordinates are the pixel coordinates of the calibration coordinates projected onto the first image by the fourth transformation relationship, the second transformation relationship when the first image was captured, the third transformation relationship, and the initial intrinsic parameters.

[0091] As can be seen above, the camera to be calibrated can capture images of a preset calibration plate, and the relative position between the camera and the preset calibration plate can change with the pose of the end effector of the target robotic arm. When calibrating the camera, the end effector of the robotic arm can be controlled to be in multiple different poses, and the camera to be calibrated can capture images of the preset calibration plate at each pose, thereby obtaining images of the preset calibration plate captured by the camera to be calibrated at various poses of the end effector of the target robotic arm.

[0092] Furthermore, based on the pixel coordinates of the calibration board feature points in each calibration board image, and the calibration coordinates of the calibration board feature points in the calibration board coordinate system of the preset calibration board, the initial intrinsic parameters of the camera to be calibrated can be calculated. Then, based on the pixel coordinates of the calibration board feature points, the calibration coordinates, and the initial intrinsic parameters of the camera to be calibrated, the following transformation relationships can be determined: a first transformation relationship between the calibration board coordinate system and the camera coordinate system of the camera to be calibrated; a second transformation relationship between the end effector coordinate system of the target robotic arm and the base coordinate system of the target robotic arm; a third transformation relationship between the camera coordinate system and the first coordinate system; and a fourth transformation relationship between the calibration board coordinate system and the second coordinate system. Wherein, the first coordinate system is the base coordinate system and the second coordinate system is the end effector coordinate system, or the first coordinate system is the end effector coordinate system and the second coordinate system is the base coordinate system.

[0093] After determining the initial intrinsic parameters and various transformation relationships, one image can be selected from the calibration board images as the first image. The pixel coordinates obtained by projecting the calibration coordinates onto the first image using the fourth transformation relationship, the second transformation relationship when capturing the first image, the third transformation relationship, and the initial intrinsic parameters can be used as the first projected coordinates of the calibration board feature points relative to the first image. Using the initial intrinsic parameters, the third transformation relationship, and the fourth transformation relationship as the first optimization terms, and the difference between the pixel coordinates of the calibration board feature points in the first image and the first projected coordinates relative to the first image as the first optimization objective, a first optimization function can be constructed. By solving the first optimization function, the target intrinsic parameters of the camera to be calibrated that satisfy the first preset requirement for the first optimization objective can be obtained.

[0094] Based on this, by applying the solution provided in the embodiments of this application, the relative position between the camera to be calibrated and the preset calibration plate can be changed by altering the pose of the robotic arm's end effector, thereby enabling the camera to acquire images of the calibration plate from different angles. Therefore, in the camera calibration process of the embodiments of this application, it is not necessary to manually place calibration plates in different poses, thus automating camera calibration, reducing the complexity of camera calibration work, and improving the efficiency of camera calibration.

[0095] Furthermore, since the pose accuracy of the robotic arm's end effector is high, the accuracy of the second transformation relationship between the target robotic arm's end effector coordinate system and the base coordinate system determined based on the end effector pose is also high. Consequently, the target intrinsic parameters obtained by optimizing the initial intrinsic parameters based on the second transformation relationship can also have high accuracy. Therefore, applying the solution provided in the embodiments of this application can also improve the calibration accuracy of camera calibration.

[0096] To better understand the camera calibration method provided in this application, the relevant concepts involved in this application embodiment will first be explained.

[0097] The purpose of camera calibration is to determine the coordinates of an object in the world coordinate system based on its coordinates in the pixel coordinate system, thereby enabling object localization based on an image of the object. Typically, camera calibration can be divided into camera intrinsic parameter calibration and camera extrinsic parameter calibration. Camera intrinsic parameters represent the transformation relationship between the camera coordinate system and the pixel coordinate system. Based on the camera intrinsic parameters, the coordinates of an object in the camera coordinate system can be determined from its pixel coordinates. The camera calibration method provided in this application is mainly used for calibrating camera intrinsic parameters.

[0098] See Figure 1 This is a schematic diagram illustrating a world coordinate system, camera coordinate system, pixel coordinate system, and image coordinate system provided in an embodiment of this application. Figure 1 In the middle, O W -X W Y W Z W For the world coordinate system, O C -X C Y C Z C Let P be the camera coordinate system, uv be the pixel coordinate system, and o-xy be the image coordinate system. Wherein, point P(X...) W Y W Z W Let P be a point in the world coordinate system. The image point p of point P in the image has coordinates (u, v) in the pixel coordinate system and (x, y) in the image coordinate system.

[0099] The world coordinate system is a three-dimensional Cartesian coordinate system that describes the spatial position of a camera and an object (for example, point P). It reflects the position of objects in the real world. The origin of the world coordinate system can be determined based on the specific circumstances.

[0100] Optionally, in this embodiment, the world coordinate system can be constructed based on a calibration plate. For example, the origin of the world coordinate system can be the upper left corner feature point of the calibration plate, and the Y-axis of the world coordinate system can be... W -Y W The plane can coincide with the calibration plate plane, and the Z-axis of the world coordinate system... W The axis can pass through the plane of the calibration plate and point vertically upwards.

[0101] Optionally, in this embodiment of the application, a calibration plate coordinate system can be established based on a preset calibration plate. When the calibration plate coordinate system established based on the preset calibration plate coincides with the world coordinate system established based on the preset calibration plate, the world coordinate system established based on the preset calibration plate can be called the calibration plate coordinate system.

[0102] The camera coordinate system is a three-dimensional Cartesian coordinate system with its origin located at the optical center of the camera lens. The x and y axes are parallel to the two sides of the image plane, respectively, and the z-axis is the optical axis of the lens, perpendicular to the image plane. The image plane is the plane on which the camera forms an image.

[0103] The pixel coordinate system is a two-dimensional Cartesian coordinate system that reflects the arrangement of pixels in the camera's charge-coupled device (CCD). The origin of the pixel coordinate system is located at the top left corner of the image, and the u-axis and v-axis are parallel to the two sides of the image plane, respectively. The unit of the coordinate axes in the pixel coordinate system is pixels (integers).

[0104] The image coordinate system is a two-dimensional Cartesian coordinate system. Because the pixel coordinate system is not conducive to coordinate transformations, the image coordinate system was established. The units of its coordinate axes can be easily transformed from those of the world coordinate system and the camera coordinate system. The origin of the image coordinate system is the intersection of the camera's optical axis and the image plane (or the principal point), which is the center point of the image. The x-axis and y-axis are parallel to the u-axis and v-axis, respectively. Therefore, the pixel coordinate system and the image coordinate system can be considered as a translation relationship, meaning they can be obtained through translation. The difference lies in the units of the coordinate axes of the pixel coordinate system and the image coordinate system.

[0105] In industrial applications, the positional relationship between the robotic arm and the camera includes "eye on the hand" and "eye outside the hand." Here, "eye" refers to the camera, and "hand" refers to the robotic arm. As shown in Figure 5(a), "eye on the hand" means the camera is fixed to the end of the robotic arm and moves with it, with the relative position between the camera and the end of the robotic arm remaining constant. As shown in Figure 5(b), "eye outside the hand" means the camera is fixed, separated from the end of the robotic arm, and the relative position between the camera and the base of the robotic arm remains constant.

[0106] When using a robotic arm to grasp, sort, and assemble target objects, it is also necessary to obtain the position of the target object in the "hand" coordinate system. Through hand-eye calibration, the positional relationship between the "eye" coordinate system and the "hand" coordinate system can be obtained, thereby determining the position of the target object in the "hand" coordinate system based on its position in the "eye" coordinate system. In the camera calibration method provided in this application embodiment, the calculated third transformation relationship can be used as the result of hand-eye calibration. In hand-eye calibration, the "eye" is the camera; when the "eye" is on the hand, the first coordinate system can be determined based on the relative positions of the camera to be calibrated, the preset calibration plate, and the end effector of the robotic arm, meaning the "hand" in hand-eye calibration is the end effector of the robotic arm; when the "eye" is outside the hand, the first coordinate system can be determined as the robotic arm base.

[0107] Next, the system involved in the camera calibration method provided in the embodiments of this application will be described.

[0108] Figure 2This is a schematic diagram of the structure of a camera calibration system provided in an embodiment of this application, as shown below. Figure 2 As shown, the system may include: electronic device 201, camera to be calibrated 202, target robotic arm 203, and preset calibration plate 204.

[0109] See Figure 2 In this embodiment, the electronic device 201 has a communication connection with the camera 202 to be calibrated and the target robotic arm 203. Furthermore, the electronic device 201 can be directly or indirectly connected to the camera 202 to be calibrated and the target robotic arm 203 via wired or wireless communication, which is reasonable, and this embodiment does not impose specific limitations on this.

[0110] The aforementioned electronic device 201 can implement any of the steps of the camera calibration method applied to the electronic device 201 provided in the embodiments of this application.

[0111] The aforementioned camera 202 to be calibrated can acquire images of the preset calibration plate 204; and the aforementioned preset calibration plate 204 is within the shooting range of the camera 202 to be calibrated.

[0112] The aforementioned target robotic arm 203 can be composed of a robotic arm base 2031 and a robotic arm end effector 2032. The robotic arm end effector 2032 can change to various poses, and the relative position between the camera to be calibrated 202 and the preset calibration plate 204 can change with the pose of the robotic arm end effector 2032.

[0113] The aforementioned preset calibration board 204 can be a checkerboard calibration board, a dot calibration board, a CharuCo calibration board, etc. This application embodiment does not specifically limit the specific type of calibration board.

[0114] The following is a detailed description of a camera calibration method provided in an embodiment of this application, with reference to the accompanying drawings.

[0115] Figure 3 This is a schematic flowchart of a camera calibration method provided in an embodiment of this application, such as... Figure 3 As shown, the method may include the following steps S301-S305.

[0116] S301: Acquire images of the calibration board obtained by taking pictures of the preset calibration board when the camera to be calibrated is in multiple poses at the end of the target robotic arm.

[0117] The relative positions of the camera to be calibrated and the preset calibration plate can change with the pose of the robotic arm end effector.

[0118] When calibrating the camera to be calibrated, the relative position of the camera and the preset calibration plate can change with the pose of the end effector of the target robotic arm. Therefore, when the end effector is in different poses, the relative position of the camera to be calibrated and the preset calibration plate are different, and the image of the calibration plate captured by the camera to be calibrated is also different. Furthermore, when calibrating the camera, the end effector of the robotic arm can be in multiple different poses, and the camera to be calibrated can capture images of the preset calibration plate when the end effector of the robotic arm is in each pose, thereby obtaining multiple poses of the end effector of the robotic arm, and each image of the calibration plate captured by the camera to be calibrated when the end effector of the robotic arm is in each pose.

[0119] Among them, the calibration board images correspond one-to-one with the poses and the number is equal, and the multiple poses are all different.

[0120] The number of calibration board images and poses can be set by those skilled in the art according to actual application conditions, and this application embodiment does not specifically limit this.

[0121] Optionally, the aforementioned multiple poses can be multiple poses of the end effector of the robotic arm that are pre-set by those skilled in the art. During the camera calibration process, the end effector of the target robotic arm can change its pose according to the program pre-set by those skilled in the art, and the camera to be calibrated can take pictures of the preset calibration plate when the end effector of the robotic arm is in each pose.

[0122] Optionally, the aforementioned multiple poses can also be multiple poses of the end effector of the robotic arm obtained by a person skilled in the art through manipulation of the target robotic arm during the camera calibration process. The camera to be calibrated can take pictures of the preset calibration plate when the end effector of the robotic arm is in each pose.

[0123] S302: Calculate the initial intrinsic parameters of the camera to be calibrated based on the pixel coordinates of the calibration board feature points of the preset calibration board in each calibration board image, and the calibration coordinates of the calibration board feature points in the calibration board coordinate system of the preset calibration board.

[0124] Since the positions of the calibration board feature points within the preset calibration board are known, the calibration coordinates of each feature point can be determined within the calibration board coordinate system established based on the preset calibration board. Furthermore, after acquiring each calibration board image, the pixel coordinates of each feature point can be obtained using an image detection algorithm. Then, based on the pixel coordinates of the calibration board feature points in each calibration board image, and the calibration coordinates of these feature points within the preset calibration board coordinate system, the initial intrinsic parameters of the camera to be calibrated can be calculated.

[0125] Optionally, the aforementioned preset calibration plate may have multiple calibration plate feature points arranged in an equally spaced array. Since the distance between any two adjacent calibration plate feature points is known and equal, the calibration coordinates of each calibration plate feature point can be determined in the calibration plate coordinate system established based on the preset calibration plate.

[0126] Optionally, the aforementioned preset calibration board can be a chessboard calibration board as shown in Figure 4(a). Furthermore, the feature points of the calibration board can be the corner points of each chessboard grid in the chessboard calibration board. In this case, the corner points of each chessboard grid can also be called the corner points of the calibration board.

[0127] Optionally, the aforementioned preset calibration plate can be a dot calibration plate as shown in Figure 4(b), and the feature points of the calibration plate can be the centers of each dot in the dot calibration plate.

[0128] Optionally, in one specific implementation, step S302 may include the following step 1.

[0129] Step 1: Based on the pixel coordinates of the calibration board feature points of the preset calibration board in each calibration board image, and the calibration coordinates of the calibration board feature points in the calibration board coordinate system of the preset calibration board, calculate the initial intrinsic parameters of the camera to be calibrated using the Zhang Zhengyou calibration method.

[0130] After obtaining each calibration board image, the Zhang Zhengyou calibration method can be used to calculate the initial intrinsic parameters of the camera to be calibrated based on the pixel coordinates of the calibration board feature points in each calibration board image and the calibration coordinates of the calibration board feature points in the calibration board coordinate system of the preset calibration board.

[0131] The principle and process of calculating the initial intrinsic parameters of the camera to be calibrated using Zhang Zhengyou's calibration method can be described as follows.

[0132] The process of transforming the target object from the calibration plate coordinate system to the camera coordinate system can be achieved through rotation and translation. Furthermore, as shown in Equation 1, the transformation matrix in this process can be represented by a homogeneous coordinate matrix composed of the rotation matrix and the translation vector.

[0133] Formula 1:

[0134]

[0135] Where R is the rotation matrix, t is the translation vector, and X C Y C Z C X represents the coordinates in the camera coordinate system. W Y W Z W This indicates the coordinates in the calibration plate coordinate system.

[0136] Since the origin of the calibration plate coordinate system can be determined as the upper left corner feature point of the calibration plate, the plane of the calibration plate can be exactly aligned with the X-axis. W -Y W Plane coincidence, Z W The axis passes through the plane of the calibration plate and points vertically upwards; at this point, the Z-axis of all feature points on the calibration plate... W Setting it to 0 simplifies subsequent calculations. Therefore, as shown above, the condition for Z can be omitted. W The rotation amount of the shaft is r3.

[0137] The mapping process of the target object from the camera coordinate system to the pixel coordinate system can be represented by Equation 2.

[0138] Formula 2:

[0139]

[0140] Where u and v represent coordinates in the pixel coordinate system, s represents the scale factor, and f x f y u0 and v0 represent the camera intrinsic parameters.

[0141] Furthermore, the mapping relationship between the pixel coordinate system and the calibration plate coordinate system can be represented by Equation 3.

[0142] Formula 3:

[0143]

[0144] In the field of computer vision, the homography matrix H can be used to describe the positional mapping relationship between objects in the world coordinate system and the pixel coordinate system. Therefore, in summary, the homography matrix H can be represented as shown in Equation 4.

[0145] Formula 4:

[0146]

[0147] Here, M is the intrinsic parameter matrix of the camera. It can be seen that the homography matrix H contains both the transformation relationship [r1r2t] between the calibration board coordinate system and the camera coordinate system, and the intrinsic parameter matrix M between the camera coordinate system and the pixel coordinate system.

[0148] Since the internal structure of the camera remains unchanged, the intrinsic parameter matrix M is a constant for different calibration board images captured by the same camera. For the same calibration board image, since the positional relationship between the calibration board and the camera remains unchanged when the image is captured, the transformation relationship [r1r2t] between the calibration board coordinate system and the camera coordinate system, as well as the intrinsic parameter matrix M, are all constants. For a single feature point on the same calibration board image, the transformation relationship [r1r2t] between the calibration board coordinate system and the camera coordinate system, the intrinsic parameter matrix M, and the scale factor s are all constants.

[0149] Let the three columns of the homography matrix H be [H1H2H3], and we can obtain Equation 5.

[0150] Formula 5:

[0151]

[0152] Eliminating the scaling factor s, we can obtain Equation 6.

[0153] Formula 6:

[0154]

[0155]

[0156] Since the scale factor s has been eliminated, Equation 6 holds true for all feature points on the same calibration board image. Here, (u, v) can be the pixel coordinates of the calibration board feature points in the pixel coordinate system, (X... W Y W The coordinates of the calibration plate feature points can be the calibration coordinates in the calibration plate coordinate system. Since the homography matrix H is a homogeneous matrix with 8 independent unknown elements, each calibration plate feature point can provide two constraint equations as shown in Equation 6. Therefore, when the number of calibration plate feature points on a calibration plate image is equal to 4, the homography matrix H corresponding to that calibration plate image can be obtained. When the number of calibration plate feature points on a calibration plate image is greater than 4, the optimal homography matrix H can be regressed using the least squares method.

[0157] Since the rotation vectors are orthogonal to each other in the construction, r1 and r2 are orthogonal to each other. Furthermore, based on the characteristic that the rotation vectors r1 and r2 are orthogonal to each other and the characteristic that the intrinsic parameter matrix M corresponding to each calibration plate image is the same, the intrinsic parameter matrix M of the camera to be calibrated can be obtained using each calibration plate image and the homography matrix H obtained above. The intrinsic parameter matrix M can then be used as the initial intrinsic parameter of the camera to be calibrated.

[0158] S303: For each calibration board image, based on the pixel coordinates, calibration coordinates, and initial intrinsic parameters of the calibration board feature points, determine the first transformation relationship between the calibration board coordinate system and the camera coordinate system of the camera to be calibrated when the calibration board image is captured, and based on the pose of the end effector of the robotic arm when the calibration board image is captured, determine the second transformation relationship between the end effector coordinate system and the base coordinate system of the target robotic arm when the calibration board image is captured.

[0159] The transformation relationship between the calibration board coordinate system and the camera coordinate system of the camera to be calibrated can be represented by a first transformation relationship. Since the relative position of the camera to be calibrated and the preset calibration board can change with the pose of the robotic arm's end effector, and each calibration board image is obtained by the camera to be calibrated capturing images of the preset calibration board when the end effector of the target robotic arm is in multiple poses, the first transformation relationship corresponding to each calibration board image can be different. Furthermore, after obtaining the initial intrinsic parameters of the camera to be calibrated, for each calibration board image, based on the pixel coordinates of the feature points on the calibration board, the calibration coordinates, and the initial intrinsic parameters, the first transformation relationship between the calibration board coordinate system and the camera coordinate system of the camera to be calibrated when capturing that calibration board image can be determined.

[0160] Since each calibration plate image is obtained by the camera to be calibrated taking pictures of the preset calibration plate when the end of the target robotic arm is in multiple poses, for each calibration plate image, the second transformation relationship between the end coordinate system of the target robotic arm and the base coordinate system of the target robotic arm can be determined based on the pose of the end of the robotic arm when the calibration plate image is taken.

[0161] S304: Based on the calculated first transformation relationship and multiple second transformation relationship, determine the third transformation relationship between the camera coordinate system and the first coordinate system, and the fourth transformation relationship between the calibration plate coordinate system and the second coordinate system.

[0162] The first coordinate system and the second coordinate system are determined based on the relative positions of the camera to be calibrated, the preset calibration plate and the end of the robotic arm. The first coordinate system is the base coordinate system and the second coordinate system is the end coordinate system, or the first coordinate system is the end coordinate system and the second coordinate system is the base coordinate system.

[0163] Based on the relative positions of the camera to be calibrated, the preset calibration plate, and the end effector of the robotic arm, a first coordinate system and a second coordinate system can be determined. Furthermore, when the first coordinate system is the base coordinate system, the second coordinate system is the end effector coordinate system; conversely, when the first coordinate system is the end effector coordinate system, the second coordinate system is the base coordinate system. Since the first and second transformation relationships corresponding to each calibration plate image can be determined from each calibration plate image, these first and second transformation relationships can be defined as a set of transformation relationships. Furthermore, based on these multiple sets of transformation relationships, a third transformation relationship between the camera coordinate system and the first coordinate system, and a fourth transformation relationship between the calibration plate coordinate system and the second coordinate system, can be determined.

[0164] Wherein, if the camera to be calibrated is fixed at the end of the robotic arm and the preset calibration plate is fixed at a specified position outside the end of the robotic arm, then the first coordinate system is the end coordinate system and the second coordinate system is the base coordinate system;

[0165] Correspondingly, if the preset calibration plate is fixed at the end of the robotic arm and the camera to be calibrated is fixed at a designated position outside the end of the robotic arm, then the first coordinate system is the base coordinate system and the second coordinate system is the end coordinate system.

[0166] In industrial applications, the camera to be calibrated can be called the "eye," the target robotic arm can be called the "hand," and the situation where the camera to be calibrated is fixed at the end of the robotic arm and the preset calibration plate is fixed at a designated position outside the end of the robotic arm can be called "eye on hand."

[0167] For example, when "the eye is in the hand", the positional relationship between the camera to be calibrated, the calibration plate, the end effector of the robotic arm and the base of the robotic arm can be as shown in Figure 5(a); the transformation relationship between the camera coordinate system of the camera to be calibrated, the calibration plate coordinate system of the preset calibration plate, the base coordinate system of the target robotic arm and the end effector coordinate system of the target robotic arm can be as shown in Figure 5(b).

[0168] As can be seen, when the "eye is in the hand", the calibration plate coordinate system can be transformed into the camera coordinate system through the first transformation relationship, the camera coordinate system can be transformed into the end coordinate system through the third transformation relationship, the end coordinate system can be transformed into the base coordinate system through the second transformation relationship, and the base coordinate system can be transformed into the calibration plate coordinate system through the fourth transformation relationship.

[0169] As shown in Figure 5(a), when the "eye is on the hand," the relative position between the camera to be calibrated and the end effector of the target robotic arm remains unchanged, as does the relative position between the preset calibration plate and the base of the target robotic arm. Therefore, when the "eye is on the hand," the third transformation relationship between the camera coordinate system and the end effector coordinate system, as well as the fourth transformation relationship between the calibration plate coordinate system and the base coordinate system, remain unchanged.

[0170] As shown in Figure 5(b), for each calibration plate image, the fourth transformation relation corresponding to that calibration plate image can be equal to the product of the first, third, and second transformation relations corresponding to that calibration plate image. Furthermore, from the above, it can be concluded that for any two calibration plate images: Image 1 and Image 2, the following equation 7 holds.

[0171] Formula 7:

[0172] base2 T end2 * end2 T camera2 * camera2 T object2 = base1 T end1 * end1 T camera1 * camera1 T object1

[0173] Where T indicates the transformation relationship, base indicates the base coordinate system, end indicates the end coordinate system, camera indicates the camera coordinate system, and object indicates the calibration plate coordinate system.

[0174] By shifting terms in Equation 7 to the left and right, we can obtain Equation 8.

[0175] Formula 8:

[0176] base1 T end1 -1 * base2 T end2 * end2 T camera2 = end1 T camera1 * camera1 T object1 * camera2 T object2 -1

[0177] Since the third transformation relationship between the camera coordinate system and the end effector coordinate system does not change with the pose of the end effector of the target robotic arm when the "eye is on the hand" is in the case of "eye on hand", the third transformation relationship corresponding to Image 2 is as follows: end2 T camera2 The third transformation relationship corresponding to Image 1 end1 T camera1 Equal. Furthermore, end1 T camera1 and end2 T camera2 All are considered as X, base1 T end1 -1 * base2T end2 Treat it as A, camera1 T object1 * camera2 T object2 -1 If we consider B as the model, we can construct the equation AX = XB. By solving the constructed equation AX = XB using the first and second transformation relationships corresponding to multiple calibration plate images, we can obtain the third transformation relationship.

[0178] For example, the situation where the preset calibration plate is fixed to the end of the robotic arm and the camera to be calibrated is fixed at a designated position outside the end of the robotic arm can be called "eye outside hand". When "eye outside hand", the positional relationship between the camera to be calibrated, the calibration plate, the end of the robotic arm and the base of the robotic arm can be shown in Figure 6(a); the transformation relationship between the camera coordinate system of the camera to be calibrated, the calibration plate coordinate system of the preset calibration plate, the base coordinate system of the target robotic arm and the end coordinate system of the target robotic arm can be shown in Figure 6(b).

[0179] As can be seen, when the "eye is outside the hand", the calibration plate coordinate system can be transformed into the camera coordinate system through the first transformation relationship, the camera coordinate system can be transformed into the base coordinate system through the third transformation relationship, the base coordinate system can be transformed into the end coordinate system through the second transformation relationship, and the end coordinate system can be transformed into the calibration plate coordinate system through the fourth transformation relationship.

[0180] As shown in Figure 6(a), when the "eye is outside the hand," the relative positions between the camera to be calibrated and the base of the target robotic arm remain unchanged, as do the relative positions between the preset calibration plate and the end effector of the target robotic arm. Therefore, when the "eye is outside the hand," the third transformation relationship between the camera coordinate system and the base coordinate system, as well as the fourth transformation relationship between the calibration plate coordinate system and the end effector coordinate system, remain unchanged.

[0181] As shown in Figure 6(b), for each calibration plate image, the fourth transformation relation corresponding to that calibration plate image can be equal to the product of the first, third, and second transformation relations corresponding to that calibration plate image. Furthermore, from the above, it can be concluded that for any two calibration plate images: Image 1 and Image 2, the following equation 9 holds.

[0182] Formula 9:

[0183] end2 T base2 * base2 T camera2 * camera2 T object2 = end1 T base1 * base1 T camera1 * camera1 T object1

[0184] By shifting terms in Equation 9 to the left and right, we can obtain Equation 10.

[0185] Formula 10:

[0186] end1 T base1 -1 * end2 T base2 * base2 T camera2 = base1 T camera1 * camera1 T object1 * camera2 T object2 -1

[0187] Since the third transformation relationship between the camera coordinate system and the base coordinate system does not change with the pose of the end effector of the target robotic arm when the "eye is outside the hand" orientation, the third transformation relationship corresponding to Image 2 is... base2 T camera2 The third transformation relationship corresponding to Image 1 base1 T camera1 Equal. Furthermore, base1 T camera1 and base2 T camera2 All are considered as X, end1 T base1 -1 * end2 T bade2 Treat it as A, camera1 T object1 * camera2 T object2 -1 If we consider B as the model, we can construct the equation AX = XB. By solving the constructed equation AX = XB using the first and second transformation relationships corresponding to multiple calibration plate images, we can obtain the third transformation relationship.

[0188] As can be seen from the above, the embodiments of this application are applicable to both situations where "the eye is on the hand" and "the eye is outside the hand".

[0189] For calculating the fourth transformation relationship between the calibration plate coordinate system and the second coordinate system, in one possible implementation, similar to the calculation method of the third transformation relationship described above, the fourth transformation relationship can be obtained by constructing the equation AX = XB and solving the constructed equation AX = XB using the first and second transformation relationships corresponding to multiple calibration plate images.

[0190] Since the fourth transformation relation is equal to the product of the first, third, and second transformation relations, in one optional implementation, the fourth transformation relation can also be determined based on the first, third, and second transformation relations calculated above.

[0191] S305: Using the initial intrinsic parameters, the third transformation relationship, and the fourth transformation relationship as the first optimization terms, and the difference between the pixel coordinates of the calibration plate feature points in the first image and the first projection coordinates of the first image as the first optimization objective, construct a first optimization function and solve the first optimization function to obtain the target intrinsic parameters of the camera to be calibrated that make the first optimization objective meet the first preset requirements.

[0192] The first image is any one of the calibration board images, and the first projection coordinates are the pixel coordinates of the calibration coordinates projected onto the first image by the fourth transformation relationship, the second transformation relationship when the first image was captured, the third transformation relationship, and the initial intrinsic parameters.

[0193] One image is selected from all the calibration board images as the first image. The pixel coordinates of the calibration board feature points, calculated using the fourth transformation relationship, the second transformation relationship when the first image was captured, the third transformation relationship, and the initial intrinsic parameters, are projected onto the first image as the first projection coordinates. It can be seen that the more accurate the calculated fourth transformation relationship, the second transformation relationship when the first image was captured, the third transformation relationship, and the initial intrinsic parameters, the smaller the difference between the pixel coordinates of the calibration board feature points in the first image and the first projection coordinates relative to the first image.

[0194] It is evident that the difference between the pixel coordinates of the calibration plate feature points in the first image and the first projected coordinates relative to the first image reflects the errors of the fourth transformation relationship calculated above, the second transformation relationship when the first image was captured, the third transformation relationship, and the initial intrinsic parameters. Since the second transformation relationship when the first image was captured is determined based on the pose of the end effector of the target robotic arm at the time of capturing the first image, and the pose of the end effector of the target robotic arm is generally quite accurate, the error of the second transformation relationship calculated when capturing the first image can be ignored. Therefore, the aforementioned difference primarily reflects the errors of the fourth transformation relationship, the third transformation relationship, and the initial intrinsic parameters calculated above.

[0195] Furthermore, in order to obtain more accurate intrinsic parameters of the camera to be calibrated, the initial intrinsic parameters, the third transformation relationship and the fourth transformation relationship obtained above can be used as the first optimization term, and the difference between the pixel coordinates of the feature points of the calibration plate in the first image and the first projection coordinates of the first image can be used as the first optimization target. A first optimization function is constructed, and by solving the first optimization function, the target intrinsic parameters of the camera to be calibrated that satisfy the first preset requirement can be obtained.

[0196] The aforementioned first preset requirement can be set by those skilled in the art according to the calibration accuracy requirements in specific applications, and this application embodiment does not impose specific limitations.

[0197] Optionally, to further improve the calibration accuracy of the camera calibration, when calculating the initial intrinsic parameters of the camera to be calibrated based on the pixel coordinates of the calibration board feature points of the preset calibration board in each calibration board image, and the calibration coordinates of the calibration board feature points in the calibration board coordinate system of the preset calibration board, the distortion coefficient of the camera to be calibrated can also be calculated. Furthermore, the aforementioned first projection coordinates can be the pixel coordinates projected onto the first image using the fourth transformation relationship, the second transformation relationship when capturing the first image, the third transformation relationship, and the initial intrinsic parameters.

[0198] Optionally, the first optimization function mentioned above can be a nonlinear least squares equation.

[0199] For example, the pixel coordinates of the calibration board feature points in the first image can be (u, v), the initial intrinsic parameters of the camera to be calibrated can be K, the distortion coefficients of the camera to be calibrated can be D, and the calibration coordinates of the calibration board feature points can be P. board .

[0200] Furthermore, when the eye is outside the hand, the first optimization function described above can be a nonlinear least squares equation as shown below.

[0201]

[0202] in, This indicates the transformation relationship between the camera coordinate system and the base coordinate system. This indicates the transformation relationship between the base coordinate system and the end coordinate system. This indicates the transformation relationship between the end coordinate system and the calibration plate coordinate system. This represents the first projected coordinates of the feature points on the calibration board relative to the first image.

[0203] When the eye is on the hand, the first optimization function described above can be a nonlinear least squares equation as shown below.

[0204]

[0205] in, This indicates the transformation relationship between the camera coordinate system and the end coordinate system. This indicates the transformation relationship between the end coordinate system and the base coordinate system. This indicates the transformation relationship between the base coordinate system and the calibration plate coordinate system. This represents the first projected coordinates of the feature points on the calibration board relative to the first image.

[0206] Optionally, in one specific implementation, the initial intrinsic parameters, the third transformation relationship, and the fourth transformation relationship obtained above can be used as the first optimization term, and the difference between the pixel coordinates of the calibration plate feature points in the first image and the first projection coordinates of the first image can be used as the first optimization target. The first optimization function is constructed based on the LM (Levenberg–Marquardt) algorithm, and the first optimization function is solved to obtain the target intrinsic parameters of the camera to be calibrated that make the first optimization target meet the first preset requirements.

[0207] Since the initial intrinsic parameters are obtained through step-by-step calculations, the final initial intrinsic parameters are highly dependent on the parameters calculated earlier, and the accuracy of the obtained initial intrinsic parameters is lower than that of the previously calculated parameters. Constructing a first optimization function based on the LM algorithm and solving this first optimization function can optimize the initial intrinsic parameters, resulting in more accurate initial intrinsic parameters, which are the target intrinsic parameters.

[0208] Based on this, by applying the solution provided in the embodiments of this application, the relative position between the camera to be calibrated and the preset calibration plate can be changed by changing the pose of the end effector of the robotic arm, thereby realizing the automation of camera calibration and improving the calibration efficiency of camera calibration.

[0209] Furthermore, due to the high accuracy of the pose of the robotic arm's end effector, the accuracy of the second transformation relationship between the target robotic arm's end effector coordinate system and the base coordinate system, determined based on the end effector's pose, is also high. Consequently, the accuracy of the third and fourth transformation relationships determined based on the second transformation relationship is also high. The target intrinsic parameters obtained by optimizing the initial intrinsic parameters based on the second, third, and fourth transformation relationships can also have high accuracy. Therefore, applying the solution provided in the embodiments of this application can further improve the calibration accuracy of camera calibration.

[0210] Optionally, in one specific implementation, the target robotic arm can locate the target object using multiple cameras. In this case, the number of cameras to be calibrated can be multiple. Furthermore, as... Figure 7 As shown in the embodiments of this application, a camera calibration method may further include the following steps S701-S702.

[0211] S701: Determine the reference camera among the cameras to be calibrated, and calculate the initial extrinsic parameters of the non-reference camera based on the third transformation relationship of the reference camera and the third transformation relationship of each non-reference camera.

[0212] When calibrating multiple cameras to be calibrated, a reference camera can be determined first. Then, based on the third transformation relationship between the camera coordinate system of the reference camera and the first coordinate system and the third transformation relationship between the camera coordinate system of each non-reference camera and the first coordinate system, the transformation relationship between the camera coordinate system of the non-reference camera and the camera coordinate system of the reference camera can be calculated, which serves as the initial external parameter of the non-reference camera.

[0213] For example, when the "eye is in the hand," the third transformation relationship between the camera coordinate system of the reference camera A and the end effector coordinate system of the target robotic arm C is matrix P, and the third transformation relationship between the camera coordinate system of the non-reference camera B and the end effector coordinate system of the target robotic arm C is matrix Q. Based on matrices P and Q, the transformation relationship between the camera coordinate system of the reference camera A and the camera coordinate system of the non-reference camera B can be calculated, serving as the initial extrinsic parameters for the non-reference camera B.

[0214] S702: For each non-reference camera, the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera are used as the second optimization terms. The difference between the pixel coordinates of the calibration plate feature points in the second image and the second projection coordinates of the second image is used as the second optimization objective. A second optimization function is constructed and solved to obtain the target extrinsic parameters of the non-reference camera that make the second optimization objective meet the second preset requirements.

[0215] The second image is any one of the calibration board images captured by the non-reference camera. The second projection coordinates are: the calibration coordinates after the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the second image, the third transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera projected onto the pixel coordinates of the second image.

[0216] For each non-reference camera, one image is selected from the calibration board images captured by that non-reference camera as the second image. The calibration coordinates of the calibration board feature points are projected onto the pixel coordinates of the second image using the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the second image is captured by the non-reference camera, the third transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera. It can be seen that the more accurate the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the second image is captured by the non-reference camera, the third transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera, the smaller the difference between the pixel coordinates of the calibration board feature points in the second image and the second projected coordinates of the second image.

[0217] It is evident that the difference between the pixel coordinates of the calibration plate feature points in the second image and their second projected coordinates relative to the second image reflects the errors of the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the second image, the third transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera. Since the second transformation relationship of the reference camera when the non-reference camera captures the second image is determined based on the pose of the end effector of the target robotic arm when the non-reference camera captures the second image, and the pose of the end effector of the target robotic arm is generally quite accurate, the error of the second transformation relationship of the reference camera when the non-reference camera captures the second image can be ignored. Therefore, the aforementioned difference primarily reflects the errors of the fourth transformation relationship of the reference camera, the third transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera.

[0218] Furthermore, in order to obtain more accurate extrinsic parameters of the non-reference camera, the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera obtained above can be used as the second optimization terms. The difference between the pixel coordinates of the calibration plate feature points in the second image and the second projection coordinates of the second image is used as the second optimization objective. A second optimization function is constructed, and by solving the second optimization function, the target extrinsic parameters of the non-reference camera that satisfy the second preset requirements can be obtained.

[0219] The aforementioned second preset requirement can be set by those skilled in the art according to the calibration accuracy requirements in specific applications, and this application embodiment does not impose specific limitations.

[0220] Optionally, in one specific implementation, the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera obtained above can be used as the second optimization terms. The difference between the pixel coordinates of the calibration plate feature points in the second image and the second projection coordinates of the second image is used as the second optimization target. A second optimization function is constructed based on the LM algorithm, and the second optimization function is solved to obtain the target extrinsic parameters of the non-reference camera that make the second optimization target meet the second preset requirements.

[0221] Since the initial extrinsic parameters are obtained through step-by-step calculations, the final initial extrinsic parameters are highly dependent on the parameters calculated earlier, and the accuracy of the obtained initial extrinsic parameters is lower than that of the previously calculated parameters. Constructing a second optimization function based on the LM algorithm and solving the second optimization function can optimize the initial extrinsic parameters, resulting in more accurate initial extrinsic parameters, which are the target extrinsic parameters.

[0222] Optionally, in one specific implementation, in step S702 above, solving the second optimization function to obtain the target extrinsic parameters of the non-reference camera that make the second optimization target meet the second preset requirements may include the following step 2.

[0223] Step 2: Solve the second optimization function to obtain the target extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera that make the second optimization objective meet the second preset requirements.

[0224] In other words, by solving the second optimization function, the target extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera can be obtained simultaneously to make the second optimization objective meet the second preset requirements. Thus, the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera can be optimized simultaneously.

[0225] Since the third transformation relationship of the reference camera is essentially the hand-eye calibration result of the reference camera, the initial extrinsic parameters of each non-reference camera can represent the transformation relationship between the camera coordinate system of the non-reference camera and the camera coordinate system of the reference camera. Therefore, for each non-reference camera, the hand-eye calibration result of the non-reference camera can be determined based on the hand-eye calibration result of the reference camera and the initial extrinsic parameters of the non-reference camera. Furthermore, for each non-reference camera, a more accurate hand-eye calibration result can be obtained based on the target extrinsic parameters of the non-reference camera that enable the second optimization objective to meet the second preset requirement and the third transformation relationship of the reference camera.

[0226] Based on this, by applying this specific implementation method, when there are multiple cameras to be calibrated, the conversion relationship between these multiple cameras, that is, the extrinsic parameters of these multiple cameras to be calibrated, can be calibrated, and the extrinsic parameter calibration results, intrinsic parameter calibration results, and third conversion relationship of these multiple cameras to be calibrated can be optimized, thereby obtaining more accurate extrinsic parameters, intrinsic parameters, and hand-eye calibration results for each camera to be calibrated.

[0227] Optionally, in one specific implementation, in step S305 above, solving the first optimization function to obtain the target intrinsic parameters of the camera to be calibrated that make the first optimization objective meet the first preset requirements may include the following step 31.

[0228] Step 31: Solve the first optimization function to obtain the target intrinsic parameters of the camera to be calibrated that make the first optimization objective meet the first preset requirements, as well as the target transformation relationship between the camera coordinate system and the first coordinate system.

[0229] Since the first optimization function is constructed with the initial intrinsic parameters, the third transformation relationship, and the fourth transformation relationship as the first optimization terms, and the difference between the pixel coordinates of the calibration plate feature points in the first image and the first projection coordinates of the first image as the first optimization objective, solving the first optimization function can not only obtain the target intrinsic parameters of the camera to be calibrated that make the first optimization objective meet the first preset requirements, but also obtain the target transformation relationship between the camera coordinate system and the first coordinate system that makes the first optimization objective meet the first preset requirements, which is the optimized third transformation relationship.

[0230] Accordingly, in step S701 above, calculating the initial extrinsic parameters of the non-reference camera based on the third transformation relationship of the reference camera and the third transformation relationship of each non-reference camera may include the following step 32.

[0231] Step 32: Calculate the initial extrinsic parameters of the non-reference camera based on the target transformation relationship of the reference camera and the target transformation relationship of each non-reference camera.

[0232] Since the target transformation relation is an optimized third transformation relation, it is more accurate than the third transformation relation. Therefore, to improve the accuracy of the calculated initial extrinsic parameters, the initial extrinsic parameters of each non-reference camera can be calculated based on the target transformation relation of the reference camera and the target transformation relation of each non-reference camera.

[0233] Accordingly, in step S702 above, the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera are the second optimization terms, and the difference between the pixel coordinates of the calibration plate feature points in the second image and the second projection coordinates of the second image is the second optimization objective. The second optimization function can include the following step 33.

[0234] Step 33: Using the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the target transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera as the second optimization terms, and using the difference between the pixel coordinates of the calibration plate feature points in the second image and the second projection coordinates of the second image as the second optimization objective, construct a second optimization function.

[0235] The second projection coordinates are: the calibration coordinates after the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the second image, the target transformation relationship of the reference camera, the pixel coordinates of the non-reference camera's initial extrinsic parameters and the non-reference camera's target intrinsic parameters projected onto the second image.

[0236] When constructing the second optimization function, the third transformation relationship of the reference camera can also be replaced with the third transformation relationship of the reference camera after optimization by the first optimization function, which is the target transformation relationship of the reference camera, thereby further improving the accuracy of the target extrinsic parameters of the non-reference camera obtained in the end.

[0237] Optionally, in one specific implementation, when there are multiple cameras to be calibrated, the camera calibration method provided in this application embodiment may further include the following steps 41-42.

[0238] Step 41: Determine the reference camera among the cameras to be calibrated, and for each non-reference camera, calculate the initial extrinsic parameters of the non-reference camera based on the third transformation relationship of the reference camera and the third transformation relationship of the non-reference camera, so as to obtain the initial extrinsic parameters of each non-reference camera.

[0239] When calibrating multiple cameras to be calibrated, a reference camera can be determined first. Then, for each non-reference camera, the transformation relationship between the camera coordinate system of the non-reference camera and the camera coordinate system of the reference camera is calculated based on the third transformation relationship between the camera coordinate system of the reference camera and the first coordinate system, and the third transformation relationship between the camera coordinate system of the non-reference camera and the first coordinate system. This transformation relationship is used as the initial extrinsic parameter of the non-reference camera, thus obtaining the initial extrinsic parameters of each non-reference camera.

[0240] Step 42: Using the initial extrinsic parameters of each non-reference camera, the target intrinsic parameters of each non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera as the third optimization terms, and the sum of the target differences corresponding to each non-reference camera as the third optimization objective, construct the third optimization function, and solve the third optimization function to obtain the target extrinsic parameters of each non-reference camera, the target intrinsic parameters of each non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera that satisfy the third preset requirements of the third optimization objective.

[0241] The target difference value corresponding to each non-reference camera is: the difference between the pixel coordinates of the calibration board feature points in the third image and the third projection coordinates of the third image. The third image is any one of the calibration board images captured by the non-reference camera. The third projection coordinates are: the calibration coordinates after the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the third image, the third transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera projected onto the pixel coordinates of the third image.

[0242] After obtaining the initial extrinsic parameters of each non-reference camera, the initial extrinsic parameters of each non-reference camera, the target intrinsic parameters of each non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera can be used as the third optimization terms. The sum of the target differences corresponding to each non-reference camera can be used as the third optimization objective. The third optimization function is constructed and solved to obtain the target extrinsic parameters of each non-reference camera, the target intrinsic parameters of each non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera that make the third optimization objective meet the third preset requirements.

[0243] In other words, by solving the third optimization function, we can simultaneously obtain the target extrinsic parameters of each non-reference camera, the target intrinsic parameters of each non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera so that the third optimization objective meets the third preset requirements. Thus, we can simultaneously optimize the initial extrinsic parameters of each non-reference camera, the target intrinsic parameters of each non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera.

[0244] For example, the pixel coordinates of the calibration board feature points in the third image captured by the reference camera 1 can be (u cam1 v cam1 The pixel coordinates of the calibration board feature points in the third image captured by the non-reference camera 2 can be (u cam2 v cam2 The pixel coordinates of the calibration board feature points in the third image captured by the non-reference camera 3 can be (u cam3 v cam3 The initial intrinsic parameters of reference camera 1 can be K. cam1 The initial intrinsic parameters of non-reference camera 2 can be K. cam2 The initial intrinsic parameters of the non-reference camera 3 can be K. cam3 The distortion coefficient of reference camera 1 can be {D} cam1 The distortion coefficients of the non-reference camera 2 can be {D}. cam2 The distortion coefficients of the non-reference camera 3 can be {D}. cam3 The calibration coordinates of the feature points on the calibration plate can be P. board .

[0245] Furthermore, when the eye is outside the hand, the third optimization function can be the nonlinear least squares equation shown below.

[0246]

[0247] in, This indicates the transformation relationship between the camera coordinate system and the base coordinate system of reference camera 1. This indicates the transformation relationship between the base coordinate system and the end coordinate system. This indicates the transformation relationship between the end coordinate system and the calibration plate coordinate system. This indicates the transformation relationship between the camera coordinate system of reference camera 2 and the camera coordinate system of reference camera 1. This indicates the transformation relationship between the camera coordinate system of reference camera 3 and the camera coordinate system of reference camera 1. This represents the third projection coordinates of the feature points on the calibration plate relative to the third image captured by reference camera 1. This represents the third projected coordinates of the feature points on the calibration board relative to the third image captured by the non-reference camera 2. This represents the third projected coordinates of the calibration plate feature points relative to the third image captured by the non-reference camera 3.

[0248] When the eye is on the hand, the third optimization function mentioned above can be a nonlinear least squares equation as shown below.

[0249]

[0250] in, This indicates the transformation relationship between the camera coordinate system and the end coordinate system of reference camera 1. This indicates the transformation relationship between the end coordinate system and the base coordinate system. This indicates the transformation relationship between the base coordinate system and the calibration plate coordinate system. This indicates the transformation relationship between the camera coordinate system of reference camera 2 and the camera coordinate system of reference camera 1. This indicates the transformation relationship between the camera coordinate system of reference camera 3 and the camera coordinate system of reference camera 1. This represents the third projection coordinates of the feature points on the calibration plate relative to the third image captured by reference camera 1. This represents the third projected coordinates of the feature points on the calibration board relative to the third image captured by the non-reference camera 2. This represents the third projected coordinates of the calibration plate feature points relative to the third image captured by the non-reference camera 3.

[0251] Based on this, by applying this specific implementation method, when there are multiple cameras to be calibrated, the extrinsic parameter calibration results, intrinsic parameter calibration results, and hand-eye calibration results of these multiple cameras can be optimized simultaneously, thereby obtaining more accurate extrinsic parameter, intrinsic parameter, and hand-eye calibration results for each camera to be calibrated.

[0252] Corresponding to the camera calibration method provided in the above embodiments of this application, this application also provides a camera calibration device.

[0253] Figure 8This is a schematic diagram of the structure of a camera calibration device provided in an embodiment of this application, as shown below. Figure 8 As shown, the camera calibration device may include the following modules:

[0254] The image acquisition module 801 is used to acquire images of the calibration plate obtained by the camera to be calibrated taking pictures of the preset calibration plate when the end of the target robot arm is in multiple poses; wherein, the relative position of the camera to be calibrated and the preset calibration plate changes with the pose of the end of the robot arm.

[0255] The intrinsic parameter calculation module 802 is used to calculate the initial intrinsic parameters of the camera to be calibrated based on the pixel coordinates of the calibration board feature points of the preset calibration board in each calibration board image, and the calibration coordinates of the calibration board feature points in the calibration board coordinate system of the preset calibration board.

[0256] The first determining module 803 is used to determine, for each calibration board image, a first transformation relationship between the calibration board coordinate system and the camera coordinate system of the camera to be calibrated when the calibration board image is captured, based on the pixel coordinates of the feature points of the calibration board, the calibration coordinates, and the initial intrinsic parameters; and to determine, based on the pose of the end effector of the robotic arm when the calibration board image is captured, a second transformation relationship between the end effector coordinate system of the target robotic arm and the base coordinate system of the target robotic arm when the calibration board image is captured.

[0257] The second determining module 804 is used to determine, based on a plurality of calculated first transformation relationships and a plurality of second transformation relationships, a third transformation relationship between the camera coordinate system and the first coordinate system, and a fourth transformation relationship between the calibration plate coordinate system and the second coordinate system; wherein, the first coordinate system and the second coordinate system are determined based on the relative positions of the camera to be calibrated, the preset calibration plate, and the end effector of the robotic arm, wherein the first coordinate system is the base coordinate system and the second coordinate system is the end effector coordinate system, or, the first coordinate system is the end effector coordinate system and the second coordinate system is the base coordinate system;

[0258] The intrinsic parameter optimization module 805 is used to construct a first optimization function with the initial intrinsic parameters, the third transformation relationship, and the fourth transformation relationship as the first optimization terms, and the difference between the pixel coordinates of the calibration board feature points in the first image and the first projection coordinates of the first image as the first optimization objective, and solve the first optimization function to obtain the target intrinsic parameters of the camera to be calibrated that make the first optimization objective meet the first preset requirements; wherein, the first image is any image among the calibration board images, and the first projection coordinates are the pixel coordinates of the calibration coordinates projected onto the first image by the fourth transformation relationship, the second transformation relationship when the first image was captured, the third transformation relationship, and the initial intrinsic parameters.

[0259] Based on this, by applying the solution provided in the embodiments of this application, the relative position between the camera to be calibrated and the preset calibration plate can be changed by changing the pose of the end effector of the robotic arm, thereby realizing the automation of camera calibration and improving the calibration efficiency of camera calibration.

[0260] Furthermore, due to the high accuracy of the pose of the robotic arm's end effector, the accuracy of the second transformation relationship between the target robotic arm's end effector coordinate system and the base coordinate system, determined based on the end effector's pose, is also high. Consequently, the accuracy of the third and fourth transformation relationships determined based on the second transformation relationship is also high. The target intrinsic parameters obtained by optimizing the initial intrinsic parameters based on the second, third, and fourth transformation relationships can also have high accuracy. Therefore, applying the solution provided in the embodiments of this application can further improve the calibration accuracy of camera calibration.

[0261] Optionally, in one specific implementation, the number of cameras to be calibrated is multiple, and the device further includes:

[0262] The extrinsic parameter calculation module is used to determine the reference camera among the cameras to be calibrated, and to calculate the initial extrinsic parameters of the non-reference camera according to the third transformation relationship of the reference camera and the third transformation relationship of each non-reference camera;

[0263] The extrinsic parameter optimization module is used to, for each non-reference camera, take the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera as the second optimization terms, and take the difference between the pixel coordinates of the calibration board feature points in the second image and the second projection coordinates of the second image as the second optimization objective, construct a second optimization function, and solve the second optimization function to obtain the target extrinsic parameters of the non-reference camera that make the second optimization objective meet the second preset requirements; wherein, the second image is any image among the calibration board images captured by the non-reference camera, and the second projection coordinates are: the pixel coordinates of the calibration coordinates projected into the second image by the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the second image, the third transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera.

[0264] Optionally, in one specific implementation, the intrinsic parameter optimization module is specifically used for:

[0265] Solve the first optimization function to obtain the target intrinsic parameters of the camera to be calibrated that make the first optimization objective meet the first preset requirements, and the target transformation relationship between the camera coordinate system and the first coordinate system;

[0266] The external parameter calculation module is specifically used for:

[0267] Based on the target transformation relationship of the reference camera and the target transformation relationship of each non-reference camera, calculate the initial extrinsic parameters of the non-reference camera;

[0268] The extrinsic parameter optimization module is specifically used for:

[0269] Using the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the target transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera as the second optimization terms, and the difference between the pixel coordinates of the calibration plate feature points in the second image and the second projection coordinates of the second image as the second optimization objective, a second optimization function is constructed; wherein, the second projection coordinates are: the pixel coordinates of the calibration coordinates projected onto the second image by the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the second image, the target transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera.

[0270] Optionally, in one specific implementation, the number of cameras to be calibrated is multiple, and the device further includes:

[0271] A camera determination module is used to determine a reference camera among the cameras to be calibrated, and for each non-reference camera, calculate the initial extrinsic parameters of the non-reference camera according to the third transformation relationship of the reference camera and the third transformation relationship of the non-reference camera, so as to obtain the initial extrinsic parameters of each non-reference camera.

[0272] The function construction module is used to construct a third optimization function with the initial extrinsic parameters of each non-reference camera, the target intrinsic parameters of each non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera as the third optimization terms, and the sum of the target differences corresponding to each non-reference camera as the third optimization objective. The module then solves the third optimization function to obtain the target extrinsic parameters of each non-reference camera, the target intrinsic parameters of each non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera that satisfy the third preset requirement for the third optimization objective.

[0273] The target difference value corresponding to each non-reference camera is: the difference between the pixel coordinates of the calibration board feature points in the third image and the third projection coordinates of the third image. The third image is any one of the calibration board images captured by the non-reference camera. The third projection coordinates are: the pixel coordinates of the calibration coordinates projected onto the third image by the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the third image, the third transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera.

[0274] Optionally, in one specific implementation, the first optimization function is a nonlinear least squares equation.

[0275] Optionally, in one specific implementation, if the camera to be calibrated is fixed at the end of the robotic arm and the preset calibration plate is fixed at a designated position outside the end of the robotic arm, then the first coordinate system is the end coordinate system and the second coordinate system is the base coordinate system;

[0276] If the preset calibration plate is fixed at the end of the robotic arm and the camera to be calibrated is fixed at a designated position outside the end of the robotic arm, then the first coordinate system is the base coordinate system and the second coordinate system is the end coordinate system.

[0277] This application also provides an electronic device, such as... Figure 9 As shown, it includes:

[0278] Memory 901 is used to store computer programs;

[0279] The processor 902, when executing the program stored in the memory 901, implements the steps of any camera calibration method provided in the embodiments of this application.

[0280] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 902, communication interface, and memory 901 communicating with each other via the communication bus.

[0281] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0282] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0283] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0284] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0285] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described camera calibration methods.

[0286] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the camera calibration methods described above.

[0287] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), etc.

[0288] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0289] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments, device embodiments, electronic device embodiments, computer-readable storage medium embodiments, and computer program product embodiments are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0290] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A camera calibration method, characterized by, The method includes: The calibration plate image is obtained by taking pictures of the preset calibration plate when the camera to be calibrated is in multiple poses at the end of the target robotic arm; wherein the relative position of the camera to be calibrated and the preset calibration plate changes with the pose of the end of the robotic arm. Based on the pixel coordinates of the calibration board feature points of the preset calibration board in each calibration board image, and the calibration coordinates of the calibration board feature points in the calibration board coordinate system of the preset calibration board, the initial intrinsic parameters of the camera to be calibrated are calculated. For each calibration board image, based on the pixel coordinates of the feature points of the calibration board, the calibration coordinates, and the initial intrinsic parameters, a first transformation relationship between the calibration board coordinate system and the camera coordinate system of the camera to be calibrated is determined when the calibration board image is captured. Based on the pose of the end effector of the robotic arm when the calibration board image is captured, a second transformation relationship between the end effector coordinate system of the target robotic arm and the base coordinate system of the target robotic arm is determined when the calibration board image is captured. Based on the calculated first and second transformation relationships, a third transformation relationship between the camera coordinate system and the first coordinate system, and a fourth transformation relationship between the calibration plate coordinate system and the second coordinate system are determined. The first and second coordinate systems are determined based on the relative positions of the camera to be calibrated, the preset calibration plate, and the end effector of the robotic arm. If the preset calibration plate is fixed at the end effector of the robotic arm and the camera to be calibrated is fixed at a specified position outside the end effector of the robotic arm, the relative position between the camera to be calibrated and the robotic arm base remains unchanged; the first coordinate system is the base coordinate system, and the second coordinate system is the end effector coordinate system. If the camera to be calibrated is fixed at the end effector of the robotic arm and the preset calibration plate is fixed at a specified position outside the end effector of the robotic arm, the relative position between the camera to be calibrated and the end effector of the robotic arm remains unchanged; the first coordinate system is the end effector coordinate system, and the second coordinate system is the base coordinate system. Using the initial intrinsic parameters, the third transformation relationship, and the fourth transformation relationship as the first optimization terms, and the difference between the pixel coordinates of the calibration board feature points in the first image and the first projection coordinates of the first image as the first optimization objective, a first optimization function is constructed and solved to obtain the target intrinsic parameters of the camera to be calibrated that satisfy the first preset requirement; wherein, the first image is any one of the calibration board images, and the first projection coordinates are the pixel coordinates of the calibration coordinates projected onto the first image by the fourth transformation relationship, the second transformation relationship when the first image was captured, the third transformation relationship, and the initial intrinsic parameters; The number of cameras to be calibrated is multiple, and the method further includes: determining a reference camera among the cameras to be calibrated, and for each non-reference camera, calculating the initial extrinsic parameters of the non-reference camera based on the third transformation relationship of the reference camera and the third transformation relationship of the non-reference camera, to obtain the initial extrinsic parameters of each non-reference camera; constructing a third optimization function with the initial extrinsic parameters of each non-reference camera, the target intrinsic parameters of each non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera as the third optimization term, and with the sum of the target differences corresponding to each non-reference camera as the third optimization objective, and solving the third optimization function to obtain the target of each non-reference camera that makes the third optimization objective satisfy the third preset requirement. The calibration parameters include extrinsic parameters, target intrinsic parameters of each non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera. The target difference for each non-reference camera is the difference between the pixel coordinates of the calibration plate feature points in the third image and the third projection coordinates of the third image. The third image is any one of the calibration plate images captured by the non-reference camera. The third projection coordinates are the pixel coordinates projected onto the third image by the calibration coordinates through the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the third image, the third transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera.

2. The method of claim 1, wherein, The number of cameras to be calibrated is multiple, and the method further includes: A reference camera is determined among the cameras to be calibrated, and the initial extrinsic parameters of the non-reference camera are calculated based on the third transformation relationship of the reference camera and the third transformation relationship of each non-reference camera. For each non-reference camera, the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera are used as the second optimization terms. The difference between the pixel coordinates of the calibration board feature points in the second image and the second projection coordinates of the second image is used as the second optimization objective. A second optimization function is constructed and solved to obtain the target extrinsic parameters of the non-reference camera that satisfy the second preset requirement. The second image is any one of the calibration board images captured by the non-reference camera, and the second projection coordinates are: the pixel coordinates of the calibration coordinates projected onto the second image by the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the second image, the third transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera.

3. The method according to claim 2, characterized in that, Solving the first optimization function to obtain the target intrinsic parameters of the camera to be calibrated that make the first optimization objective satisfy the first preset requirement includes: Solve the first optimization function to obtain the target intrinsic parameters of the camera to be calibrated that make the first optimization objective meet the first preset requirements, and the target transformation relationship between the camera coordinate system and the first coordinate system; The step of calculating the initial extrinsic parameters of a non-reference camera based on the third transformation relationship of the reference camera and the third transformation relationship of each non-reference camera includes: Based on the target transformation relationship of the reference camera and the target transformation relationship of each non-reference camera, calculate the initial extrinsic parameters of the non-reference camera; The second optimization function is constructed using the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera as the second optimization terms, and the difference between the pixel coordinates of the calibration board feature points in the second image and the second projection coordinates of the second image as the second optimization objective. The second optimization function includes: Using the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the target transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera as the second optimization terms, and the difference between the pixel coordinates of the calibration plate feature points in the second image and the second projection coordinates of the second image as the second optimization objective, a second optimization function is constructed; wherein, the second projection coordinates are: the pixel coordinates of the calibration coordinates projected onto the second image by the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the second image, the target transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera.

4. The method according to any one of claims 1 to 3, characterized in that, The first optimization function is a nonlinear least squares equation.

5. A camera calibration system characterized by, The system includes: a camera to be calibrated, a target robotic arm, a preset calibration plate, and processing equipment; The camera to be calibrated is used to acquire images of the preset calibration plate; The target robotic arm includes a robotic arm base and a robotic arm end effector, and the robotic arm end effector can change into various poses. The relative position between the camera to be calibrated and the preset calibration plate can change with the pose of the robotic arm end effector. The preset calibration plate is within the shooting range of the camera to be calibrated; The processing device is used to implement the method according to any one of claims 1-4.

6. A camera calibration apparatus characterized by comprising: The device includes: The image acquisition module is used to acquire images of the calibration plate taken by the camera to be calibrated when the end of the target robotic arm is in multiple poses; wherein the relative position of the camera to be calibrated and the preset calibration plate changes with the pose of the end of the robotic arm. The intrinsic parameter calculation module is used to calculate the initial intrinsic parameters of the camera to be calibrated based on the pixel coordinates of the calibration board feature points of the preset calibration board in each calibration board image, and the calibration coordinates of the calibration board feature points in the calibration board coordinate system of the preset calibration board. The first determining module is used to determine, for each calibration board image, a first transformation relationship between the calibration board coordinate system and the camera coordinate system of the camera to be calibrated when the calibration board image is captured, based on the pixel coordinates of the feature points of the calibration board, the calibration coordinates, and the initial intrinsic parameters; and to determine, based on the pose of the end effector of the robotic arm when the calibration board image is captured, a second transformation relationship between the end effector coordinate system and the base coordinate system of the target robotic arm when the calibration board image is captured. The second determining module is used to determine, based on a plurality of calculated first transformation relationships and a plurality of second transformation relationships, a third transformation relationship between the camera coordinate system and the first coordinate system, and a fourth transformation relationship between the calibration plate coordinate system and the second coordinate system; wherein, the first coordinate system and the second coordinate system are determined based on the relative positions of the camera to be calibrated, the preset calibration plate, and the end effector of the robotic arm. If the preset calibration plate is fixed at the end effector of the robotic arm and the camera to be calibrated is fixed at a specified position outside the end effector of the robotic arm, the relative position between the camera to be calibrated and the robotic arm base remains unchanged, the first coordinate system is the base coordinate system, and the second coordinate system is the end effector coordinate system; if the camera to be calibrated is fixed at the end effector of the robotic arm and the preset calibration plate is fixed at a specified position outside the end effector of the robotic arm, the relative position between the camera to be calibrated and the end effector of the robotic arm remains unchanged, the first coordinate system is the end effector coordinate system, and the second coordinate system is the base coordinate system; An intrinsic parameter optimization module is used to construct a first optimization function with the initial intrinsic parameters, the third transformation relationship, and the fourth transformation relationship as the first optimization terms, and the difference between the pixel coordinates of the calibration board feature points in the first image and the first projection coordinates of the first image as the first optimization objective. The module then solves the first optimization function to obtain the target intrinsic parameters of the camera to be calibrated that satisfy the first preset requirement. The first image is any one of the calibration board images, and the first projection coordinates are the pixel coordinates of the calibration coordinates projected onto the first image using the fourth transformation relationship, the second transformation relationship when capturing the first image, the third transformation relationship, and the initial intrinsic parameters. The number of cameras to be calibrated is multiple, and the device further includes: A camera determination module is used to determine a reference camera among the cameras to be calibrated, and for each non-reference camera, calculate the initial extrinsic parameters of the non-reference camera according to the third transformation relationship of the reference camera and the third transformation relationship of the non-reference camera, so as to obtain the initial extrinsic parameters of each non-reference camera. The function construction module is used to construct a third optimization function with the initial extrinsic parameters of each non-reference camera, the target intrinsic parameters of each non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera as the third optimization terms, and the sum of the target differences corresponding to each non-reference camera as the third optimization objective. The module then solves the third optimization function to obtain the target extrinsic parameters of each non-reference camera, the target intrinsic parameters of each non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera that satisfy the third preset requirement for the third optimization objective. The target difference corresponding to each non-reference camera is: the difference between the pixel coordinates of the calibration board feature points in the third image and the third projection coordinates of the third image. The third image is any one of the calibration board images captured by the non-reference camera. The third projection coordinates are: the pixel coordinates of the calibration coordinates projected onto the third image through the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the third image, the third transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera.

7. The apparatus according to claim 6, characterized in that, The number of cameras to be calibrated is multiple, and the device further includes: The extrinsic parameter calculation module is used to determine the reference camera among the cameras to be calibrated, and to calculate the initial extrinsic parameters of the non-reference camera according to the third transformation relationship of the reference camera and the third transformation relationship of each non-reference camera; An extrinsic parameter optimization module is used to, for each non-reference camera, take the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the third transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera as the second optimization terms, and take the difference between the pixel coordinates of the calibration board feature points in the second image and the second projection coordinates of the second image as the second optimization objective, construct a second optimization function, and solve the second optimization function to obtain the target extrinsic parameters of the non-reference camera that make the second optimization objective meet the second preset requirements; wherein, the second image is any image among the calibration board images captured by the non-reference camera, and the second projection coordinates are: the pixel coordinates of the calibration coordinates projected into the second image by the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the second image, the third transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera; And / or, The intrinsic parameter optimization module is specifically used for: Solve the first optimization function to obtain the target intrinsic parameters of the camera to be calibrated that make the first optimization objective meet the first preset requirements, and the target transformation relationship between the camera coordinate system and the first coordinate system; The external parameter calculation module is specifically used for: Based on the target transformation relationship of the reference camera and the target transformation relationship of each non-reference camera, calculate the initial extrinsic parameters of the non-reference camera; The extrinsic parameter optimization module is specifically used for: Using the initial extrinsic parameters of the non-reference camera, the target intrinsic parameters of the non-reference camera, the target transformation relationship of the reference camera, and the fourth transformation relationship of the reference camera as the second optimization terms, and the difference between the pixel coordinates of the calibration plate feature points in the second image and the second projection coordinates of the second image as the second optimization objective, a second optimization function is constructed; wherein, the second projection coordinates are: the pixel coordinates of the calibration coordinates projected onto the second image through the fourth transformation relationship of the reference camera, the second transformation relationship of the reference camera when the non-reference camera captures the second image, the target transformation relationship of the reference camera, the initial extrinsic parameters of the non-reference camera, and the target intrinsic parameters of the non-reference camera; And / or, The first optimization function is a nonlinear least squares equation.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-4.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-4.

Images