Camera Calibration Method, Camera Calibration Device, Electronic Device, and Storage Medium

Abstract

The present disclosure provides a camera calibration method, a camera calibration device, an electronic device, and a storage medium. The method includes: taking any one of the sub-calibration plates in the combined calibration plate as a reference calibration plate, and obtaining a calibration plate image of the combined calibration plate; wherein the combined calibration plate includes: a plurality of interconnected sub-calibration plates, and the rotation directions and / or rotation angles of the plurality of sub-calibration plates are different; based on the calibration plate image, obtaining the pixel coordinates of the feature points of the plurality of sub-calibration plates; based on the relative position relationship between the reference calibration plate and the plurality of sub-calibration plates in the calibration plate image, determining the world coordinates of the feature points of the plurality of sub-calibration plates; and based on the pixel coordinates and world coordinates of the feature points of the plurality of sub-calibration plates, determining the calibration parameters of the camera.

Description

Technical Field

[0001] The present disclosure relates to the technical field of machine vision calibration, and in particular, to a camera calibration method, a camera calibration device, an electronic device, and a storage medium. Background Art

[0002] Camera calibration is a basic problem in machine vision. Due to the reasons of production, processing and its own characteristics, there will be errors in the camera. These errors will cause the original image to be distorted. Therefore, camera calibration needs to be carried out before visual measurement to ensure the accuracy of subsequent image acquisition work.

[0003] In the prior art, although the image of the planar calibration board is collected by the camera, then the pixel coordinates and world coordinates corresponding to the feature points of the planar calibration board are determined, and finally the calibration parameters of the camera are obtained, the camera can be calibrated, but this method cannot ensure the consistency and stability of the calibration. Summary of the Invention

[0004] In view of this, the present disclosure provides a camera calibration method, a camera calibration device, an electronic device, and a storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a camera calibration method, the method including:

[0006] Taking any one of the sub-calibration boards in the combined calibration board as a reference calibration board, and obtaining a calibration board image of the combined calibration board; wherein, the combined calibration board includes: a plurality of interconnected sub-calibration boards, and the rotation directions and / or rotation angles of the plurality of sub-calibration boards are different;

[0007] Based on the calibration board image, determining pixel coordinates of feature points of the plurality of sub-calibration boards;

[0008] Based on the relative position relationship between the reference calibration board and the plurality of sub-calibration boards in the calibration board image, determining world coordinates of feature points of the plurality of sub-calibration boards;

[0009] Based on the pixel coordinates and world coordinates of the feature points of the plurality of sub-calibration boards, determining calibration parameters of the camera.

[0010] Optionally, the taking any one of the sub-calibration boards in the combined calibration board as a reference calibration board and obtaining a calibration board image of the combined calibration board includes:

[0011] Determining the reference calibration board from the plurality of sub-calibration boards of the combined calibration board;

[0012] Based on the reference calibration board, adjusting the camera; after adjustment, the image acquisition surface of the camera is parallel to the reference calibration board, and the field of view of the camera covers the combined calibration board;

[0013] Based on the adjusted camera, acquire the calibration board image of the combined calibration board.

[0014] Optionally, determining the pixel coordinates of the feature points of multiple sub-calibration boards based on the calibration board image includes:

[0015] Based on the calibration board image of the combined calibration board, obtain sub-images of multiple sub-calibration boards within the calibration board image; wherein, the relative positional relationship between the image acquisition plane of the camera and the sub-calibration boards corresponding to multiple different sub-images is different;

[0016] Based on the sub-images of multiple sub-calibration boards, determine the pixel coordinates of the feature points within the sub-image of each sub-calibration board.

[0017] Optionally, determining the pixel coordinates of the feature points within the sub-image of each sub-calibration board based on the sub-images of multiple sub-calibration boards includes:

[0018] Detect candidate feature points of the sub-images of multiple sub-calibration boards and determine the distribution of the candidate feature points;

[0019] Based on the distribution, select feature point data;

[0020] Based on the distance relationship between adjacent feature points, determine the pixel coordinates of the feature points within the sub-image of each sub-calibration board.

[0021] Optionally, determining the world coordinates of the feature points of multiple sub-calibration boards based on the relative positional relationship between the reference calibration board and multiple sub-calibration boards in the calibration board image includes:

[0022] Determine the world coordinates corresponding to each feature point in the reference calibration board;

[0023] Based on the relative positional relationship between the reference calibration board and each sub-calibration board, determine the rotation matrix and translation matrix corresponding to each sub-calibration board;

[0024] Based on the rotation matrix and translation matrix corresponding to each sub-calibration board, and the world coordinates of each feature point in the reference calibration board, respectively determine the world coordinates of the feature points within each sub-calibration board.

[0025] Optionally, determining the calibration parameters of the camera based on the pixel coordinates and world coordinates of the feature points of multiple sub-calibration boards includes:

[0026] Based on multiple sub-calibration boards, determine a training data set; wherein, the training data set includes: the pixel coordinates and world coordinates of the feature points of at least one sub-calibration board;

[0027] Based on the world coordinates of the feature points of the sub-calibration board in the training dataset and the initial calibration parameters of the camera, determine the first predicted pixel coordinates of the feature points of the sub-calibration board;

[0028] Construct a first objective function based on the error between the first predicted pixel coordinates of the feature points and the pixel coordinates;

[0029] Iteratively optimize the initial calibration parameters based on the first objective function to obtain the calibration parameters.

[0030] Optionally, the iteratively optimizing the initial calibration parameters based on the first objective function to obtain the calibration parameters includes:

[0031] Based on the first calibration parameters optimized from the first objective function, determine the rotation matrix and translation matrix between the reference calibration board and the camera;

[0032] Obtain a validation dataset; wherein, the validation dataset includes: the pixel coordinates and world coordinates of the feature points of at least one sub-calibration board; the validation dataset is different from the training dataset;

[0033] Based on the world coordinates of the feature points of the sub-calibration board in the validation dataset, the rotation matrix and translation matrix between the sub-calibration board and the reference calibration board, and the rotation matrix and translation matrix between the reference calibration board and the camera, determine the second predicted pixel coordinates of the feature points of the sub-calibration board;

[0034] Based on the difference between the second predicted pixel coordinates of the sub-calibration board in the validation dataset and the pixel coordinates, determine the loss function corresponding to the validation dataset;

[0035] Construct a second objective function based on the first objective function and the loss function;

[0036] Iteratively optimize the first calibration parameters based on the second objective function to obtain the calibration parameters.

[0037] In a second aspect, an embodiment of the present disclosure provides a camera calibration device, the device includes:

[0038] An acquisition module, configured to obtain a calibration board image of the combined calibration board with any one of the sub-calibration boards in the combined calibration board as a reference calibration board; wherein, the combined calibration board includes: a plurality of interconnected sub-calibration boards, and the rotation directions and / or rotation angles of the plurality of sub-calibration boards are different;

[0039] A first determination module, configured to determine the pixel coordinates of the feature points of the plurality of sub-calibration boards based on the calibration board image;

[0040] A second determination module, configured to determine the world coordinates of the feature points of the multiple sub-calibration plates based on the relative position relationship between the reference calibration plate and the multiple sub-calibration plates in the calibration plate image.

[0041] A third determination module, configured to determine the calibration parameters of the camera based on the pixel coordinates and world coordinates of the feature points of the multiple sub-calibration plates.

[0042] In a third aspect, an embodiment of the present disclosure provides an electronic device, including a processor and a memory, where the memory is used to store code instructions; the processor is used to run the code instructions to execute the steps in any of the camera calibration methods.

[0043] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which stores a computer program, and the computer program includes instructions for implementing any of the camera calibration methods.

[0044] In the embodiment of the present disclosure, based on any one of the sub-calibration plates in the combined calibration plate as the reference calibration plate, the calibration plate image of the combined calibration plate is obtained; wherein, the combined calibration plate includes: multiple interconnected sub-calibration plates, and the rotation directions and / or rotation angles of the multiple sub-calibration plates are different; based on the calibration plate image, the pixel coordinates of the feature points of the multiple sub-calibration plates are determined; based on the relative position relationship between the reference calibration plate and the multiple sub-calibration plates in the calibration plate image, the world coordinates of the feature points of the multiple sub-calibration plates are determined; based on the pixel coordinates and world coordinates of the feature points of the multiple sub-calibration plates, the calibration parameters of the camera are determined. In this way, compared with the existing camera calibration methods, since the calibration feature points are all on the same plane, resulting in the lack of dimensional information of the feature point coordinates and causing the problem of low camera calibration accuracy. On the one hand, in the embodiment of the present disclosure, by collecting the calibration plate image of the combined calibration plate, the image information of multiple calibration plates at different angles and / or different distances can be obtained, without repeatedly adjusting the angle and / or distance between the camera and the calibration plate, improving the calibration efficiency; on the other hand, the non-coplanar characteristic of the combined calibration plate provides spatial three-dimensional information, so that more accurate pixel coordinates and world coordinates can be obtained, which is beneficial to the stability and consistency of camera calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a schematic flowchart of a camera calibration method shown according to an exemplary embodiment;

[0046] Figure 2 is a schematic structural diagram of a combined calibration plate shown according to an exemplary embodiment;

[0047] Figure 3 is a schematic diagram of the positional relationship between a reference calibration plate and a sub-calibration plate shown according to an exemplary embodiment;

[0048] Figure 4 is a block diagram of a camera calibration device shown according to an exemplary embodiment;

[0049] Figure 5 is a block diagram of the hardware structure of an electronic device shown according to an exemplary embodiment. Detailed implementation manners

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the following will further describe the specific technical solutions of the invention in detail with reference to the accompanying drawings in the embodiments of the present disclosure. The following embodiments are used to illustrate the present disclosure but are not intended to limit the scope of the present disclosure.

[0051] The embodiments of the present disclosure provide a camera calibration method. Figure 1 is a schematic flowchart of a camera calibration method shown according to an exemplary embodiment, as Figure 1 shown; the method includes:

[0052] In step S1, based on any one of the sub-calibration plates in the combined calibration plate as a reference calibration plate, a calibration plate image of the combined calibration plate is obtained; wherein, the combined calibration plate includes: a plurality of interconnected sub-calibration plates, and the rotation directions and / or rotation angles of the plurality of sub-calibration plates are different;

[0053] In step S2, based on the calibration plate image, the pixel coordinates of the feature points of the plurality of sub-calibration plates are determined;

[0054] In step S3, based on the relative position relationship between the reference calibration plate and the plurality of sub-calibration plates in the calibration plate image, the world coordinates of the feature points of the plurality of sub-calibration plates are determined;

[0055] In step S4, based on the pixel coordinates and world coordinates of the feature points of the plurality of sub-calibration plates, the calibration parameters of the camera are determined.

[0056] It can be understood that before obtaining the combined calibration plate image, a plurality of sub-calibration plates should be prepared first, and multiple planar sub-calibration plates are spliced at a preset angle so that the multiple planar calibration plates are fixed to each other as a combined calibration plate.

[0057] In the embodiments of the present disclosure, when preparing the combined calibration plate, the edge positions of the plurality of sub-calibration plates should be flat so that the plurality of sub-calibration plates are closer when pasted, facilitating subsequent image acquisition work.

[0058] Here, the angular relationship between the plurality of sub-reference calibration plates can be fixedly set or flexibly set, and the embodiments of the present disclosure do not limit this.

[0059] It should be noted that the type, graphic shape, and arrangement of the sub-calibration plates in the embodiments of the present disclosure are not limited, as long as it is satisfied that the combined calibration plate is formed by splicing multiple sub-calibration plates.

[0060] In some embodiments, in order to facilitate obtaining the pixel coordinates and world coordinates of the corresponding feature points of the calibration plate image, the sub-calibration plate can be set as a calibration plate including a checkerboard with multiple black and white squares arranged at intervals.

[0061] It should be explained that the corner points on the checkerboard are coplanar and lack three-dimensional information, resulting in the checkerboard being unable to be accurately used in pose estimation. However, in the embodiments of the present disclosure, by splicing multiple sub-calibration plates with checkerboard patterns at an angle, three-dimensional spatial information is provided to achieve accurate detection of feature points, thereby improving the accuracy of camera calibration.

[0062] It can be understood that the combined calibration plate is formed by splicing multiple sub-calibration plates. In order to obtain a complete calibration plate image, a rail fixture and a control system can be used to enable the camera to move and take pictures, so as to ensure that the field of view of the camera covers the combined calibration plate.

[0063] Specifically, any one of the sub-calibration plates in the combined calibration plate is selected as the reference calibration plate. The combined calibration plate is installed at one end of the rail, and the camera is assembled parallel to the reference calibration plate on the fixture. The controller controls the movement of the camera so that the field of view of the camera can cover the combined calibration plate to obtain a complete calibration plate image of the combined calibration plate with high imaging accuracy.

[0064] Here, the camera for collecting the calibration image of the combined calibration plate can be a monocular camera, a multiocular camera, or a panoramic camera, and the embodiments of the present disclosure do not limit this.

[0065] It should be noted that after obtaining the calibration plate image corresponding to the combined calibration plate, image processing should be performed on the calibration plate image to extract the feature points in the calibration plate image.

[0066] Here, the extracted feature points can include: checkerboard corner points, where the checkerboard corner points are the points at the four corners of the black or white squares.

[0067] In some embodiments, in order to further improve the accuracy of camera calibration, during the process of extracting feature points, the obtained calibration plate image of the combined calibration plate can first be binarized by using the maximum inter-class variance method to eliminate the interference of factors such as ambient brightness on the feature points; then the Harris algorithm is selected for feature point detection to determine the feature point data of the calibration plate image.

[0068] Here, the distance between adjacent feature points in the feature point data can be equal or unequal, and the embodiments of the present disclosure do not limit this.

[0069] Based on the calibration board image, the pixel coordinates of the feature points of multiple sub-calibration boards are determined. It can be understood that a certain feature point is selected as the coordinate origin on the obtained calibration board image, and a pixel coordinate system is established. According to the arrangement and distribution of each feature point, the pixel coordinates of the feature points of multiple sub-calibration boards are calculated.

[0070] In some embodiments, according to the grid point distribution of the chessboard, the detected feature points are sorted. The sorting method is to sort along the horizontal direction from left to right to determine the image pixel coordinates (u ij , v ij ) corresponding to each feature point, where i = 1, 2,... n and j = 1, 2,... n.

[0071] It should be noted that multiple interconnected sub-calibration boards are spliced into a combined calibration board, and the rotation directions and / or rotation angles of multiple sub-calibration boards are different. Therefore, the relative position information of multiple sub-calibration boards with respect to the reference calibration board is different.

[0072] Here, the relative position information includes: angle information and / or distance information.

[0073] In the embodiments of the present disclosure, in order to facilitate the calculation of the world coordinates corresponding to the feature points of the calibration board image, the reference calibration board can be set as a planar calibration board. After calculating the world coordinates corresponding to the feature points of the reference calibration board, the relative position relationship between the reference calibration board and the sub-calibration board is obtained, that is, the spatial three-dimensional information is obtained, and then the world coordinates corresponding to the feature points of each sub-calibration board are determined, which is beneficial to improving the calibration accuracy of the camera.

[0074] It can be understood that the calibration parameters of the camera at least include at least one of internal parameters, external parameters, and distortion parameters. Among them, the internal parameters include image focal length, image principal point coordinates, and offset parameters; the external parameters include an external reference rotation matrix and an external reference translation matrix. The external reference translation matrix is used to describe the rotation relationship from the world coordinate system to the image coordinate system, and the external reference translation matrix is used to describe the translation relationship from the world coordinate system to the image coordinate system. The distortion parameters include radial distortion parameters and tangential distortion parameters.

[0075] It should be noted that the world coordinates corresponding to the feature points of multiple obtained sub-calibration boards reflect the real physical coordinates of the feature points. The initial calibration parameters can be preset to generate a camera calibration function, and the world coordinates corresponding to the feature points of multiple sub-calibration boards are brought into the camera calibration function to calculate the predicted pixel coordinates. By comparing the pixel coordinates corresponding to the feature points with the predicted pixel coordinates, the initial calibration parameters of the camera are adjusted to determine the target calibration parameters of the camera.

[0076] Exemplarily, Figure 2 is a schematic structural diagram of a combined calibration board shown according to an exemplary embodiment. As Figure 2As shown in the figure; taking calibration board 1 as the reference calibration board, the included angle between calibration board 1 and calibration board 2 is angle1, the included angle between calibration board 1 and calibration board 3 is angle2, the included angle between calibration board 1 and calibration board 4 is angle3, the included angle between calibration board 1 and calibration board 5 is angle4, the length of each calibration board is H (unit: millimeter), and the width is W (unit: millimeter).

[0077] To collect complete calibration board images, the combined calibration board is installed at one end of the guide rail. The 3D iToF camera is assembled on the fixture parallel to the surface of the reference calibration board. When the controller controls the 3D iToF camera device to move to a distance Z0 (unit: millimeter) from the reference calibration board, the 3D iToF camera collects the image data of the combined calibration board, where Z0 can be reasonably adjusted according to the imaging situation of the lens and image sensor of the 3D iToF camera.

[0078] In the embodiment of the present disclosure, based on any one of the sub-calibration boards in the combined calibration board as the reference calibration board, the calibration board image of the combined calibration board is obtained; wherein, the combined calibration board includes: multiple interconnected sub-calibration boards, and the rotation directions and / or rotation angles of the multiple sub-calibration boards are different; based on the calibration board image, the pixel coordinates of the feature points of the multiple sub-calibration boards are determined; based on the relative position relationship between the reference calibration board and the multiple sub-calibration boards in the calibration board image, the world coordinates of the feature points of the multiple sub-calibration boards are determined; based on the pixel coordinates and world coordinates of the feature points of the multiple sub-calibration boards, the calibration parameters of the camera are determined. Thus, compared with the existing camera calibration methods, since the calibration feature points are all on the same plane, resulting in the lack of dimensional information of the feature point coordinates and causing the problem of low camera calibration accuracy. On the one hand, in the embodiment of the present disclosure, by collecting the calibration board image of the combined calibration board, the image information of multiple calibration boards at different angles and / or different distances can be obtained, without repeatedly adjusting the angle and / or distance between the camera and the calibration board, improving the calibration efficiency; on the other hand, using the non-coplanar characteristics of the combined calibration board provides spatial three-dimensional information, so that more accurate pixel coordinates and world coordinates can be obtained, which is beneficial to the stability and consistency of camera calibration.

[0079] Optionally, the obtaining the calibration board image of the combined calibration board based on any one of the sub-calibration boards in the combined calibration board as the reference calibration board includes:

[0080] Determine the reference calibration board from the multiple sub-calibration boards of the combined calibration board;

[0081] Based on the reference calibration board, adjust the camera; after adjustment, the image acquisition plane of the camera is parallel to the reference calibration board, and the field of view of the camera covers the combined calibration board;

[0082] Based on the adjusted camera, collect the calibration board image of the combined calibration board.

[0083] It should be noted that, in order to improve the accuracy of camera calibration, before collecting the combined calibration board images, a reference calibration board needs to be selected from the combined calibration board, and by adjusting the position information of the camera, ensure that the line connecting the center of the camera lens and the center of the reference calibration board is perpendicular to the horizontal plane, so as to ensure that the camera imaging plane is parallel to the reference calibration board plane, and then obtain the corresponding calibration images.

[0084] It can be understood that when collecting the combined calibration board images, the distance and angle between the camera and the combined calibration board should be appropriately adjusted to ensure that the field of view of the camera covers the combined calibration board, so as to obtain the calibration board images of the complete combined calibration board and avoid the time-consuming problem of collecting calibration board images multiple times.

[0085] In some embodiments, in order to ensure the accuracy of camera calibration, it is necessary to collect the calibration board images of the combined calibration board with complete and clear imaging accuracy, so as to obtain complete coordinate dimension information, thereby laying a foundation for determining accurate pixel coordinates and world coordinates.

[0086] In the embodiments of the present disclosure, by selecting a reference calibration board from the combined calibration board, adjusting the camera to be parallel to the reference calibration board, and ensuring that the field of view of the camera covers the combined calibration board, on the one hand, the camera can collect the calibration board images of the complete combined calibration board, so that subsequent images of sub-calibration boards at multiple different angles and / or different distances can be obtained according to the calibration board images without multiple image acquisitions; on the other hand, the depth information of each feature point in the reference calibration board can be directly determined according to the distance between the reference calibration board and the camera, so as to facilitate the subsequent determination of the world coordinates and pixel coordinates corresponding to the feature points, which is beneficial to improving the accuracy of camera calibration.

[0087] Optionally, determining the pixel coordinates of the feature points of the multiple sub-calibration boards based on the calibration board images includes:

[0088] Based on the calibration board images of the combined calibration board, obtaining sub-images of multiple sub-calibration boards within the calibration board images; wherein, the relative position relationship between the image acquisition surface of the camera and the sub-calibration boards corresponding to the multiple different sub-images is different;

[0089] Based on the sub-images of the multiple sub-calibration boards, determining the pixel coordinates of the feature points within the sub-images of each sub-calibration board.

[0090] It should be noted that when splicing the combined calibration board, feature identifiers ABCD can be set around the multiple sub-calibration boards. After obtaining the images of the combined calibration board, based on the connected regions constructed by the feature identifiers ABCD, the obtained calibration board images are split into sub-images of multiple sub-calibration boards within the calibration board images.

[0091] Here, the relative position relationship between the image acquisition surface of the camera and the sub-calibration plates corresponding to multiple different sub-images is different. It should be noted that the multiple different sub-images are equivalent to the images acquired by the camera at different angles and different distances.

[0092] Therefore, in the embodiments of the present disclosure, by splitting the calibration plate image of the combined calibration plate, sub-images of multiple sub-calibration plates at different angles within the calibration plate image can be obtained, without the need to adjust the camera position for multiple acquisitions.

[0093] It can be understood that after obtaining the sub-images of multiple sub-calibration plates, feature point detection can be performed on the sub-images to facilitate determining the pixel coordinates corresponding to the feature points within the sub-images of each sub-calibration plate.

[0094] In the embodiments of the present disclosure, only one calibration plate image of a multi-angle combined calibration plate needs to be acquired and split, and sub-images of sub-calibration plates at multiple angles can be obtained, thus eliminating the need to adjust the camera angle for multiple image acquisitions, which is beneficial to saving the time of camera acquisition and improving the efficiency of camera calibration.

[0095] Optionally, determining the pixel coordinates of the feature points within the sub-images of each sub-calibration plate based on the sub-images of the multiple sub-calibration plates includes:

[0096] Detecting candidate feature points of the sub-images of the multiple sub-calibration plates and determining the distribution of the candidate feature points;

[0097] Based on the distribution, selecting feature point data;

[0098] Based on the distance relationship between adjacent feature points, determining the pixel coordinates of the feature points within the sub-images of each sub-calibration plate.

[0099] It should be noted that in order to ensure the accuracy of the pixel coordinates and world coordinates corresponding to all sub-image feature points, it is necessary to detect the candidate feature points of the sub-images of each sub-calibration, and based on the detection results of all candidate feature points, select the candidate feature points presenting an m*n arrangement distribution as the feature point data.

[0100] It can be understood that the pixel coordinates are the coordinates of the feature points in the sub-images of the sub-calibration plates in the pixel coordinate system, and the pixel coordinate system is a two-dimensional coordinate system. Therefore, by selecting the coordinate origin on the calibration plate image and establishing a two-dimensional coordinate system, the pixel coordinates corresponding to each feature point within the calibration plate image can be determined.

[0101] In some embodiments, in order to facilitate the calculation of the pixel coordinates corresponding to the feature points, the coordinate origin is usually set at the upper left corner of the calibration plate image. The pixel coordinates of the feature points are (m, n), indicating that the feature point is m pixels away from the coordinate origin in the horizontal direction and n pixels away from the coordinate origin in the vertical direction.

[0102] Here, based on the distance relationship between adjacent feature points, the pixel coordinates of the feature points within the sub-image of each sub-calibration plate are determined. It can be understood that after determining the coordinate origin of the sub-image, a two-dimensional pixel coordinate system can be established based on the coordinate origin, and then based on the distance between each adjacent feature point, the pixel coordinates of the feature points within the sub-image can be calculated.

[0103] In the embodiments of the present disclosure, by detecting the distribution of candidate feature points in the sub-images of multiple sub-calibration plates, selecting the feature point data with regular distribution, and then based on the distance relationship between adjacent feature points, the pixel coordinates of the feature points within the sub-image of each sub-calibration plate are calculated, thereby ensuring the accuracy of the pixel coordinates corresponding to the feature points and further improving the accuracy of camera calibration.

[0104] Optionally, the determining of the world coordinates of the feature points of the multiple sub-calibration plates based on the relative position relationship between the reference calibration plate and the multiple sub-calibration plates in the calibration plate image includes:

[0105] Determining the world coordinates corresponding to each feature point in the reference calibration plate;

[0106] Based on the relative position relationship between the reference calibration plate and each sub-calibration plate, determining the rotation matrix and translation matrix corresponding to each sub-calibration plate;

[0107] Based on the rotation matrix and the translation matrix corresponding to each sub-calibration plate, and the world coordinates of each feature point in the reference calibration plate, respectively determining the world coordinates of the feature points within each sub-calibration plate.

[0108] It should be noted that the world coordinates are the coordinates of the feature points in the sub-image of the sub-calibration plate in the world coordinate system, and the world coordinate system is a three-dimensional coordinate system. Therefore, by selecting the coordinate origin on the sub-calibration plate and establishing a three-dimensional coordinate system, the world coordinates corresponding to each feature point within the sub-calibration plate can be determined.

[0109] In some embodiments, in order to facilitate the calculation of the world coordinates corresponding to the feature points, the coordinate origin is usually set at the upper left corner of the calibration plate. The world coordinates corresponding to the feature points are (m, n, k), indicating that the feature point is m millimeters away from the coordinate origin in the horizontal direction, n millimeters away from the coordinate origin in the vertical direction, and k millimeters away from the coordinate origin in the direction perpendicular to the horizontal and vertical coordinate planes.

[0110] Referring to the world coordinates corresponding to each feature point on the reference calibration board, it can be understood that the coordinate origin is set on the reference calibration board, a three-dimensional world coordinate system is established with the coordinate origin, and based on the distance position relationship between adjacent feature points, the world coordinates corresponding to each feature point on the reference calibration board are calculated.

[0111] It should be explained that each sub-calibration board can make each rotated and / or translated sub-calibration board coincide with the reference calibration board by rotating the relative angle and translating the relative distance. Therefore, the rotation matrix and translation matrix corresponding to each sub-calibration board can be calculated based on the relative position relationship between the reference calibration board and each sub-calibration board.

[0112] Here, the relative position relationship between each sub-calibration board and the reference calibration board is different. Therefore, the rotation matrix and translation matrix of each sub-calibration board relative to the reference calibration board are also different.

[0113] It can be understood that each sub-calibration board can coincide with the reference calibration board through rotation and / or translation. Therefore, after the sub-calibration board coincides with the reference calibration board, the feature points in each sub-calibration board can also coincide with the feature points in the reference calibration board.

[0114] In this way, the world coordinates of the feature points in each sub-calibration board can be calculated respectively based on the rotation matrix and translation matrix corresponding to each sub-calibration board and the world coordinates of each feature point in the reference calibration board.

[0115] Exemplarily, Figure 3 is a schematic diagram showing the positional relationship between a reference calibration board and a sub-calibration board according to an exemplary embodiment. As Figure 3 shown, calibration board 1 is the reference calibration board, the length of each sub-calibration board is H, and the list list1 of the world coordinates corresponding to the feature points on the reference calibration board is [(x0, y0, z0), (x N , y N , z N )], N = m * n. The angle between calibration board 2 and the reference calibration board is equivalent to rotating 180° - angle1 around the y-axis, rotating 0 degrees around the x-axis and rotating 0 degrees around the z-axis, and then translating.

[0116] It can be seen from this that the list list2 of the world coordinates corresponding to the feature points in calibration board 2 can be obtained by calculating the rotation matrix and translation matrix; its rotation matrix: R X = [1, 0, 0; 0, 1, 0; 0, 0, 1], R Y = [cos(180° - angle1), 0, sin(180° - angle1); 0, 1, 0; sin(180° - angle1), 0, cos(180° - angle1)], R Z= [1, 0, 0; 0, 1, 0; 0, 0, 1]; R = R X *R Y *R Z ; Translation matrix: T = [H * cos(180° - angle1); 0; H * sin(180° - angle1)];

[0117] Therefore, [X i ; Y i ; Z i in list2 = R * [x i ; y i ; z i + T; i ∈ [0, N].

[0118] The corresponding world coordinates in other sub - calibration plates can all be calculated according to the rotation and translation relationship between them and the reference calibration plate. Specifically:

[0119] When making the combined calibration plate, the angles of each sub - calibration plate relative to the reference calibration plate are: the angles around the X, Y, and Z axes are α, β, and γ respectively; then the rotation matrices for the three rotations are:

[0120]

[0121] R = R Z (γ) * R y (β) * R x (α); T = [x t ; y t ; z t ;

[0122] That is, the world coordinates corresponding to the feature points of each sub - calibration plate are [X W ; Y W ; Z W = R * [X r ; Y r ; Z r + T; where [X r ; Y r ; Z r are the world coordinates of the corresponding feature points in the reference calibration plate, which are constructed based on the physical distances between the feature points in the reference calibration plate and the position of the corresponding coordinate system origin.

[0123] In the embodiments of the present disclosure, by calculating the world coordinates corresponding to each feature point in the reference calibration plate, and then based on the relative position relationship between the reference calibration plate and each sub - calibration plate, the rotation matrix and translation matrix corresponding to each sub - calibration plate are determined; finally, the world coordinates of the feature points within each sub - calibration plate are calculated. In this way, by introducing the spatial position relationship, more accurate three - dimensional world coordinates are obtained, thereby improving the consistency and stability of camera calibration.

[0124] Optionally, determining the calibration parameters of the camera based on the pixel coordinates and world coordinates of the feature points of the multiple sub-calibration plates includes:

[0125] Determining a training data set based on the multiple sub-calibration plates; wherein, the training data set includes: the pixel coordinates and world coordinates of the feature points of at least one sub-calibration plate;

[0126] Determining the first predicted pixel coordinates of the feature points of the sub-calibration plate based on the world coordinates of the feature points of the sub-calibration plate in the training data set and the initial calibration parameters of the camera;

[0127] Constructing a first objective function based on the error between the first predicted pixel coordinates of the feature points and the pixel coordinates;

[0128] Iteratively optimizing the initial calibration parameters based on the first objective function to obtain the calibration parameters.

[0129] It should be noted that in addition to the reference calibration plate, for the combined calibration plate, the pixel coordinates and world coordinates of the feature points of a certain proportion of sub-calibration plates should be selected from the remaining sub-calibration plates as the training data set, and the remaining ones are the test data set and the validation data set.

[0130] Here, the division ratios of the training data set, the test data set, and the validation data set can be set according to actual needs, and the embodiments of the present disclosure do not limit this. For example, the division ratios of the training data set, the test data set, and the validation data set can be 6:2:2, 7:2:1, 7:1:2, 8:1:1, or 9:0:1, etc.

[0131] It should be explained that through the initial calibration parameters of the camera, that is, the internal parameters, external parameters, and distortion coefficients, an initial calibration function can be determined.

[0132] Determining the first predicted pixel coordinates of the feature points of the sub-calibration plate based on the world coordinates of the feature points of the sub-calibration plate in the training data set and the initial calibration parameters of the camera. It can be understood that by substituting the world coordinates of the feature points of the sub-calibration plate in the training data set into the initial calibration function, the first predicted pixel coordinates corresponding to the feature points of the sub-calibration plate can be obtained.

[0133] Here, after obtaining the first predicted pixel coordinates, calculate the error between the pixel coordinates corresponding to the feature points of the sub-calibration plate in the training data set and the first predicted pixel coordinates, and construct a first objective function based on the error between the two.

[0134] It should be noted that after determining the first objective function, by continuously adjusting the initial calibration parameters of the camera, the world coordinates corresponding to the feature points of multiple sub-calibration plates in the training dataset are brought into the initial calibration function to obtain the first predicted pixel coordinates corresponding to the feature points of the sub-calibration plates; then the pixel coordinates in the training dataset are subtracted from the first predicted pixel coordinates, and iterative optimization is continuously performed until the difference between the two reaches the minimum, thereby determining the target calibration parameters of the camera.

[0135] Exemplarily, the internal parameters, external parameters, and distortion parameters of the camera are constructed as unknown parameter variables, with argmin∑ i＝1...n (||U i -U’(M, R i , T i , k1, k2, p1, p2, k3)|| 2 ) as the first objective function, and iterative optimization is performed using the world coordinates and pixel coordinates of the feature points of the sub-calibration plates in the training dataset to determine the target calibration parameters, that is, the internal parameters, external parameters, and distortion parameters.

[0136] It should be noted that M is the camera internal parameter matrix, R i is the camera rotation matrix of each sub-calibration plate, T i is the camera translation matrix of each sub-calibration plate; k1, k2, k3 are the radial distortion coefficients of the camera; p1, p2 are the tangential distortion coefficients of the camera; U i is the pixel coordinate corresponding to the feature point inside each sub-calibration plate; U’ is the first predicted pixel coordinate of the feature point determined by the camera internal parameters, external parameters, and distortion coefficients.

[0137] In the embodiments of the present disclosure, a training dataset is determined from multiple sub-calibration plates. Based on the world coordinates of the feature points of the sub-calibration plates in the training dataset and the initial calibration parameters of the camera, the first predicted pixel coordinates of the feature points of the sub-calibration plates are calculated; then the error between the first predicted pixel coordinates and the pixel coordinates in the training dataset is used to construct the first objective function; with minimizing the first objective function as the optimization goal, the initial calibration parameters are iteratively optimized until the calibration parameters of the camera are obtained, and the difference between the first predicted pixel coordinates of the feature points determined based on the calibration parameters and the pixel coordinates is minimized, improving the stability of camera calibration.

[0138] Optionally, the iterative optimization of the initial calibration parameters based on the first objective function to obtain the calibration parameters includes:

[0139] Based on the first calibration parameters optimized from the first objective function, determine the rotation matrix and translation matrix between the reference calibration plate and the camera;

[0140] Obtain a validation dataset; wherein, the validation dataset includes: pixel coordinates and world coordinates of feature points of at least one sub-calibration board; the validation dataset is different from the training dataset;

[0141] Based on the world coordinates of the feature points of the sub-calibration board in the validation dataset, the rotation matrix and translation matrix between the sub-calibration board and the reference calibration board, and the rotation matrix and translation matrix between the reference calibration board and the camera, determine the second predicted pixel coordinates of the feature points of the sub-calibration board;

[0142] Based on the difference between the second predicted pixel coordinates and the pixel coordinates of the sub-calibration board in the validation dataset, determine the loss function corresponding to the validation dataset;

[0143] Based on the first objective function and the loss function, construct a second objective function;

[0144] Based on the second objective function, iteratively optimize the first calibration parameters to obtain the calibration parameters.

[0145] It should be noted that in order to further improve the accuracy of camera calibration, after obtaining the calibration parameters by optimizing with the first objective function, secondary optimization can be performed based on the data of the sub-calibration board in the validation dataset.

[0146] Based on the first calibration parameters obtained from the first optimization (i.e., optimization with the first objective function), determine the rotation matrix and translation matrix between the reference calibration board and the camera; and according to the rotation matrix and translation matrix between each sub-calibration board in the multiple sub-calibration boards in the validation dataset and the reference calibration board, determine the rotation matrix and translation matrix between each sub-calibration board in the multiple sub-calibration boards and the camera.

[0147] According to the world coordinates of the feature points of each sub-calibration board in the multiple sub-calibration boards in the validation dataset, and the rotation matrix and translation matrix between the sub-calibration board and the camera, determine the corresponding second predicted pixel coordinates of the feature points of the sub-calibration board.

[0148] It can be understood that after determining the rotation matrix and translation matrix between each sub-calibration board in the multiple sub-calibration boards in the validation dataset and the camera, the world coordinates of the feature points in the sub-calibration board can be used to re-project the feature points into the image coordinate system of the camera to determine the pixel coordinates of the projection points (i.e., the second predicted pixel coordinates).

[0149] Here, after obtaining the second predicted pixel coordinates, calculate the difference between the pixel coordinates corresponding to the feature points of the sub-calibration board in the training dataset and the second predicted pixel coordinates, and construct a loss function based on the difference between the two.

[0150] To obtain more accurate camera calibration parameters, a second objective function can be constructed based on the first objective function and the loss function, with the optimization objective of minimizing the first objective function and the loss function, and iteratively optimizing the first calibration parameters to obtain the target calibration parameters of the camera.

[0151] It can be understood that the difference between the first predicted pixel coordinates and the pixel coordinates of the feature points determined based on the target calibration parameters obtained from the second objective function, and the difference between the second predicted pixel coordinates and the pixel coordinates is minimized.

[0152] Exemplarily, let [u w , v w = U’(M, R1, T1, k1, k2, p1, p2, k3)(X W , Y W , Z W ) be the second predicted pixel coordinate function, where w ∈ [0, N];

[0153] According to the rotation relationship R j and the translation relationship T j between the randomly selected verification calibration board and the reference calibration board, and the rotation relationship R1 and the translation relationship T1 between the reference calibration board and the camera calculated in the optimization equation, the world coordinates of the verification calibration board are reprojected into the image coordinate system, and the projection error between the projection point and the detected corner point is used as the loss function.

[0154] Therefore, let Loss = (∑ i＝1...N |UR W - uw| + |VR W - vw|)N be the loss function.

[0155] Meanwhile, let argmin∑ i＝1...n (||U i - U’(M, R i , T i , k1, k2, p1, p2, k3)|| 2 ) + λ * Loss be the second objective function; the target calibration parameters of the camera are obtained by iteratively optimizing the first calibration parameters.

[0156] It should be noted that M is the camera internal parameter matrix, R i is the camera rotation matrix of each sub - calibration board, T i is the camera translation matrix of each sub - calibration board; k1, k2, k3 are the radial distortion coefficients of the camera; p1, p2 are the tangential distortion coefficients of the camera; U i is the pixel coordinate corresponding to the feature point inside each sub - calibration board; U’ is the first predicted pixel coordinate of the feature point determined by the camera internal parameters, external parameters and distortion coefficients; [Xw ; Y w ; Z w are the pixel coordinates corresponding to the feature points of the sub-calibration board in the verification dataset.

[0157] Based on the calibration parameters optimized by the first objective function in the embodiments of the present disclosure, the rotation matrix and translation matrix between the reference calibration board and the camera are determined; then a verification dataset is selected, and based on the world coordinates corresponding to the feature points of the sub-calibration board in the verification dataset, the rotation matrix and translation matrix between the sub-calibration board and the reference calibration board, and the rotation matrix and translation matrix between the reference calibration board and the camera, the second predicted pixel coordinates of the projection points of the feature points of the sub-calibration board in the verification dataset re-projected into the image coordinate system are determined; then a loss function is constructed using the second predicted pixel coordinates and the pixel coordinates in the verification dataset; finally, based on the loss function and the first objective function, a second objective function is constructed to make the iteratively obtained camera parameters more accurate and further improve the accuracy of camera calibration.

[0158] The embodiments of the present disclosure also provide a camera calibration device, Figure 4 which is a block diagram of a camera calibration device shown according to an exemplary embodiment, as Figure 4 shown. The device 100 includes:

[0159] An acquisition module 101, configured to obtain a calibration board image of the combined calibration board based on any one of the sub-calibration boards in the combined calibration board as the reference calibration board; wherein, the combined calibration board includes: a plurality of interconnected sub-calibration boards, and the rotation directions and / or rotation angles of the plurality of sub-calibration boards are different;

[0160] A first determination module 102, configured to determine the pixel coordinates of the feature points of the plurality of sub-calibration boards based on the calibration board image;

[0161] A second determination module 103, configured to determine the world coordinates of the feature points of the plurality of sub-calibration boards based on the relative position relationship between the reference calibration board and the plurality of sub-calibration boards in the calibration board image;

[0162] A third determination module 104, configured to determine the calibration parameters of the camera based on the pixel coordinates and world coordinates of the feature points of the plurality of sub-calibration boards.

[0163] Figure 5 is a block diagram of the hardware structure of an electronic device shown according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0164] Referring to Figure 5, Device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0165] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above methods. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0166] The memory 804 is configured to store various types of data to support the operation of the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0167] The power component 806 provides power to the various components of the device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0168] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can not only sense the boundaries of the touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0169] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

[0170] The I / O interface 812 provides an interface between the processing component 802 and a peripheral interface module, and the peripheral interface module can be a keyboard, a click wheel, buttons, etc. These buttons can include but are not limited to: a home button, a volume button, a power-on button, and a lock button.

[0171] The sensor component 814 includes one or more sensors for providing status assessments of various aspects of the device 800. For example, the sensor component 814 can detect the on / off state of the device 800, the relative positioning of components, such as the display and keypad of the device 800. The sensor component 814 can also detect a change in the position of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor component 814 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 814 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0172] The communication component 816 is configured to facilitate communication between the device 800 and other devices in a wired or wireless manner. The device 800 can access a wireless network based on communication standards, such as WiFi, 3G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0173] In an exemplary embodiment, the device 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.

[0174] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions, such as a memory 804 including instructions, is also provided. The above instructions may be executed by a processor 820 of the device 800 to complete the above method. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0175] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a network processing device, enables the network processing device to execute a camera calibration method, the method including:

[0176] Based on any one of the sub-calibration plates in the combined calibration plate as a reference calibration plate, obtaining a calibration plate image of the combined calibration plate; wherein, the combined calibration plate includes: a plurality of interconnected sub-calibration plates, and the rotation directions and / or rotation angles of the plurality of sub-calibration plates are different;

[0177] Based on the calibration plate image, determining the pixel coordinates of the feature points of the plurality of sub-calibration plates;

[0178] Based on the relative position relationship between the reference calibration plate and the plurality of sub-calibration plates in the calibration plate image, determining the world coordinates of the feature points of the plurality of sub-calibration plates;

[0179] Based on the pixel coordinates and world coordinates of the feature points of the plurality of sub-calibration plates, determining the calibration parameters of the camera.

[0180] In the examples of the present disclosure, the camera calibration method, the camera calibration device, the electronic device, and the storage medium described are only taken as examples of the embodiments of the present disclosure, but are not limited thereto. As long as it relates to the camera calibration method and the camera calibration device, it is within the protection scope of the present disclosure.

[0181] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, the "in one embodiment" or "in an embodiment" that appears throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present disclosure, the magnitude of the sequence numbers of the above processes does not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present disclosure. The sequence numbers of the embodiments of the present disclosure above are only for description and do not represent the advantages and disadvantages of the embodiments.

[0182] It should be noted that in this text, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or device including that element.

[0183] As described above, it is only the implementation manner of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present disclosure can easily think of changes or substitutions, which should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A camera calibration method, characterized in that, The method includes: Based on any one of the sub-calibration plates in the combined calibration plate as a reference calibration plate, obtaining a calibration plate image of the combined calibration plate; wherein, the combined calibration plate includes: a plurality of interconnected sub-calibration plates, and the rotation directions and / or rotation angles of the plurality of sub-calibration plates are different; the image acquisition surface of the camera for acquiring the calibration plate image is parallel to the reference calibration plate, and the field of view of the camera covers the combined calibration plate; the distance information between the reference calibration plate and the camera is used to determine the depth information of each feature point in the reference calibration plate; Based on the calibration plate image, determining the pixel coordinates of the feature points of the plurality of sub-calibration plates; Based on the relative position relationship between the reference calibration plate and the plurality of sub-calibration plates in the calibration plate image, determining the world coordinates of the feature points of the plurality of sub-calibration plates; Based on the world coordinates of the feature points of the sub-calibration plates in the training dataset and the initial calibration parameters of the camera, determining the first predicted pixel coordinates of the feature points of the sub-calibration plates; wherein, the training dataset includes: the pixel coordinates and world coordinates of the feature points of at least one sub-calibration plate; Constructing a first objective function based on the error between the first predicted pixel coordinates and the pixel coordinates of the feature points; Iteratively optimizing the initial calibration parameters based on the first objective function to obtain first calibration parameters; Based on the difference between the second predicted pixel coordinates and the pixel coordinates of the feature points of the sub-calibration plates in the validation dataset, determining the loss function corresponding to the validation dataset; wherein, the validation dataset includes: the pixel coordinates and world coordinates of the feature points of at least one sub-calibration plate; the validation dataset is different from the training dataset; Constructing a second objective function based on the first objective function and the loss function; Iteratively optimizing the first calibration parameters based on the second objective function to obtain the calibration parameters of the camera; wherein, the calibration parameters include internal parameters, external parameters, and distortion parameters.

2. The method according to claim 1, characterized in that, The step of obtaining the calibration plate image of the combined calibration plate based on any one of the sub-calibration plates in the combined calibration plate as a reference calibration plate includes: Determining the reference calibration plate from the plurality of sub-calibration plates of the combined calibration plate; Adjusting the camera based on the reference calibration plate; the image acquisition surface of the adjusted camera is parallel to the reference calibration plate, and the field of view of the camera covers the combined calibration plate; Based on the adjusted camera, acquiring the calibration plate image of the combined calibration plate.

3. The method according to claim 1, wherein The step of determining the pixel coordinates of the feature points of the plurality of sub-calibration plates based on the calibration plate image includes: Based on the calibration plate image of the combined calibration plate, obtaining sub-images of the plurality of sub-calibration plates within the calibration plate image; wherein, the relative position relationship between the image acquisition surface of the camera and the sub-calibration plates corresponding to the plurality of different sub-images is different; Based on the sub-images of the plurality of sub-calibration plates, determining the pixel coordinates of the feature points within the sub-images of each sub-calibration plate.

4. The method according to claim 3, characterized in that The step of determining the pixel coordinates of the feature points within the sub-images of each sub-calibration plate based on the sub-images of the plurality of sub-calibration plates includes: Detect candidate feature points of sub-images of the multiple sub-calibration plates, and determine the distribution of the candidate feature points; Select feature point data based on the distribution; Based on the distance relationship between adjacent feature points, determine the pixel coordinates of the feature points within the sub-image of each sub-calibration plate.

5. The method according to claim 1, wherein The determining of the world coordinates of the feature points of the multiple sub-calibration plates based on the relative position relationship between the reference calibration plate and the multiple sub-calibration plates in the calibration plate image includes: Determine the world coordinates corresponding to each feature point in the reference calibration plate; Based on the relative position relationship between the reference calibration plate and each sub-calibration plate, determine the rotation matrix and translation matrix corresponding to each sub-calibration plate; Based on the rotation matrix and the translation matrix corresponding to each sub-calibration plate, and the world coordinates of each feature point in the reference calibration plate, respectively determine the world coordinates of the feature points within each sub-calibration plate.

6. The method according to claim 1, characterized in that The method includes: Based on the first calibration parameters optimized by the first objective function, determine the rotation matrix and translation matrix between the reference calibration plate and the camera; Based on the world coordinates of the feature points of the sub-calibration plate in the validation data set, the rotation matrix and translation matrix between the sub-calibration plate and the reference calibration plate, and the rotation matrix and translation matrix between the reference calibration plate and the camera, determine the second predicted pixel coordinates of the feature points of the sub-calibration plate.

7. A camera calibration device, characterized in that, The apparatus includes: An acquisition module, configured to acquire a calibration plate image of the combined calibration plate with any one of the sub-calibration plates in the combined calibration plate as the reference calibration plate; wherein, the combined calibration plate includes: multiple interconnected sub-calibration plates, and the rotation directions and / or rotation angles of the multiple sub-calibration plates are different; the image acquisition plane of the camera for acquiring the calibration plate image is parallel to the reference calibration plate, and the field of view of the camera covers the combined calibration plate; the distance information between the reference calibration plate and the camera is used to determine the depth information of each feature point in the reference calibration plate; A first determination module, configured to determine the pixel coordinates of the feature points of the multiple sub-calibration plates based on the calibration plate image; A second determination module, configured to determine the world coordinates of the feature points of the multiple sub-calibration plates based on the relative position relationship between the reference calibration plate and the multiple sub-calibration plates in the calibration plate image; A third determination module, configured to determine first predicted pixel coordinates of feature points of the sub-calibration board based on world coordinates of the feature points of the sub-calibration board in the training dataset and initial calibration parameters of the camera; wherein the training dataset includes: pixel coordinates and world coordinates of feature points of at least one sub-calibration board; construct a first objective function based on an error between the first predicted pixel coordinates of the feature points and the pixel coordinates; iteratively optimize the initial calibration parameters based on the first objective function to obtain first calibration parameters; determine a loss function corresponding to the validation dataset based on a difference between second predicted pixel coordinates of the feature points of the sub-calibration board in the validation dataset and the pixel coordinates; wherein the validation dataset includes: pixel coordinates and world coordinates of feature points of at least one sub-calibration board; the validation dataset is different from the training dataset; construct a second objective function based on the first objective function and the loss function; iteratively optimize the first calibration parameters based on the second objective function to obtain calibration parameters of the camera; wherein the calibration parameters include internal parameters, external parameters, and distortion parameters.

8. An electronic device, characterized in that, Comprising a processor and a memory, the memory is used to store code instructions; the processor is used to run the code instructions to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, For storing a computer program, the computer program includes instructions for implementing the method according to any one of claims 1 to 6.

Images