Camera calibration method, device and equipment and computer readable storage medium

Abstract

The invention relates to a camera calibration method, device and equipment and a computer readable storage medium. The method comprises the following steps: acquiring a first checkerboard image acquired by a to-be-calibrated camera for a first checkerboard and a second checkerboard image acquired by a to-be-calibrated camera for a second checkerboard, performing corner detection on the first checkerboard image to obtain a first corner, and performing corner detection on the second checkerboard image to obtain a second corner; determining an initial calibration model according to the first image coordinate and the first physical coordinate of the first angular point; according to the second image coordinate of the second corner point and the initial calibration model, determining an alternative physical coordinate corresponding to the second corner point; aligning a second physical coordinate of the second angular point with the alternative physical coordinate to obtain a target physical coordinate corresponding to the second physical coordinate; and determining a target calibration model according to the target physical coordinate and the second image coordinate. According to the invention, high-precision calibration of the camera can be realized.

Description

Technical Field

[0001] The present application relates to the technical field of device calibration, and particularly to a camera calibration method, device, equipment and computer-readable storage medium. Background Art

[0002] In computer vision technology, in order to determine the mapping relationship between the three-dimensional position of a certain point in the world coordinate system and the two-dimensional position in the image coordinate system, it is necessary to establish a geometric model of camera imaging. The parameters of the geometric model are the camera calibration parameters, and the process of solving the camera calibration parameters is camera calibration.

[0003] In the traditional technology, based on the three-dimensional reconstruction method of images, a number of disordered images are taken from several perspectives in a room, and the camera to be calibrated also takes one image. These images are combined and the structure from motion algorithm for three-dimensional reconstruction is used to determine the poses corresponding to all the images, and the accuracy of camera calibration is relatively low. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a camera calibration method, device, equipment, computer-readable storage medium and computer program product, which can achieve high-precision calibration of the camera.

[0005] In a first aspect, the present application provides a camera calibration method, including:

[0006] Obtain a first checkerboard image collected by a camera to be calibrated for a first checkerboard, and a second checkerboard image collected for a second checkerboard, perform corner detection on the first checkerboard image to obtain first corners, and perform corner detection on the second checkerboard image to obtain second corners; the accuracy of the second checkerboard is higher than that of the first checkerboard;

[0007] Determine an initial calibration model according to the first image coordinates and first physical coordinates of the first corners;

[0008] Determine alternative physical coordinates corresponding to the second corners according to the second image coordinates of the second corners and the initial calibration model;

[0009] Align the second physical coordinates of the second corners with the alternative physical coordinates to obtain target physical coordinates corresponding to the second physical coordinates;

[0010] Determine a target calibration model according to the target physical coordinates and the second image coordinates.

[0011] In a second aspect, the present application provides a camera calibration device, including:

[0012] A corner detection module, configured to obtain a first checkerboard image acquired by a camera to be calibrated for a first checkerboard, and a second checkerboard image acquired for a second checkerboard, perform corner detection on the first checkerboard image to obtain first corners, and perform corner detection on the second checkerboard image to obtain second corners; the accuracy of the second checkerboard is higher than that of the first checkerboard;

[0013] An initial model determination module, configured to determine an initial calibration model according to the first image coordinates and first physical coordinates of the first corners;

[0014] A physical coordinate calculation module, configured to determine alternative physical coordinates corresponding to the second corners according to the second image coordinates of the second corners and the initial calibration model;

[0015] A physical coordinate alignment module, configured to align the second physical coordinates of the second corners with the alternative physical coordinates to obtain target physical coordinates corresponding to the second physical coordinates;

[0016] A target model determination module, configured to determine a target calibration model according to the target physical coordinates and the second image coordinates.

[0017] In a third aspect, the present application provides a camera calibration device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps in the above method are implemented.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method are implemented.

[0019] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in the above method are implemented.

[0020] For the above camera calibration method, device, equipment, computer-readable storage medium and computer program product, by using the checkerboard images sampled by the camera to be calibrated for the first checkerboard and the second checkerboard with different accuracies respectively to calibrate the camera to be calibrated, it is possible to obtain an initial calibration model from the first checkerboard image acquired from the first checkerboard with lower accuracy, and then optimize the parameters of the initial calibration model through the second checkerboard image acquired from the second checkerboard with higher accuracy to obtain a target calibration model with higher accuracy, thereby realizing high-precision calibration of the camera to be calibrated and enabling high-precision positioning measurement of the target. Description of the Drawings

[0021] Figure 1An application environment diagram of a camera calibration method provided by an embodiment of the present application;

[0022] Figure 2 A flowchart of a camera calibration method provided by an embodiment of the present application;

[0023] Figure 3 A schematic diagram of a calibration checkerboard provided by an embodiment of the present application;

[0024] Figure 4 A schematic diagram of checkerboard corner points provided by an embodiment of the present application;

[0025] Figure 5 A schematic diagram of a checkerboard including a marker code provided by an embodiment of the present application;

[0026] Figure 6 A flowchart of another camera calibration method provided by an embodiment of the present application;

[0027] Figure 7 A structural block diagram of a camera calibration device provided by an embodiment of the present application;

[0028] Figure 8 An internal structure diagram of a camera calibration device provided by an embodiment of the present application;

[0029] Figure 9 An internal structure diagram of another camera calibration device provided by an embodiment of the present application;

[0030] Figure 10 An internal structure diagram of a computer-readable storage medium provided by an embodiment of the present application. Detailed implementation manners

[0031] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0032] The camera calibration method provided by the embodiments of the present application can be applied to, for example Figure 1In the application environment shown. Among them, the camera 102 to be calibrated can collect the first checkerboard image for the first checkerboard, and then place the second checkerboard at the position of the first checkerboard, or place the second checkerboard and the first checkerboard on the same plane. The accuracy of the second checkerboard is higher than that of the first checkerboard; the camera 102 to be calibrated collects the second checkerboard image for the second checkerboard, and then can transmit the first checkerboard image and the second checkerboard image collected by the camera 102 to be calibrated to the server 104. After the server 104 obtains the first checkerboard image and the second checkerboard image, it performs corner detection on the first checkerboard image to obtain the first corner, performs corner detection on the second checkerboard image to obtain the second corner, determines the initial calibration model according to the first image coordinates and the first physical coordinates of the first corner, determines the alternative physical coordinates corresponding to the second corner according to the second image coordinates of the second corner and the initial calibration model, aligns the second physical coordinates of the second corner with the alternative physical coordinates to obtain the target physical coordinates corresponding to the second physical coordinates, and determines the target calibration model according to the target physical coordinates and the second image coordinates. It should be noted that the first checkerboard image and the second checkerboard image collected by the camera to be calibrated can also be transmitted to the terminal, and the terminal processes the first checkerboard image and the second checkerboard image. The terminal can be but is not limited to various personal computers, laptops, smartphones, tablets, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Or, the processor of the camera to be calibrated can also process the first checkerboard image and the second checkerboard image, and finally obtain the target calibration model of the camera to be calibrated, so that the camera to be calibrated can convert the image coordinates at any position in the captured image into the coordinates of the same physical coordinate system.

[0033] As Figure 2 shown, an embodiment of the present application provides a camera calibration method, taking the method applied to Figure 1 the server 104 in it as an example for illustration. It can be understood that the camera calibration device can include at least one of a terminal and a server. The method includes the following steps:

[0034] S202. Obtain the first checkerboard image collected by the camera to be calibrated for the first checkerboard, and the second checkerboard image collected for the second checkerboard, perform corner detection on the first checkerboard image to obtain the first corner, and perform corner detection on the second checkerboard image to obtain the second corner; where the accuracy of the second checkerboard is higher than that of the first checkerboard.

[0035] Among them, the camera to be calibrated refers to the camera that needs to be calibrated, and there can be one or more cameras to be calibrated. If there are multiple cameras to be calibrated, multiple first checkerboard images can be respectively acquired for the first checkerboard by the multiple cameras to be calibrated, and multiple second checkerboard images can be respectively acquired for the second checkerboard. The first checkerboard and the second checkerboard are calibration checkerboards with different precisions, and the second checkerboard has a higher precision. For example, it can be a professionally customized checkerboard or a checkerboard that has been corrected. Exemplarily, the schematic diagram of the calibration checkerboard is as shown in Figure 3 shown. The precision of the second checkerboard can be higher than the precision threshold, and the precision threshold can include at least one of a scale threshold, a straightness threshold, a flatness threshold, etc. Among them, the scale threshold is the threshold characterizing the accuracy of the checkerboard scale, the straightness threshold is the threshold characterizing the straightness of the straight lines in the checkerboard, and the flatness threshold is the threshold characterizing the flatness of the checkerboard surface. The precision of the second checkerboard is higher than that of the first checkerboard. For example, it can mean that the second checkerboard is higher than the first checkerboard in at least one of the checkerboard scale precision, straight line precision, or surface flatness. For example, if the checkerboard scale precision, straight line height, and surface flatness of the second checkerboard are all higher than those of the first checkerboard, it indicates that the precision of the second checkerboard is higher than that of the first checkerboard.

[0036] The first checkerboard image is the image acquired by the camera to be calibrated for the first checkerboard. Among them, the first checkerboard image can be the image acquired by the camera to be calibrated for the entire first checkerboard, or the image acquired for a part of the first checkerboard. Correspondingly, the second checkerboard image is the image acquired by the camera to be calibrated for the second checkerboard. The second checkerboard image can be the image acquired by the camera to be calibrated for the entire second checkerboard, or the image acquired for a part of the second checkerboard. In an actual application scenario, the camera to be calibrated can first acquire the first checkerboard image for the first checkerboard, then place the second checkerboard at the position of the first checkerboard, or cover the first checkerboard with the second checkerboard, and then acquire the second checkerboard image for the second checkerboard. Or, the first checkerboard and the second checkerboard can be placed on the same plane, and the camera to be calibrated sequentially acquires the first checkerboard and the second checkerboard, and correspondingly obtains the first checkerboard image and the second checkerboard image in sequence. Exemplarily, both the first checkerboard and the second checkerboard include marker codes. In the same checkerboard, there can be multiple marker codes, and each marker code is used to represent the physical coordinate information corresponding to the position of the corresponding marker code. When the camera to be calibrated acquires an image of the first checkerboard or the second checkerboard, it is necessary to acquire the corresponding marker code. That is to say, the acquired first checkerboard image and second checkerboard image both include the corresponding marker code information.

[0037] In an exemplary embodiment, the corner detection algorithm can be used to detect the corners of the checkerboard image, and the detected corners are as shown in Figure 4 (the vertex positions in the black circles in the figure). A corner can be the intersection of two lines or a point on two adjacent things with different main directions. The operators included in the corner detection algorithm can be at least one of, for example, the Harris operator, the Moravec operator, the Forstner operator, or the Susan operator, etc. Alternatively, the findChessboardCorners function of opencv can also be used for corner detection. The corner detection algorithm can be used to detect the corners of the first checkerboard image to obtain the first corners, and the corner detection algorithm can be used to detect the corners of the second checkerboard image to obtain the second corners. That is, the first corners are the corners in the first checkerboard image, and the second corners are the corners in the second checkerboard image. It is easy to understand that the first checkerboard image and the second checkerboard image can be respectively detected for corners by the same corner detection algorithm or different corner detection algorithms to correspondingly obtain the first corners and the second corners.

[0038] S204. Determine the initial calibration model according to the first image coordinates and the first physical coordinates of the first corners.

[0039] Among them, the first image coordinates refer to the image coordinates of the first corners in the first checkerboard image, and the first physical coordinates refer to the physical coordinates of the first corners. The physical coordinates refer to the coordinates on the physical imaging plane. The physical coordinates are equivalent to describing the position of pixels using physical units and are two-dimensional coordinates. The physical coordinate system can be set according to the actual application scenario, that is, the origin and the axis directions of the physical coordinate system can be set according to actual needs. The initial calibration model is the calibration model determined according to the first image coordinates and the first physical coordinates. The image coordinates refer to the coordinates of the imaged image. It is easy to understand that the image coordinates can be easily obtained from the collected image, that is, the first image coordinates can be obtained from the first checkerboard image, and the second image coordinates can be obtained from the second checkerboard image. The image coordinate systems corresponding to the images collected by the same camera are the same.

[0040] Exemplarily, the initial calibration model can be determined according to the multiple first image coordinates and the multiple first physical coordinates of multiple first corners. Specifically, the conversion relationship between the first image coordinates and the first physical coordinates can be determined according to the multiple first image coordinates and the multiple first physical coordinates, and the initial calibration model can be determined according to the conversion relationship and the calibration difference condition. Among them, the calibration difference condition is used to constrain the accuracy of the initial calibration model. For example, the calibration difference condition is that the image coordinates corresponding to the physical coordinates are calculated according to the conversion relationship, and the difference between the calculated image coordinates and the actual image coordinates is less than the difference threshold.

[0041] S206. Determine the alternative physical coordinates corresponding to the second corner point according to the second image coordinates of the second corner point and the initial calibration model.

[0042] Among them, the second image coordinates refer to the image coordinates of the second corner point in the second checkerboard image. Specifically, the second image coordinates of the second corner point can be input into the initial calibration model to obtain the alternative physical coordinates corresponding to the second corner point. It is easy to understand that the alternative physical coordinates calculated according to the initial calibration model may be the same as or different from the second physical coordinates of the second corner point.

[0043] S208. Align the second physical coordinates of the second corner point with the alternative physical coordinates to obtain the target physical coordinates corresponding to the second physical coordinates.

[0044] Among them, aligning the second physical coordinates of the second corner point with the alternative physical coordinates is equivalent to aligning the physical coordinate system of the second corner point with the physical coordinate system of the first corner point. For example, the second physical coordinates and the alternative physical coordinates can be aligned through certain rotation operations and translation operations to obtain the aligned target physical coordinates.

[0045] Exemplarily, an alignment vector can be determined according to the second physical coordinates and the alternative physical coordinates, and then the target physical coordinates corresponding to the second physical coordinates can be obtained according to the second physical coordinates and the alignment vector.

[0046] S210. Determine the target calibration model according to the target physical coordinates and the second image coordinates.

[0047] Among them, the accuracy of the target calibration model is usually higher than that of the initial calibration model. After the camera to be calibrated is calibrated through the target calibration model, the image coordinates of different images located can be converted into physical coordinates in the same physical coordinate system to achieve high-precision measurement.

[0048] It is easy to understand that the method of determining the target calibration model according to the target physical coordinates and the second image coordinates can be implemented by referring to the method of determining the initial calibration model according to the first image coordinates and the first physical coordinates of the first corner point, and replacing the first physical coordinates with the target physical coordinates and the first image coordinates with the second image coordinates.

[0049] It can be seen that in the embodiments of the present application, the camera to be calibrated is calibrated by using the checkerboard images obtained by the camera to be calibrated sampling the first checkerboard and the second checkerboard with different precisions respectively. It is possible to obtain an initial calibration model from the first checkerboard image collected from the first checkerboard with lower precision, and then optimize the parameters of the initial calibration model by using the second checkerboard image collected from the second checkerboard with higher precision to obtain a target calibration model with higher precision, thereby realizing high-precision calibration of the camera to be calibrated and high-precision positioning measurement of the measurement target.

[0050] In some embodiments, aligning the second physical coordinates of the second corner points with the alternative physical coordinates to obtain the target physical coordinates corresponding to the second physical coordinates in S208 includes:

[0051] Fitting the second physical coordinates of the second corner points with the alternative physical coordinates to obtain a coordinate fitting relationship; determining the target rotation amount and the target translation amount according to the coordinate fitting relationship; and aligning the second physical coordinates with the alternative physical coordinates according to the target rotation amount and the target translation amount to obtain the target physical coordinates corresponding to the second physical coordinates.

[0052] Among them, the coordinate fitting relationship is used to represent the conversion relationship between the first physical coordinates and the alternative physical coordinates. The target rotation amount and the target translation amount are equivalent to the alignment parameters for aligning the second physical coordinates with the alternative physical coordinates.

[0053] Exemplarily, the relationship between the second physical coordinates of the second corner points and the alternative physical coordinates can be fitted by means of a rigid transformation or a non-rigid transformation, so as to obtain a coordinate fitting relationship. The target rotation amount and the target translation amount can be determined from the coordinate fitting relationship. After rotating the second physical coordinates by the target rotation amount and translating them by the target translation amount, the target physical coordinates corresponding to the second physical coordinates are obtained, that is, the alignment of the second physical coordinates with the alternative physical coordinates is realized.

[0054] In an exemplary embodiment, the second physical coordinates and the alternative physical coordinates of multiple second corner points are fitted by a 2D (two-dimensional) rigid transformation, and the obtained coordinate fitting relationship is shown in the following formula (1).

[0055] Formula (1)

[0056] Among them, R represents the target rotation amount, t represents the target translation amount, N represents the number of second corner points, i represents the i-th second corner point, represents the second physical coordinates of the i-th second corner point, represents the alternative physical coordinates of the i-th second corner point. It should be noted that the target rotation amount includes the angle and direction of rotation, that is, the target rotation amount can be represented in the form of a rotation vector.

[0057] According to the coordinate fitting relationship in formula (1), the target rotation amount R and the target translation amount t can be obtained, and then according to R and t, the target physical coordinates corresponding to the specific second physical coordinates can be determined. For example, the second physical coordinates of the j-th second corner point The corresponding target physical coordinates can be determined by the following formula (2).

[0058] Formula (2)

[0059] It should be noted that the specific fitting process of the coordinate fitting relationship can be selected according to the actual application scenario and is not specifically limited here.

[0060] It can be seen that in this embodiment, by fitting the second physical coordinates of the second corner points with the alternative physical coordinates, the target rotation amount and the target translation amount for aligning the second physical coordinates of the second corner points with the alternative physical coordinates are obtained, and then alignment is performed according to the target rotation amount and the target translation amount to obtain the aligned target physical coordinates, so that the second physical coordinates can be accurately aligned with the alternative physical coordinates.

[0061] In some embodiments, there are multiple first corner points; according to the first image coordinates and the first physical coordinates of the first corner points, determining the initial calibration model includes:

[0062] According to the first image coordinates and the first physical coordinates of each first corner point, determining the conversion relationship between the image coordinates and the physical coordinates; according to the conversion relationship and the first physical coordinates, determining the alternative image coordinates corresponding to the first physical coordinates; and optimizing the conversion relationship according to the difference between the alternative image coordinates and the first image coordinates to obtain the initial calibration model.

[0063] Exemplarily, the first image coordinates and the first physical coordinates of multiple first corner points can be calibrated to determine the conversion relationship between the image coordinates and the physical coordinates, substitute the first physical coordinates into the conversion relationship to obtain the alternative image coordinates corresponding to the first physical coordinates, and optimize the parameters of the conversion relationship according to the difference between the alternative image coordinates and the first image coordinates, so that the difference between the alternative image coordinates and the first image coordinates is less than the difference threshold, obtain the optimized conversion relationship, and use the optimized conversion relationship as the initial calibration model. The optimized conversion relationship parameters are the model parameters of the initial calibration model.

[0064] In one example, the first image coordinates and the first physical coordinates of multiple first corner points can be expressed as shown in the following formula (3):

[0065] Formula (3)

[0066] Where M represents the number of first corner points.

[0067] Assume that the conversion relationship between image coordinates and physical coordinates is θ. Then, the first physical coordinate of any first corner point can be input into θ to obtain the alternative image coordinates corresponding to the first physical coordinate. Compare the alternative image coordinates with the first image coordinates. If the difference between the alternative image coordinates and the first image coordinates is greater than the difference threshold, continue to adjust the parameters of θ until the difference between the alternative image coordinates and the first image coordinates is less than or equal to the difference threshold, and an optimized conversion relationship is obtained. , use as the initial calibration model. Exemplarily, It can be expressed as shown in the following formula (4).

[0068] Formula (4)

[0069] Where represents the first physical coordinate of the i-th first corner point, represents the first image coordinate of the i-th corner point, represents the alternative image coordinates corresponding to the first physical coordinate.

[0070] In other words, the initial calibration model can make the difference between the alternative image coordinates and the first image coordinates less than the difference threshold.

[0071] It can be seen that in this embodiment, through the conversion relationship between the image coordinates and physical coordinates corresponding to the first corner points, the alternative image coordinates corresponding to the first physical coordinates are determined, and the parameters of the conversion relationship are optimized according to the difference between the alternative image coordinates and the first image coordinates. After meeting the optimization conditions, an initial calibration model is obtained, which can obtain a more accurate initial calibration model and lay a good foundation for reducing the camera calibration error.

[0072] In some embodiments, the first checkerboard image includes marker code information; before determining the initial calibration model according to the first image coordinates and the first physical coordinates of the first corner points, the above method further includes: identifying the marker code information in the first checkerboard image to determine the physical coordinates of the first marker code; and determining the first physical coordinates of the first corner points according to the positional relationship between the first marker code and the first corner points and the physical coordinates of the first marker code.

[0073] Understandably, the second checkerboard image may or may not include marker code information. If the second checkerboard image does not include marker code information, after detecting the second corner points, the second physical coordinates of the second corner points can be determined in sequence according to the positions of the second corner points in the second checkerboard image and the physical sizes of the checkerboard squares in the second checkerboard. For example, if the second physical coordinates of the first second corner point in the upper left corner of the second checkerboard image can be set to (0, 0), and if the size of the checkerboard square is 5 mm, then the second physical coordinates of the second corner point adjacent to the right of the first second corner point can be set to (5, 0), and the second physical coordinates of the second corner point adjacent to the bottom of the first second corner point can be set to (0, 5), and so on, the second physical coordinates of all second corner points can be obtained.

[0074] Exemplarily, the second checkerboard image includes marker code information. Before aligning the second physical coordinates of the second corner points with the alternative physical coordinates to obtain the target physical coordinates corresponding to the second physical coordinates, the marker code information in the second checkerboard image can be recognized to determine the physical coordinates of the second marker code; according to the positional relationship between the second marker code and the second corner points, and the physical coordinates of the second marker code, the second physical coordinates of the second corner points can be determined.

[0075] Among them, the marker code information refers to the position information corresponding to the marker code. The marker code information included in the first checkerboard image refers to the position information corresponding to the first marker code included in the first checkerboard, and the marker code information included in the second checkerboard image refers to the position information corresponding to the second marker code included in the second checkerboard. The marker code can be characterized by, for example, a two-dimensional QR (2-dimensional bar code) code, an aruco (Hamming code grid graph) code, or other forms of marker codes, etc. According to the marker code, the physical coordinates of the marker code position can be determined. Understandably, the first marker code and the second marker code can be of the same type or different types. Exemplarily, the checkerboard including the marker code is as Figure 5 shown Figure 5 in which the black square 502 represents the marker code position. Among them, the marker code can be located at any position in the checkerboard, and one or more marker codes can be included in the same checkerboard. That is to say, the first marker code can be one or more, and the second marker code can also be one or more.

[0076] Exemplarily, by recognizing the marker code information in the checkerboard image, the physical coordinates of the marker code can be determined. Based on the positional relationship between the marker code and the checkerboard corner points, when the physical coordinates of the marker code are determined, the origin and axis directions of the corresponding physical coordinate system can be determined. Based on this physical coordinate system, the physical coordinates of the corresponding checkerboard corner points can be determined. It should be noted that the position (physical coordinates) of the marker code can represent any position in the corresponding physical coordinate system. As Figure 5 shown, if the marker code information is recognized to determine the origin and axis directions of the physical coordinate system, that is, it is determined that the upper left corner of the checkerboard image is the origin of the coordinate system, the horizontal arrow is the X-axis direction, and the vertical arrow is the Y-axis direction. Assuming that the physical length and physical width corresponding to each checkerboard are both 15, then the physical coordinates of the first corner point in the upper left corner are (15, 15), the physical coordinates of the second corner point horizontally are (30, 15), the physical coordinates of the second corner point vertically are (15, 30), and the physical coordinates of other corner points can be obtained by analogy, and the physical coordinates of any corner point in the corresponding checkerboard image can be obtained. Alternatively, the marker code information includes the X-direction offset and Y-direction offset of the center position of the marker code relative to the origin of the physical coordinate system, so that the coordinates of the corner points around the marker code in this physical coordinate system can be determined. For example, by scanning the marker code and recognizing the marker code information, (19.5, 6.5, P10, M2) is obtained. Among them, 19.5 represents that the center position of the marker code is offset by 19.5 checkerboard squares in the X-axis direction relative to the origin of the physical coordinate system, 6.5 represents that the center position of the marker code is offset by 6.5 checkerboard squares in the Y-axis direction relative to the origin of the physical coordinate system, M2 represents that the width of the marker code is 2 checkerboard squares, and the upper, lower, left, and right edge transition areas are 0.5 checkerboard squares, and P10 represents that the physical size of one checkerboard square is 10 mm. Then, the center position of the marker code occupies one checkerboard square and 0.5 checkerboard squares horizontally and vertically respectively. Except for 1.5 checkerboard squares, it can be obtained that the upper left corner point of the marker code is offset by 19.5 - 1.5 = 18 checkerboard squares in the X-axis direction and 6.5 - 1.5 checkerboard squares in the Y-axis direction, and the coordinates of the upper left corner point of the marker code can be obtained as (18 * 10 mm, 5 * 10 mm). Among them, the checkerboard square is the smallest component unit of the checkerboard, that is, the smallest square in the checkerboard.

[0077] Among them, the above marker code information can be the marker code information in the first checkerboard image or the marker code information in the second checkerboard image. If the marker code information is the marker code information in the first checkerboard image, the physical coordinates of the first marker code in the first checkerboard can be correspondingly determined, so as to determine the first physical coordinates of the first corner point; if the marker code information is the marker code information in the second checkerboard image, the physical coordinates of the second marker code in the second checkerboard can be correspondingly determined, so as to determine the second physical coordinates of the second corner point.

[0078] It can be seen that in this embodiment, by identifying the marker code information in the first checkerboard image, the physical coordinates of the first marker code can be determined, and according to the positional relationship between the first marker code and the first corner point, the first physical coordinates of the first corner point can be determined, which can accurately and quickly determine the first physical coordinates of the first corner point and lay a foundation for accurately determining the target calibration model subsequently.

[0079] In some embodiments, there are multiple cameras to be calibrated; obtaining the first checkerboard image collected by the cameras to be calibrated for the first checkerboard and the second checkerboard image collected for the second checkerboard includes:

[0080] Multiple cameras to be calibrated simultaneously photograph the first checkerboard to obtain multiple first checkerboard images; multiple cameras to be calibrated simultaneously photograph the second checkerboard to obtain multiple second checkerboard images; wherein, each first checkerboard image includes marker code information.

[0081] In an actual application scenario, multiple cameras to be calibrated can be calibrated simultaneously. Specifically, multiple cameras to be calibrated can simultaneously photograph the same first checkerboard to obtain the first checkerboard image corresponding to each camera to be calibrated, thereby obtaining multiple first checkerboard images. Correspondingly, by multiple cameras to be calibrated simultaneously photographing the same second checkerboard, the second checkerboard image corresponding to each camera to be calibrated is obtained, thereby obtaining multiple second checkerboard images.

[0082] Exemplarily, multiple cameras to be calibrated can simultaneously photograph the first checkerboard. During the photographing process, each camera to be calibrated needs to collect the first marker code in the first checkerboard to obtain multiple first checkerboard images. Each first checkerboard image includes the marker code information corresponding to the first marker code. Then, the second checkerboard is placed at the position of the first checkerboard, or the second checkerboard is placed on the plane where the first checkerboard is located, that is, the first checkerboard and the second checkerboard are in the same plane, and multiple cameras to be calibrated simultaneously photograph the second checkerboard. The second checkerboard image may or may not include marker code information. During the photographing process, if the second checkerboard image includes marker code information, then each camera to be calibrated needs to collect the second marker code in the second checkerboard to obtain multiple second checkerboard images. Correspondingly, each second checkerboard image includes the marker code information corresponding to the second marker code.

[0083] It can be seen that in this embodiment, by multiple cameras to be calibrated simultaneously photographing the checkerboard to obtain the checkerboard image, multiple cameras to be calibrated can be calibrated simultaneously, so that multiple cameras to be calibrated can convert the image coordinates to the same physical coordinate system. When using the cameras to perform positioning measurement on a relatively large target, it is possible to achieve high-precision measurement of a relatively large target through networking of two or more cameras.

[0084] In some embodiments, the first checkerboard is a printed checkerboard, and the second checkerboard is a standard checkerboard.

[0085] Among them, the printed checkerboard refers to a checkerboard obtained by printing with a printer. The printer can be an office or home printer, such as an inkjet printer or a laser printer. In other words, the printed checkerboard can be obtained in a relatively simple way. For example, the checkerboard pattern can be preset through office software and then printed with a printer to obtain a paper printed checkerboard.

[0086] The standard checkerboard refers to a checkerboard with a checkerboard accuracy higher than the accuracy threshold. For example, it can be a customized checkerboard specifically for calibration, such as a standard checkerboard engraved on glass or acrylic materials using a high-precision engraving laser. It is easy to understand that the checkerboard accuracy can be manifested in aspects such as checkerboard scale accuracy, straight line accuracy, and surface flatness. In other words, if at least one of the checkerboard scale accuracy, straight line accuracy, or surface flatness of the checkerboard is higher than the corresponding threshold, it can be stated that the checkerboard accuracy of the checkerboard is higher than the accuracy threshold. For example, if the checkerboard scale accuracy of the checkerboard is higher than the scale threshold, it indicates that the checkerboard accuracy of the checkerboard is higher than the accuracy threshold. Or, if the checkerboard scale accuracy of the checkerboard is higher than the scale threshold and the straight line accuracy is higher than the straightness threshold, it indicates that the checkerboard accuracy of the checkerboard is higher than the accuracy threshold. Or, if the checkerboard scale accuracy of the checkerboard is higher than the scale threshold, the straight line accuracy is higher than the straight line threshold, and the surface flatness is higher than the flatness threshold, it indicates that the checkerboard accuracy of the checkerboard is higher than the accuracy threshold.

[0087] It can be seen that in this embodiment, the coordinate systems of multiple cameras to be calibrated can be calibrated to the same physical coordinate system through the printed checkerboard, and then the calibration accuracy of the initial calibration model can be improved through the standard checkerboard, enabling high-precision calibration of the cameras. That is, high-precision calibration of the cameras can be achieved through the printed checkerboard with low cost and easy acquisition and the standard checkerboard required, which can reduce the calibration cost as a whole.

[0088] In an exemplary embodiment, the schematic diagram of the camera calibration process is as Figure 6 shown. Taking the case where there are two cameras to be calibrated as an example, the camera calibration method includes:

[0089] S602. Simultaneously capture the printed checkerboard with the first marker code by two cameras to be calibrated to obtain the first checkerboard image.

[0090] S604. Detect the corner points of the first checkerboard image to obtain the first corner points, and determine the initial calibration model of each camera according to the first image coordinates and the first physical coordinates of the first corner points.

[0091] S606. Simultaneously photograph a standard checkerboard using two cameras to be calibrated, obtaining a second checkerboard image.

[0092] S608. Perform corner detection on the second checkerboard image to obtain second corners, and determine alternative physical coordinates of the second corners based on the second image coordinates of the second corners and an initial calibration model.

[0093] S610. Align the second physical coordinates of the second corners corresponding to each camera with the alternative physical coordinates to obtain target physical coordinates corresponding to the second physical coordinates.

[0094] S612. Update the initial calibration model according to the target physical coordinates to obtain a target calibration model.

[0095] Determine the conversion relationship between the target physical coordinates and the second image coordinates based on the target physical coordinates and the second image coordinates of the second corners, obtaining a target calibration model.

[0096] It can be seen that in the above embodiments, an initial camera geometric model is determined by printing a checkerboard, and then the calibration model is optimized using a standard checkerboard. High-precision calibration of the camera can be achieved with a relatively low calibration cost. In addition, the calibration models of all cameras correspond to the same physical coordinate system. After all cameras complete their respective detection and positioning tasks, the image coordinates obtained from positioning can be converted into physical coordinates in the same physical coordinate system, enabling high-precision positioning and measurement of large-size targets.

[0097] It should be understood that although the various steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless specifically stated in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The execution order of these steps or stages does not necessarily have to be sequential, but can be executed alternately or in rotation with at least some of the steps or stages in other steps or other steps.

[0098] Based on the same inventive concept, an embodiment of the present application also provides a camera calibration device. The implementation solution provided by this device for solving problems is similar to the implementation solution described in the above method. Therefore, the specific limitations in one or more embodiments of the camera calibration device provided below can refer to the limitations on the camera calibration method in the above text, and will not be elaborated here.

[0099] As Figure 7As shown in the figure, an embodiment of the present application provides a camera calibration device, including:

[0100] A corner point detection module 702, configured to obtain a first checkerboard image collected by the camera to be calibrated for a first checkerboard, and a second checkerboard image collected for a second checkerboard, perform corner point detection on the first checkerboard image to obtain first corner points, and perform corner point detection on the second checkerboard image to obtain second corner points; the accuracy of the second checkerboard is higher than that of the first checkerboard;

[0101] An initial model determination module 704, configured to determine an initial calibration model according to the first image coordinates and first physical coordinates of the first corner points;

[0102] A physical coordinate calculation module 706, configured to determine alternative physical coordinates corresponding to the second corner points according to the second image coordinates of the second corner points and the initial calibration model;

[0103] A physical coordinate alignment module 708, configured to align the second physical coordinates of the second corner points with the alternative physical coordinates to obtain target physical coordinates corresponding to the second physical coordinates;

[0104] A target model determination module 710, configured to determine a target calibration model according to the target physical coordinates and the second image coordinates.

[0105] In some embodiments, in terms of aligning the second physical coordinates of the second corner points with the alternative physical coordinates to obtain target physical coordinates corresponding to the second physical coordinates, the physical coordinate alignment module 708 is specifically configured to fit the second physical coordinates of the second corner points with the alternative physical coordinates to obtain a coordinate fitting relationship; determine a target rotation amount and a target translation amount according to the coordinate fitting relationship; and align the second physical coordinates with the alternative physical coordinates according to the target rotation amount and the target translation amount to obtain target physical coordinates corresponding to the second physical coordinates.

[0106] In some embodiments, there are multiple first corner points; in terms of determining an initial calibration model according to the first image coordinates and first physical coordinates of the first corner points, the initial model determination module 704 is specifically configured to determine a conversion relationship between the image coordinates and the physical coordinates according to the first image coordinates and first physical coordinates of each first corner point; determine alternative image coordinates corresponding to the first physical coordinates according to the conversion relationship and the first physical coordinates; and optimize the conversion relationship according to the difference between the alternative image coordinates and the first image coordinates to obtain an initial calibration model.

[0107] In some embodiments, the first checkerboard image includes marker code information; the apparatus further includes a physical coordinate determination module, which is specifically configured to identify the marker code information in the first checkerboard image and determine the physical coordinates of the first marker code before determining the initial calibration model based on the first image coordinates and the first physical coordinates of the first corner points; and determine the first physical coordinates of the first corner points according to the positional relationship between the first marker code and the first corner points and the physical coordinates of the first marker code.

[0108] In some embodiments, there are multiple cameras to be calibrated; in terms of obtaining the first checkerboard image collected by the cameras to be calibrated for the first checkerboard and the second checkerboard image collected for the second checkerboard, the corner point detection module 702 is specifically configured to have multiple cameras to be calibrated simultaneously photograph the first checkerboard to obtain multiple first checkerboard images; and have multiple cameras to be calibrated simultaneously photograph the second checkerboard to obtain multiple second checkerboard images; wherein, each first checkerboard image includes marker code information.

[0109] Each module in the above camera calibration apparatus can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in the processor in the computer device in hardware form or be independent of it, or can be stored in the memory in the computer device in software form so that the processor can call and execute the operations corresponding to each of the above modules.

[0110] In some embodiments, a camera calibration device is provided. The camera calibration device can be a server, and its internal structure diagram can be as Figure 8 shown. The camera calibration device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the camera calibration device is used to provide computing and control capabilities. The memory of the camera calibration device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the camera calibration device is used to store camera calibration data. The input / output interface of the camera calibration device is used to exchange information between the processor and external devices. The communication interface of the camera calibration device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the above camera calibration method are implemented.

[0111] In some embodiments, a camera calibration device is provided. The camera calibration device can be a terminal, and its internal structure diagram can be as Figure 9As shown in the figure. The camera calibration device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the camera calibration device is used to provide computing and control capabilities. The memory of the camera calibration device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the camera calibration device is used to exchange information between the processor and external devices. The communication interface of the camera calibration device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, the steps in the above-mentioned camera calibration method are implemented. The display unit of the camera calibration device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen; the input device of the camera calibration device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the camera calibration device, or an external keyboard, touchpad, or mouse, etc.

[0112] Those skilled in the art can understand that Figure 8 or Figure 9 the structure shown in the figure is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the camera calibration device to which the solution of this application is applied. The specific camera calibration device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0113] In some embodiments, a camera calibration device is provided. The camera calibration device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented.

[0114] In some embodiments, as Figure 10 shown, an internal structure diagram of a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. When the computer program is executed by the processor, the steps in the above-mentioned method embodiments are implemented.

[0115] In some embodiments, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by the processor, the steps in the above-mentioned method embodiments are implemented.

[0116] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0117] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., and are not limited thereto.

[0118] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0119] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A camera calibration method, characterized in that: include: Acquire a first chessboard image acquired by the camera to be calibrated for the first chessboard, and a second chessboard image acquired for the second chessboard, and perform corner point detection on the first chessboard image to obtain a first corner point, and perform corner point detection on the second chessboard image to obtain a second corner point; the accuracy of the second chessboard is higher than the accuracy of the first chessboard; Determining an initial calibration model according to the first image coordinates and the first physical coordinates of the first corner point; Determining candidate physical coordinates corresponding to the second corner point according to the second image coordinates of the second corner point and the initial calibration model; Aligning the second physical coordinate of the second corner point with the candidate physical coordinate to obtain the target physical coordinate corresponding to the second physical coordinate; A target calibration model is determined according to the target physical coordinates and the second image coordinates.

2. The method according to claim 1, characterized in that The aligning the second physical coordinate of the second corner point with the candidate physical coordinate to obtain the target physical coordinate corresponding to the second physical coordinate includes: Fitting the second physical coordinates of the second corner point with the candidate physical coordinates to obtain a coordinate fitting relationship; Determine the target rotation amount and the target translation amount according to the coordinate fitting relationship; The second physical coordinate is aligned with the candidate physical coordinate according to the target rotation amount and the target translation amount to obtain the target physical coordinate corresponding to the second physical coordinate.

3. The method according to claim 1, characterized in that The first corner points include a plurality of first corner points; and determining the initial calibration model according to the first image coordinates and the first physical coordinates of the first corner points includes: Determining a conversion relationship between the image coordinates and the physical coordinates according to the first image coordinates and the first physical coordinates of each of the first corner points; Determine, according to the conversion relationship and the first physical coordinate, candidate image coordinates corresponding to the first physical coordinate; The conversion relationship is optimized according to the difference between the candidate image coordinates and the first image coordinates to obtain an initial calibration model.

4. The method according to claim 1, characterized in that: The first chessboard image includes marking code information; Before determining the initial calibration model according to the first image coordinates and the first physical coordinates of the first corner point, the method further includes: Identify the marking code information in the first chessboard image to determine the physical coordinates of the first marking code; The first physical coordinates of the first corner point are determined according to the positional relationship between the first marking code and the first corner point, and the physical coordinates of the first marking code.

5. The method according to claim 1, characterized in that The camera to be calibrated includes a plurality of cameras; the step of obtaining a first chessboard image acquired by the camera to be calibrated for the first chessboard and a second chessboard image acquired for the second chessboard includes: The plurality of cameras to be calibrated simultaneously photograph the first chessboard to obtain a plurality of first chessboard images; The plurality of cameras to be calibrated simultaneously photograph the second chessboard to obtain a plurality of second chessboard images; each of the first chessboard images includes marking code information.

6. The method according to any one of claims 1 to 5, characterized in that The first chessboard is a printed chessboard, and the second chessboard is a standard chessboard.

7. A camera calibration device, characterized in that: include: A corner point detection module, used to obtain a first chessboard image acquired by the camera to be calibrated for the first chessboard, and a second chessboard image acquired for the second chessboard, and perform corner point detection on the first chessboard image to obtain a first corner point, and perform corner point detection on the second chessboard image to obtain a second corner point; the accuracy of the second chessboard is higher than the accuracy of the first chessboard; An initial model determination module, used to determine an initial calibration model according to the first image coordinates and the first physical coordinates of the first corner point; a physical coordinate calculation module, configured to determine candidate physical coordinates corresponding to the second corner point according to the second image coordinates of the second corner point and the initial calibration model; A physical coordinate alignment module, used for aligning the second physical coordinate of the second corner point with the candidate physical coordinate to obtain a target physical coordinate corresponding to the second physical coordinate; The target model determination module is used to determine the target calibration model according to the target physical coordinates and the second image coordinates.

8. The device according to claim 7, characterized in that In terms of aligning the second physical coordinates of the second corner point with the alternative physical coordinates to obtain the target physical coordinates corresponding to the second physical coordinates, the physical coordinate alignment module is specifically used to fit the second physical coordinates of the second corner point with the alternative physical coordinates to obtain a coordinate fitting relationship; determine a target rotation amount and a target translation amount based on the coordinate fitting relationship; and align the second physical coordinates with the alternative physical coordinates based on the target rotation amount and the target translation amount to obtain the target physical coordinates corresponding to the second physical coordinates.

9. A camera calibration device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images