Camera calibration method and device, computer equipment and computer readable storage medium

Abstract

The invention relates to a camera calibration method and device, computer equipment and a computer readable storage medium. The method comprises the following steps: in response to a condition that a calibration board image is a sparse two-dimensional code chessboard, determining angular point image coordinates of each angular point in a local area where a two-dimensional code is located in the sparse two-dimensional code chessboard to obtain a local angular point image coordinate set; constructing an initial seed chessboard based on angular point image coordinates selected in the local angular point image coordinate set, a cell size determined by a two-dimensional code corresponding to the local angular point image coordinate set, a code direction and expansion direction information; the to-be-selected seed chessboard with the minimum chessboard energy in the to-be-selected seed chessboards obtained after the initial seed chessboards grow according to a preset chessboard growth strategy serves as a target seed chessboard; and determining a target conversion matrix based on the angular point image coordinates and the angular point physical coordinates corresponding to the angular points in the target seed chessboard. According to the method, the determination efficiency and accuracy of the conversion matrix between the physical coordinate system and the local image coordinate system in the camera calibration process can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a camera calibration method, apparatus, computer device, and computer-readable storage medium. Background Technology

[0002] Camera calibration is a fundamental and crucial technique in computer vision and photogrammetry. Its core purpose is to establish an accurate mathematical relationship between the real three-dimensional world and two-dimensional images, that is, to establish a transformation matrix between the physical coordinate system and the image coordinate system.

[0003] Currently, the commonly used camera calibration method is as follows: First, determine the corner coordinates of each cell corner point in the black and white checkerboard in the image coordinate system to obtain the image coordinate set. Then, establish a physical coordinate system (world coordinate system) with a certain point on the black and white checkerboard (such as the upper left corner point) and calculate the corresponding corner point physical coordinates to obtain the physical coordinate set. Finally, fit the image coordinate set and the physical coordinate set to determine the transformation matrix between the two.

[0004] However, when determining corner points in a local image coordinate system, it is necessary to judge each pixel in the black and white checkerboard image (usually by gradient change) to determine whether the pixel is a corner point. However, the number of pixels in a black and white checkerboard image is generally extremely large, which leads to low efficiency in determining cell corner points, and consequently low efficiency in determining the transformation matrix. In addition, noise that may exist in the black and white checkerboard image is very easy to be misidentified as cell corner points, which leads to low accuracy in determining the image coordinate set and physical coordinate set, and consequently low accuracy in determining the transformation matrix. Summary of the Invention

[0005] Therefore, it is necessary to provide a camera calibration method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the above-mentioned technical problems, which can improve the efficiency and accuracy of determining the transformation matrix between the physical coordinate system and the local image coordinate system during the camera calibration process.

[0006] In a first aspect, this application provides a camera calibration method, including: In response to the calibration board image being a sparse QR code chessboard, the corner point image coordinates of each corner point in the local area where the QR code is located in the sparse QR code chessboard are determined, and the local corner point image coordinate set is obtained. Based on the coordinates of at least two selected corner points in the local corner point image coordinate set, the cell size and code direction determined by the QR code corresponding to the local corner point image coordinate set, and the preset expansion direction information, an initial seed chessboard corresponding to each selected corner point image coordinate is constructed. After growing each initial seed chessboard according to the preset chessboard grid growth strategy, the candidate seed chessboard with the lowest chessboard energy among the candidate seed chessboards is taken as the target seed chessboard. Based on the image coordinates and physical coordinates of each corner point in the target seed chessboard, the target transformation matrix is ​​determined; the target transformation matrix is ​​used to realize the transformation between physical coordinates and image coordinates.

[0007] Secondly, this application provides a camera calibration device, comprising: The first determining module is used to determine the corner point image coordinates of each corner point in the local area where the QR code is located in the sparse QR code chessboard in response to the calibration board image being a sparse QR code chessboard, and obtain the local corner point image coordinate set. The construction module is used to construct the initial seed chessboard corresponding to each selected corner image coordinate based on the coordinates of at least two selected corner images in the local corner image coordinate set, the cell size and code direction determined by the QR code corresponding to the local corner image coordinate set, and the preset expansion direction information. The growth module is used to grow each initial seed chessboard according to a preset chessboard grid growth strategy, and then select the candidate seed chessboard with the lowest chessboard energy among the candidate seed chessboards as the target seed chessboard. The second determining module is used to determine the target transformation matrix based on the corner image coordinates and corner physical coordinates corresponding to each corner point in the target seed chessboard; the target transformation matrix is ​​used to realize the transformation between physical coordinates and image coordinates.

[0008] Thirdly, this application provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the method described above.

[0009] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method.

[0010] Fifthly, this application provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the method described above.

[0011] The aforementioned camera calibration method, apparatus, computer equipment, computer-readable storage medium, and computer program product, on the one hand, determine the target seed chessboard based on the corner image coordinates of each corner point within a local area corresponding to a QR code in a sparse QR code chessboard. Firstly, the local area only occupies a small portion of the sparse QR code chessboard. Secondly, the selected pixels (local corner image coordinate set) are only the corner points of each chessboard cell within the local area, not all pixels within the local area. Furthermore, the corner points in the target seed chessboard only correspond to a portion of the corner points within the local corner image coordinate set that meet certain conditions; that is, the number of corner points corresponding to the target seed chessboard is less than the local corner image coordinate set. Therefore, the number of corner image coordinates corresponding to the corner points in the target seed chessboard used to calculate the target transformation matrix, and the number of corner point physical coordinates calculated from the corner points and the obtained cell physical dimensions, are also limited. Compared to existing technologies, this method significantly reduces costs, thereby effectively improving the efficiency of determining the target transformation matrix. Furthermore, after the initial seed chessboard is grown, candidate seed chessboards are obtained. If the candidate seed chessboard has the lowest chessboard energy among all candidate seed chessboards, it indicates that the overlap between the corner points of the candidate seed chessboard (target seed chessboard) and the corresponding corner points in the sparse QR code chessboard is as high as possible, and the number of corner points with high overlap in the candidate seed chessboard is as large as possible. This means that the accuracy and quantity of the corner point image coordinates and physical coordinates corresponding to each corner point in the target seed chessboard are both high. Since the corner point image coordinates and physical coordinates are used to further calculate the target transformation matrix, the accuracy of determining the target transformation matrix is ​​improved. In summary, using this application can effectively improve the efficiency and accuracy of determining the target transformation matrix during camera calibration. Attached Figure Description

[0012] Figure 1 An application environment diagram of a camera calibration method provided in this application embodiment; Figure 2 A schematic flowchart illustrating a camera calibration method provided in an embodiment of this application; Figure 3 A schematic diagram of a dense / sparse QR code chessboard provided in an embodiment of this application; Figure 4 This application provides a schematic diagram of a single growth process of an initial seed chessboard in an embodiment of the present application. Figure 5 A structural block diagram of a camera calibration device provided in an embodiment of this application; Figure 6 An internal structural diagram of a computer device provided in an embodiment of this application; Figure 7 An internal structural diagram of another computer device provided in an embodiment of this application; Figure 8This is an internal structural diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0014] The camera calibration method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a communication network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0015] like Figure 2 As shown, this application embodiment provides a camera calibration method, which is applied to... Figure 1 The method will be illustrated using terminal 102 or server 104 as examples. It is understood that the computer device may include at least one of a terminal and a server. The method includes the following steps: S110. In response to the calibration board image being a sparse QR code chessboard, determine the corner point image coordinates of each corner point in the local area where the QR code is located in the sparse QR code chessboard, and obtain the local corner point image coordinate set.

[0016] It should be noted that before camera calibration, a calibration board image for camera calibration must be determined in advance. Subsequently, based on the coordinates of specific points in the calibration board image in the image coordinate system and the coordinates in the physical coordinate system (world coordinate system), the coordinate transformation matrix corresponding to the camera needs to be calculated to achieve camera calibration. The coordinate transformation matrix is ​​used to transform the coordinates in the physical coordinate system to the coordinates in the image coordinate system, that is, to establish an accurate mathematical relationship between the real three-dimensional world and the two-dimensional image.

[0017] Before performing step S110, this embodiment pre-creates a calibration board image. The optional calibration board image in this embodiment includes at least a dense QR code checkerboard and a sparse QR code checkerboard. The dense QR code checkerboard consists of setting a corresponding QR code in each white cell of a black-and-white checkerboard, while the sparse QR code checkerboard consists of setting a corresponding QR code in a 2×2 cell group at a specific location in a black-and-white checkerboard. Compared to the dense QR code checkerboard, the sparse QR code checkerboard has fewer QR codes, and the distance between different QR codes is greater. For example, see the schematic diagrams of the dense and sparse QR code checkerboards. Figure 3 .

[0018] Specifically, taking the creation of a sparse QR code chessboard as an example, the pre-defined chessboard layout information is obtained. This layout information includes at least: the total number of rows (rows), the total number of columns (cols), the distance between codes (gap), and the physical size of each cell (cell_size). Based on this layout information, a black-and-white grid chessboard is generated. The black-and-white grid chessboard is composed of alternating black and white cells. During the generation of this black-and-white grid chessboard, each cell has its chessboard coordinates (row, col) in the chessboard coordinate system. Here, row is the row number of the cell in the black-and-white grid chessboard, and col is the column number of the cell in the black-and-white grid chessboard. Taking one cell as an example, the chessboard coordinates (row, col) of that cell are obtained. In response to row%gap=0 or 1 and col%gap=0 or 1, the cell is rendered as a white cell. See also... Figure 3 The sparse QR code chessboard shown has one QR code occupying four white cells. These four white cells form a 2×2 cell group, which are: the first white cell with row%gap=0 and col%gap=0, the second white cell with row%gap=0 and col%gap=1, the third white cell with row%gap=1 and col%gap=0, and the fourth white cell with row%gap=1 and col%gap=1. In this embodiment, a QR code is generated in each 2×2 cell group in the black and white chessboard grid to obtain a sparse QR code chessboard. For this purpose, a QR code corresponding to each 2×2 cell group needs to be generated first.

[0019] Taking the generation process of one of the QR codes as an example, firstly, the center of the 2×2 cell group corresponding to the QR code is determined, denoted as the grid center. The grid center is the center coordinate (Row, Col) of the 2×2 cell group in the corner coordinate system. Further, the grid center (Row, Col) and the code gap and cell physical size (cell_size) in the chessboard layout information are encoded into QR code information text according to a preset encoding format. For example, the QR code information text is "Col, RowGgapScell_size". This embodiment also presets a DM encoding library, which is used to process the QR code information text (text) into the corresponding QR code (image). The initial output of the DM encoding library... The QR code is designated as the initial QR code, with a side length of 2 × cell_size. In this embodiment, to ensure that after the initial QR code is concentrically filled with 2×2 cell groups, a white border remains around it, the initial QR code needs to be scaled, with a scaling ratio r1 = 0.88 × (2 × cell_size). The new QR code obtained after scaling the initial QR code according to this scaling ratio is designated as the target QR code. This target QR code is then concentrically filled into the corresponding 2×2 cell groups in the black and white checkerboard grid, thereby creating a sparse QR code checkerboard. In the sparse QR code checkerboard, the side length of each cell is the cell's physical size, cell_size, and the distance between two adjacent QR codes is the code gap.

[0020] To determine whether the calibration board image used is a dense or sparse QR code chessboard, a QR code is first detected in the calibration board image. Then, the QR code information text is parsed. Next, the characters at specific encoding positions in the QR code information text are used to determine whether the calibration board image is a dense or sparse QR code chessboard. For example, taking the QR code information text "Col,RowGgapScell_size" as an example, the characters at specific encoding positions in this QR code information text are "G" and "S". In response to detecting that the characters at specific encoding positions in the QR code information text are "G" and "S", it can be determined that the calibration board image containing the corresponding QR code is a sparse QR code chessboard. The determination logic for a dense QR code chessboard is similar and will not be repeated here.

[0021] This embodiment uses a calibration board image as a sparse QR code chessboard as an example. To reduce the computational cost of calculating the coordinate transformation matrix corresponding to the camera, this embodiment aims to define a local area on the sparse QR code chessboard centered on the QR code. Subsequently, the image coordinates and physical coordinates corresponding to specific points in this local area are used to calculate the initial coordinate transformation matrix corresponding to the camera. The specific points in the local area are the corner points of each cell in the local area. These corner points are represented in the form of row and column indices (x, y), which are the coordinates in the index coordinate system of the sparse QR code chessboard. The index coordinate system takes the top-left corner of the first cell in the top-left corner of the sparse QR code chessboard as the origin, the horizontal axis as the x-axis, and the vertical axis as the y-axis. Each corner point in each cell of the sparse QR code chessboard can be determined through this index coordinate system. For example, taking the x-axis as an example, the corner point at the origin is (0, 0), and along the x-axis direction, the corner points are (0, 1), (0, 2), (0, 3), and so on.

[0022] In the sparse QR code chessboard, the coordinates of each corner point in each cell under the image coordinate system can be obtained by a preset corner point extraction algorithm, and the coordinates of the identified corner points under the image coordinate system are recorded as corner point image coordinates.

[0023] Taking a QR code in a sparse QR code chessboard as an example, the corner point image coordinates corresponding to each corner point in the local area where the QR code is located are also known as the local corner point image coordinate set.

[0024] S120. Based on the coordinates of at least two selected corner points in the local corner point image coordinate set, the cell size and code direction determined by the QR code corresponding to the local corner point image coordinate set, and the preset expansion direction information, construct the initial seed chessboard corresponding to the selected corner point image coordinates.

[0025] In this embodiment, the local corner point image coordinate set corresponds to a local region, and a QR code exists in this local region. By processing the QR code, the cell size p_size corresponding to the QR code can be obtained. By detecting and decoding the QR code, the direction of the QR code can be obtained, and this direction is recorded as the code direction. In this embodiment, the first column and the last row of the QR code are both black squares. The intersection of the black squares in the first column and the last row is taken as the origin, and the direction away from the origin in the last row is taken as the code direction. Among them, there are multiple corner image coordinates in the local corner image coordinate set. Taking one of the corner image coordinates as the selected corner image coordinate as an example, a planar coordinate system is established with the corner image coordinate as the origin and the code direction as the x-axis direction. It should be noted that this embodiment also presets the expansion direction information for the coordinate system. The expansion direction information includes 8 angles, namely 0°, 45°, 90°, 135°; 180°, 225°, 270°, 315°. In this embodiment, the angle shown in the x-axis direction is 0°. Taking the selected corner point image coordinates as the starting point, after extending along each angle by the length of a cell size p_size, an extension point can be determined at each angle. Taking one of the extension points as an example, the corner point image coordinates closest to the extension point are further determined from the local corner point image coordinate set as the target point. The 8 target points correspond to a 2×2 cell group in the local area. The 8 target points are distributed on the corner points of each cell in the 2×2 cell group located at the edge, and the selected corner point image coordinates are the center of the 2×2 cell group. The 2×2 cell group containing the selected corner point image coordinates is recorded as the initial seed chessboard. Each corner point image coordinate in the local corner point image coordinate set can be used to calculate a corresponding initial seed chessboard.

[0026] S130. After growing each initial seed chessboard according to the preset chessboard grid growth strategy, the candidate seed chessboard with the smallest chessboard energy among the candidate seed chessboards is taken as the target seed chessboard.

[0027] It should be noted that the initial seed chessboard has a small number of corner points. To ensure the accuracy of the coordinate transformation matrix corresponding to the camera, a sufficient number of corner points are required. Therefore, this embodiment intends to use the initial seed chessboard as a basis and control the growth of the initial seed chessboard according to a preset chessboard grid growth strategy, so that the initial seed chessboard grows into a chessboard with more corner points. The new seed chessboard formed after the initial seed chessboard has grown according to the above-mentioned preset chessboard grid growth strategy is recorded as the candidate seed chessboard.

[0028] It should also be noted that the candidate seed chessboard contains multiple corner points, but the positional correspondence between some of these corner points and the corresponding corner points in the sparse QR code chessboard may not be high. In other words, there may be a certain difference in their positions. This may cause multiple corner points in the candidate seed chessboard that were originally on the same straight line or the same vertical line to no longer be on the same straight line or the same vertical line, resulting in poor structural reliability of the candidate seed chessboard.

[0029] This embodiment pre-defines a chessboard energy function for calculating the chessboard energy of the candidate seed chessboard. This function is used to evaluate whether the number of corner points in the candidate seed chessboard is relatively large and whether the structural reliability of the candidate seed chessboard is good. The smaller the calculated chessboard energy, the more corner points the candidate seed chessboard has and the better the structural reliability of the candidate seed chessboard.

[0030] In this embodiment, each corner image coordinate in the local corner image coordinate set can be used to calculate a corresponding candidate seed chessboard, and the candidate seed chessboard with the lowest chessboard energy is taken as the target seed chessboard.

[0031] It should be noted that, since the target seed chessboard has a relatively large number of corner points, the number of corner points used to calculate the coordinate transformation matrix is ​​guaranteed, thus ensuring the accuracy of the calculated coordinate transformation matrix. Furthermore, the corner points in the target seed chessboard and their corresponding corner points in the sparse QR code chessboard have a high degree of positional agreement, ensuring the accuracy of the corner points in the target seed chessboard, which further guarantees the accuracy of the calculated coordinate transformation matrix. In addition, this embodiment does not use all the pixels (which are extremely numerous) in the target seed chessboard for calculating the coordinate transformation matrix, but only uses the corner points of the cells within the target seed chessboard (which are relatively fewer than all pixels). Therefore, the subsequent calculation of the coordinate transformation matrix based on the corner points of the cells within the target seed chessboard effectively ensures the calculation efficiency of the coordinate transformation matrix.

[0032] S140. Based on the corner image coordinates and corner physical coordinates corresponding to each corner point in the target seed chessboard, determine the target transformation matrix; the target transformation matrix is ​​used to realize the transformation between physical coordinates and image coordinates.

[0033] In this sparse QR code chessboard, each QR code can only be used to calculate a corresponding target seed chessboard. Taking a corner point in the target seed chessboard as an example, the corner point's corresponding corner point image coordinates (in the image coordinate system) can be determined through a corner point extraction algorithm. The product of the corner point index (x, y) and the cell physical size (cell_size) can be used as the corner point's physical coordinates (in the physical coordinate system). By fitting the corner point image coordinates and corner point physical coordinates of each corner point in each target seed chessboard in the sparse QR code chessboard, the coordinate transformation matrix corresponding to the camera's calibration using the sparse QR code chessboard can be calculated, and this coordinate transformation matrix is ​​denoted as the target transformation matrix. This target transformation matrix is ​​used to realize the transformation between coordinates in the physical coordinate system and coordinates in the image coordinate system, and the coordinates in the physical coordinate system are designated as physical coordinates, and the coordinates in the image coordinate system are designated as image coordinates.

[0034] As can be seen, in this embodiment, on the one hand, the target seed chessboard is determined based on the corner image coordinates of each corner point within a local area corresponding to a QR code in the sparse QR code chessboard. First, the local area only occupies a small portion of the sparse QR code chessboard. Second, the selected pixels (local corner image coordinate set) are only the corner points of each chessboard cell in the sparse QR code chessboard within the local area, not all the pixels within the local area. Furthermore, the corner points in the target seed chessboard only correspond to a portion of the corner points within the local corner image coordinate set that meet certain conditions. That is, the number of corner points corresponding to the target seed chessboard is less than that of the local corner image coordinate set. Therefore, the number of corner image coordinates corresponding to the corner points in the target seed chessboard used to calculate the target transformation matrix, and the number of corner point physical coordinates calculated by the corner points and the obtained cell physical dimensions, are significantly improved compared to the prior art. This significantly reduces the efficiency of determining the target transformation matrix. Furthermore, after the initial seed chessboard is grown, candidate seed chessboards are obtained. If the candidate seed chessboard has the lowest chessboard energy among all candidate seed chessboards, it indicates that the overlap between the corner points of the candidate seed chessboard (target seed chessboard) and the corresponding corner points in the sparse QR code chessboard is as high as possible, and the number of corner points with high overlap in the candidate seed chessboard is as large as possible. This means that the accuracy and quantity of the corner point image coordinates and physical coordinates corresponding to each corner point in the target seed chessboard are both high. Since the corner point image coordinates and physical coordinates corresponding to each corner point are used to further calculate the target transformation matrix, the accuracy of determining the target transformation matrix is ​​improved. In summary, using this application can effectively improve the efficiency and accuracy of determining the target transformation matrix during camera calibration.

[0035] In some embodiments, the corner image coordinates of each corner point within the local area where the QR code is located in the sparse QR code chessboard are determined to obtain a set of local corner image coordinates, including: S111. Detect and decode the QR codes in the sparse QR code chessboard to obtain the detection and decoding results; the detection and decoding results shall include at least: code center, code size, and code distance.

[0036] It should be noted that the sparse QR code chessboard contains multiple QR codes. In this embodiment, multiple initial seed chessboards corresponding to each QR code will be established. Then, a target seed chessboard will be selected from each initial seed chessboard. After deduplication and fusion of the corner indexes of adjacent target seed chessboards, the coordinate transformation matrix will be determined by the corner image coordinates and the corner physical coordinates of the corner points in the target seed chessboard. Taking one of the QR codes as an example, in order to determine the initial seed chessboard corresponding to the QR code, a corresponding local region needs to be determined on the sparse QR code chessboard. The initial seed chessboard will be generated in this local region. In this embodiment, the local region is established based on the corresponding QR code.

[0037] Specifically, taking a QR code as an example, the QR code is detected and decoded using a preset QR code detection and decoding algorithm to obtain the detection and decoding result. Here, the code center is the coordinate of the QR code in the image coordinate system, used to represent the position of the center point of the QR code; the code size c_size is the side length of the QR code's bounding box, in pixels, and there is a preset size ratio r2 between the code size and the cell size p_size. For example, this size ratio r2 is 0.8. The code distance gap is the spacing between two adjacent QR codes, which can be obtained by decoding the QR code text information.

[0038] S112. Based on the detection and decoding results, determine the local area where the QR code is located.

[0039] In the sparse QR code chessboard, each QR code can be identified as a corresponding local region. Taking one such local region as an example, this local region is specifically determined based on the detection and decoding result obtained from detecting and decoding the corresponding QR code. This local region is square, with its center being the code center in the detection and decoding result, and its side length is... Where, gap is the inter-symbol distance in the detection decoding result, and p_size is the cell size calculated based on the code size c_size in the detection decoding result; where, .

[0040] S113. Perform corner detection on the sparse QR code chessboard to obtain a set of candidate corner image coordinates.

[0041] It should be noted that, taking a local area in a sparse QR code chessboard as an example, in order to generate the initial seed chessboard in that local area, it is necessary to first determine the coordinates of the corner points of each cell in that local area in the image coordinate system. To determine the coordinates of the corner points of each cell in the sparse QR code chessboard in the image coordinate system, this embodiment pre-sets a corner detection algorithm based on Hessian response. The algorithm process is roughly as follows: First, the sparse QR code chessboard image is normalized to grayscale and smoothed by Gaussian to suppress noise and obtain a denoised image. Then, the second derivatives and mixed second derivatives of the denoised image in the x and y directions are calculated to construct a Hessian matrix. Next, each pixel in the denoised image is processed using the Hessian matrix to obtain a saddle point response map. Then, the absolute value of the saddle point response map is taken, and local extrema are selected by thresholding and non-maximum suppression (NMS) as candidate image corner points. The selected candidate image corner points are recorded as a set of candidate corner point image coordinates.

[0042] S114. Perform sub-pixel refinement on the candidate corner point image coordinate set to obtain a high-precision corner point image coordinate set.

[0043] It should be noted that the accuracy of the coordinates of the candidate image corner points in the image coordinate system is not high. The coordinate transformation matrix required for subsequent camera calibration depends on the candidate image corner points. If the coordinate accuracy of the candidate image corner points themselves is not high, the accuracy of the final calculated coordinate transformation matrix will be poor. Therefore, this embodiment first improves the coordinate accuracy of each candidate image corner point in the candidate corner point image coordinate set.

[0044] Specifically, in this embodiment, subpixel refinement is performed on each candidate corner point in the candidate corner point image coordinate set to improve the coordinate accuracy of each candidate corner point. For example, the subpixel refinement algorithm can be an iterative optimization algorithm based on local gray-level gradient, and there is no specific limitation. The candidate corner point image coordinate set after subpixel refinement is denoted as the high-precision corner point image coordinate set.

[0045] S115. Determine the high-precision corner image coordinates located within the local area in the high-precision corner image coordinate set to obtain the local corner image coordinate set.

[0046] In this case, a local area in a sparse QR code chessboard is taken as an example, and the high-precision corner point image coordinates are set in the local area, which is denoted as the local corner point image coordinate set.

[0047] As can be seen, in this embodiment, by refining the candidate corner point image coordinate set to subpixel level, the coordinate accuracy of each candidate image corner point in the candidate corner point image coordinate set is effectively improved. Since each corner point in the candidate corner point image coordinate set is subsequently used to establish the coordinate transformation matrix corresponding to camera calibration, the accuracy of the finally calculated coordinate transformation matrix can be effectively improved.

[0048] In some embodiments, based on the coordinates of at least two selected corner points in the local corner point image coordinate set, the cell size and code direction determined by the QR code corresponding to the local corner point image coordinate set, and preset expansion direction information, an initial seed chessboard corresponding to each selected corner point image coordinate is constructed, including: S121. Based on the selected corner image coordinates in the local corner image coordinate set and the code direction parsed from the QR code corresponding to the local corner image coordinate set, establish a local image coordinate system with the selected corner image coordinates as the origin.

[0049] It should be noted that, taking a local area in a sparse QR code chessboard as an example, in order to ensure the comprehensiveness of the initial seed chessboard generated in this local area, this embodiment intends to generate a one-to-one corresponding initial seed chessboard for each corner point image coordinate in the local corner point image coordinate set corresponding to the local area, so as to facilitate the subsequent selection of the target seed chessboard from each initial seed chessboard.

[0050] A QR code exists in a local area. The detection and decoding result obtained after detecting and decoding the QR code also includes the code direction, which is used to characterize the direction of the corresponding QR code.

[0051] The selected corner image coordinate is taken as one of the corner image coordinates in the local corner image coordinate set. The corner image coordinate is taken as the origin and the code direction is taken as the x-axis direction to establish a coordinate system. This coordinate system is denoted as the local image coordinate system. In this embodiment, a local image coordinate system is established for each corner image coordinate in the local corner image coordinate set.

[0052] S122. Based on the local image coordinate system, the cell size determined by the QR code corresponding to the local corner image coordinate set, and the preset expansion direction information, determine the coordinates of multiple predicted corner images.

[0053] It should be noted that the cell size p_size can be determined by comparing the decoded QR code size c_size with the preset size ratio r2; where, To determine the initial seed chessboard for the coordinates of each selected image corner point, this embodiment also presets expansion direction information; wherein, the expansion direction information includes 8 expansion direction angles, namely 0°, 45°, 90°, 135°; 180°, 225°, 270°, 315°.

[0054] Taking the coordinates of one of the selected image corner points as an example, we take the coordinates of the selected image corner point as the starting point, and... p_size is the step size, which determines the endpoint of the angle along each of the above expansion directions, and the endpoint is recorded as the predicted corner point image coordinate. The coordinates of a selected image corner point can determine the coordinates of 8 predicted corner points.

[0055] S123. Based on the image coordinates of each predicted corner point and the local corner point image coordinate set, determine the actual corner point image coordinates corresponding to each predicted corner point image coordinate.

[0056] Taking a predicted corner image coordinate as an example, in the image coordinate system, there is generally a nearest corner image coordinate in the local corner image coordinate set near the predicted corner image coordinate. The predicted corner image coordinate may not be accurately located at a corner on the sparse QR code chessboard, but the corner image coordinates in the local corner image coordinate set accurately represent a corner on the sparse QR code chessboard. The initial seed chessboard created subsequently is centered on the selected corner image coordinate and contains corners corresponding to the angles of the eight expansion directions. To ensure the accuracy of the corners corresponding to the angles of the eight expansion directions, this embodiment determines the nearest corner image coordinate in the local corner image coordinate set for each predicted corner image coordinate, which is used as the actual corner image coordinate corresponding to each predicted corner image coordinate.

[0057] S124. Determine the initial seed chessboard based on the selected corner point image coordinates and the corresponding multiple actual corner point image coordinates.

[0058] The initial seed chessboard is a 2×2 cell group. The center point of the cell group is a selected corner point image coordinate from the local corner point image coordinate set. The eight actual corner point image coordinates corresponding to the selected corner point image coordinates are distributed at the four corners and the four midpoints of the cell group.

[0059] As can be seen, in this embodiment, by using the actual corner image coordinates corresponding to the selected corner image coordinates to establish the initial seed chessboard, instead of using the predicted corner image coordinates corresponding to the selected corner image coordinates, the accuracy of the initial seed chessboard can be effectively improved, thereby effectively improving the accuracy of the subsequently calculated coordinate transformation matrix.

[0060] In some embodiments, after growing each initial seed chessboard according to a preset chessboard grid growth strategy, the candidate seed chessboard with the lowest chessboard energy among the resulting candidate seed chessboards is used as the target seed chessboard, including: S131. In response to the initial seed chessboard energy being less than a preset energy threshold, the corner points of the predicted growth image are determined based on the cell size determined by the QR code corresponding to the initial seed chessboard and the preset chessboard grid growth direction.

[0061] The initial seed chessboard energy is used to measure whether the number of corner points in the initial seed chessboard is relatively large and whether the structural reliability of the initial seed chessboard is relatively high. The higher the structural reliability of the initial seed chessboard, the more the initial seed chessboard appears as a chessboard-like grid. In this embodiment, a chessboard energy calculation formula is preset to calculate the chessboard energy of the initial seed chessboard.

[0062] It should be noted that the smaller the chessboard energy, the more corner points the corresponding initial seed chessboard has, and the higher the structural reliability of the initial seed chessboard is.

[0063] Specifically, by comparing the decoded QR code size c_size with a preset size ratio r2, the cell size p_size can be determined; where, .

[0064] It should be noted that the number of corner points in the initial seed chessboard is relatively small. As shown in the above embodiment, the initial seed chessboard generally contains 9 corner points (1 corner point in the center and 8 corner points on the edges). In order to ensure the accuracy of the coordinate transformation matrix calculated later, a sufficient number of corner points is required. Therefore, this embodiment also controls the growth of the seed chessboard in a local area according to a certain growth strategy based on the initial seed chessboard until the preset growth stopping condition is reached. In order to realize the growth of the initial seed chessboard, this embodiment presets the chessboard growth direction, which includes four growth directions: up, down, left, and right.

[0065] Taking one growth process in one of the growth directions as an example, assuming that the growth direction is downward, see [link to relevant documentation]. Figure 4 Each corner point (A1, B1, C1) on the bottom edge of the initial seed chessboard is used as a starting point. The determined cell size p_size is used as the step size, and the growth direction is downwards. A corresponding endpoint (A2, B2, C2) is determined for each starting point. The endpoint is located below the corresponding starting point, and the distance between them is one cell size p_size. The endpoint is recorded as a corner point of the predicted growth image. For an example, see [link to example]. Figure 4 The corner points of the predicted growth image generated from the initial seed chessboard are A2, B2, and C2, respectively.

[0066] S132. Determine the coordinates of the corner image that is closest to the corner point in the local corner image coordinate set and the corner point in the predicted growth image, and use them as the actual corner point in the growth image corresponding to the corner point in the predicted growth image.

[0067] To ensure the accuracy of the grown corner points, taking one of the predicted growth image corner points as an example, the coordinates of the corner point image closest to the predicted growth image corner point are determined in the local corner point image coordinate set, and the coordinates of this corner point image are used as the actual growth image corner point corresponding to the predicted growth image corner point.

[0068] S133. In response to the fact that the distance between the actual growth image corner point and the corresponding predicted growth image corner point is less than a preset distance threshold, the actual growth image corner point is added to the initial seed chessboard to obtain a new initial seed chessboard.

[0069] There is a distance between the actual growth image corner point and the corresponding predicted growth image corner point. This distance is the distance in the image coordinate system and is recorded as the image distance. To measure whether the actual growth image corner point can replace the corresponding predicted growth image corner point in the initial seed chessboard, this embodiment presets an image distance threshold. If the image distance is less than the image distance threshold, it means that the actual growth image corner point is the corner point that should grow correctly if the initial seed chessboard is not skewed. At this time, the actual growth image corner point can be added to the initial seed chessboard. If the image distance is not less than the image distance threshold, the predicted growth image corner point is added to the initial seed chessboard, and the new seed chessboard after the corner point is added is recorded as the new initial seed chessboard.

[0070] S134. Iterate through the new initial seed chessboard until the iteration stops, and obtain the candidate seed chessboard.

[0071] It should be noted that the above steps describe the process of the initial seed chessboard growing in one direction. After completing this process, this embodiment needs to continue repeating the process along the original growth direction to grow a new initial seed chessboard, thereby realizing the iterative growth of the initial seed chessboard. The iteration stopping condition for iterative growth along a growth direction is the detection of a decrease in the board energy of the new initial seed chessboard. After reaching the iteration stopping condition corresponding to the growth direction, the growth in other growth directions is continued based on the currently obtained new initial seed chessboard. If the iteration stopping condition corresponding to the last growth direction is met, the number of corner points in the new initial seed chessboard no longer increases, the initial seed chessboard has completed its growth, and the new initial seed chessboard at this time is recorded as the candidate seed chessboard.

[0072] S135. Among the candidate seed chessboards corresponding to the selected corner point image coordinates, the candidate seed chessboard with the smallest chessboard energy is taken as the target seed chessboard.

[0073] In this embodiment, each corner point image coordinate in the local corner point image coordinate set is a selected corner point image coordinate. Each corner point image coordinate can be used to calculate a corresponding candidate seed chessboard through the above steps. In this embodiment, the candidate seed chessboard with the smallest chessboard energy is selected as the target seed chessboard.

[0074] As can be seen, in this embodiment, firstly, by adding the actual growth image corner point that has the closest image spacing to the corner point of the predicted growth image and whose image spacing is less than the image spacing threshold to the initial seed chessboard, the accuracy of the corner point position in the initial seed chessboard can be improved, thereby facilitating the improvement of the accuracy of the subsequently calculated coordinate transformation matrix; secondly, by selecting the candidate seed chessboard with the smallest chessboard energy from each candidate seed chessboard as the target seed chessboard, the accuracy of the corner point position in the seed chessboard can be further improved, thereby facilitating the improvement of the accuracy of the subsequently calculated coordinate transformation matrix.

[0075] In some embodiments, the candidate seed chessboard with the lowest chessboard energy among the candidate seed chessboards corresponding to the selected corner point image coordinates is selected as the target seed chessboard, including: S135A. Calculate the corner point quantity energy and chessboard structure energy of the candidate seed chessboard corresponding to the selected corner point image coordinates.

[0076] S135B: Determine the board energy of the candidate seed chessboard based on the corner point energy and board structure energy.

[0077] It should be noted that the formula for calculating the board energy of the initial seed board and the candidate seed board is as follows: E = Ec + Es; Where E is the chessboard energy, Ec is the corner number energy, and Es is the chessboard structure energy; the corner number energy is the negative value of the number of corners in the seed chessboard, and the chessboard structure energy is used to measure the structural reliability of the seed chessboard; the smaller the chessboard energy, the more corners the seed chessboard has, and the higher the structural reliability of the seed chessboard is. The smaller the chessboard energy, the more suitable the seed chessboard is as a seed chessboard for subsequent calculation of the coordinate transformation matrix.

[0078] S135C: Select the candidate seed board with the lowest board energy among the candidate seed boards corresponding to the selected corner point image coordinates, and use it as the target seed board.

[0079] As can be seen, in this embodiment, since the target seed chessboard has the minimum chessboard energy, on the one hand, the number of corner points in the target seed chessboard is as large as possible, so as to facilitate the improvement of the accuracy of the coordinate transformation matrix calculated subsequently; on the other hand, the accuracy of the corner point positions in the target seed chessboard is as optimal as possible (the structural reliability is as high as possible), so as to further improve the accuracy of the coordinate transformation matrix calculated subsequently.

[0080] In some embodiments, the target transformation matrix is ​​determined based on the corner image coordinates and physical coordinates of each corner point in the target seed chessboard, including: S141. Based on the corner point image coordinates corresponding to each corner point in the target seed chessboard, obtain the original corner point image coordinate set.

[0081] It should be noted that the sparse QR code chessboard contains multiple QR codes. Taking any two adjacent QR codes as an example, the local regions corresponding to the two QR codes partially overlap, and the side length of the local region is... In this case, the overlapping area is located between the centers of the two QR codes. Taking one of the QR codes as an example, the corner indexes (coordinates in the corner coordinate system, such as (1,1), (1,2), (2,1)...) of each corner point in the local area corresponding to that QR code are established based on the QR code. In the aforementioned overlapping area, for the same corner point, using different QR codes as references, that corner point has two corner indexes, and these two corner indexes may be different. It should also be noted that there is a certain distance between each corner point in the overlapping area and the two QR codes, and this distance is recorded as the code point distance. That is, each corner point in the overlapping area has two code point distances. Generally speaking, the smaller the code point distance between a corner point and a QR code in the overlapping area, the more accurate the corner index determined by that corner point based on that QR code is.

[0082] To ensure that each corner point corresponds to a corner point index and to guarantee the accuracy of the corner point index, taking a corner point in an overlapping area as an example, the code spacing between the corner point and the two QR codes corresponding to the overlapping area is first determined. Then, the corner point index created based on the QR code corresponding to the smaller code spacing is used as the target corner point index. In this way, the deduplication of different corner point indices corresponding to the same corner point in the overlapping area can be unified, and the accuracy of the corner point index can be guaranteed.

[0083] Since the target seed chessboard exists in a local area of ​​the QR code, if some corner points of the target seed chessboard are located in the aforementioned overlapping area, the corner points located in the overlapping area of ​​the target seed chessboard can also achieve deduplication and unification of corner point indexes, thereby facilitating the improvement of the accuracy of the corner point indexes of the target seed chessboard.

[0084] Taking a corner point in the target seed chessboard as an example, the corner point has a corresponding corner point index, and the corner point index has a corresponding corner point image coordinate in the local corner point image coordinate set. The corner point image coordinates corresponding to the corner point index of each corner point in the target seed chessboard in the local corner point image coordinate set are denoted as the original corner point image coordinate set.

[0085] S142. Based on each corner point in the target seed chessboard and the QR code corresponding to the target seed chessboard, determine the physical coordinates of each corner point to obtain the original set of corner point physical coordinates.

[0086] It should be noted that, taking a QR code in a sparse QR code chessboard as an example, by detecting and decoding the QR code, we can obtain the cell physical size (cell_size) in the corresponding QR code text information, the corner index of a corner point in the target seed chessboard that belongs to the same local area as the QR code, and the cell physical size (cell_size) as the product of the corner index and the cell physical size (cell_size), which is the coordinate of the corner point in the physical coordinate system (world coordinate system), and is recorded as the corner point physical coordinates; and the corner point physical coordinates corresponding to each corner point in each target seed chessboard are recorded as the original corner point physical coordinate set.

[0087] S143. Fit the original corner point image coordinate set to the original corner point physical coordinate set to obtain the original transformation matrix.

[0088] In this system, the coordinates of the original corner point image coordinate set and the original corner point physical coordinate set are in one-to-one correspondence. By fitting the original corner point image coordinate set and the original corner point physical coordinate set, a corresponding transformation matrix can be obtained, and this transformation matrix is ​​denoted as the original transformation matrix. The original transformation matrix is ​​used to transform the coordinates in the physical coordinate system to the image coordinate system. The specific fitting method for fitting the original corner point image coordinate set and the original corner point physical coordinate set is either an affine transformation or a radial transformation.

[0089] S144. Based on the original transformation matrix, add the corner image coordinates and corner physical coordinates corresponding to each missing corner point in the sparse QR code chessboard to the original corner image coordinate set and the original corner physical coordinate set, respectively, to obtain the middle corner image coordinate set and the middle corner physical coordinate set.

[0090] It should be noted that before generating the target seed chessboard in the local area corresponding to the QR code, when detecting the corner points in the local area, some corner points may not be detected due to dirt or other reasons in their position on the sparse QR code chessboard. These undetected corner points are recorded as missed corner points.

[0091] To ensure the completeness of the corner points required to generate the final coordinate transformation matrix, this embodiment intends to add the corner point image coordinates and corner point physical coordinates corresponding to each missing corner point to the original corner point image coordinate set and the original corner point physical coordinate set, respectively.

[0092] Specifically, the aforementioned original transformation matrix can map the coordinates of each corner point in the original corner point image coordinate set to the physical coordinate system, thereby obtaining the original reprojected corner point physical coordinates corresponding to each corner point in the original corner point image coordinate set. Taking one of the original reprojected corner point physical coordinates as an example, this original reprojected corner point physical coordinate has a corresponding original corner point physical coordinate in the original corner point physical coordinate set, and the difference between the original reprojected corner point physical coordinate and the corresponding original corner point physical coordinate is denoted as the original reprojection deviation. The original reprojection deviation corresponding to each original reprojected corner point physical coordinate can be calculated in the above way, and the mean of each original reprojection deviation is denoted as the overall average deviation d.

[0093] This embodiment identifies missing corners from a sparse QR code chessboard through manual annotation and / or corner re-detection. Then, using the corner index and cell size of the missing corner, its image coordinates are calculated. Next, the original transformation matrix is ​​used to map these coordinates to a physical coordinate system, yielding the missing corner's physical coordinates. Finally, the physical coordinates of the nearest original corner to the missing corner are determined from the original set of physical coordinates, and the physical coordinates of this original corner and the missing corner are compared. If the distance between them is less than 2d (that is, twice the overall average deviation d), it means that the physical coordinates of the missing corner point are qualified to be added to the original physical coordinate set of corner points. At the same time as adding the physical coordinates of the missing corner point to the original physical coordinate set of corner points, the corner point image coordinates of the missing corner point are also added to the original corner point image coordinate set of corner points. The coordinate set after adding the physical coordinates of the missing corner point to the original physical coordinate set of corner points is denoted as the intermediate physical coordinate set of corner points. The coordinate set after adding the corner point image coordinates to the original corner point image coordinate set of corner points is also denoted as the intermediate corner point image coordinate set of corner points.

[0094] S145. Determine the target transformation matrix based on the image coordinate set and physical coordinate set of the intermediate corner point.

[0095] Among them, by fitting the image coordinate set and the physical coordinate set of the middle corner point, the corresponding coordinate transformation matrix can be obtained, which is denoted as the target transformation matrix.

[0096] As can be seen, in this embodiment, on the one hand, by deduplicating and unifying the corner indexes of the corner points in the overlapping areas of the two QR codes to retain more accurate corner indexes, the accuracy of the corner points in the subsequently generated target seed chessboard can be improved, thereby facilitating the improvement of the accuracy of the final calculated target transformation matrix; on the other hand, by supplementing the corner image coordinates and corner physical coordinates of the missed corner points to the original corner image coordinate set and the original corner physical coordinate set respectively, the comprehensiveness of the corner points included in the generated intermediate corner image coordinate set and the intermediate corner physical coordinate set can be improved, thereby facilitating the improvement of the accuracy of the final calculated target transformation matrix.

[0097] In some embodiments, determining the target transformation matrix based on the image coordinate set and the physical coordinate set of the intermediate corner points includes: S145A. Fit the image coordinate set and the physical coordinate set of the middle corner point to obtain the intermediate transformation matrix.

[0098] S145B: Based on the intermediate transformation matrix, the physical coordinate set of the intermediate corner points is reprojected to obtain the reprojected corner point image coordinate set.

[0099] The intermediate transformation matrix can be used to map physical coordinates to corresponding image coordinates. In other words, the intermediate transformation matrix can be used to map the physical coordinates of each corner point in the intermediate corner point physical coordinate set to the image coordinate system, thereby obtaining the corner point image coordinates that correspond one-to-one with the physical coordinates of each corner point in the intermediate corner point physical coordinate set. This achieves the reprojection of the intermediate corner point physical coordinate set, and the reprojected corner point image coordinates are marked as the reprojected corner point image coordinate set.

[0100] S145C. Based on the reprojection corner point image coordinate set and the intermediate corner point image coordinate set, determine the reprojection error of each intermediate corner point image coordinate in the intermediate corner point image coordinate set.

[0101] Taking a reprojected corner image coordinate in the reprojected corner image coordinate set as an example, the reprojected corner image coordinate has an intermediate corner image coordinate corresponding to the same corner in the intermediate corner image coordinate set, and the difference between the reprojected corner image coordinate and the intermediate corner image coordinate is denoted as the reprojection error.

[0102] S145D: Based on the reprojection error, filter the intermediate corner point image coordinate set and the intermediate corner point physical coordinate set to obtain the target corner point image coordinate set and the target corner point physical coordinate set.

[0103] Among them, the reprojection error can be used to filter out the coordinates of the middle corner points in the image coordinate set that have poor coordinate accuracy. In order to achieve this filtering, this embodiment sets a corresponding error threshold for the reprojection error.

[0104] If the error threshold is positive, then the corresponding reprojection errors of the intermediate corner point image coordinates that are greater than the error threshold are filtered out from the intermediate corner point image coordinate set to obtain the target corner point image coordinate set. The filtered intermediate corner point image coordinates are also marked as the screened corner point image coordinates. Each screened corner point image coordinate has a one-to-one correspondence with the intermediate corner point physical coordinates in the intermediate corner point physical coordinate set. The intermediate corner point physical coordinates corresponding to the screened corner point image coordinates in the intermediate corner point physical coordinate set are also filtered out to obtain the target corner point physical coordinate set.

[0105] If the error threshold is non-positive, then a 2D Chi-square distribution is determined based on the mean and variance of each reprojection error. The intermediate corner point image coordinates that deviate from this distribution are then filtered out from the intermediate corner point image coordinate set to obtain the target corner point image coordinate set. The filtered intermediate corner point image coordinates are then marked as the screened corner point image coordinates. Each screened corner point image coordinate has a one-to-one correspondence with the intermediate corner point physical coordinates in the intermediate corner point physical coordinate set. The intermediate corner point physical coordinates that correspond to the screened corner point image coordinates in the intermediate corner point physical coordinate set are also filtered out to obtain the target corner point physical coordinate set.

[0106] S145E: Fit the target corner point image coordinate set and the target corner point physical coordinate set to obtain the target transformation matrix.

[0107] In this embodiment, the coordinate transformation matrix obtained by fitting the target corner point image coordinate set with the target corner point physical coordinate set is also the target transformation matrix.

[0108] In other embodiments, based on the obtained target transformation matrix, the target transformation matrix can also be used as a new intermediate transformation matrix, and the target corner point image coordinate set and the target corner point physical coordinate set can be used as the new intermediate corner point image coordinate set and the new intermediate corner point physical coordinate set, respectively. Then, the above steps S145B-S145E are executed again to realize one iteration of the target transformation matrix. Then, the target transformation matrix is ​​iterated multiple times until the preset number of iterations is reached. Then, the latest target transformation matrix obtained when the number of iterations is reached is used as the final target transformation matrix.

[0109] As can be seen, in this embodiment, coordinates with poor accuracy in the intermediate corner image coordinate set and the intermediate corner physical coordinate set can be filtered out by the reprojection error, thereby improving the accuracy of the determined target corner image coordinate set and the target corner physical coordinate set. Since the target transformation matrix is ​​obtained by fitting the target corner image coordinate set and the target corner physical coordinate set, the accuracy of the determined target transformation matrix can be improved. In the other embodiments mentioned above, during the iterative generation of the target transformation matrix, coordinates with lower accuracy in the new intermediate corner image coordinate set and the new intermediate corner physical coordinate set are continuously filtered out, so that the accuracy of each coordinate in the new intermediate corner image coordinate set and the new intermediate corner physical coordinate set required to fit the final target transformation matrix is ​​better, thereby further improving the accuracy of the determined final target transformation matrix.

[0110] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0111] Based on the same inventive concept, this application also provides a camera calibration device. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more camera calibration device embodiments provided below can be found in the limitations of the camera calibration method above, and will not be repeated here.

[0112] like Figure 4 As shown, this application embodiment provides a camera calibration device 400, including: The first determining module 410 is used to determine the corner point image coordinates of each corner point in the local area where the QR code is located in the sparse QR code chessboard in response to the calibration board image being a sparse QR code chessboard, and obtain the local corner point image coordinate set. The construction module 420 is used to construct the initial seed chessboard corresponding to each selected corner image coordinate based on the coordinates of at least two selected corner images in the local corner image coordinate set, the cell size and code direction determined by the QR code corresponding to the local corner image coordinate set, and the preset expansion direction information. The growth module 430 is used to grow each initial seed chessboard according to a preset chessboard grid growth strategy, and then select the candidate seed chessboard with the smallest chessboard energy among the candidate seed chessboards as the target seed chessboard. The second determining module 440 is used to determine the target transformation matrix based on the corner image coordinates and corner physical coordinates corresponding to each corner point in the target seed chessboard; the target transformation matrix is ​​used to realize the transformation between physical coordinates and image coordinates.

[0113] In some embodiments, in determining the corner image coordinates of each corner point within the local area where the QR code is located in a sparse QR code chessboard, and obtaining a set of local corner image coordinates, the first determining module 410 is specifically used for: Detect and decode the QR codes in a sparse QR code chessboard to obtain the detection and decoding results; the detection and decoding results include at least: code center, code size, and code spacing. Based on the detection and decoding results, the local area where the QR code is located is determined; Corner detection is performed on a sparse QR code chessboard to obtain a set of candidate corner image coordinates; Subpixel refinement is performed on the candidate corner point image coordinate set to obtain a high-precision corner point image coordinate set; The high-precision corner image coordinates located within the local area in the high-precision corner image coordinate set are determined to obtain the local corner image coordinate set.

[0114] In some embodiments, regarding the construction of the initial seed chessboard corresponding to each selected corner image coordinate based on at least two corner image coordinates selected from the local corner image coordinate set, the cell size and code direction determined by the QR code corresponding to the local corner image coordinate set, and preset expansion direction information, the construction module 420 is specifically used for: Based on the selected corner image coordinates in the local corner image coordinate set and the code direction parsed from the QR code corresponding to the local corner image coordinate set, a local image coordinate system with the selected corner image coordinates as the origin is established. Based on the cell size determined by the QR code corresponding to the local image coordinate system and the local corner image coordinate set, and the preset expansion direction information, multiple predicted corner image coordinates are determined. Based on the image coordinates of each predicted corner point and the local corner point image coordinate set, determine the actual corner point image coordinates corresponding to each predicted corner point image coordinate; The initial seed chessboard is determined based on the selected corner point image coordinates and the corresponding multiple actual corner point image coordinates.

[0115] In some embodiments, regarding the selection of the candidate seed board with the lowest board energy among the candidate seed boards obtained after growing each initial seed board according to a preset board grid growth strategy, the growth module 430 is specifically used for: In response to the initial seed chessboard having a chessboard energy less than a preset energy threshold, the corner points of the predicted growth image are determined based on the cell size determined by the QR code corresponding to the initial seed chessboard and the preset chessboard grid growth direction. Determine the coordinates of the corner image that is closest to the corner point in the local corner image coordinate set and the corner point in the predicted growth image, and use them as the actual corner point in the growth image corresponding to the corner point in the predicted growth image. In response to the fact that the distance between the actual growth image corner point and the corresponding predicted growth image corner point is less than a preset distance threshold, the actual growth image corner point is added to the initial seed chessboard to obtain a new initial seed chessboard. The new initial seed chessboard is iteratively grown until the iteration stops, resulting in a candidate seed chessboard. The candidate seed chessboard with the lowest chessboard energy among the candidate seed chessboards corresponding to the selected corner point image coordinates is taken as the target seed chessboard.

[0116] In some embodiments, the growth module 430 is specifically used to select the seed chessboard with the lowest chessboard energy from the candidate seed chessboards corresponding to the selected corner point image coordinates, as the target seed chessboard. Calculate the corner point quantity energy and chessboard structure energy of the candidate seed chessboard corresponding to the selected corner point image coordinates; The board energy of the candidate seed chessboard is determined based on the corner point energy and board structure energy. The candidate seed board with the lowest board energy among the candidate seed boards corresponding to the selected corner point image coordinates is taken as the target seed board.

[0117] In some embodiments, regarding the determination of the target transformation matrix based on the corner image coordinates and physical coordinates corresponding to each corner point in the target seed chessboard, the second determining module 440 is specifically used for: Based on the corner point image coordinates corresponding to each corner point in the target seed chessboard, the original corner point image coordinate set is obtained; Based on each corner point in the target seed chessboard and the QR code corresponding to the target seed chessboard, the physical coordinates of each corner point are determined, and the original set of corner point physical coordinates is obtained. The original corner point image coordinate set and the original corner point physical coordinate set are fitted to obtain the original transformation matrix; Based on the original transformation matrix, the corner image coordinates and corner physical coordinates corresponding to each missing corner point in the sparse QR code chessboard are added to the original corner image coordinate set and the original corner physical coordinate set, respectively, to obtain the middle corner image coordinate set and the middle corner physical coordinate set. The target transformation matrix is ​​determined based on the image coordinate set and the physical coordinate set of the middle corner point.

[0118] In some embodiments, regarding determining the target transformation matrix based on the intermediate corner image coordinate set and the intermediate corner physical coordinate set, the second determining module 440 is specifically used for: By fitting the image coordinate set and the physical coordinate set of the middle corner point, the intermediate transformation matrix is ​​obtained; The physical coordinate set of the intermediate corner points is reprojected based on the intermediate transformation matrix to obtain the reprojected corner point image coordinate set; Based on the reprojection corner point image coordinate set and the intermediate corner point image coordinate set, determine the reprojection error of each intermediate corner point image coordinate in the intermediate corner point image coordinate set; Based on the reprojection error, the image coordinate set and physical coordinate set of the intermediate corner points are filtered to obtain the image coordinate set and physical coordinate set of the target corner points; The target corner point image coordinate set and the target corner point physical coordinate set are fitted to obtain the target transformation matrix.

[0119] Each module in the aforementioned camera calibration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0120] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to the camera calibration method. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the aforementioned camera calibration method.

[0121] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it performs the steps in the aforementioned camera calibration method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen; the input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs or touchpads set on the casing of the computer device, or external keyboards, touchpads or mice, etc.

[0122] Those skilled in the art will understand that Figure 6 or Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0123] In some embodiments, a computer device is provided, the computer device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0124] In some embodiments, such as Figure 8 The diagram shows the internal structure of a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the above-described method embodiments.

[0125] In some embodiments, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0126] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0127] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0128] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A camera calibration method, characterized in that, include: In response to the calibration board image being a sparse QR code chessboard, the corner point image coordinates of each corner point in the local area where the QR code is located in the sparse QR code chessboard are determined, and a local corner point image coordinate set is obtained. Based on at least two selected corner image coordinates from the local corner image coordinate set, the cell size and code direction determined by the QR code corresponding to the local corner image coordinate set, and the preset expansion direction information, an initial seed chessboard corresponding to each selected corner image coordinate is constructed. After each initial seed chessboard is grown according to the preset chessboard grid growth strategy, the candidate seed chessboard with the smallest chessboard energy among the candidate seed chessboards is taken as the target seed chessboard. Based on the corner image coordinates and corner physical coordinates corresponding to each corner point in the target seed chessboard, a target transformation matrix is ​​determined; the target transformation matrix is ​​used to realize the transformation between physical coordinates and image coordinates.

2. The method according to claim 1, characterized in that, The step of determining the corner image coordinates of each corner point within the local area where the QR code is located in the sparse QR code chessboard, to obtain a local corner image coordinate set, includes: The QR codes in the sparse QR code chessboard are detected and decoded to obtain the detection and decoding results; the detection and decoding results include at least: code center, code size, and code spacing. Based on the detection and decoding results, the local area where the QR code is located is determined; Corner detection is performed on the sparse QR code chessboard to obtain a set of candidate corner image coordinates; The candidate corner point image coordinate set is refined to subpixel level to obtain a high-precision corner point image coordinate set; The high-precision corner image coordinates located within the local area in the high-precision corner image coordinate set are determined to obtain the local corner image coordinate set.

3. The method according to claim 1, characterized in that, The step of constructing an initial seed chessboard corresponding to each selected corner image coordinate based on at least two corner image coordinates selected from the local corner image coordinate set, the cell size and code direction determined by the QR code corresponding to the local corner image coordinate set, and preset expansion direction information includes: Based on the selected corner image coordinates in the local corner image coordinate set and the code direction parsed from the QR code corresponding to the local corner image coordinate set, a local image coordinate system is established with the selected corner image coordinates as the origin; Based on the local image coordinate system, the cell size determined by the QR code corresponding to the local corner image coordinate set, and the preset expansion direction information, multiple predicted corner image coordinates are determined; Based on the predicted corner point image coordinates and the local corner point image coordinate set, the actual corner point image coordinates corresponding to each predicted corner point image coordinate are determined. The initial seed chessboard is determined based on the selected corner point image coordinates and the corresponding multiple actual corner point image coordinates.

4. The method according to claim 1, characterized in that, The step of growing each of the initial seed chessboards according to a preset chessboard grid growth strategy, and then selecting the candidate seed chessboard with the lowest chessboard energy among the candidate seed chessboards, as the target seed chessboard, includes: In response to the initial seed chessboard having a chessboard energy less than a preset energy threshold, the corner points of the predicted growth image are determined based on the cell size determined by the QR code corresponding to the initial seed chessboard and the preset chessboard grid growth direction. Determine the coordinates of the corner image that is closest to the predicted growth image corner in the local corner image coordinate set, and use it as the actual growth image corner corresponding to the predicted growth image corner. In response to the fact that the distance between the actual growth image corner point and the corresponding predicted growth image corner point is less than a preset distance threshold, the actual growth image corner point is added to the initial seed chessboard to obtain a new initial seed chessboard. The new initial seed chessboard is iteratively grown until the iteration stops, resulting in a candidate seed chessboard. The candidate seed chessboard with the lowest chessboard energy among the candidate seed chessboards corresponding to the selected corner point image coordinates is taken as the target seed chessboard.

5. The method according to claim 4, characterized in that, The step of selecting the candidate seed chessboard with the lowest chessboard energy from the candidate seed chessboards corresponding to the selected corner point image coordinates, and using it as the target seed chessboard, includes: Calculate the corner point quantity energy and chessboard structure energy of the candidate seed chessboard corresponding to the selected corner point image coordinates; The chessboard energy of the candidate seed chessboard is determined based on the energy of the number of corner points and the energy of the chessboard structure. The candidate seed chessboard with the lowest chessboard energy among the candidate seed chessboards corresponding to the selected corner point image coordinates is taken as the target seed chessboard.

6. The method according to claim 1, characterized in that, The determination of the target transformation matrix based on the corner image coordinates and physical coordinates of each corner point in the target seed chessboard includes: Based on the corner point image coordinates corresponding to each corner point in the target seed chessboard, the original corner point image coordinate set is obtained; Based on each corner point in the target seed chessboard and the QR code corresponding to the target seed chessboard, the physical coordinates of each corner point are determined to obtain the original set of corner point physical coordinates. The original corner point image coordinate set and the original corner point physical coordinate set are fitted to obtain the original transformation matrix; Based on the original transformation matrix, the corner image coordinates and corner physical coordinates corresponding to each missing corner point in the sparse QR code chessboard are added to the original corner image coordinate set and the original corner physical coordinate set respectively, to obtain the middle corner image coordinate set and the middle corner physical coordinate set. The target transformation matrix is ​​determined based on the image coordinate set and the physical coordinate set of the intermediate corner points.

7. The method according to claim 6, characterized in that, The step of determining the target transformation matrix based on the image coordinate set and the physical coordinate set of the intermediate corner points includes: The intermediate corner point image coordinate set and the intermediate corner point physical coordinate set are fitted to obtain the intermediate transformation matrix; Based on the intermediate transformation matrix, the physical coordinate set of the intermediate corner points is reprojected to obtain the reprojected corner point image coordinate set. Based on the reprojected corner image coordinate set and the intermediate corner image coordinate set, determine the reprojection error of each intermediate corner image coordinate in the intermediate corner image coordinate set; Based on the reprojection error, the intermediate corner point image coordinate set and the intermediate corner point physical coordinate set are filtered to obtain the target corner point image coordinate set and the target corner point physical coordinate set; The target corner point image coordinate set and the target corner point physical coordinate set are fitted to obtain the target transformation matrix.

8. A camera calibration device, characterized in that, include: The first determining module is used to determine the corner point image coordinates of each corner point in the local area where the QR code is located in the sparse QR code chessboard in response to the calibration board image being a sparse QR code chessboard, and to obtain a set of local corner point image coordinates. The construction module is used to construct an initial seed chessboard corresponding to each selected corner image coordinate based on at least two corner image coordinates selected from the local corner image coordinate set, the cell size and code direction determined by the QR code corresponding to the local corner image coordinate set, and preset expansion direction information. The growth module is used to grow each of the initial seed chessboards according to a preset chessboard grid growth strategy, and then select the candidate seed chessboard with the lowest chessboard energy among the candidate seed chessboards as the target seed chessboard. The second determining module is used to determine the target transformation matrix based on the corner image coordinates and corner physical coordinates corresponding to each corner point in the target seed chessboard; the target transformation matrix is ​​used to realize the transformation between physical coordinates and image coordinates.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images