Camera calibration method, device, equipment, system and medium

Abstract

The invention discloses a camera calibration method, device, equipment, system and medium, the method is applied to calibration of a target camera group, the target camera group comprises a plurality of cameras, the method comprises the following steps: obtaining a plurality of candidate relative extrinsic parameters, the candidate relative extrinsic parameters being relative extrinsic parameters between candidate camera pairs formed by any two cameras; performing credibility screening on the plurality of candidate relative external parameters to obtain a target relative external parameter; and determining actual external parameters of each camera based on the target relative external parameters. According to the scheme, a mode for flexibly selecting the candidate relative extrinsic parameters is provided.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a camera calibration method, device, equipment, system and medium. Background Art

[0002] 2D-3D lifting refers to converting two-dimensional data or information into three-dimensional data or information, and camera calibration has an important impact on this conversion process. For example, the accuracy of camera calibration will affect the accuracy of the conversion.

[0003] Camera calibration technology includes the use of one-dimensional calibration objects for calibration. In the process of using one-dimensional calibration objects for calibration, generally speaking, it is necessary to first determine the relative extrinsic parameters, and then convert the relative extrinsic parameters into actual extrinsic parameters. How to determine the relative extrinsic parameters has become an urgent problem to be solved. Summary of the invention

[0004] The present application at least provides a camera calibration method, device, equipment, system and medium.

[0005] The present application provides a camera calibration method, which is applied to the calibration of a target camera group, wherein the target camera group includes multiple cameras. The method includes: obtaining multiple candidate relative extrinsic parameters, where the candidate relative extrinsic parameters are relative extrinsic parameters between a candidate camera pair consisting of any two cameras; performing credibility screening on the multiple candidate relative extrinsic parameters to obtain target relative extrinsic parameters; and determining the actual extrinsic parameters of each camera based on the target relative extrinsic parameters.

[0006] Therefore, when calibrating the target camera group, multiple candidate relative extrinsic parameters are first obtained. The candidate relative extrinsic parameters are the relative extrinsic parameters between any two cameras. The multiple candidate relative extrinsic parameters are screened for credibility, so as to select the target relative extrinsic parameter for determining the actual extrinsic parameter. This provides a flexible way to select candidate relative extrinsic parameters and performs credibility screening, which can improve the accuracy of the target relative extrinsic parameter and the accuracy of the camera group parameter calibration.

[0007] Among them, credibility screening of multiple candidate relative extrinsic parameters to obtain the target relative extrinsic parameters includes: obtaining common view parameters corresponding to each candidate relative extrinsic parameter, the common view parameters are used to characterize the field of view overlap between the candidate camera pairs; based on the common view parameters, screening multiple candidate relative extrinsic parameters to obtain the target relative extrinsic parameters.

[0008] Therefore, the common view parameter can characterize the overlap of the field of view between the camera pair, which can indicate the accuracy of the relative extrinsic parameter calculation between the two cameras. Thus, by screening the candidate relative extrinsic parameters according to the common view parameter, more accurate target relative extrinsic parameters can be selected, thereby improving the accuracy of parameter calibration.

[0009] Among them, the common view parameter is the number of common view points between the candidate camera pairs; based on the common view parameter, multiple candidate relative extrinsic parameters are screened to obtain the target relative extrinsic parameter, including: taking each camera as a node, connecting edges between two nodes corresponding to each candidate camera pair, and determining the weight of the corresponding edge based on the number of common view points of each candidate camera pair to form a target graph; solving the maximum spanning tree for the target graph; and taking the candidate relative extrinsic parameter corresponding to the edge contained in the maximum spanning tree as the target relative extrinsic parameter.

[0010] Therefore, by taking the camera as a node and the common view parameter as the weight of the edge connecting the nodes, and solving the maximum spanning tree of the graph, it is possible to select accurate target relative extrinsic parameters and improve the accuracy of parameter calibration.

[0011] The number of common viewing angle points between the candidate camera pairs is determined based on a number of calibration image pairs captured by the candidate camera pairs; the calibration image pairs are obtained by capturing the target scene by two cameras included in the candidate camera pair, and at least one calibration point is set in the target scene; the common viewing angle points in the calibration image pair include two first feature points from two calibration images in the calibration image pair, respectively, and the two first feature points represent the same calibration point.

[0012] Therefore, the number of common viewing angle points between the candidate camera pairs is determined through the candidate relatively captured calibration image pairs, so that the common viewing situation between the candidate camera pairs can be characterized to indicate the accuracy of the relative external parameters of the two.

[0013] Among them, the actual extrinsic parameters are the extrinsic parameters of the camera relative to the first world coordinate system; based on the target relative extrinsic parameters, determining the actual extrinsic parameters of each camera includes: obtaining at least one group of calibration point position information groups, the calibration point position information groups include the position information of the calibration points obtained based on the calibration images collected by each camera, the calibration images are collected for the target scene, the target scene is provided with at least one calibration point, and all calibration images corresponding to the same calibration point position information group are collected when the position of at least one calibration point is fixed; based on at least one group of calibration point position information groups, the variables to be optimized are optimized relative extrinsic parameters of each camera are obtained; wherein the variables to be optimized include the target relative extrinsic parameters, the optimized relative extrinsic parameters are the extrinsic parameters relative to the second world coordinate system, the second world coordinate system is determined based on the camera coordinate system of the reference camera, and the reference camera is any camera in the target camera group; determining the conversion relationship between the first world coordinate system and the second world coordinate system; based on the conversion relationship, the optimized relative extrinsic parameters are converted to obtain the actual extrinsic parameters.

[0014] Therefore, by optimizing the relative extrinsic parameters, the accuracy of the relative extrinsic parameters can be improved, and then the relative extrinsic parameters are converted to the first world coordinate system to achieve the calibration of the extrinsic parameters.

[0015] Among them, a small batch stochastic gradient descent algorithm is used as an optimization algorithm for optimizing the variables to be optimized.

[0016] Therefore, the small-batch stochastic gradient descent algorithm can use multiple processes for optimization to improve optimization efficiency, and randomly select a small batch of data to participate in the optimization each time, introduce randomness, avoid local optimality, and improve the optimization effect.

[0017] In which, at least three calibration points are set in the target scene, and the calibration images corresponding to different calibration point position information groups are acquired when the positions of at least three calibration points are different; the optimization is performed based on the target loss, and the target loss includes at least one of the length loss and the straight line loss, and the step of obtaining the target loss includes: in response to the target loss including the length loss, for each calibration point position information group, the predicted spatial distance between two calibration points in at least three calibration points is obtained, and the length loss corresponding to the calibration point position information group is determined based on the difference between the predicted spatial distance and the corresponding real spatial distance; in response to the target loss including the straight line loss, for each calibration point position information group, the spatial position information of the first calibration point, the second calibration point and the third calibration point in at least three calibration points is obtained; based on the spatial position information of the first calibration point and the spatial position information of the third calibration point, the predicted position information of the second calibration point is predicted, and based on the difference between the spatial position information of the second calibration point and the predicted position information, the straight line loss corresponding to the calibration point position information group is determined; wherein, the first calibration point, the second calibration point and the third calibration point are collinear, and the second calibration point is between the first calibration point and the third calibration point.

[0018] Therefore, according to the characteristics of the calibration points, at least one of the length loss and the straight line loss is optimized to improve the accuracy of the relative external parameters.

[0019] Among them, the small batch stochastic gradient descent algorithm is implemented using the torch framework.

[0020] Therefore, the optimization algorithm under the torch framework is used to improve the optimization efficiency and optimization effect.

[0021] Wherein, determining the conversion relationship between the first world coordinate system and the second world coordinate system includes: obtaining the planar position information of the second feature point contained in the alignment image captured by each camera; the alignment image is acquired by capturing the target scene, and a plurality of alignment calibration points are arranged in the target scene, and a second feature point represents an alignment calibration point in the alignment image; based on the planar position information of the second feature point corresponding to each alignment calibration point, determining the first spatial position information of each alignment calibration point; based on the first spatial position information and the second spatial position information of each alignment calibration point, determining the conversion relationship, wherein the first spatial position information is the position information of the alignment calibration point in the second world coordinate system, and the second spatial position information is the position information of the alignment calibration point in the first world coordinate system.

[0022] Therefore, the information collected by multiple cameras is used in combination with the information from multiple perspectives to determine the conversion relationship between the two world coordinate systems and improve the accuracy of the conversion relationship.

[0023] Among them, obtaining the planar position information of the second feature points contained in the alignment images captured by each camera includes: obtaining the annotation information of each alignment image; the annotation information includes the planar position information of the second feature points contained in the alignment image, and the correspondence between the alignment calibration point and the second feature point; based on the planar position information of the second feature points corresponding to each alignment calibration point, determining the first spatial position information of each alignment calibration point includes: for each alignment calibration point, triangulating the planar position information of the second feature points corresponding to the alignment calibration point to obtain the first spatial position information of the alignment calibration point.

[0024] Therefore, by adopting the marking method to obtain the first spatial position information of the alignment calibration point, the alignment calibration point can be arbitrarily selected, thereby increasing the flexibility of selecting the alignment calibration point.

[0025] Among them, obtaining the planar position information of the second feature points contained in the alignment images captured by each camera includes: performing feature point detection on the alignment images to obtain the planar position information of the second feature points contained in the alignment images; determining the first spatial position information of each alignment calibration point based on the planar position information of the second feature points corresponding to each alignment calibration point includes: clustering all the second feature points; wherein the second feature points of the same category represent the same alignment calibration point; triangulating the planar position information of the second feature points of the same category to obtain the spatial position information to be matched corresponding to each category; based on the matching relationship between the first spatial distance and the second spatial distance, determining the matching relationship between the spatial position information to be matched and the alignment calibration point; the first spatial distance is the distance between each two spatial position information to be matched, and the second spatial distance is the distance between each two alignment calibration points; for each alignment calibration point, the spatial position information to be matched that matches the alignment calibration point is used as the first spatial position information of the alignment calibration point.

[0026] Therefore, by acquiring the first spatial position information of the alignment calibration point by image processing, it is possible to automatically determine the conversion relationship according to the image and reduce manual participation.

[0027] Among them, every two cameras in the target camera group form a candidate camera pair.

[0028] Therefore, every two cameras form a candidate camera pair, so that all optional relative external parameters can be obtained, and the candidate relative external parameters used are richer.

[0029] Among them, obtaining multiple candidate relative extrinsic parameters includes: obtaining calibration images captured by each camera; preprocessing the calibration images, and performing calibration point detection on the preprocessed calibration images to obtain position information of calibration points corresponding to the calibration images; for each candidate camera pair, determining the candidate relative extrinsic parameters between the candidate camera pairs based on the position information of the calibration points captured by the candidate camera pairs.

[0030] Therefore, based on the position information of the calibration points in the calibration image acquired by the candidate camera pair, the relative extrinsic parameters between the two cameras are obtained.

[0031] Among them, the calibration image is obtained by collecting the target scene, and at least one calibration point is set in the target scene; preprocessing the calibration image includes at least one of the following: constructing a mask image based on a reference image corresponding to the calibration image, and applying the mask image to the calibration image; performing feature enhancement on the calibration image for color information of the calibration points; and performing a morphological closing operation on the calibration image.

[0032] Therefore, preprocessing operations are performed to enhance the features of the calibration points in the calibration image and improve the accuracy of calibration point detection.

[0033] The present application provides a camera calibration device, which is used for calibrating a target camera group, wherein the target camera group includes multiple cameras. The device includes an acquisition module, a screening module and an extrinsic parameter adjustment module. The acquisition module is used to acquire multiple candidate relative extrinsic parameters, wherein the candidate relative extrinsic parameters are relative extrinsic parameters between a candidate camera pair consisting of any two cameras; the screening module is used to perform credibility screening on the multiple candidate relative extrinsic parameters to obtain target relative extrinsic parameters; and the extrinsic parameter adjustment module is used to determine the actual extrinsic parameters of each camera based on the target relative extrinsic parameters.

[0034] The present application provides an electronic device, including a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement any of the aforementioned camera calibration methods.

[0035] The present application provides a camera calibration system, comprising a plurality of cameras and the electronic device as described above.

[0036] The present application provides a computer-readable storage medium having program instructions stored thereon, and when the program instructions are executed by a processor, any of the above-mentioned camera calibration methods is implemented.

[0037] The above scheme first obtains multiple candidate relative extrinsic parameters, where the candidate relative extrinsic parameters are the relative extrinsic parameters between any two cameras, and performs credibility screening on the multiple candidate relative extrinsic parameters to select the target relative extrinsic parameter for determining the actual extrinsic parameter. This provides a flexible way to select candidate relative extrinsic parameters and performs credibility screening, which can improve the accuracy of the target relative extrinsic parameter and the accuracy of the camera group parameter calibration.

[0038] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0040] Figure 1 It is a flowchart of an embodiment of a camera calibration method of the present application;

[0041] Figure 2 It is a schematic diagram of an embodiment of the maximum spanning tree of the present application;

[0042] Figure 3 This is a flowchart of an embodiment of camera parameter optimization of the present application;

[0043] Figure 4 This is a flow chart of an embodiment of external reference conversion of the present application;

[0044] Figure 5 This is a flowchart of an embodiment of step S410 of the present application;

[0045] Figure 6 is a flowchart of another embodiment of step S410 of the present application;

[0046] Figure 7 is a flowchart of another embodiment of step S110 of the present application;

[0047] Figure 8 is a schematic diagram of another embodiment of the pre-processing step of the present application;

[0048] Fig. 9 is a flowchart of another embodiment of the camera calibration method of the present application;

[0049] Fig.10 It is a schematic diagram of the framework of an embodiment of a camera calibration device of the present application;

[0050] Fig.11 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0051] Fig.12 It is a schematic diagram of the framework of an embodiment of a camera calibration system of the present application;

[0052] Fig.13 A schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0053] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0054] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0055] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0056] See also Figure 1 , Figure 1 It is a flowchart of an embodiment of a camera calibration method of the present application.

[0057] The camera calibration method of the present application can be used for parameter calibration of a target camera group, where the target camera group includes multiple cameras. The number of cameras included in the target camera group can be set according to actual application needs, for example, 2, 6, etc. Among them, the type of cameras in the target camera group can be set according to actual application needs, for example, it can include at least one of a black and white camera, an infrared camera, and an RGB camera.

[0058] The camera calibration method of the present application uses a one-dimensional calibration object during the calibration process. Specifically, the method may include the following steps:

[0059] Step S110: Acquire multiple candidate relative extrinsic parameters.

[0060] The relative external parameter is the external parameter of another camera relative to the reference coordinate system when one camera is used as the reference coordinate system. The candidate relative external parameter is the relative external parameter between the candidate camera pairs, and the candidate camera pair is composed of any two cameras in the target camera group.

[0061] In some embodiments, every two cameras in the target camera group form a candidate camera pair, and a candidate camera pair can determine a candidate relative extrinsic parameter. The multiple candidate relative extrinsic parameters include the relative extrinsic parameters between every two cameras. Thus, the multiple candidate relative extrinsic parameters include all the available relative extrinsic parameters. The options of the target relative extrinsic parameters are enriched. On this basis, screening is performed to improve the comprehensiveness of the target relative extrinsic parameter selection and the accuracy of the target relative extrinsic parameters.

[0062] Step S120: screening multiple candidate relative external parameters for credibility to obtain a target relative external parameter.

[0063] Through credibility screening, more accurate target relative extrinsic parameters can be selected from multiple candidate relative extrinsic parameters, thereby improving the accuracy of camera group parameter calibration.

[0064] In some embodiments, the basis for credibility screening may be a common view parameter corresponding to the candidate relative extrinsic parameter. The common view parameter corresponding to the candidate relative extrinsic parameter may characterize the overlap of the field of view between the candidate camera pairs.

[0065] It should be noted that the more the fields of view overlap between the two cameras, the higher the accuracy of the relative external parameters between the two cameras, and the less the fields of view overlap, the lower the accuracy of the relative external parameters. Therefore, using the common view parameter can achieve the screening of more accurate target relative external parameters.

[0066] In one implementation scenario, a common view parameter corresponding to each candidate relative extrinsic parameter is obtained, and multiple candidate relative extrinsic parameters are screened based on the common view parameter to obtain a target relative extrinsic parameter.

[0067] In some embodiments, the aforementioned target relative extrinsic parameter is a relative extrinsic parameter between a pair of cameras. Different target relative extrinsic parameters have different reference coordinate systems. Exemplarily, the target relative extrinsic parameters include the relative extrinsic parameters between camera a and camera b, the relative extrinsic parameters between camera a and camera c, and the relative extrinsic parameters between camera c and camera d. After determining the target relative extrinsic parameters, the target relative extrinsic parameters can be converted to a coordinate system for subsequent processing. Specifically, one of all cameras can be selected as a reference camera, and the relative extrinsic parameters between each camera and the reference camera can be converted. Continuing with the previous example, camera a is used as the reference camera, and the results obtained by the conversion include the relative extrinsic parameters of camera b relative to camera a, the relative extrinsic parameters of camera c relative to camera a, and the relative extrinsic parameters of camera d relative to camera a.

[0068] Step S130: Determine the actual extrinsic parameters of each camera based on the target relative extrinsic parameters.

[0069] Among them, the target relative external parameters are determined by taking the camera as the reference coordinate system, and the actual external parameters are the external parameters of the camera relative to the first world coordinate system, wherein the first world coordinate system can be predetermined, and its origin and coordinate axes can be fixed.

[0070] In some embodiments, the above relative external parameter screening process can be implemented by the following steps: taking each camera as a node, connecting edges between two nodes corresponding to each candidate camera pair, and determining the weight of the corresponding edge based on the common view parameters of each candidate camera pair to form a target graph. Solving the maximum spanning tree for the target graph, and taking the candidate relative external parameters corresponding to the edges included in the maximum spanning tree as the target relative external parameters.

[0071] The spanning tree is a very small connected subgraph of the target graph, which contains all n nodes in the graph, but only has n-1 edges that constitute a tree. Based on the spanning tree, the unique relative extrinsic parameters of each camera can be determined. The target graph is a weighted connected graph, and the maximum spanning tree is the spanning tree with the largest edge weight among all spanning trees of the target graph. The maximum spanning tree has the largest weight, indicating that the target relative extrinsic parameters are the most accurate, thereby improving the accuracy of parameter calibration.

[0072] In some embodiments, the common viewing parameter may be the number of common viewing angle points.

[0073] It should be noted that the images captured by the two cameras can reflect the overlap of the cameras' fields of view. For example, if the images captured by the two cameras contain the same points in the physical world, it means that the fields of view of the two cameras overlap. Therefore, the image pairs captured by the two cameras can be used to determine the common view parameters between the two cameras.

[0074] In a specific application scenario, two cameras respectively capture the target scene to obtain two calibration images, forming a pair of calibration image pairs. At least one calibration point is set in the target scene. The number of common viewing angle points between the two cameras can be determined based on the calibration image pairs. For a pair of calibration image pairs, if there is a first feature point in each of the two calibration images, and the two first feature points represent the same calibration point, then the two first feature points can be used as common viewing angle points. The number of common viewing angle points in each pair of calibration image pairs can be determined based on the image, and the number of common viewing angle points between the candidate camera pairs can be the sum of the number of common viewing angle points in all calibration image pairs.

[0075] See also Figure 2 , Figure 2 It is a schematic diagram of an embodiment of the maximum spanning tree of the present application.

[0076] In this embodiment, a total of 6 cameras are set in the target scene to form a target camera group. Each camera is used as a node, and edges are connected between two nodes. The common view parameter between the two cameras is used as the corresponding edge weight to form a target graph. Figure 2 As shown in a, Figure 2 In the above code, b represents the process of solving the maximum spanning tree.

[0077] In some embodiments, converting the target relative extrinsic parameter to the actual extrinsic parameter of each camera may include two steps: optimizing the camera parameters and converting the optimized target relative extrinsic parameter to the actual extrinsic parameter.

[0078] See also Figure 3 , Figure 3 It is a flowchart of an embodiment of camera parameter optimization of the present application.

[0079] Specifically, this step may include:

[0080] Step S310: Obtain at least one set of calibration point position information.

[0081] The calibration point position information group includes the position information of the calibration points obtained based on the calibration images collected by each camera. The calibration image is collected from the target scene, and at least one calibration point is set in the target scene. All calibration images corresponding to the same calibration point position information group are collected when the positions of all calibration points in the target scene do not change.

[0082] In a specific application scenario, a calibration image is obtained by performing multiple synchronous acquisitions by a target camera group. Synchronous acquisition may refer to that when there is no change in the calibration point in the target scene, each camera performs an acquisition. During a synchronous acquisition, a frame of image captured by each camera is used as a calibration image group. Calibration point detection is performed on each calibration image in a group, and the position information of the calibration point in each image is obtained to form a calibration point position information group.

[0083] In some embodiments, the target relative extrinsic parameter includes a rotation matrix, and before optimization, the rotation matrix is ​​converted into a rotation vector.

[0084] In a specific application scenario, the rotation matrix has 9 degrees of freedom. Converting the rotation matrix into a rotation vector can reduce the 9 degrees of freedom to 3 degrees of freedom, which can simplify the variables to be optimized, stabilize the gradient descent process, and search for the optimal solution.

[0085] Step S320: Optimizing the variables to be optimized based on at least one set of calibration point position information groups to obtain optimized relative external parameters of each camera.

[0086] The variables to be optimized include target relative extrinsic parameters. It should be noted that all target relative extrinsic parameters are converted to use the same camera as a reference for ease of calculation.

[0087] In some embodiments, the above optimization is a bundled adjustment, and the variables to be optimized also include the internal parameters of each camera. Among them, the internal parameters can be pre-calibrated, and the specific calibration method can be selected according to the actual application needs. For example, the Zhang Zhengyou calibration method can be used to calibrate through a calibration board (chessboard, ChArUco calibration board, etc.).

[0088] In some embodiments, the optimization involved in step S320 uses a small batch stochastic gradient descent algorithm as an optimization algorithm. Each time the optimization is performed, a portion of all the calibration point position information groups is randomly selected for optimization.

[0089] In a specific application scenario, the small batch stochastic gradient descent algorithm is implemented using the torch framework.

[0090] In the above scheme, by using the small batch stochastic gradient descent algorithm, the complete optimization process can be divided into multiple small batches, and all batches can be assigned to multiple processes, which is conducive to improving the optimization efficiency. In addition, each time a part of the data is randomly selected to participate in the calculation, randomness is introduced to avoid local optimality.

[0091] In some embodiments, the optimization objective is to minimize a loss function. The optimization process includes a step of determining a target loss based on the loss function, and the target loss may include at least one of a length loss and a straight line loss.

[0092] In some embodiments, at least three calibration points are set in the target scene. It should be noted that a calibration object may be set in the target scene to provide the calibration points. For example, the calibration object is a T-bar on which three calibration points are fixedly set.

[0093] In a specific application scenario, light bulbs are used as calibration points, and three light bulbs are fixed on the crossbar of a T-bar as three calibration points. The position of the T-bar is adjusted so that the position of the calibration point is different each time the synchronous acquisition is performed. The relative position between the calibration points remains unchanged. The calibration images of different calibration point position information groups are acquired under different calibration point positions.

[0094] In a specific application scenario, calibration point detection is performed on a calibration image to determine the calibration points contained in the image and the plane position information of each calibration point, that is, the pixel coordinates.

[0095] Among them, obtaining the length loss may include: for a group, obtaining the predicted spatial distance between two calibration points among at least three calibration points, and determining the length loss corresponding to the calibration point position information group based on the difference between the predicted spatial distance and the corresponding real spatial distance.

[0096] Wherein, obtaining the straight line loss may include: for a group, obtaining the spatial position information of the first calibration point, the second calibration point and the third calibration point, predicting the predicted position information of the second calibration point based on the spatial position information of the first calibration point and the spatial position information of the third calibration point, and determining the straight line loss corresponding to the calibration point position information group based on the difference between the spatial position information of the second calibration point and the predicted position information. Wherein, the first calibration point, the second calibration point and the third calibration point are collinear, and the second calibration point is between the first calibration point and the third calibration point.

[0097] In some embodiments, the target loss may also include a reprojection loss.

[0098] Among them, obtaining the reprojection loss can include: for a group, obtaining the spatial position information of the first calibration point, the second calibration point and the third calibration point, reprojecting the spatial position information of each calibration point onto each camera plane respectively to obtain the reprojection position, and determining the reprojection loss based on the difference between the reprojection position and the pixel coordinates of the calibration point in the camera captured image.

[0099] In a specific application scenario, d1 and d2 represent the distance between the first calibration point and the second calibration point, and between the second calibration point and the third calibration point on the T-bar. i represents the parameters of the i-th camera, A j , B j , C j represents the spatial coordinates of the three points of the T-bar at the jth moment. ij , b ij 、c ij represents the pixel coordinates of the three points of the T-bar in the field of view of the i-th camera at the j-th time. The target loss can be calculated using the following formula:

[0100]

[0101] loss_length ij =(A j -C j ) 2 -(d1+d2) 2

[0102] loss_reproj ij =(a ij -P i A j ) 2+(b ij -P i B j ) 2 +(c ij -P i C j ) 2

[0103] Among them, loss_line ij Indicates linear loss, loss_length ij Represents length loss, loss_reproj ij Represents the reprojection loss. Further, the three losses can be weighted as follows:

[0104] loss_iter ij =weight line *loss_line ij +weight reproj *loss_reproj ij +weight length *loss_length ij

[0105] loss=∑ i ∑ j loss_iter ij

[0106] Among them, weight line 、weight reproj 、weight length They represent the corresponding weights respectively, and finally the losses of all cameras and all groups are summed up to get the final loss.

[0107] Among them, A j , B j , C j Represents the spatial coordinates of the three points of the T-bar at the jth moment, and is also used as the variable to be optimized. It can be initialized by triangulating the position information of the calibration points at the jth moment.

[0108] See also Figure 4 , Figure 4 1 is a flow chart of an embodiment of external reference conversion of the present application. Specifically, the steps may include:

[0109] Step S410: Determine the conversion relationship between the first world coordinate system and the second world coordinate system.

[0110] It can be understood that the relative external parameter is an external parameter determined based on the camera coordinate system, while the actual external parameter is an external parameter determined based on the first world coordinate system, and the origin and coordinate axes of the first world coordinate system can be relatively fixed.

[0111] Specifically, the optimized relative external parameters are the external parameters of the camera relative to the second world coordinate system, which is the coordinate system currently used. The second world coordinate system is determined based on the camera coordinate system of the reference camera, and the reference camera can be any camera in the target camera group.

[0112] Among them, the actual external parameter is the external parameter of the camera relative to the first world coordinate system, which is the target coordinate system for the external parameter conversion.

[0113] Specifically, multiple alignment calibration points may be used to determine the conversion relationship between the two world coordinate systems based on spatial position information of the alignment calibration points in the first world coordinate system and spatial position information of the alignment calibration points in the second world coordinate system.

[0114] The first world coordinate system may be predetermined, and the second spatial position information of the alignment calibration point in the first world coordinate system may also be predetermined. The first spatial position information of the alignment calibration point in the second world coordinate system may be analyzed and obtained through the image of the alignment calibration point, thereby determining the conversion relationship.

[0115] In some embodiments, the plane position information of the second feature points contained in the multiple alignment images is obtained, wherein each camera captures an alignment image of the target scene. Multiple alignment calibration points are set in the target scene, and a second feature point represents an alignment calibration point in the alignment image. Based on the plane position information of the second feature points corresponding to each alignment calibration point, the first spatial position information of each alignment calibration point is determined. The first spatial position information is the position information of the alignment calibration point in the second world coordinate system.

[0116] In a specific application scenario, multiple alignment calibration points are set in the target scene, and each camera collects the target scene respectively to obtain the alignment images collected by each camera. Feature point detection is performed on each alignment image to obtain the planar position information of the second feature point in the alignment image. It should be noted that due to the limited field of view of the camera, there may be a situation where the alignment calibration points are not included in the alignment image collected by the camera.

[0117] In some embodiments, before performing feature point detection on the aligned images, the aligned images may be preprocessed, for example, by applying corresponding masks, feature enhancement, etc.

[0118] Step S420: converting the optimized relative extrinsic parameter based on the conversion relationship to obtain the actual extrinsic parameter.

[0119] See also Figure 5 , Figure 5 4 is a flow chart of an embodiment of step S410 of the present application. Specifically, step S410 may include:

[0120] Step S511: Obtaining annotation information of each aligned image.

[0121] In this embodiment, the alignment image may be annotated with annotation information, including the planar position information of the second feature point contained in the alignment image, and the correspondence between the alignment calibration point and the second feature point.

[0122] Based on the annotation information of each aligned image, the plane position information of all the second feature points corresponding to each alignment calibration point may be determined.

[0123] In a specific application scenario, the device can display the alignment image to the user for viewing, and determine the planar position information of the second feature point in the alignment image and the alignment calibration point corresponding to the second feature point in response to the user's operation instruction.

[0124] The second feature points included in the alignment image may be feature points of the target scene itself, and there is no need to additionally set calibration objects to provide alignment calibration points.

[0125] In a specific application scenario, there is a table tennis table in the target scene. The upper left corner of the table tennis table is the first alignment calibration point, the lower left corner is the second alignment calibration point, and the upper right corner is the third alignment calibration point. The feature point representing the upper left corner of the table tennis table in the alignment image represents the first alignment calibration point, and its plane position information is used as the plane position information of the first alignment calibration point. There is no need to set up additional calibration objects in this scene to provide alignment calibration points.

[0126] In this embodiment, the plane position information of the alignment calibration point can be directly obtained through the annotation information to determine its first spatial position information. The alignment calibration point can be arbitrarily selected by the annotation method, which can improve the flexibility of setting the alignment calibration point. For example, the feature corner points existing in the scene can be used as alignment calibration points, and the alignment calibration points that are more suitable for the target scene can be selected, which can improve the accuracy of parameter calibration.

[0127] In one implementation scenario, the additional calibration objects set in the target scene do not match the target scene size. For example, the scene size is very different from the size of the additional calibration objects, which will lead to a decrease in parameter calibration accuracy. Directly selecting feature corner points in the field as alignment calibration points does not require additional calibration objects, and can flexibly select calibration points that can adapt to the target scene size, thereby improving calibration accuracy.

[0128] In a specific application scenario, the target scene is a golf course. The size of the additional calibration object is usually small, for example, 530mm, which cannot adapt to the calibration of the camera in the golf course. It is expensive to prepare a calibration object that matches the size of the golf course, and it is also troublesome to use. By selecting and marking alignment calibration points from the image, the alignment annotation points can be flexibly set, and the appropriate alignment calibration points can be selected from the site feature points, which can conveniently realize parameter calibration and improve the accuracy of parameter calibration.

[0129] In some embodiments, the labeling information is obtained using the labelme labeling tool.

[0130] Step S512: obtaining first spatial position information of the alignment calibration point based on the plane position information of the second feature point corresponding to the alignment calibration point.

[0131] For each alignment calibration point, its first spatial position information can be determined respectively. Specifically, the plane position information of the second feature point corresponding to the same alignment calibration point is triangulated to obtain the first spatial position information of the alignment calibration point in the second world coordinate system. The triangulation requires the plane position information of at least two second feature points.

[0132] In some embodiments, other methods may be used to convert the planar position information into spatial position information.

[0133] In a specific application scenario, the plane position information of all second feature points corresponding to the same alignment calibration point and the parameters of the camera that collected the second feature points are used as the input of triangulation. The external parameters of the camera used at this time are the external parameters in the second world coordinate system, and the spatial position information of the alignment calibration point in the second world coordinate system is obtained.

[0134] Step S513: Determine a conversion relationship based on the first spatial position information and the second spatial position information of each alignment calibration point.

[0135] In some embodiments, the annotation information of the aligned image may further include second spatial position information of the aligned calibration points.

[0136] In some embodiments, the transformation relationship may be determined by using point cloud matching.

[0137] In a specific application scenario, after determining the first spatial position information and the second spatial position information corresponding to each alignment calibration point, an iterative closest point (ICP) algorithm may be used to determine the conversion relationship between the two world coordinate systems.

[0138] See also Figure 6 , Figure 6It is a flowchart of another embodiment of step S410 of the present application. In this embodiment, an alignment calibration point is set in the target scene, and the first spatial position information of the alignment calibration point is determined by image processing.

[0139] In a specific application scenario, an L-shaped rod may be placed in the target scene, and alignment calibration points are provided at the turning point and two end points of the L-shaped rod. In some cases, a light bulb may be used as an alignment calibration point.

[0140] Specifically, step S410 may include:

[0141] Step S611: performing feature point detection on the aligned image to obtain the planar position information of the second feature point contained in the aligned image.

[0142] In this embodiment, the plane position information of the feature points contained in the aligned image is obtained by performing feature point detection on the image. In some embodiments, the plane position information of the second feature point can be obtained by spot detection.

[0143] Step S612: clustering all second feature points.

[0144] The purpose of clustering is to determine the second feature points representing the same alignment calibration point. The second feature points of the same category represent the same alignment calibration point.

[0145] It is understandable that two second feature points acquired by different cameras may represent the same alignment calibration point. All second feature points can be clustered to determine all second feature points corresponding to the same alignment calibration point. Exemplarily, the distance between the second feature points of the same category is smaller than the distance between the two second feature points of different categories. Therefore, the second feature points can be clustered using the distance between the second feature points.

[0146] It should be noted that, at this time, the second feature points of a category represent the same alignment calibration point, but which alignment calibration point it specifically represents has not yet been determined.

[0147] In some implementation scenarios, the number of alignment calibration points is three, and correspondingly, the second feature points are divided into three categories, namely, the first category, the second category, and the third category. Exemplarily, the first category indicates which of the three alignment calibration points has not yet been determined.

[0148] In some embodiments, two second feature points from different cameras form a second feature point pair, the second feature points and the camera that collects the second feature points form a ray, and the distance between the two rays in space can be used to cluster the second feature points.

[0149] In some embodiments, the symmetric epipolar distance can also be used to measure the distance between two second feature points from different cameras. The symmetric epipolar distance is obtained by projecting the distance between two rays in space onto two camera planes. The distance between the second feature points acquired by the same camera can be set to a preset value to avoid interference with clustering. Exemplarily, the preset value can be infinite. The cost matrix is ​​constructed using the symmetric epipolar distance, and the second feature points are clustered based on the symmetric epipolar distance to obtain "clusters", and the second feature points in each cluster are the same type of second feature points, representing the same spatial point.

[0150] Step S613: triangulate the plane position information of the second feature points of the same category to obtain the to-be-matched spatial position information corresponding to each category.

[0151] The spatial position information to be matched is the position information of the spatial point represented by the second feature point of this category in the second world coordinate system.

[0152] In a specific application scenario, the planar position information of the second feature point of the same category and the parameters of the camera that captured the second feature point are used as inputs of triangulation to obtain the spatial position information of the corresponding spatial point to be matched.

[0153] It can be understood that the second feature point of a category represents the same spatial point. Based on the spatial distance between the spatial points and the spatial distance between the alignment calibration points, the matching relationship between the spatial point and the alignment calibration point can be determined, so that the spatial position information to be matched can be associated with the alignment calibration point.

[0154] Step S614: Based on the matching relationship between the first spatial distance and the second spatial distance, determine the matching relationship between the spatial position information to be matched and the alignment calibration point, and use the spatial position information to be matched that matches the alignment calibration point as the first spatial position information of the alignment calibration point.

[0155] The first spatial distance is the distance between every two pieces of spatial position information to be matched, and the second spatial distance is the distance between every two alignment calibration points.

[0156] In a specific application scenario, the alignment calibration point includes a first alignment calibration point, a second alignment calibration point, and a third alignment calibration point. If the first spatial distance a is the same as a second spatial distance A, the first spatial distance b is the same as the second spatial distance B. The first spatial distance a is determined based on the spatial position information 1 to be matched and the spatial position information 2 to be matched, and the second spatial distance b is determined based on the spatial position information 2 to be matched and the spatial position information 3 to be matched. The second spatial distance A is determined based on the first alignment calibration point and the second alignment calibration point, and the second spatial distance B is determined based on the first alignment calibration point and the third alignment calibration point.

[0157] From this comparison, it can be seen that the spatial position information 2 to be matched is involved in the calculation of the first spatial distance a and the first spatial distance b at the same time, and the first alignment mark point is involved in the calculation of the second spatial distance A and the second spatial distance B at the same time. The spatial position information 2 to be matched is matched with the first alignment mark point. Therefore, the spatial position information 2 to be matched is used as the first spatial position information of the first alignment mark point.

[0158] Step S615: Determine a conversion relationship based on the first spatial position information and the second spatial position information of each alignment calibration point.

[0159] See also Figure 7 , Figure 7 1 is a flow chart of another embodiment of step S110 of the present application. Specifically, this step may include:

[0160] Step S711: Acquire the calibration images captured by each camera.

[0161] The calibration image is acquired from the target scene, and at least one calibration point is set in the target scene.

[0162] In some embodiments, calibration points are provided by setting additional calibration objects in the target scene. In a specific application scenario, a T-bar is used as the calibration object, a light bulb is used as the calibration point, and three calibration points are fixedly set on the crossbar of the T-bar.

[0163] Furthermore, suppose the distances between the three points of the T-bar are d1 and d2, and d1>d2. The closest distance between the middle point of the T-bar and the camera is Z min To ensure that the projections d*1 and d*2 of d1 and d2 satisfy the inequality d*1>d*2, d1 and d2 need to satisfy In this solution, d1=510mm and d2=250mm.

[0164] The target camera group performs at least one synchronous acquisition on the target scene, and one synchronous acquisition obtains a set of calibration image groups. All calibration images in a set of calibration image groups are acquired when the position of the calibration point is fixed. Different groups are acquired when the positions of the calibration points are different.

[0165] Step S712: pre-process the calibration image.

[0166] Wherein, each calibration image is preprocessed respectively, and the preprocessing steps can be set according to actual application requirements.

[0167] In some embodiments, preprocessing includes at least one of the following steps: constructing a mask image based on a reference image corresponding to the calibration image, and applying the mask image to the calibration image; performing feature enhancement on the calibration image for color information of calibration points; and performing a morphological closing operation on the calibration image.

[0168] In some embodiments, all calibration images captured by the same camera correspond to the same reference image. The reference image is an empty field image captured by the camera when no calibration points are set in the target scene. Constructing a mask map based on the reference image may include: extracting an area that may interfere with the detection of the calibration points from the reference image, and using the area as the background area. Specifically, the area whose color information represented by the pixels is consistent with the color information of the calibration points can be extracted from the reference image as the background area, and other areas are used as the foreground area. Therefore, after the mask map is applied to the corresponding calibration image, the image that interferes with the detection of the calibration points is used as the background and is not paid attention to.

[0169] In a specific application scenario, the calibration point is red. The red area in the empty field image is extracted as the background area, so that in the subsequent processing of the calibration image acquired by the camera, the red area in the empty field image will not interfere with the calibration point detection.

[0170] It should be noted that the color information of the calibration points can be set according to the actual application needs, and the feature enhancement step can be adjusted accordingly. For example, if the camera is an RGB camera and the calibration image is an RGB image, the color of the calibration points can be any one of red, blue, and green. In some cases, the camera can also be a black and white camera or an infrared camera.

[0171] In some embodiments, a light bulb is used to represent a calibration point, and a morphological closing operation can fill in the holes in the light bulb image to facilitate subsequent spot detection and improve detection accuracy.

[0172] See also Figure 8 , Figure 8 It is a schematic diagram of another embodiment of the pre-processing step of the present application.

[0173] In this embodiment, the camera is an RGB camera, and the color of the calibration point is red. The feature enhancement of the calibration image for the color information of the calibration point includes: obtaining a red channel map, a blue channel map, and a green channel map, comparing each pixel value in the blue channel map and the green channel map, taking the larger value of the two to form a new grayscale map, and using the red channel map to subtract the above grayscale map to obtain a feature map.

[0174] In a specific application scenario, when no calibration points are set in the target scene, an empty field image captured by each camera is obtained for use in the mask image corresponding to the camera. When calibration points are set in the target scene, a calibration image captured by the camera is obtained. The mask image is applied to the corresponding calibration image, and the calibration image after the mask is applied is subjected to feature enhancement for the color information of the calibration points. A morphological closing operation is performed on the calibration image after feature enhancement, and the obtained image is binarized for use in calibration point detection.

[0175] Step S713: performing calibration point detection on the preprocessed calibration image to obtain calibration point position information corresponding to the calibration image.

[0176] It should be noted that by performing feature point detection on the calibration image, the plane position information of the first feature point in the calibration image can be obtained. Further, according to the plane position information of the first feature point in the calibration image, the calibration points corresponding to each first feature point can be determined, and the calibration point position information of the calibration points existing in the calibration image is obtained.

[0177] In some cases, due to the limited field of view of the camera, some calibration images may not contain calibration points, and some calibration images may contain partial calibration points. In one implementation scenario, three calibration points are set in the target scene. For calibration images that cannot collect complete calibration points, the first feature points detected from the image cannot be associated with the calibration points. For example, if two first feature points are detected in the image, since the positional relationship between the three first feature points is required to determine the corresponding relationship between the first feature points and the calibration points, it is currently impossible to determine which calibration points the two first feature points represent, and the calibration point position information cannot be obtained, and the first feature points cannot be matched, and cannot be used as matching point pairs.

[0178] In a specific application scenario, the calibration point position information includes the pixel coordinates corresponding to each calibration point in the image. There is a first feature point in the calibration image, that is, the projection of the calibration point in the camera captured image, and the pixel of the first feature point is used as the pixel coordinate of the calibration point.

[0179] Step S714: Determine candidate relative extrinsic parameters between the candidate camera pairs based on the calibration point position information acquired by the candidate camera pairs.

[0180] For each candidate camera pair, the candidate relative extrinsic parameters are determined based on the calibration point position information collected by the two cameras contained therein.

[0181] Specifically, each time synchronous acquisition is performed, the two cameras included in the candidate camera pair respectively capture a calibration image to form a pair of calibration images. Based on these two images, the matching point pairs in the two images can be determined. The matching point pairs refer to two first feature points from the two images respectively, and the two first feature points represent the same calibration point, that is, the common viewing angle point. Exemplarily, image a is subjected to calibration point detection to obtain the position information of the first calibration point, the second calibration point and the third calibration point contained therein, and image b is subjected to calibration point detection to obtain the position information of the first calibration point, the second calibration point and the third calibration point contained therein, so that the matching point pairs representing the same calibration point in the two images can be determined.

[0182] Based on the position information of the matching point pairs collected by the candidate camera pairs (i.e., the calibration point position information), the initial relative external parameters of the candidate camera pairs can be obtained. In some embodiments, the recoverpose function is called to process the position information of the matching point pairs to obtain the initial relative external parameters between the candidate camera pairs. The initial relative external parameters include rotation parameters and translation parameters. The translation parameters in the initial relative external parameters are the result of scaling by the scaling factor.

[0183] Furthermore, the above initial relative external parameters can be used to estimate the distance between the calibration points, and the scaling factor can be determined based on the estimated distance and the actual distance.

[0184] In some embodiments, at least two calibration points are set in the target scene, and two calibration points are selected from them. For multiple pairs of calibration image pairs acquired by the candidate camera pair, if the two calibration points are acquired in the calibration image pair, the calibration image pair is used as the target calibration image pair. For each pair of target calibration image pairs, the pixel coordinates of the same calibration point in the two images are triangulated using the initial relative external parameters to estimate the spatial position information of the calibration point, and the estimated spatial distance between the two calibration points is estimated based on the spatial position information of the two calibration points. Based on the estimated spatial distances corresponding to all image pairs, an estimated spatial distance reference value is determined, and the scaling factor corresponding to the candidate camera pair is determined based on the ratio of the estimated spatial distance reference value to the actual distance between the two calibration points.

[0185] In a specific application scenario, the candidate camera pair collects n pairs of target calibration image pairs, and both images in each pair of target calibration image pairs collect calibration points A and C. Triangulation is performed based on the pixel coordinates of calibration point A in the two images to estimate the spatial position information of calibration point A. Similarly, the estimated spatial distance between calibration point A and calibration point C is further calculated. An estimated spatial distance can be calculated for each pair of target calibration images, forming a set {d1, d2, ..., dn}. The estimated spatial distance reference value is fitted using the random sampling consensus (RANSAC) algorithm, and the actual distance between calibration point A and calibration point C is predetermined. The ratio of the calculated estimated spatial distance reference value to the actual distance is used as a scaling factor. At this time, the scaling factor can be used to restore the translation parameter in the corresponding initial relative external parameter to the value before scaling.

[0186] In some embodiments, the above calibration image pair can also be used to determine the common view parameter between the two cameras. Specifically, the matching point pairs included in the calibration image pair indicate that there is a common view between the two cameras, and the common view parameter can be determined based on the number of matching point pairs.

[0187] Exemplarily, all calibration image pairs captured by the candidate camera pair are counted, and the number of matching point pairs contained in all calibration image pairs is counted as the number of common viewing angle points of the candidate camera pair, so as to serve as the common viewing parameter of the candidate camera pair. Exemplarily, a common viewing condition can be set to determine whether the calibration image pair satisfies the common viewing condition, and the number of image pairs satisfying the common viewing condition is used as the common viewing parameter, or the ratio of the number of image pairs satisfying the common viewing condition to the total number of image pairs is used as the common viewing parameter. Among them, the common viewing condition can be that the number of matching point pairs contained in the calibration image pair is equal to the number of calibration points, that is, both images have completely captured all calibration points.

[0188] It should be noted that in the aforementioned embodiment, multiple synchronous acquisitions are performed using the target camera group. Each time synchronous acquisition is performed, all cameras capture one frame of image as a set of calibration image groups. Calibration point detection is performed on the calibration image to obtain calibration point position information. The calibration point position information is used for both determination of candidate relative extrinsic parameters and bundling optimization. In some embodiments, the calibration image used in the bundling optimization stage and the calibration image used in the candidate relative extrinsic parameter determination stage may also be different images collected separately, and accordingly, the calibration information is also different.

[0189] Those skilled in the art can understand that in the above methods of the specific implementation, the order of writing each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic. The contents of different embodiments can be combined on the basis of no contradiction.

[0190] See also Fig. 9 , Fig. 9 It is a flowchart of another embodiment of the camera calibration method of the present application.

[0191] In this embodiment, the camera internal parameters are pre-calibrated. By collecting empty field pictures and establishing a mask map, the robustness of corner detection is further enhanced. Using a T-bar as a calibration object, the camera is initialized based on the maximum spanning tree and the ransac algorithm, which can better initialize the camera pose. Based on the torch algorithm framework (an automatic differentiation framework), a loss function formed by weighting three losses: calibration rod length loss + calibration rod straight line loss + reprojection error loss is designed to stabilize the gradient descent process, and a better solution can be found compared to other methods. Using an L-bar as a calibration object, first match it through the symmetric limit distance to form a clique (for example, if the calibration object contains three calibration points, there will be three clumps in the end). Triangulate each clumping, and restore the identification information of the marking point through the distance of the three-dimensional point. Finally, align it with the world coordinate system through point cloud matching. This method combines multi-perspective information and can better align the stereo vision system with the real world.

[0192] See also Fig.10 , Fig.10 It is a schematic diagram of the framework of an embodiment of a camera calibration device of the present application.

[0193] In this embodiment, the camera calibration device 100 is used for calibrating a target camera group, which includes multiple cameras. The camera calibration device 100 includes an acquisition module 101, a screening module 102 and an extrinsic parameter adjustment module 103. The acquisition module 101 is used to acquire multiple candidate relative extrinsics, where the candidate relative extrinsics are relative extrinsics between candidate camera pairs consisting of any two cameras; the screening module 102 is used to perform credibility screening on the multiple candidate relative extrinsics to obtain the target relative extrinsics; the extrinsic parameter adjustment module 103 is used to determine the actual extrinsics of each camera based on the target relative extrinsics.

[0194] See also Fig.11 , Fig.11 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application.

[0195] In this embodiment, the electronic device 110 includes a memory 111 and a processor 112 coupled to each other, and the processor 112 is used to execute program instructions stored in the memory 111 to implement any of the above-mentioned camera calibration methods. In a specific implementation scenario, the electronic device 110 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 110 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0196] Specifically, the processor 112 is used to control itself and the memory 111 to implement any of the above-mentioned camera calibration methods. The processor 112 may also be referred to as a CPU (Central Processing Unit). The processor 112 may be an integrated circuit chip having the ability to process signals. The processor 112 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 112 may be implemented by an integrated circuit chip.

[0197] See also Fig.12 , Fig.12 It is a schematic diagram of the framework of an embodiment of a camera calibration system of the present application.

[0198] In this embodiment, the camera calibration system 120 includes a plurality of cameras 121 and an electronic device 122. The electronic device 122 may be Fig.11 Electronic devices in.

[0199] See also Fig.13 , Fig.13 A schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application.

[0200] In this embodiment, the computer-readable storage medium 130 stores program instructions 131 that can be executed by a processor, and the program instructions 131 are used to implement any of the above-mentioned camera calibration methods.

[0201] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0202] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0203] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0204] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0205] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0206] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A camera calibration method, characterized in that: The method is applied to the calibration of a target camera group, wherein the target camera group includes a plurality of cameras, and the method includes: Acquire a plurality of candidate relative extrinsic parameters, where the candidate relative extrinsic parameters are relative extrinsic parameters between any two candidate camera pairs formed by the cameras; Performing credibility screening on the multiple candidate relative external parameters to obtain a target relative external parameter; Based on the target relative extrinsic parameters, actual extrinsic parameters of each of the cameras are determined.

2. The method according to claim 1, characterized in that The credibility screening of the plurality of candidate relative external parameters to obtain the target relative external parameter comprises: Obtaining common view parameters corresponding to each of the candidate relative extrinsic parameters, wherein the common view parameters are used to characterize the overlap of the fields of view between the candidate camera pairs; The plurality of candidate relative extrinsic parameters are screened based on the common view parameter to obtain the target relative extrinsic parameter.

3. The method according to claim 2, characterized in that The common view parameter is the number of common view angle points between the candidate camera pair; and the screening of the plurality of candidate relative extrinsic parameters based on the common view parameter to obtain the target relative extrinsic parameter includes: Taking each of the cameras as a node, connecting edges between two nodes corresponding to each of the candidate camera pairs, and determining the weight of the corresponding edge based on the number of common viewing angles of each of the candidate camera pairs, so as to form a target graph; Solving the maximum spanning tree for the target graph; The candidate relative extrinsic parameter corresponding to the edge included in the maximum spanning tree is used as the target relative extrinsic parameter.

4. The method according to claim 3, characterized in that The number of common viewing angle points between the candidate camera pairs is determined based on a number of calibration image pairs captured by the candidate camera pairs; The calibration image pair is obtained by respectively capturing a target scene by two cameras included in the candidate camera pair, and at least one calibration point is set in the target scene; the common viewing angle point in the calibration image pair includes two first feature points respectively from two calibration images in the calibration image pair, and the two first feature points represent the same calibration point.

5. The method according to any one of claims 1 to 4, characterized in that: The actual extrinsic parameter is an extrinsic parameter of the camera relative to the first world coordinate system; and determining the actual extrinsic parameter of each camera based on the target relative extrinsic parameter includes: Acquire at least one group of calibration point position information groups, the calibration point position information groups comprising position information of calibration points respectively obtained based on calibration images acquired by each of the cameras, the calibration images being acquired from a target scene, the target scene being provided with at least one calibration point, and all the calibration images corresponding to the same calibration point position information group being acquired when the position of the at least one calibration point is fixed; Based on the at least one group of calibration point position information groups, the variables to be optimized are optimized to obtain the optimized relative extrinsic parameters of each camera; wherein the variables to be optimized include the target relative extrinsic parameters, the optimized relative extrinsic parameters are extrinsic parameters relative to a second world coordinate system, the second world coordinate system is determined based on the camera coordinate system of a reference camera, and the reference camera is any camera in the target camera group; Determining a conversion relationship between the first world coordinate system and the second world coordinate system; The optimized relative extrinsic parameter is converted based on the conversion relationship to obtain the actual extrinsic parameter.

6. The method according to claim 5, characterized in that Using a small batch stochastic gradient descent algorithm as the optimization algorithm for optimizing the variable to be optimized; and / or, At least three calibration points are set in the target scene, and the calibration images corresponding to different calibration point position information groups are acquired when the positions of the at least three calibration points are different; the optimization is performed based on the target loss, and the target loss includes at least one of a length loss and a straight line loss. The step of acquiring the target loss includes: In response to the target loss including the length loss, for each of the calibration point position information groups, obtaining a predicted spatial distance between two of the at least three calibration points, and determining the length loss corresponding to the calibration point position information group based on a difference between the predicted spatial distance and a corresponding real spatial distance; In response to the target loss including the straight-line loss, for each of the calibration point position information groups, the spatial position information of the first calibration point, the second calibration point and the third calibration point among the at least three calibration points is obtained; based on the spatial position information of the first calibration point and the spatial position information of the third calibration point, the predicted position information of the second calibration point is predicted; based on the difference between the spatial position information of the second calibration point and the predicted position information, the straight-line loss corresponding to the calibration point position information group is determined; wherein, the first calibration point, the second calibration point and the third calibration point are collinear, and the second calibration point is between the first calibration point and the third calibration point.

7. The method according to claim 5, characterized in that Determining the conversion relationship between the first world coordinate system and the second world coordinate system includes: Acquire the planar position information of the second feature point contained in the alignment image captured by each camera; the alignment image is acquired by capturing a target scene, a plurality of alignment calibration points are set in the target scene, and one of the second feature points represents one of the alignment calibration points in the alignment image; Determining first spatial position information of each alignment calibration point based on the plane position information of the second feature point corresponding to each alignment calibration point; Based on the first spatial position information and the second spatial position information of each alignment mark point, the conversion relationship is determined, wherein the first spatial position information is the position information of the alignment mark point in the second world coordinate system, and the second spatial position information is the position information of the alignment mark point in the first world coordinate system.

8. The method according to claim 7, characterized in that The obtaining of the planar position information of the second feature points contained in the aligned images captured by each of the cameras comprises: Acquire annotation information of each of the aligned images; the annotation information includes planar position information of a second feature point contained in the aligned image, and a corresponding relationship between the alignment calibration point and the second feature point; The determining the first spatial position information of each alignment calibration point based on the plane position information of the second feature point corresponding to each alignment calibration point comprises: For each alignment calibration point, triangulation is performed on the plane position information of the second feature point corresponding to the alignment calibration point to obtain the first spatial position information of the alignment calibration point.

9. The method according to claim 7, characterized in that: The obtaining of the planar position information of the second feature points contained in the aligned images captured by each of the cameras comprises: Performing feature point detection on the aligned image to obtain plane position information of a second feature point contained in the aligned image; The determining the first spatial position information of each alignment calibration point based on the plane position information of the second feature point corresponding to each alignment calibration point comprises: Clustering all the second feature points; wherein the second feature points of the same category represent the same alignment calibration point; Performing triangulation on the plane position information of the second feature points of the same category to obtain the to-be-matched spatial position information corresponding to each category; Based on the matching relationship between the first spatial distance and the second spatial distance, determining the matching relationship between the spatial position information to be matched and the alignment calibration points; the first spatial distance is the distance between every two of the spatial position information to be matched, and the second spatial distance is the distance between every two of the alignment calibration points; For each alignment mark point, the to-be-matched spatial position information that matches the alignment mark point is used as the first spatial position information of the alignment mark point.

10. The method according to any one of claims 1 to 9, characterized in that: Every two cameras in the target camera group form a pair of candidate camera pairs; and / or, The obtaining of multiple candidate relative external parameters includes: Obtain the calibration images collected by each camera; Preprocessing the calibration image, and performing calibration point detection on the preprocessed calibration image to obtain position information of the calibration points corresponding to the calibration image; For each of the candidate camera pairs, a candidate relative extrinsic parameter between the candidate camera pairs is determined based on the position information of the calibration points acquired by the candidate camera pairs.

11. The method according to claim 10, characterized in that The calibration image is acquired from a target scene, and at least one calibration point is set in the target scene; and the preprocessing of the calibration image includes at least one of the following: constructing a mask image based on a reference image corresponding to the calibration image, and applying the mask image to the calibration image; Performing feature enhancement on the calibration image with respect to color information of the calibration points; A morphological closing operation is performed on the calibration image.

12. A camera calibration device, characterized in that: The device is used for calibrating a target camera group, the target camera group includes a plurality of cameras, and the device includes: An acquisition module, used for acquiring a plurality of candidate relative extrinsic parameters, wherein the candidate relative extrinsic parameters are relative extrinsic parameters between candidate camera pairs formed by any two of the cameras; A screening module, used for screening the plurality of candidate relative external parameters for credibility to obtain a target relative external parameter; The extrinsic parameter adjustment module is used to determine the actual extrinsic parameters of each of the cameras based on the target relative extrinsic parameters.

13. An electronic device, characterized in that: The invention comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the camera calibration method according to any one of claims 1 to 11.

14. A camera calibration system, characterized in that: The system comprises: A plurality of cameras and an electronic device as claimed in claim 13.

15. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the camera calibration method according to any one of claims 1 to 11 is implemented.

Images