A multi-camera calibration method in a large scene, information processing terminal and storage medium

Abstract

The present invention belongs to the technical field of multi-view geometry under computer vision, and discloses a multi-camera calibration method, an information processing terminal and a storage medium in a large scene. The multi-camera in the large scene calibration uses a scene clustering algorithm and an abnormal trajectory removal algorithm to perform ambiguous matching filtering; the ambiguous matching filtering image uses different fusion modes to enhance weak texture information; the image after weak texture information enhancement uses an ICP registration algorithm combined with a camera configuration registration algorithm to perform coordinate system registration. The algorithm in the present invention has the ability to adapt to texture similarity and texture sparsity in large scenes. The accuracy of calibration is greatly improved. High-precision multi-camera pose calibration in large scenes can be achieved. The present invention obtains a transformation matrix by performing plane modeling on the layout and solving the spherical center coordinates. The above two methods complement each other and enhance the ease of use of the coordinate system registration algorithm.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-view geometry under computer vision, and in particular to a multi-camera calibration method in a large scene, an information processing terminal and a storage medium. Background Art

[0002] At present, one of the basic tasks of computer vision is to calculate the geometric information of objects in three-dimensional space from the image information obtained by the camera, and thus reconstruct and identify the objects. The relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image is determined by the geometric model of camera imaging. These geometric model parameters are the camera parameters. Under most conditions, these parameters must be obtained through experiments and calculations. Whether in image measurement or machine vision applications, the calibration of camera parameters is a very critical link. The accuracy of the calibration results and the stability of the algorithm directly affect the accuracy of the results produced by the camera. Therefore, doing a good job of camera calibration is the prerequisite for doing a good job in subsequent work, and improving the calibration accuracy is the focus of scientific research.

[0003] In open large scenes, cameras are deployed in a long-distance and dispersed manner, which brings great challenges to high-precision camera pose estimation. At present, the camera pose is generally obtained by placing significant markers during the camera pose calibration process. However, significant markers have the problem of balancing the difficulty of production and visibility in large scenes. At the same time, with the increase of usage time, significant markers are worn out, which in turn affects the calibration accuracy.

[0004] In order to overcome the shortcomings of landmark calibration, the current patent "Camera self-calibration method and device based on multiple photos" proposes a method for calibrating camera pose using scene information. The patent "Method and system for automatically calibrating camera extrinsics" proposes to calibrate camera pose in combination with lane information in a specific scene. However, the methods proposed in the above patents cannot completely solve the large scene situation, and there are still three problems that need to be solved. First, in scenes with similar texture information, mismatches will occur. How to study an efficient algorithm to filter out mismatches is crucial. Secondly, in low-texture situations, the extracted feature point constraint information is insufficient, and then the camera pose constraint equation is insufficient, so the calculated camera pose has low accuracy or cannot be calculated. Finally, the camera pose obtained by the self-calibration algorithm is in the self-calibration coordinate system and has not been converted to the real world coordinate system. It is necessary to design a method to efficiently convert the coordinate system through scene constraint information. Therefore, it is necessary to design a high-efficiency and high-precision multi-camera calibration algorithm for large scenes.

[0005] Through the above analysis, the problems and defects of the prior art are as follows:

[0006] (1) In the existing technology, the calibration accuracy is low during matching due to ambiguity caused by similar textures in the ambiguous matching filtering of images from different perspectives.

[0007] (2) The existing technology cannot avoid the situation where the feature points in the same area are too dense when enhancing the weak texture image information, resulting in very low calibration accuracy under weak texture.

[0008] (3) In the prior art, regarding image coordinate system registration, the existing enhanced coordinate system registration methods have poor complementarity, resulting in low accuracy of the registered images. Summary of the invention

[0009] In view of the problems existing in the prior art, the present invention provides a multi-camera calibration method in a large scene, an information processing terminal and a storage medium.

[0010] The present invention is implemented as follows: a method for calibrating multiple cameras in a large scene includes: performing ambiguous matching filtering on multiple cameras in the large scene calibration using a scene clustering algorithm and an abnormal trajectory removal algorithm;

[0011] The ambiguous matched filter image is subjected to weak texture information enhancement using different fusion modes; the image after weak texture information enhancement is subjected to coordinate system registration using the ICP registration algorithm combined with the camera configuration registration algorithm.

[0012] Furthermore, the ambiguous matched filtering includes: in multi-camera calibration of a large scene, using breadth-first search to establish estimation information, dividing the trajectories in the scene into unique trajectories and ambiguous trajectories through scene consistency, penalty synchronization, and length weighting, and obtaining the unique trajectory ratio of each perspective image; searching for neighbor nodes of each perspective image to perform a unique trajectory ratio scoring metric, and performing trajectory filtering based on the unique trajectory ratio scoring metric.

[0013] Furthermore, the scene clustering algorithm includes: a set of images is represented as I = {I1, ..., I n}, the visible trajectory on each image is T i ; The set of iconic image tracks is defined as The ambiguous trajectory is defined as The unique trajectory is defined as T U =T A -T C ; Define the iconic image set as U; The scene consistency criterion is: As much as possible; the penalty synchronization criteria are: As small as possible; the length weighted criterion is: define the metric function as:

[0014]

[0015]

[0016] The two metric functions in the above formula are 0 at the beginning. As the image is traversed, the largest R(U) and R(U score ) as the initial set, and then search for this large value in the remaining images one by one until T A The proportion of the image exceeds the threshold of the proportion of the entire image; and the unique track of each image is separated from the aggregate track;

[0017] The algorithm for removing abnormal trajectories includes: after generating a unique trajectory, using the unique trajectory to determine whether the current image matching pair can be retained, and the screening rules are:

[0018]

[0019] In the above formula, uniqueTrack ij uniqueTrack i and uniqueTrack j The intersection of .

[0020] Furthermore, the different fusion modes include fusion of region selection and fusion of filtering strategies; the fusion of region selection includes:

[0021] Perform semantic segmentation on images at multiple moments to obtain feature points corresponding to matching relationships within the semantic information of the human body; fuse the semantic prior feature points at multiple moments into the reference moment;

[0022] The fusion of the filtering strategy includes removing false positive points in the fusion multi-time semantic prior internal matching relationship and removing redundant repeated points during fusion;

[0023] The process of removing false positive points, i.e., outliers, in the fused multi-time semantic prior region includes:

[0024] (1) First, randomly select the minimum number of points that satisfies the image dual-view model, where the H matrix requires a minimum of 4 pairs of points and the F matrix group requires a minimum of 8 pairs of points;

[0025] (2) Calculate the model M based on the selected points and choose which model to use based on the symmetric transfer error;

[0026] (3) Calculate the number of inliers among all points that meet the model; dynamically update the threshold of inliers screened by the RANSAC algorithm. The update formula is:

[0027]

[0028] Use the current model to measure the similarity of the points and add it to the total loss calculation as a loss term;

[0029] (4) Compare totalCost with the recorded minimum minCost and select the coordinates of the inner point position with the minimum loss;

[0030] (5) Re-use the recorded internal points to update the M model, and then iterate the above steps (1) to (4).

[0031] Furthermore, the ICP registration algorithm includes: 1) reconstructing the sparse 3D point cloud pose into a dense 3D point cloud to obtain dense point cloud information of the scene; manually selecting at least four dense point cloud coordinates in the dense point cloud, counted as P = p1, ..., p n , the coordinates obtained at this time are the coordinates in the self-calibration coordinate system;

[0032] 2) Use triangulation to restore the 3D world coordinates in the self-calibrated coordinate system;

[0033] 3) In the real scene, according to the specified coordinate system, select the real world point coordinates corresponding to the dense point cloud or triangulation point, and calculate them as

[0034] 4) Use the Euclidean transformation s, R, t to make: Use Iterative Closest Point (ICP) to solve.

[0035] (3a5) Closest point solution: Calculate the centroid positions of two sets of corresponding points Then calculate the de-centroided coordinates of each point:

[0036] q i =p i -p,q′ i =p′ i -p′

[0037] Define the loss function and convert it into an optimization problem to solve the rotation matrix:

[0038]

[0039] According to the R matrix solved by the above formula, the t matrix is ​​calculated. * =p-Rq′; restore the corresponding scale information:

[0040] p sub =p i -p i+1

[0041] p′ sub =p′ subi -p′ subi+1

[0042]

[0043] pass to perform coordinate system alignment.

[0044] Furthermore, the camera configuration registration algorithm includes:

[0045] (I) Set the origin of the world coordinate system of the sphere center coordinates of the camera layout position, defined as P ballcenter ;

[0046] (II) According to the camera deployment configuration, select a camera that is in the same plane and parallel to the scene ground, and solve the world coordinates of the camera optical center based on the camera external parameters of the self-calibrated coordinate system:

[0047] (III) performing plane fitting for all points on the same plane to determine the plane parameter equation and the normal vector perpendicular to the plane;

[0048] (IV) generating a rotation matrix according to the coordinates of the sphere center and the normal vector of the vertical plane;

[0049] (V) Restore the scale and translation matrix T, and output the final registration matrix for coordinate system registration.

[0050] Furthermore, the multi-camera calibration method in a large scene specifically includes the following steps:

[0051] Step 1, feature extraction and matching verification;

[0052] Step 2, ambiguous matched filtering;

[0053] Step 3: Semantic information fusion of weak textures

[0054] Step 4, construction of image connection graph: linking the matching relationships between multiple images together;

[0055] Step 5, initialization: two views with enough matching points are selected, the number of matching points is greater than 50 pairs; the triangulated angle between the two views is greater than 5°; the number of matching points in the H matrix is ​​less than 60% of the total number of points; the number of successfully triangulated matching pairs is greater than 50%;

[0056] Step 6, image registration: select the view with the largest number of visible tracks with the current reconstructed camera pose from all the view angles that have not been estimated as the new view angle; use the RANSAC-PnP algorithm to solve the new camera pose; after determining the camera pose, triangulate the current camera view angle and the other estimated camera pose view angles to reconstruct a new 3D world point;

[0057] Step 7, triangulation: restore the coordinates of the 3D points through known camera pose parameters and matching points;

[0058] Step 8, bundle adjustment optimization: reproject each world point onto the image plane, construct a reprojection error loss function, iteratively optimize the camera pose and the three-dimensional space point position of the feature points, and converge to the optical center of the camera;

[0059] Step 9: Coordinate system registration of metric reconstruction: After matching filtering and semantic information fusion, the camera pose is estimated by incremental motion recovery structure;

[0060] Step 10: After trajectory clustering, semantic information fusion, and pose registration, the camera pose is estimated.

[0061] Another object of the present invention is to provide an information data processing terminal, the information data processing terminal includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following steps: performing ambiguous matching filtering on multiple cameras in large scene calibration using a scene clustering algorithm and an abnormal trajectory removal algorithm;

[0062] Different fusion modes are used to enhance weak texture information of ambiguous matched filter images; the image after weak texture information enhancement is registered in the coordinate system using the ICP registration algorithm combined with the camera configuration registration algorithm.

[0063] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps: performing ambiguous matched filtering on multiple cameras in large scene calibration using a scene clustering algorithm and an abnormal trajectory removal algorithm;

[0064] The ambiguous matched filter image is subjected to weak texture information enhancement using different fusion modes; the image after weak texture information enhancement is subjected to coordinate system registration using the ICP registration algorithm combined with the camera configuration registration algorithm.

[0065] Another object of the present invention is to provide a camera, which performs the following steps: in large scene calibration, ambiguous matched filtering is performed using a scene clustering algorithm and an abnormal trajectory removal algorithm; weak texture information is enhanced for the ambiguous matched filtered image using different fusion modes; and coordinate system registration is performed for the image after weak texture information enhancement using an ICP registration algorithm combined with a camera configuration registration algorithm.

[0066] In combination with the above technical solutions and the technical problems solved, please analyze the advantages and positive effects of the technical solutions to be protected by the present invention from the following aspects:

[0067] First, in view of the technical problems existing in the above-mentioned prior art and the difficulty of solving the problems, the technical solutions to be protected by the present invention and the results and data during the research and development process are closely combined to analyze in detail and deeply how the technical solutions of the present invention solve the technical problems, and some creative technical effects brought about after solving the problems. The specific description is as follows:

[0068] Aiming at the problem of mismatching of similar textures in large scenes, the trajectory clustering algorithm studied in the present invention can effectively filter out mismatches to improve the accuracy of estimating camera pose. Specifically, when establishing trajectory information by breadth-first search, the trajectory information with a trajectory length less than 3 is filtered out to increase the proportion of longer trajectories. When clustering trajectories, a greedy algorithm is used to select the landmark image, that is, the image with the most visible trajectories in the current perspective is preferentially selected as the landmark perspective image, and the coefficient is set to 0.1 to ensure that the number of unique trajectories is greater than the number of ambiguous trajectories, and the trajectory length is used as a penalty item to reduce the proportion of mismatched points in long trajectories. Finally, the unique trajectory accounts for 0.9 of all trajectories as the end condition of the greedy algorithm. When filtering ambiguous matches, 0.15 is used as the threshold for mismatches, and mismatched relationship pairs with a unique trajectory ratio coefficient less than 0.15 are filtered out. Finally, the algorithm of the present invention is verified on an ice hockey rink. Of the 19 cameras surrounding the scene, 10 cameras are calibrated before the mismatch is eliminated, and there are some misestimated camera poses. After filtering out false matches, 19 cameras were fully calibrated and the accuracy of the camera’s estimated pose was greatly improved.

[0069] In order to solve the problem of insufficient weak texture constraint information in large scenes, the semantic information fusion algorithm studied in this paper can effectively enhance the constraint information, increase the number of constraint equations when estimating the camera pose, and thus improve the accuracy of estimating the camera pose. This algorithm fuses different numbers of moments for different scenes. In basketball scenes, the texture information is relatively sparse, and the accurate camera pose can be estimated using 5 moment data. In ice hockey scenes, 10 moment data are required to estimate a relatively accurate camera pose. When semantically fusing multiple moment data, redundant internal points need to be filtered out during the fusion process. Specifically, there are six cases: (1) There are no other points within the domain, which proves that there is no feature point coverage in the domain at this time, and this fusion point is added. (2) There is an original point within a small distance range of the fusion point, and the distance threshold is generally set to 1, which proves that there is already a feature point covering this domain. In this case, the fusion point is discarded. (3) There is an original point within a long distance range of the fusion point, and the coverage point is far away from the current area. In this case, the fusion point is added. (4) When the difference between the point to be fused and the nearest point and the second nearest point is large, and the distance to the nearest point is greater than 1, it proves that there is only one similar point nearby in the current domain, and the fusion point is added. (5) When the distance between the fusion point and the nearest neighbor and the second nearest neighbor is similar, and the distance threshold between the two is set to 2, the fusion point is not added. (6) When there are two points with similar distances near the fusion point, and the distance threshold exceeds 2, the fusion point is added. In the algorithm for removing fusion points, the neighborhood range r is set to 5, and the distance similarity ratio threshold is 0.8. At the same time, it is necessary to perform secondary RANSAC removal on the fused matching relationship point pairs to remove the existing outliers. In the process of removing outliers, the threshold is dynamically updated according to the number of current inliers to try to eliminate the influence of the threshold. The algorithm in this section has been verified in basketball and ice hockey scenes, and the mean square error of the absolute pose error and the relative pose error are reduced to less than 1. At the same time, the number of sparse three-dimensional points after fusing multi-time semantic information is effectively expanded. The number of 3D points in the basketball scene after fusion of 5 moments is expanded to 16568, and the number of 3D points in the basketball court data at a single moment is 9520. The number of 3D points in the ice hockey court after fusion of 10 moments is expanded to 19940, and the number of 3D points at a single moment is 6523. The above can effectively prove the effectiveness of the semantic information fusion algorithm in the present invention.

[0070] Regarding the coordinate system registration algorithm, the ICP-based registration algorithm is more applicable and can be applied to any scene, but it is necessary to manually establish the three-dimensional point pairs of the calibration coordinate system and the real-world coordinate system, and then solve the registration matrix. The camera configuration-based registration algorithm is suitable for the more common surround camera layout. The algorithm only needs to provide the radius of the camera layout surround point to automatically solve the registration matrix without manually establishing the corresponding relationship. The above two algorithms have been verified in basketball scenes, ice hockey scenes, and dance scenes, proving that no secondary errors will be introduced, and greatly enhancing the practical applicability of the present invention.

[0071] Second, considering the technical solution as a whole or from the perspective of the product, the technical effects and advantages of the technical solution to be protected by the present invention are described in detail as follows:

[0072] In terms of ambiguous matching filtering, the present invention designs a trajectory clustering algorithm, which establishes estimation information through breadth-first search, and then divides the trajectories in the scene into unique trajectories and ambiguous trajectories through scene consistency, penalty synchronization, and length weighting, aiming to obtain the unique trajectory ratio of each view image. By searching the neighbor nodes of each view image for unique trajectory ratio scoring metric, trajectory filtering is performed based on the metric to reduce the impact of ambiguous matching caused by similar textures on calibration accuracy.

[0073] In terms of weak texture information enhancement, the present invention first selects the characters in the scene as the significant scene area for matching relationship point fusion, so as to achieve the effect of enhancing constraint information. Secondly, when fusing matching relationship pairs, the fused external points and internal points are optimized. When filtering out the external points, the threshold is adaptively modified to reduce the impact of the threshold on system performance in order to address the problem of random sampling consistency (RANSAC) being too sensitive to the threshold. In terms of redundant and repeated internal point filtering, the internal points of different degrees are filtered through the field search algorithm to avoid the occurrence of too dense feature points in the same area. Through the above measures, the calibration accuracy of the calibration algorithm under weak texture is enhanced.

[0074] With regard to coordinate system registration, the present invention proposes an ICP-based registration algorithm and a camera configuration-based registration algorithm for different scenarios. The ICP-based registration algorithm obtains the world coordinates in the calibration coordinate system through triangulation, associates this world coordinate with the world coordinates in the real world coordinate system, and solves the transformation matrix. The registration algorithm combined with the camera configuration is mainly aimed at the surround layout. The transformation matrix is ​​obtained by plane modeling of the layout and solving the spherical center coordinates. The above two methods complement each other and enhance the ease of use of the coordinate system registration algorithm.

[0075] Simulation experiments show that the algorithm in the present invention has the ability to adapt to texture similarity and texture sparseness in large scenes, greatly improves the calibration accuracy, and can achieve high-precision multi-camera pose calibration in large scenes.

[0076] Third, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0077] The technical solution of the present invention fills the technical gap in the industry at home and abroad: At present, for the problems of similar texture mismatch and weak texture constraint information in large scenes, the common solutions are to place markers with rich texture information in the scene, use deep learning-based methods to improve the extraction and matching capabilities of feature points, or fuse feature point information at multiple times. The above three solutions have limitations: Solution 1: Texture-rich markers need to be specially customized and are not easy to transport and carry. Secondly, the use of such markers for calibration limits the use scenario of this algorithm. When the camera is not facing the ground, the texture marker cannot be placed. Solution 2: The feature points extracted by the deep learning method are closely related to the distribution of the deep learning data set. When the scene distribution is inconsistent with the data set distribution, there will be a large deviation in the position of the extracted feature points. Secondly, when the deep learning method is used to extract feature points, the number of feature points is related to the threshold setting, and the threshold coefficient cannot be determined well. Solution 3: The solution based on feature point fusion is easy to introduce erroneous feature points in the fusion process, which in turn produces erroneous matching, thereby placing greater requirements on the back-end optimization part, and the accuracy is difficult to guarantee. At the same time, none of the above three solutions proposes an effective solution for erroneous matching.

[0078] The algorithm proposed by the present invention fuses the matching relationships after filtering at multiple moments, which can not only ensure that the fused information eliminates erroneous matches, but also expand the number of constraint equations. At the same time, when fusing, the present invention uses the main body of the moving person in the scene as the fusion information, removes the constraints of the texture markers, and greatly expands the application scenarios of the present invention. At the same time, the present invention also contributes two coordinate system registration algorithms, so that the estimated camera pose can be quickly converted to the real-world coordinate system, further improving the ease of use of the present invention.

[0079] The technical solution of the present invention overcomes technical prejudice: the calibration algorithm based on significant markers is currently more reliable and has higher calibration accuracy. In the past, the accuracy of the camera pose calibrated based on kinematic recovery structure was not very trusted. The present invention verified and compared the accuracy of camera pose estimation in experiments, proving that the estimated camera pose of the algorithm of the present invention is not inferior to the calibration algorithm based on significant markers, and the estimated camera pose can also meet the needs of large scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 is a flow chart of a multi-camera calibration method in a large scene provided by an embodiment of the present invention;

[0081] Figure 2This is a flowchart of the overall method provided by the embodiment of the present invention, taking the multi-camera calibration method in a large scene based on trajectory clustering and semantic information fusion as an example. It includes feature extraction and matching, trajectory filtering, multi-time semantic information fusion, pose estimation, and coordinate system registration modules, and finally outputs the camera pose;

[0082] Figure 3 It is a principle diagram of a clustering algorithm provided by an embodiment of the present invention. The algorithm is mainly divided into trajectory establishment, trajectory clustering, and trajectory filtering;

[0083] Figure 4 It is a principle diagram of a semantic information fusion filtering algorithm provided by an embodiment of the present invention, and the algorithm is mainly divided into matching relationship external point filtering and matching relationship integration repeated internal point filtering;

[0084] Figure 5 is a schematic diagram of a coordinate system registration algorithm provided by an embodiment of the present invention, which is mainly divided into an ICP-based registration algorithm and a camera configuration prior information registration algorithm;

[0085] Figure 6 The embodiment of the present invention provides a camera pose estimation effect diagram by fusing multi-time semantic information. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0087] 1. Explanatory Examples In order to enable those skilled in the art to fully understand how to implement the present invention, this section provides an illustrative example that expands and describes the technical solution of the claims.

[0088] like Figure 1 As shown, the multi-camera calibration method in a large scene provided by the present invention includes:

[0089] S101, ambiguous matching filter: In multi-camera calibration of large scenes, repeated structures can cause ambiguous matches.

[0090] S102, semantic information fusion of weak textures: It is difficult to extract enough feature points from a single texture, which results in insufficient constraint equations to solve a better camera pose. When there are not enough constraint points in the scene, the semantic information generated during the movement of the characters can be used to fuse these semantic matching information together, thereby increasing the number of constraint points and ensuring a better camera pose is solved.

[0091] S103, coordinate system registration is performed through an ICP-based registration algorithm and a registration algorithm using camera configuration prior information: after matching filtering and semantic information fusion, the camera pose can be estimated through incremental motion recovery structure.

[0092] The present invention will now further describe the technical solution of the present invention by taking a multi-camera calibration method in a large scene based on trajectory clustering and semantic information fusion as an example.

[0093] The present invention provides a multi-camera calibration method in a large scene based on trajectory clustering and semantic information fusion, including the following steps: Step 1, feature extraction and matching verification: when a group of unordered images are input, feature points should be extracted for all images first. There are two requirements for feature points. The first is difference: feature points should show obvious features that are different from non-feature points. The second is repeatability: feature points corresponding to the same three-dimensional point should be repeatedly detected in different perspectives. The algorithm in this patent uses the SuperPoint algorithm to extract significant feature points in the scene and generate descriptors of the feature points. The descriptor can provide unique identity authentication for each feature point, and can ensure that feature points of the same spatial point at different perspectives have highly similar descriptors. The descriptors of different feature points are relatively different, and the general descriptor is a vector. This algorithm uses SuperGlue for feature matching. For details (see Figure 2 ) to realize information extraction from multi-time and multi-view images.

[0094] The matching relationship is calculated through the feature descriptors in the two images, and dual-view geometry verification is performed.

[0095] Step 2: Ambiguous matching filter: In multi-camera calibration of large scenes, repeated structures can cause ambiguous matching. Figure 2 ) in the trajectory filtering module. Detailed algorithm design (see Figure 3 ), which is mainly divided into two parts: establishing trajectory information and trajectory clustering.

[0096] (2a) Scene clustering algorithm:

[0097] A set of images is represented as I = {I1,…,I n}, the visible trajectory on each image is T i The set of iconic image trajectories is defined as The ambiguous trajectory is defined as The unique trajectory is defined as T U =T A -T C The set of iconic images is defined as U. The scene consistency criterion is: In order to ensure that the entire scene is covered as much as possible, the number of tracks covered in U should be as large as possible, that is, As many as possible. The penalty synchronization criterion is: avoid the number of ambiguous trajectories as small as possible, that is, As small as possible. The length-weighted criterion is: the probability of ambiguous trajectories appearing in long trajectories is high, and the trajectory length in long trajectories needs to be penalized, so the metric function is defined as:

[0098]

[0099]

[0100] The two metric functions in the above formula are 0 at the beginning, and then as the image is traversed, the largest R(U) and R(U score ) as the initial set, and then search for this large value in the remaining images one by one until T A The proportion exceeds the threshold of the proportion of the entire image. Finally, the unique track of each image is separated from the aggregate track.

[0101] (2b) Abnormal trajectory removal algorithm:

[0102] After the unique track is generated, it is used to determine whether the current image matching pair can be retained. The screening rules are as follows:

[0103]

[0104] In the above formula, uniqueTrack ij uniqueTrack i and uniqueTrack j The above two formulas respectively indicate that the similarity of the unique trajectories between the two images should be as large as possible, and the proportion of long trajectories in the unique trajectories should be as small as possible.

[0105] Step 3, semantic information fusion of weak textures: It is difficult to extract a sufficient number of feature points from a single texture, which results in the constraint equation being insufficient to solve a better camera pose. At the same time, note that it is assumed that the cameras around the venue have a fixed perspective and will not move for a long time, and the characters in the scene are always moving. With the above characteristics, it can be explained that when there are not enough constraint points in the scene, the semantic information generated during the movement of the characters can be used to fuse these semantic matching information together, thereby increasing the number of constraint points and ensuring that a better camera pose is solved. The overall process (see Figure 2 ) matching relationship fusion and filtering module. This includes filtering outlier algorithms and removing redundant points algorithms. The detailed process (see Figure 4 ) as shown.

[0106] (3a) Fusion of regional selection:

[0107] The lines in the scene can indeed detect many feature points, but for a large scene, most of the coverage area is smooth and without significant landmarks. The feature points that can be detected in these areas are extremely rare. Therefore, the feature points brought by understanding the scene from the overall distribution are insufficient. The characters in the scene bring rich texture information because they are wearing clothes with rich textures. At the same time, as the characters move, they can bring enough feature points to the entire scene, just making up for the disadvantage of sparse texture of the scene.

[0108] Therefore, human semantic information is given priority when selecting the fusion area, and semantic segmentation is performed on multi-time images to obtain feature points corresponding to matching relationships within human semantic information. Multi-time semantic prior feature points are fused to the reference time, thereby increasing the number of constraint points and constraint point areas, thereby improving the accuracy of calibration.

[0109] (3b) Fusion of filtering strategies: The fusion filtering strategy mainly includes two parts. The first part is to remove false positive points, i.e., outliers, after fusing the multi-time semantic prior regions. The second part is to remove redundant repeated points during fusion.

[0110] The removal of outliers is mainly done by RANSAC algorithm, and the present invention improves the situation that the traditional RANSAC is too sensitive to the threshold. The process of removing outliers by the improved RANSAC algorithm is as follows:

[0111] (3b1) First, randomly select the minimum number of points that satisfies the image dual-view model, where the H matrix requires a minimum of 4 pairs of points and the F matrix group requires a minimum of 8 pairs of points.

[0112] (3b2) Calculate the model M based on the selected points and choose which model to use based on the symmetric transfer error mentioned above.

[0113] (3b3) Calculate the number of inliers among all points that meet the model. To improve this, the traditional RANSAC algorithm uses the following strategy to select inliers:

[0114]

[0115]

[0116] The above strategy for filtering inliers is too sensitive to the threshold ε. If the threshold is too small, the algorithm will be unstable. If the threshold is too large, the algorithm will fail. Therefore, the above formula is improved to compensate for the negative impact of the threshold:

[0117]

[0118] The above formula is equivalent to using the current model to measure the similarity of the points and adding it as a loss term to the total loss calculation, which reduces the dependence on the threshold and is equivalent to dynamically updating the threshold of the data.

[0119] (3b4) Compare totalCost with the recorded minimum minCost and select the coordinates of the inner point position with the minimum loss.

[0120] (3b5) Re-use the recorded internal points to update the M model, and then iterate the above steps.

[0121] Step 4: Construction of the image connection graph: The image connection graph is an undirected graph in which vertices represent images and boundaries represent the common visible areas between two images. This step is the basis for estimating the camera pose later, and links the matching relationships between multiple images together. Figure 2 ) in the image connection graph.

[0122] Step 5, initialization: Incremental 3D reconstruction is crucial to the selection of the initial two frames. First, two perspectives with enough matching points must be selected. Generally, more than 50 pairs of matching points are selected. Secondly, the baseline between the two perspectives must be long enough, that is, the triangulation angle is greater than 5°. Generally, the mean or median angle of the triangulated angles of all matching points can be compared with the threshold. Then the number of matching points in the H matrix must be as small as possible. Here, the number of internal points is generally required to be less than 60% of the total number of points. Finally, the number of successfully triangulated matching pairs must be greater than 50%. (See Figure 2 ) in the initialization module.

[0123] Step 6, image registration: (see Figure 2 ) in the reconstruction new view module, select the view with the most visible Tracks with the current reconstructed camera pose as the new view in all view graphs that have not been estimated. Because of the 3D-2D correspondence based on Tracks, the RANSAC-PnP algorithm can be used to solve the new camera pose. In order to improve the robustness of the current newly registered view, the nonlinear optimization of the camera pose of the newly registered camera is performed to further adjust the camera pose. After determining the camera pose, the current camera view and the other estimated camera pose views are triangulated to reconstruct a new 3D world point.

[0124] Step 7, triangulation: that is, restore the coordinates of the three-dimensional points through known camera pose parameters and matching points.

[0125] Step 8, bundle adjustment optimization: Since an incorrect camera pose estimation will produce an incorrect 3D world point, an incorrect 3D world point will also produce an incorrect camera pose. That is, each world point is reprojected onto the image plane, and a reprojection error loss function is constructed in the process. While reducing the loss, the camera pose and the 3D space point position of the feature point are iteratively optimized, and finally converged to the optical center of the camera.

[0126] Step 9, coordinate system registration of metric reconstruction: After matching filtering and semantic information fusion, the camera pose can be estimated through incremental motion recovery structure. However, the current camera pose is in a self-calibrated coordinate system, and because the direction of the XYZ axis will be randomly selected during self-calibration, it is uncertain. Therefore, in order to ensure that the estimated camera pose is available, the current camera pose needs to be converted to an artificially defined world coordinate system. In essence, it is to solve a T transformation matrix, which includes three degrees of freedom rotation, three degrees of freedom translation, and scale. Therefore, it can be determined that this transformation has seven degrees of freedom, and at least four pairs of points are required to provide constraint equations to solve it. This transformation can be understood as a pseudo-rigid body motion, because the angle between the coordinate systems does not change during the transformation, but the scale of the corresponding coordinate system will change. Specific algorithm block diagram (see Figure 5 ), which mainly includes the ICP-based registration algorithm and the registration algorithm using the camera configuration prior information.

[0127] (9a) ICP registration algorithm:

[0128] (9a1) Perform dense 3D point cloud reconstruction on the sparse 3D point cloud pose to obtain the dense point cloud information of the scene. Manually select at least four dense point cloud coordinates in the dense point cloud, denoted as P = p1,…,p n , the coordinates obtained at this time are the coordinates in the self-calibration coordinate system.

[0129] (9a2) The dense 3D point cloud reconstruction in step 1 takes a long time and is not suitable for practical deployment. This step can also be used to triangulate the 3D world coordinates in the self-calibrated coordinate system. Because the camera internal and external parameters in the self-calibrated coordinate system are known, the pixel coordinates of two or more images can be selected to triangulate and restore the 3D world coordinates.

[0130] (9a3) In the real scene, according to the specified coordinate system, the coordinates of the real world points corresponding to the dense point cloud or triangulation are selected and calculated as

[0131] (9a4) Now we want to find a Euclidean transformation s, R, t such that: The iterative closest point (ICP) can be used to solve the problem.

[0132] (9a5)ICP solution process: Calculate the centroid positions of two sets of corresponding points Then calculate the de-centroided coordinates of each point:

[0133] q i =p i -p,q′ i =p′ i -p′

[0134] Define the loss function and convert it into an optimization problem to solve the rotation matrix:

[0135]

[0136] According to the R matrix solved by the above formula, the t matrix is ​​calculated, that is, t * =p-Rq′. The next step is to recover the scale information. We only need to calculate the distance between each point pair and recover the corresponding scale information through the distance ratio:

[0137] p sub =p i -p i+1

[0138] p′ sub =p′ subi -p′ subi+1

[0139]

[0140] Next, you can pass to perform coordinate system alignment.

[0141] (9b) Camera configuration prior information registration algorithm: Based on the multi-camera layout as prior information, the transformation matrix T is solved. In actual camera deployment, cameras are generally deployed around the site, so the algorithm will be explained according to this example. The mathematical abstract description of the site deployment is: all cameras are assumed to be on a spherical surface with a diameter of r at the center of the sphere. The specific algorithm flow is as follows:

[0142] (9b1) Set the origin of the world coordinate system of the sphere center coordinates of the camera layout position, defined as P ballcenter .

[0143] (9b2) According to the camera deployment configuration, select a camera that is in the same plane and parallel to the scene ground, and solve the world coordinates of the camera optical center based on the camera extrinsic parameters of the self-calibrated coordinate system:

[0144] (9b3) Perform plane fitting on all points in the same plane and determine the plane parameter equation and the normal vector perpendicular to the plane.

[0145] (9b4) Generate a rotation matrix based on the coordinates of the center of the sphere and the normal vector of the perpendicular plane.

[0146] (9b5) Restore the scale and translation matrix T, and output the final registration matrix for coordinate system registration.

[0147] Step 10: After trajectory clustering, semantic information fusion, and pose registration, the camera pose is finally estimated as shown in (see Figure 6 ). The figure shows the effect of fusion at different moments. It can be seen that after cluster filtering and fusion of multi-moment semantic information, the scene constraint information points continue to increase, and the estimated camera pose loss gradually disappears. This further proves the effectiveness of the present invention.

[0148] Figure 2 This is the main block diagram of the present invention, which is mainly divided into the steps of input image, feature extraction, feature matching, trajectory filtering, matching relationship fusion, image connection graph construction, incremental motion estimation, and coordinate system registration. The specific process is: first input multi-time multi-view images, then perform feature extraction on the images to obtain the feature points of each image. Then perform feature matching to obtain matching point pairs between images of each view. Trajectory filtering is to remove erroneous ambiguous matches. Matching relationship fusion is to fuse the matching relationships after multi-time filtering to obtain the fused matching relationship point pairs. Image connection graph construction is to convert the matching relationship point pairs into trajectory information. Incremental motion estimation is to perform pose estimation operations based on trajectory information to obtain the final camera pose. The coordinate system registration step is to convert the estimated camera pose into the real world coordinate system.

[0149] Figure 3 for Figure 2 The detailed expansion process of trajectory filtering in . It is mainly divided into three steps: breadth-first search to establish trajectory information, trajectory clustering, and trajectory filtering. First, the multi-time matching relationship point pairs are converted into trajectory information through the breadth-first search algorithm to facilitate further processing later. Then in the trajectory clustering stage, a set of iconic images are selected according to the scene consistency to represent the entire three-dimensional scene. Then, the trajectory information on the iconic images is intersected according to the penalty synchronization. According to the length weighting, the trajectory length is scored to determine the unique trajectory and ambiguous trajectory. Finally, the unique trajectory ratio is scored to filter out the wrong matching relationship.

[0150] Figure 4 for Figure 2The detailed expansion process of the matching relationship fusion in Figure 1. The fusion process is mainly divided into two parts, namely, filtering outliers and removing redundant inliers. The filtering outliers algorithm first estimates the relationship model between the two perspectives, and then selects which model to use based on the symmetric transfer error. When making a threshold judgment, it is not necessary to specify a specific threshold, but to adaptively select the threshold based on the number of current matching relationship pairs. Then iterate the above process to select all inlier matching relationship point pairs. The removing redundant inliers algorithm is a domain deduplication algorithm that avoids repeated addition of inliers during the fusion process, which leads to uneven distribution of feature points. The specific process is: when there is no original point in the fusion area, add a fusion point. When there is an original point in the fusion area, determine whether to add the current fusion point based on the distance threshold. When there are multiple original points in the fusion area, select the two original points closest to the fusion point to determine whether to add the fusion point.

[0151] Figure 5 for Figure 2 Detailed expansion diagram of coordinate system registration in the paper. Two coordinate system registration algorithms are introduced respectively. The registration algorithm based on ICP is to reconstruct the sparse point cloud into a dense point cloud or triangulate multiple views to obtain the world coordinates of the current pose estimation coordinate system, and then find the world coordinates in the real world coordinate system. Establish a matching relationship and then solve the transformation matrix. The camera configuration prior information registration algorithm mathematically models the circular deployment configuration camera model, obtains the spherical center coordinates and plane normal vectors of the multi-camera surround, and then restores the rotation and translation relationship. Finally, according to the scale information of the surround layout, the scale transformation between the current pose estimation coordinate system and the real world coordinate system is obtained.

[0152] Figure 6 For the final effect, Figure 6 The left side is the original camera pose estimated by the Colmap 3D reconstruction framework. It can be seen that there are problems with camera pose loss and inaccurate camera pose estimation. Figure 6 The right side shows the algorithm effect of the present invention after eliminating ambiguous matching and fusing semantic information through trajectory clustering. It can be seen that after fusing multiple moments of information, the problem of camera pose estimation loss no longer occurs, and at the same time, the three-dimensional sparse point cloud information in the scene continues to increase.

[0153] 2. Application Examples: In order to prove the creativity and technical value of the technical solution of the present invention, this section provides application examples of the technical solution of the claims on specific products or related technologies.

[0154] The method introduced in the present invention can be applied in basketball courts, ice hockey courts, football fields, and large dance scenes, and can effectively solve the problem of camera pose loss or inaccurate camera pose estimation caused by repeated and similar scene textures and sparse scene textures. Experimental verification has been carried out in basketball scenes and ice hockey scenes. The basketball scene is 28 meters long and 15 meters wide. 20 cameras are arranged around the venue and aimed at the center of the scene for shooting. The ice hockey scene is 61 meters long and 30 meters wide. 19 cameras are used for shooting, and the cameras are aimed at the half-court area. During the verification process, the traditional camera calibration algorithm has the problem of camera loss. The algorithm of the present invention does not have the problem of camera loss, and at the same time effectively improves the accuracy of camera pose calibration.

[0155] The present invention relates to the fields of pose estimation, semantic fusion and information filtering of multi-view images, and in particular to a multi-camera calibration method that uses trajectory clustering and semantic information fusion to enhance calibration accuracy, which can be used for high-precision camera pose calibration in large scenes.

[0156] 3. Evidence of the effects of the embodiments The embodiments of the present invention have achieved some positive effects during the development or use process, and indeed have great advantages over the prior art, which are described below in conjunction with data, charts, etc. from the simulation test process.

[0157] Simulation conditions:

[0158] The hardware platform is: HP-Z840 workstation, TITAN-X-12GB-GPU, 64GB RAM.

[0159] The software platform is: Python, C++11, OpenMP.

[0160] Simulation content and results:

[0161] The images of the experiment of the present invention are taken of basketball courts, ice hockey rinks and figure skating data. A total of 20 cameras were aimed at the center of the venue to shoot, and the image resolution was 3840*2160. After ambiguous matching filtering and multi-time semantic information fusion, the adaptability of the algorithm in the present invention to texture similarity and texture sparsity in large scenes is greatly improved. The accuracy of the calibration pose is qualitatively measured by the absolute trajectory error ATE and the relative pose error RPE. The root mean square error RMSE of the two indicators is reduced to less than 1, which proves that the present invention greatly improves the accuracy of calibration. It is proved that the algorithm can be migrated from the data source, and high-precision multi-camera pose calibration in large scenes can be achieved.

[0162] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0163] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A multi-camera calibration method in a large scene, characterized in that: The multi-camera calibration method in a large scene includes: performing ambiguous matching filtering on the multi-cameras in the large scene calibration using a scene clustering algorithm and an abnormal trajectory removal algorithm; The ambiguous matched filter image is enhanced with weak texture information using different fusion modes; the image after weak texture information enhancement is registered with the coordinate system using the ICP registration algorithm combined with the camera configuration registration algorithm; The scene clustering algorithm includes: a set of images is represented as I = {I1, ..., I n }, the visible trajectory on each image is T i ; The set of iconic image tracks is defined as The ambiguous trajectory is defined as The unique trajectory is defined as T U =T A -T C ; Define the iconic image set as U; The scene consistency criterion is: As much as possible; the penalty synchronization criteria are: As small as possible; the length weighted criterion is: define the metric function as: The two metric functions in the above formula are 0 at the beginning. As the image is traversed, the largest R(U) and R(U score ) as the initial set, and then search for this large value in the remaining images one by one until T A The proportion of the image exceeds the threshold of the proportion of the entire image; and the unique track of each image is separated from the aggregate track; The algorithm for removing abnormal trajectories includes: after generating a unique trajectory, using the unique trajectory to determine whether the current image matching pair can be retained, and the screening rules are: In the above formula, uniqueTrack ij uniqueTrack i and uniqueTrack j The intersection of; The different fusion modes include fusion of region selection and fusion of filtering strategies; the fusion of region selection includes: Perform semantic segmentation on images at multiple moments to obtain feature points corresponding to matching relationships within the semantic information of the human body; fuse the semantic prior feature points at multiple moments into the reference moment; The fusion of the filtering strategies includes removing false positive points in the fusion multi-time semantic prior internal matching relationship and removing redundant repeated points during fusion.

2. The multi-camera calibration method in a large scene as claimed in claim 1, characterized in that: The ambiguous matched filtering comprises: in multi-camera calibration of a large scene, using breadth-first search to establish estimation information, dividing the trajectories in the scene into unique trajectories and ambiguous trajectories through scene consistency, penalty synchronization, and length weighting, and obtaining the unique trajectory ratio of each view image; performing unique trajectory ratio scoring metric by searching neighbor nodes of each view image, and performing trajectory filtering according to the unique trajectory ratio scoring metric.

3. The multi-camera calibration method in a large scene as claimed in claim 1, characterized in that: The process of removing false positive points, i.e., outliers, in the fused multi-time semantic prior region includes: (1) First, randomly select the minimum number of points that satisfies the image dual-view model, where the H matrix requires a minimum of 4 pairs of points and the F matrix group requires a minimum of 8 pairs of points; (2) Calculate the model M based on the selected points and choose which model to use based on the symmetric transfer error; (3) Calculate the number of inliers among all points that meet the model; dynamically update the threshold of inliers screened by the RANSAC algorithm. The update formula is: Use the current model to measure the similarity of the points and add it to the total loss calculation as a loss term; (4) Compare totalCost with the recorded minimum minCost and select the coordinates of the inner point position with the minimum loss; (5) Re-use the recorded internal points to update the M model, and then iterate the above steps (1) to (4).

4. The multi-camera calibration method in a large scene as claimed in claim 1, characterized in that: The ICP registration algorithm includes: 1) reconstructing a dense 3D point cloud from a sparse 3D point cloud pose to obtain dense point cloud information of the scene; manually selecting at least four dense point cloud coordinates in the dense point cloud, which are counted as P = p1, ..., p n , the coordinates obtained at this time are the coordinates in the self-calibration coordinate system; 2) Use triangulation to restore the 3D world coordinates in the self-calibrated coordinate system; 3) In the real scene, according to the specified coordinate system, select the real world point coordinates corresponding to the dense point cloud or triangulation point, and calculate them as 4) Use the Euclidean transformation s, R, t to make: Solve using iterative closest point ICP; (3a5) Closest point solution: Calculate the centroid positions of two sets of corresponding points Then calculate the de-centroided coordinates of each point: q i =p i -p,q i ′=p i ′-p′ Define the loss function and convert it into an optimization problem to solve the rotation matrix: According to the R matrix solved by the above formula, the t matrix is ​​calculated. * =p-Rq′; restore the corresponding scale information: p sub =p i -p i+1 p s ′ ub =p s ′ ubi -p s ′ ubi+1 pass to perform coordinate system alignment.

5. The multi-camera calibration method in a large scene as claimed in claim 1, characterized in that: The camera configuration registration algorithm includes: (I) Set the origin of the world coordinate system of the sphere center coordinates of the camera layout position, defined as P ballcenter ; (II) According to the camera deployment configuration, select a camera that is in the same plane and parallel to the scene ground, and solve the world coordinates of the camera optical center based on the camera external parameters of the self-calibrated coordinate system: (III) performing plane fitting for all points on the same plane to determine the plane parameter equation and the normal vector perpendicular to the plane; (IV) generating a rotation matrix according to the coordinates of the sphere center and the normal vector of the vertical plane; (V) Restore the scale and translation matrix T, and output the final registration matrix for coordinate system registration.

6. The multi-camera calibration method in a large scene as claimed in claim 1, characterized in that: The multi-camera calibration method in a large scene specifically includes the following steps: Step 1, feature extraction and matching verification; Step 2, ambiguous matched filtering; Step 3: Semantic information fusion of weak textures Step 4, construction of image connection graph: linking the matching relationships between multiple images together; Step 5, initialization: two views with enough matching points are selected, the number of matching points is greater than 50 pairs; the triangulated angle between the two views is greater than 5°; the number of matching points in the H matrix is ​​less than 60% of the total number of points; the number of successfully triangulated matching pairs is greater than 50%; Step 6, image registration: select the view with the largest number of visible tracks with the current reconstructed camera pose from all the view angles that have not been estimated as the new view angle; use the RANSAC-PnP algorithm to solve the new camera pose; after determining the camera pose, triangulate the current camera view angle and the other estimated camera pose view angles to reconstruct a new 3D world point; Step 7, triangulation: restore the coordinates of the 3D points through known camera pose parameters and matching points; Step 8, bundle adjustment optimization: reproject each world point onto the image plane, construct a reprojection error loss function, iteratively optimize the camera pose and the three-dimensional space point position of the feature points, and converge to the optical center of the camera; Step 9: Coordinate system registration of metric reconstruction: After matching filtering and semantic information fusion, the camera pose is estimated by incremental motion recovery structure; Step 10: After trajectory clustering, semantic information fusion, and pose registration, the camera pose is estimated.

7. An information data processing terminal, characterized in that: The information data processing terminal includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the multi-camera calibration method in a large scene as described in any one of claims 1-6.

8. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the multi-camera calibration method in a large scene as described in any one of claims 1 to 6.

9. A camera, characterized in that: The camera executes the steps of the multi-camera calibration method in a large scene as described in any one of claims 1-6.

Images