A method and system for self-calibration of array cameras

By using an array camera self-calibration method, leveraging complementary information and feature matching techniques between array cameras, and combining the PIPO-SLAM algorithm to optimize camera extrinsic parameters, the robustness and accuracy issues of traditional camera calibration in complex scenarios are resolved. This achieves efficient and lightweight camera self-calibration, improving spatial resolution and signal-to-noise ratio.

CN118505822BActive Publication Date: 2024-11-19NAT UNIV OF DEFENSE TECH
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410955309.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-19
Estimated Expiration
2044-07-17

Smart Images

  • Figure CN118505822B_ABST
    Figure CN118505822B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and system for self-calibration of an array camera. The method includes data collection of the array camera to obtain pictures of all cameras in different positions at the same time; extraction of Superpoint features to cope with blurred edges and low signal-to-noise ratio of motion images to achieve feature extraction with strong robustness; LightGlue feature matching to meet fast matching in complex environments; calculation of internal and external parameters of the camera, using homography matrix or essential matrix to solve the internal parameters of the camera and relative external parameters; and further using the pose optimization algorithm in the PIPO-SLAM algorithm to optimize the final external parameters of the camera to further improve the calibration quality as needed. The present invention combines the information complementarity between array cameras with the most advanced technology to achieve rapid self-calibration of cameras during motion and in complex changing environments. The algorithm has the advantages of high efficiency, light weight, and strong robustness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a self-calibration method and system for an array camera. Background Art

[0002] With the increasing expansion of the application field of optical imaging technology, it is particularly important to develop a high-precision optical measurement method with wide universality. When the camera is shooting, it is often accompanied by objective factors such as optical distortion, pixel grid offset of the imaging sensor, and changes in the position relationship between the camera and the earth coordinate system, which makes the generated image not conform to the projection model. Therefore, camera calibration is required, which is the basis for achieving subsequent visual tasks. However, in complex scenes, due to unknown conditions such as environmental uncertainty and edge blur caused by motion, the traditional calibration process becomes very complicated and cumbersome, with poor robustness, which seriously limits the efficiency of the camera in performing high-precision tasks. Summary of the invention

[0003] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a method and system for self-calibration of an array camera are provided. The present invention aims to realize rapid self-calibration of the camera in a complex and changing environment during motion, so as to improve the spatial resolution of the camera and improve the signal-to-noise ratio.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A camera array self-calibration method comprises the following steps:

[0006] S1, collects images taken by all cameras in the array camera at the same time;

[0007] S2, extracts image features of all images and completes feature matching between multiple views of images taken by all cameras;

[0008] S3, using the matched image features to extract the homography matrix H or the essential matrix E and solve the camera extrinsic parameters, wherein the camera extrinsic parameters include the rotation matrix R and the translation vector t; wherein using the matched image features to extract the homography matrix H and solve the camera extrinsic parameters includes:

[0009] S3.1, for cameras other than the reference camera, use the matched image features to extract the homography matrix H:

[0010] ,

[0011] In the above formula, is the homography matrix H of the i-th camera other than the reference camera, ~ are the coefficients to be solved, is the reference camera image coordinate in the homogeneous coordinate system, It is The homogeneous coordinates of the camera images, , is the number of cameras other than the reference camera;

[0012] S3.2, write the homography matrix H as a column vector representation ,in ~ are the 1st to 3rd three-dimensional column vectors of the homography matrix H, thereby obtaining the camera intrinsic parameters With the homography matrix The relationship is shown as follows:

[0013] ,

[0014] ,

[0015] In the above formula, for The focal length on axis, for The focal length on axis, Main point Axis coordinates, is the tilt factor, Main point Axis coordinates;

[0016] S3.3, let the intermediate variable B be:

[0017] ,

[0018] In the above formula, ~ is the coefficient to be solved, since is a symmetric matrix, which is simplified and stacked into a matrix using the 6-dimensional eigenvector b. , the camera internal parameters With the homography matrix The relationship equation is written as the constraint equation shown below:

[0019] ,

[0020] In the above formula, and are the i-th and j-th three-dimensional column vectors of the homography matrix H, , is an intermediate variable, and there are: , ~ are three-dimensional column vectors The three elements of ~ are three-dimensional column vectors The three elements of

[0021] S3.4, according to the constraint equation, we can solve , and then solve the camera intrinsic parameters according to the following formula :

[0022] ,

[0023] in is the scale factor;

[0024] S3.5, combined with the obtained camera intrinsic parameters, the rotation matrix of the camera extrinsic parameters is solved based on the following formula: and the translation vector t:

[0025] ,

[0026] In the above formula, are the camera internal parameters, ~ They are the 1st to 3rd three-dimensional column vectors of the homography matrix H respectively.

[0027] Optionally, the array camera used in step S1 is arranged in a 3×3 plane. The camera numbers are set with the first camera in the upper corner as the reference camera, numbered 1, increasing from left to right, the upper right camera parameter numbered 3, and the numbers increasing from top to bottom, the lower left camera numbered 7, and the lower right camera numbered 9. The images collected are 9 images taken by 9 cameras.

[0028] Optionally, step S2 includes:

[0029] S2.1, the images captured by all cameras are subjected to image enhancement processing to increase the signal-to-noise ratio and contrast;

[0030] S2.2, extract Superpoint features from all processed images using a deep neural network;

[0031] S2.3, a camera designated in the array camera is set as a reference camera, an image captured by the reference camera is used as a reference image, and the Superpoint features in the reference image are matched with the Superpoint features extracted from the images captured by the remaining cameras using a preset feature matching method.

[0032] Optionally, the feature matching method preset in step S2.3 is the LightGlue algorithm.

[0033] Optionally, the feature matching threshold filter_threshold used by the LightGlue algorithm is set to 0.8 and the maximum number of feature points used is 2048.

[0034] Optionally, in step S3, extracting the essential matrix E using the matched image features and solving the camera extrinsic parameters includes:

[0035] For cameras other than the reference camera, the essential matrix E is extracted using the matched image features according to the following formula:

[0036] ,

[0037] In the above formula, They are the first The pixels on the normalized plane between the camera and the reference camera, , is the number of cameras other than the reference camera;

[0038] According to the obtained essential matrix E, the rotation matrix R and translation vector t of the camera extrinsic parameters are solved based on the following formula:

[0039] ,

[0040] In the above formula, is an orthogonal matrix, is a diagonal matrix, and is the matrix obtained by performing singular value decomposition of the essential matrix E, represents the antisymmetric matrix of the translation vector t, is the rotation matrix.

[0041] Optionally, after step S3, the step further includes optimizing the calculated camera extrinsic parameters using a posture optimization algorithm in the PIPO-SLAM algorithm to obtain final camera extrinsic parameters.

[0042] In addition, the present invention also provides an array camera self-calibration system, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the array camera self-calibration method.

[0043] In addition, the present invention also provides a computer-readable storage medium, in which a computer program or instruction is stored. The computer program or instruction is programmed or configured to execute the array camera self-calibration method through a processor.

[0044] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the array camera self-calibration method through a processor.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] 1. The present invention makes full use of the complementary information between array cameras based on the characteristic that multiple array cameras have complementary information, makes up for the disadvantage of missing information of a single camera, and effectively improves the spatial resolution and signal-to-noise ratio.

[0047] 2. The present invention combines feature extraction and matching technology to achieve stable feature extraction and matching in complex scenes such as motion, solving the problem of low robustness of self-calibration in the motion process of traditional methods.

[0048] 3. The present invention can further optimize the calculated camera extrinsic parameters using the posture optimization algorithm in the PIPO-SLAM algorithm to obtain the final camera extrinsic parameters, thereby realizing a fast and lightweight optimization process, greatly improving the self-calibration accuracy in complex scenes, and solving the problems of large computational complexity, long computational time, and large space occupation of traditional optimization algorithms. This makes the technology widely applicable and can realize fast self-calibration of cameras in complex and changing environments during motion, with the advantages of high efficiency, lightweight, and strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The figure is a basic flow chart of a method according to a first embodiment of the present invention.

[0050] Figure 2 Schematic diagram of the arrangement and numbering sequence of array cameras in Embodiment 1 of the present invention.

[0051] Figure 3 It is a schematic diagram of the principle of feature extraction and feature matching in the first embodiment of the present invention.

[0052] Figure 4 The figure is a complete flowchart of the method according to the first embodiment of the present invention.

[0053] Figure 5 A schematic diagram of visualizing the estimated camera pose in the first embodiment of the present invention.

[0054] Figure 6 It is an enlarged schematic diagram of the visualization of the estimated camera pose in the first embodiment of the present invention.

[0055] Figure 7 This is a camera reprojection root mean square error diagram for evaluating calibration accuracy in Example 1 of the present invention.

[0056] Figure 8 Schematic diagram of the basic flow of the method of Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0057] Embodiment 1:

[0058] The following uses a certain infrared array camera as an example to further explain the array camera self-calibration method and system of the present invention in detail. Figure 1 As shown, the array camera self-calibration method of this embodiment includes the following steps:

[0059] S1, collects images taken by all cameras in the array camera at the same time;

[0060] S2, extracts image features of all images and completes feature matching between multiple views of images taken by all cameras;

[0061] S3, using the matched image features to extract the homography matrix H or the essential matrix E and solve the camera extrinsic parameters, wherein the camera extrinsic parameters include the rotation matrix R and the translation vector t.

[0062] When the array camera needs to perform calibration tasks in motion, each camera only needs to collect photos at the same time, which can improve the utilization efficiency of spatial information and increase the signal-to-noise ratio through complementary information from different perspectives at the same time, thereby promoting high-precision camera calibration. Figure 2 As shown, the array camera used in step S1 of this embodiment is arranged in a 3×3 plane. The camera numbering is set with the first camera in the upper corner as the reference camera, numbered 1, increasing from left to right, the upper right camera parameter numbered 3, and the numbers increasing from top to bottom, the lower left camera numbered 7, and the lower right camera numbered 9. The acquired images are 9 images taken by 9 cameras.

[0063] Based on the feature that multiple array cameras have complementary information, this embodiment specifically adopts a method of extracting and matching Superpoint features to improve the accuracy of extracting the homography matrix H. Specifically, step S2 in this embodiment includes:

[0064] S2.1, the images captured by all cameras are subjected to image enhancement processing to increase the signal-to-noise ratio and contrast;

[0065] S2.2, extract Superpoint features from all processed images using a deep neural network;

[0066] S2.3, a camera designated in the array camera is set as a reference camera, an image captured by the reference camera is used as a reference image, and the Superpoint features in the reference image are matched with the Superpoint features extracted from the images captured by the remaining cameras using a preset feature matching method.

[0067] like Figure 3As shown, the preset feature matching method in step S2.3 of this embodiment is the LightGlue algorithm, and the feature matching threshold filter_threshold used by the LightGlue algorithm is set to 0.8, and the maximum number of feature points used is 2048. Figure 2 As shown, the Superpoint feature points in the reference image are matched with the Superpoint features extracted from the other 8 images in turn, and the matching method uses the LightGlue algorithm to achieve the matching of Superpoint features. Figure 3 Only the matching process between two matching images using the LightGlue algorithm is shown.

[0068] In this embodiment, step S3 includes using the matched image features to extract the homography matrix H and solve the camera extrinsic parameters:

[0069] S3.1, for cameras other than the reference camera, use the matched image features to extract the homography matrix H:

[0070] ,

[0071] In the above formula, is the homography matrix H of the i-th camera other than the reference camera, ~ are the coefficients to be solved, is the reference camera image coordinate in the homogeneous coordinate system, It is The homogeneous coordinates of the camera images, , is the number of cameras other than the reference camera;

[0072] S3.2, write the homography matrix H as a column vector representation ,in ~ are the 1st to 3rd three-dimensional column vectors of the homography matrix H, thereby obtaining the camera intrinsic parameters With the homography matrix The relationship is shown as follows:

[0073] ,

[0074] ,

[0075] In the above formula, for The focal length on axis, for The focal length on axis, Main point Axis coordinates, is the tilt factor, Main point Axis coordinates;

[0076] S3.3, let the intermediate variable B be:

[0077] ,

[0078] In the above formula, ~ is the coefficient to be solved, since is a symmetric matrix, which is simplified and stacked into a matrix using the 6-dimensional eigenvector b. , the camera internal parameters With the homography matrix The relationship equation is written as the constraint equation shown below:

[0079] ,

[0080] In the above formula, and are the i-th and j-th three-dimensional column vectors of the homography matrix H, , is an intermediate variable, and there are: , ~ are three-dimensional column vectors The three elements of ~ are three-dimensional column vectors The three elements of

[0081] S3.4, according to the constraint equation, we can solve , and then solve the camera intrinsic parameters according to the following formula :

[0082] ,

[0083] in is the scale factor;

[0084] S3.5, combined with the obtained camera intrinsic parameters, the rotation matrix of the camera extrinsic parameters is solved based on the following formula: and the translation vector t:

[0085] ,

[0086] In the above formula, are the camera internal parameters, ~ They are the 1st to 3rd three-dimensional column vectors of the homography matrix H respectively.

[0087] like Figure 4 As shown, as an optional implementation, after step S3, it also includes optimizing the camera extrinsic parameters obtained by the solution using the posture optimization algorithm (PA algorithm) in the PIPO-SLAM algorithm to obtain the final camera extrinsic parameters, including: the file format required by the PA algorithm, such as the setting; Python running of the PA algorithm; importing and evaluating the optimization results of the PA algorithm. The setting of the file format required by the PA algorithm includes: normalizing the solved translation vector, and writing the translation vector of a camera in each line in the order from left to right and from top to bottom, where the reference camera translation vector is 0, and the data is output in a text file; writing the translation matrix of a camera in each line in the order from left to right and from top to bottom, and the reference camera rotation matrix is ​​the unit matrix, and the data is output in a text file; for the setting of the BAL.txt file required by the PA algorithm, the number of cameras, the number of matched feature points, and the total number of lines in the file are written in the first line, separated by spaces, and then each line is written in the order of the matching point number from small to large, the camera number, the matching point number, and the coordinates of the matching point in the camera. Among them, the Python operation of the PA algorithm includes using the pure pose estimation optimization part of the PIPO-SLAM algorithm originally used in visual inertial navigation for camera parameter optimization, and implementing PA optimization in C++ language for the rotation matrix and translation vector optimization part; using Python to call the PA algorithm written in C++ language to further optimize the external parameter matrix.

[0088] In order to verify the array camera self-calibration method of this embodiment, this embodiment uses infrared cameras that are difficult to calibrate with high precision as experimental objects. This embodiment uses data sets collected by 9 long-wave infrared cameras to perform self-calibration tests on photos in different scenes and under different lighting environments. In the selection of evaluation indicators, some studies believe that the root mean square error RMSE is more suitable as an indicator of camera calibration model performance than the mean absolute error (MAE) represented by the mean error MRE, especially when the error distribution is expected to follow a Gaussian trend. Therefore, this embodiment uses the root mean square reprojection error RMSE as an evaluation indicator, which means that the degree to which the control points and reprojection points detected in the image deviate from the pattern is given, and the formula is as follows:

[0089] ,

[0090] in, is the coordinate of the i-th feature point, is the estimated coordinate of the ith feature point, is the number of matched feature points. The experiment of this embodiment only uses 9 photos at the same time for each test, and a total of 20 tests are performed. The test results are as follows Figure 5 and Figure 6 shown. Figure 5 and Figure 6 The position relationship between the cameras after the array camera is calibrated is shown. Since the translation matrix between the array cameras is normalized, the arrangement gap is very small, such as Figure 5 As shown in the figure, the cameras are almost overlapped, which is not convenient for observation and judgment. Therefore, in order to observe the camera posture more carefully, the spacing between the cameras is increased by 100 times in this embodiment. The result is as follows Figure 6 As shown in the figure, the posture relationship between the cameras can be intuitively observed, and the changes of the cameras during the movement can also be intuitively felt. Figure 7 The average root mean square reprojection error (RMSE) of cameras other than the reference camera is shown. From the results, the average root mean square reprojection error (RMSE) is less than 0.3 pixels, and the calibration error between cameras does not change much, indicating that the calibration stability is relatively good and the complementary information between cameras is well integrated. The minimum error is only 0.242 pixels, which is even better than many traditional calibration results based on infrared calibration plates.

[0091] In order to further verify the performance of the method of this embodiment on complex tasks, it is compared with the more classic existing methods for comparison. The existing methods for comparison include Zhang Zhengyou calibration method, self-calibration method based on classic feature descriptor and self-calibration method based on bundle adjustment (BA), wherein: Zhang Zhengyou calibration method can be found in the literature: Zhang Z. A flexible new technique for camera calibration [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22. Self-calibration method based on classic feature descriptor can be found in the literature: Li Canlin, Ding Hanqing. Self-Calibration of Camera Based on SIFT Matching and Modulus Constraint [J]. Science Technology and Engineering, 2014, 14 (19). Self-calibration method based on BA optimization can be found in the literature: Shan Jie. Brief History and Summary of Bundle Adjustment [J]. Journal of Wuhan University (Information Science Edition), 2018, 43 (12): 1797-1810. The final experimental results are shown in Table 1.

[0092] Table 1 Comparison results between the method of this embodiment and the classical method

[0093]

[0094] In Table 1, letters A, B, C, and D represent Zhang Zhengyou's calibration method, the self-calibration method based on the classic feature descriptor, the self-calibration method based on BA optimization, and the self-calibration method of this embodiment, respectively. In order to more reasonably compare the differences between different feature descriptors and different optimization methods, the LightGlue algorithm is used in all places where feature matching methods are required, and the common nonlinear optimization method Levenberg-Marquardt (LM) is used as the optimization algorithm of Zhang Zhengyou's calibration method. From the experimental results in Table 1, it can be seen that the self-calibration method used in this embodiment has the smallest RMSE error and the best calibration result. Compared with Zhang Zhengyou's calibration method directly used for infrared camera calibration, the RMSE error of this embodiment is reduced by 0.5 pixels, and it is also far superior to the self-calibration methods based on classic feature extraction and BA optimization. From the perspective of time efficiency, the method using the Superpoint feature descriptor is much faster than the SIFT feature descriptor. The feature extraction speed is nearly three times that of the SIFT method, but the parameter solution speed is faster using the SIFT feature than the Superpoint feature. This is because the SIFT feature extracts fewer feature points, so the SIFT feature brings fewer calculations into the parameter estimation. However, more feature points are conducive to high-precision calibration and reduce calibration errors. Considering the calibration accuracy and parameter solution time, the SIFT feature is not as good as the Superpoint operator in the high-precision calibration of infrared cameras. From the perspective of parameter optimization methods, the PA optimization method is four times faster than the BA optimization method, and from the calibration results, the PA optimization algorithm can be closer to the real parameters, and the error generated is smaller, which is better than the BA optimization algorithm in terms of quality and efficiency.

[0095] In summary, based on the problems of low calibration accuracy and complex calibration process of cameras in complex scenes, this embodiment uses complementary information between array cameras to enhance the signal-to-noise ratio and spatial information of images; at the same time, it combines the feature descriptor Superpoint with good generalization performance and the LightGlue feature matching method with fast feature matching to achieve efficient feature extraction and matching links, and also achieves good results in motion scenes, providing a large amount of data for high-precision parameter solution; the solved initial internal and external parameters are optimized quickly and accurately through the PA optimization algorithm, further improving the calibration quality; the array camera self-calibration method and system adopted in this embodiment have the advantages of high efficiency, lightweight, strong robustness, etc.

[0096] In addition, this embodiment also provides an array camera self-calibration system, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the array camera self-calibration method.

[0097] In addition, this embodiment further provides a computer-readable storage medium, in which a computer program or instruction is stored. The computer program or instruction is programmed or configured to execute the array camera self-calibration method through a processor.

[0098] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the array camera self-calibration method through a processor.

[0099] Embodiment 2:

[0100] This embodiment is basically the same as the first embodiment, and the main difference is: Figure 8 As shown, the first embodiment uses the matched image features to extract the homography matrix H and solve the camera extrinsic parameters, while the present embodiment uses the matched image features to extract the essential matrix E and solve the camera extrinsic parameters. In step S3, using the matched image features to extract the essential matrix E and solve the camera extrinsic parameters includes:

[0101] For cameras other than the reference camera, the essential matrix E is extracted using the matched image features according to the following formula:

[0102] ,

[0103] In the above formula, They are the first The pixels on the normalized plane between the camera and the reference camera, , is the number of cameras other than the reference camera;

[0104] According to the obtained essential matrix E, the rotation matrix R and translation vector t of the camera external parameters are solved based on the following formula:

[0105] ,

[0106] In the above formula, is an orthogonal matrix, is a diagonal matrix, and is the matrix obtained by performing singular value decomposition on the essential matrix E, represents the antisymmetric matrix of the translation vector t, is the rotation matrix. Through the above method, the camera external parameters can also be solved.

[0107] In addition, this embodiment also provides an array camera self-calibration system, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the array camera self-calibration method.

[0108] In addition, this embodiment further provides a computer-readable storage medium, in which a computer program or instruction is stored. The computer program or instruction is programmed or configured to execute the array camera self-calibration method through a processor.

[0109] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the array camera self-calibration method through a processor.

[0110] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0111] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A camera array self-calibration method for realizing fast camera self-calibration during motion, characterized in that: The steps include: S1, collects images taken by all cameras in the array camera at the same time; S2, extracting image features of all images, and completing feature matching between multiple views of images taken by all cameras, including: S2.1, performing image enhancement processing on images taken by all cameras to increase signal-to-noise ratio and contrast; S2.2, extracting Superpoint features from all processed images using a deep neural network; S2.3, setting a camera specified in the array camera as a reference camera, taking the image taken by the reference camera as a reference image, and matching the Superpoint features in the reference image with the Superpoint features extracted from the images taken by the remaining cameras using a preset feature matching method; the preset feature matching method in step S2.3 is the LightGlue algorithm; S3, using the matched image features to extract the homography matrix H or the essential matrix E and solve the camera extrinsic parameters, wherein the camera extrinsic parameters include the rotation matrix R and the translation vector t; wherein using the matched image features to extract the homography matrix H and solve the camera extrinsic parameters includes: S3.1, for cameras other than the reference camera, use the matched image features to extract the homography matrix H: , In the above formula, is the homography matrix H of the i-th camera other than the reference camera, ~ are the coefficients to be solved, is the reference camera image coordinate in the homogeneous coordinate system, It is The homogeneous coordinates of the camera images, , is the number of cameras other than the reference camera; S3.2, write the homography matrix H as a column vector representation ,in ~ are the 1st to 3rd three-dimensional column vectors of the homography matrix H, thereby obtaining the camera intrinsic parameters With the homography matrix The relationship is shown as follows: , , In the above formula, for The focal length on axis, for The focal length on axis, Main point Axis coordinates, is the tilt factor, Main point Axis coordinates; S3.3, let the intermediate variable B be: , In the above formula, ~ is the coefficient to be solved, since is a symmetric matrix, which is simplified and stacked into a matrix using the 6-dimensional eigenvector b. , the camera internal parameters With the homography matrix The relationship equation is written as the constraint equation shown below: , In the above formula, and are the i-th and j-th three-dimensional column vectors of the homography matrix H, , is an intermediate variable, and there are: , ~ are three-dimensional column vectors The three elements of ~ are three-dimensional column vectors The three elements of S3.4, according to the constraint equation, we can solve , and then solve the camera intrinsic parameters according to the following formula : , in is the scale factor; S3.5, combined with the obtained camera intrinsic parameters, the rotation matrix of the camera extrinsic parameters is solved based on the following formula: and the translation vector t: , In the above formula, are the camera internal parameters, ~ They are the 1st to 3rd three-dimensional column vectors of the homography matrix H respectively; After step S3, the method further includes optimizing the calculated camera extrinsic parameters using a posture optimization algorithm in the PIPO-SLAM algorithm to obtain the final camera extrinsic parameters.

2. The array camera self-calibration method according to claim 1, characterized in that: The array camera used in step S1 is arranged in a 3×3 plane. The camera numbering is set with the first camera in the upper corner as the reference camera, numbered 1, increasing from left to right, the upper right camera parameter numbered 3, and the numbers increasing from top to bottom, the lower left camera numbered 7, and the lower right camera numbered 9. The acquired images are 9 images taken by 9 cameras.

3. The array camera self-calibration method according to claim 1, characterized in that: The feature matching threshold filter_threshold used by the LightGlue algorithm is set to 0.8, and the maximum number of feature points used is 2048.

4. The array camera self-calibration method according to claim 1, characterized in that: In step S3, using the matched image features to extract the essential matrix E and solve the camera extrinsic parameters includes: For cameras other than the reference camera, the essential matrix E is extracted using the matched image features according to the following formula: , In the above formula, They are the first The pixels on the normalized plane between the camera and the reference camera, , is the number of cameras other than the reference camera; According to the obtained essential matrix E, the rotation matrix R and translation vector t of the camera extrinsic parameters are solved based on the following formula: , In the above formula, is an orthogonal matrix, is a diagonal matrix, and is the matrix obtained by performing singular value decomposition of the essential matrix E, represents the antisymmetric matrix of the translation vector t, is the rotation matrix.

5. An array camera self-calibration system, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the array camera self-calibration method according to any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the array camera self-calibration method according to any one of claims 1 to 4 through a processor.

7. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the array camera self-calibration method according to any one of claims 1 to 4 through a processor.

Citation Information

Patent Citations

  • Camera array optimization calibration method for non-overlapping visual field

    CN107133987A

  • A structured light three-dimensional reconstruction method and device based on array cameras

    CN107860337A

  • External parameter calibration method for 3D camera group

    CN108288294A

  • Monocular SLAM method combined with SuperPoint network feature extraction

    CN114266823A

  • Synchronous calibration method and system for camera array of oversized power equipment

    CN114299153A