Multi-camera image registration method, apparatus, device and storage medium

Abstract

This application discloses a multi-camera image registration method, apparatus, device, and storage medium, relating to the field of 3D reconstruction technology. The technical solution provided in this application includes acquiring a first image captured by a first camera and a second image captured by a second camera; registering the first image and the second image based on pre-calibrated first intrinsic parameters of the first camera, second intrinsic parameters of the second camera, and extrinsic parameters between the first and second cameras; wherein the cost function used in pre-calibrating the first intrinsic parameters, second intrinsic parameters, and extrinsic parameters includes a regularized cost function, which is constructed based on the infinity homography parameter between the first and second cameras. Through the above technical means, the calibration error is reduced to improve the accuracy of real-time image registration, thereby solving the problem that the real-time image registration error in the prior art cannot meet the requirements.

Description

Technical Field

[0001] This application relates to the field of image registration technology, and in particular to a multi-camera image registration method, apparatus, device and storage medium. Background Technology

[0002] Real-time image registration refers to the process of converting multiple images captured simultaneously by multiple cameras into the same pixel space. Currently, the more mature image registration techniques are based on block or feature point registration, but both of these methods involve too much computation and have low registration efficiency, making them unsuitable for real-time registration scenarios.

[0003] In existing technologies, images captured by various cameras are uniformly converted to the pixel space of a single camera by pre-calibrating the intrinsic and extrinsic parameter matrices of multiple cameras. Image registration methods based on calibrated intrinsic and extrinsic parameters eliminate the need for matching feature points or blocks, significantly reducing registration time and enabling real-time multi-camera image registration. However, while pre-calibrating the intrinsic and extrinsic parameters of multiple cameras involves acquiring checkerboard-patterned images from all cameras, the distance between the cameras and the checkerboard is relatively short during calibration, whereas in actual operation, the distance between the cameras and the subject is much greater. Therefore, using checkerboard-calibrated intrinsic and extrinsic parameters for real-time image registration introduces calibration errors, resulting in insufficient image registration accuracy and ultimately, image registration failure. Summary of the Invention

[0004] This application provides a multi-camera image registration method, apparatus, device, and storage medium, so that the calibrated intrinsic and extrinsic parameter models simultaneously satisfy the geometric constraints of the calibration board and the spatial constraints of the long-distance shooting scene, thereby reducing the calibration error and improving the accuracy of real-time image registration, thus solving the problem that the real-time image registration error in the prior art cannot meet the requirements.

[0005] Firstly, this application provides a multi-camera image registration method, including:

[0006] Acquire a first image captured by a first camera and a second image captured by a second camera;

[0007] Based on the pre-calibrated first intrinsic parameters of the first camera, the second intrinsic parameters of the second camera, and the extrinsic parameters between the first and second cameras, the first image and the second image are registered; wherein, the cost function used when pre-calibrating the first intrinsic parameters, the second intrinsic parameters, and the extrinsic parameters includes a regularized cost function, which is constructed based on the infinity homography parameter between the first camera and the second camera.

[0008] Secondly, this application provides a multi-camera image registration apparatus, comprising:

[0009] The image acquisition module is configured to acquire a first image captured by a first camera and a second image captured by a second camera.

[0010] The image registration module is configured to register the first image and the second image based on pre-calibrated first intrinsic parameters of the first camera, second intrinsic parameters of the second camera, and extrinsic parameters between the first camera and the second camera; wherein the cost function used when pre-calibrating the first intrinsic parameters, the second intrinsic parameters, and the extrinsic parameters includes a regularized cost function, which is constructed based on the infinity homography parameter between the first camera and the second camera.

[0011] Thirdly, this application provides a multi-camera image registration device, comprising:

[0012] One or more processors; a storage device storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the multi-camera image registration method as described in the first aspect.

[0013] Fourthly, this application provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the multi-camera image registration method as described in the first aspect.

[0014] This application constrains the solution space of the calibration cost function by adding a regularization term to the calibration cost function, preventing the intrinsic and extrinsic parameter model calculated based on the calibration cost function from overfitting close-range shooting scenarios. A regularized cost function is constructed using the infinity homography parameter calibrated at a long distance. This regularized cost function constrains the solution space of the calibration cost function to tend towards the infinity homography parameter, ensuring that the intrinsic and extrinsic parameter model calculated by the calibration cost function meets the physical model requirements of long-range shooting scenarios in actual operations, thus improving the generalization ability of the intrinsic and extrinsic parameter model. Therefore, the cost function used in the pre-calibration of multi-camera intrinsic and extrinsic parameters includes a calibration cost function constructed based on a calibration board and a regularization function constructed based on the infinity homography parameter. This ensures that the calibrated intrinsic and extrinsic parameter model simultaneously satisfies the geometric constraints of the calibration board and the spatial constraints of the long-range shooting scenario, reducing the calibration error of multi-camera intrinsic and extrinsic parameters, thereby improving the accuracy of real-time image registration and solving the problem of insufficient real-time image registration error in existing technologies. Attached Figure Description

[0015] Figure 1 This is a flowchart of a multi-camera image registration method provided in an embodiment of this application;

[0016] Figure 2 This is a schematic diagram of a multispectral camera provided in an embodiment of this application;

[0017] Figure 3 This is a flowchart of pre-calibration of multiple cameras provided in an embodiment of this application;

[0018] Figure 4 This is a flowchart of determining the homography parameter at infinity provided in the embodiments of this application;

[0019] Figure 5 This is a flowchart of constructing the calibration cost function provided in the embodiments of this application;

[0020] Figure 6 This is a flowchart of image registration based on intrinsic and extrinsic parameters provided in an embodiment of this application;

[0021] Figure 7 This is a schematic diagram of the structure of a multi-camera image registration device provided in an embodiment of this application;

[0022] Figure 8 This is a schematic diagram of the structure of a multi-camera image registration device provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. A process can be terminated when its operation is completed, but it may also have additional steps not included in the drawings. A process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0024] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0025] The multi-camera image registration method provided in this embodiment can be executed by a multi-camera image registration device. This device can be implemented through software and / or hardware, and can consist of two or more physical entities, or a single physical entity. For example, the multi-camera image registration device can be an intelligent device equipped with multiple cameras, such as an unmanned device, or it can be the processor of an intelligent device. Here, unmanned device refers to a device such as a drone that can automatically execute tasks based on preset parameters. The multiple cameras can be multispectral cameras or dual-light cameras, etc.

[0026] The multi-camera image registration device is equipped with at least one type of operating system. The multi-camera image registration device can install at least one application based on the operating system. This application can be a built-in application of the operating system or an application downloaded from a third-party device or server. In this embodiment, the multi-camera image registration device has at least one application capable of executing the multi-camera image registration method; therefore, the multi-camera image registration device itself can also be the application itself.

[0027] For ease of understanding, this embodiment uses a drone as the main body for performing the multi-camera image registration method as an example for description.

[0028] In one embodiment, the intrinsic and extrinsic parameters of the multiple cameras are pre-calibrated using the Zhang Zhengyou calibration algorithm. During high-altitude operations, the UAV controls multiple cameras to capture images of the survey area, and the images are registered based on the pre-calibrated intrinsic and extrinsic parameters of the multiple cameras. The Zhang Zhengyou calibration algorithm uses multiple cameras to capture calibration images of a checkerboard pattern at close range, and constructs a calibration cost function based on the checkerboard coordinates in multiple calibration images. By solving the calibration cost function, the intrinsic and extrinsic parameters of the multiple cameras that satisfy the geometric constraints of the checkerboard pattern and the spatial constraints of the close-range shooting scene are obtained. However, during high-altitude operations, the UAV captures images of the survey area from a distance. When registering the images captured during high-altitude operations, the pre-calibrated intrinsic and extrinsic parameters of the multiple cameras may introduce significant calibration errors because they do not meet the spatial constraints of the long-distance shooting scene during high-altitude operations. This results in the image registration error failing to meet the requirements, leading to image registration failure. If the surveyed area is farmland, pixel-level registration errors may lead to incorrect crop growth analysis. Therefore, when the image registration error does not meet the requirements, the image needs to be re-registered, which affects the efficiency of real-time image registration.

[0029] To address the aforementioned issues, this embodiment provides a multi-camera image registration method to reduce calibration errors and improve the accuracy of image registration.

[0030] Figure 1 A flowchart of a multi-camera image registration method provided in an embodiment of this application is given. (Reference) Figure 1The multi-camera image registration method specifically includes:

[0031] S110, Acquire the first image captured by the first camera and the second image captured by the second camera.

[0032] In this embodiment, the first camera is any one of the multiple cameras, and the second camera is any of the other cameras besides the first camera. This embodiment uses a multispectral camera as an example for description. Figure 2 This is a schematic diagram of a multispectral camera provided in an embodiment of this application. Figure 2 As shown, the multispectral camera includes a first spectral camera 11, a second spectral camera 12, a third spectral camera 13, and a fourth spectral camera 14. The first spectral camera can be used as the first camera, and the second, third, and fourth spectral cameras can be used as the second cameras.

[0033] When the drone operates at high altitude, it controls the first, second, third, and fourth spectral cameras to simultaneously capture images of the surveyed area, resulting in a first spectral image, a second spectral image, a third spectral image, and a fourth spectral image. The first spectral image is designated as the first image, while the second, third, and fourth spectral images are all designated as the second images.

[0034] S120. Based on the pre-calibrated first intrinsic parameter of the first camera, the second intrinsic parameter of the second camera, and the extrinsic parameter between the first camera and the second camera, the first image and the second image are registered; wherein, the cost function used when pre-calibrating the first intrinsic parameter, the second intrinsic parameter, and the extrinsic parameter includes a regularized cost function, which is constructed based on the infinity homography parameter between the first camera and the second camera.

[0035] In this embodiment, the first intrinsic parameter is the intrinsic parameter of the first spectroscopic camera, and the second intrinsic parameter includes the intrinsic parameters of the second, third, and fourth spectroscopic cameras. The extrinsic parameters include the extrinsic parameters between the first and second spectroscopic cameras, between the first and third spectroscopic cameras, and between the first and fourth spectroscopic cameras.

[0036] When registering the first spectral image and the second spectral image, the second spectral image is converted to the pixel space of the first spectral image by using pre-calibrated intrinsic parameters of the first spectral camera, intrinsic parameters of the second spectral camera, and the first extrinsic parameter, thereby achieving registration of the first and second spectral images. Similarly, the registration process between the first and third spectral images, as well as the registration process between the first and fourth spectral images, is the same.

[0037] The regularization cost function is a regularization term added to the calibration cost function, which is constructed based on the calibration board during intrinsic and extrinsic parameter calibration. For example, the calibration cost function in this embodiment can be constructed based on Zhang Zhengyou's calibration algorithm. Since the intrinsic and extrinsic parameter model calculated by the calibration cost function satisfies the geometric constraints of the calibration board and the spatial constraints of the close-up shooting scene, this embodiment proposes to constrain the solution space of the calibration cost function through a regularization term to prevent the intrinsic and extrinsic parameter model calculated by the calibration cost function from overfitting the close-up shooting scene.

[0038] In this embodiment, when a multispectral camera captures images of the surveyed area from a high altitude, the distance between the surveyed area and the multispectral camera exceeds the depth range that the camera can measure due to the short baseline of the multispectral camera. Therefore, the multispectral camera cannot determine the image depth, and the surveyed area within the image is considered as an infinity plane. The homography parameter calibrated based on the image containing the infinity plane, i.e., the image captured by the multispectral camera at a distance, is the infinity homography parameter in this embodiment. The homography parameter refers to the relative transformation matrix of the pixel spaces of the two cameras, and its expression is H = K1R. 2,1 K2 -1 H represents the homography parameter, K1 represents the intrinsic parameter of the first camera, K2 represents the intrinsic parameter of the second camera, and R represents the extrinsic rotation parameter between the first and second cameras. In summary, the intrinsic and extrinsic parameters of the multispectral camera in the infinity homography parameter satisfy the spatial constraints of long-distance shooting scenarios during high-altitude multispectral camera operations. Therefore, this embodiment constructs a regularization term based on the infinity homography parameter to construct the calibration cost function, constraining the solution space of the calibration cost function to tend towards the infinity homography parameter. This ensures that the intrinsic and extrinsic parameter models calculated by the calibration cost function meet the physical model requirements of long-distance shooting scenarios during actual operations, improving the generalization ability of the intrinsic and extrinsic parameter models.

[0039] It should be noted that since the infinity homography parameter only satisfies the spatial constraints of the long-distance shooting scene and not the geometric constraints of the calibration board, the calibration error of the infinity homography parameter is greater than the calibration error of the calibration cost function. Therefore, the infinity homography parameter is suitable as a regularization term of the calibration cost function to constrain its solution space, but not as the dominant cost function.

[0040] In one embodiment, Figure 3 This is a flowchart illustrating the pre-calibration of a multispectral camera provided in an embodiment of this application. For example... Figure 3 As shown, the pre-calibration steps for the multispectral camera specifically include S210-S270:

[0041] S210. Acquire a first calibration image and a second calibration image. The first calibration image is captured by a first camera at a preset height, and the second calibration image is captured by a second camera at a preset height.

[0042] In this embodiment, the preset altitude is greater than the ranging depth of the multispectral camera, typically set to 100 meters. For example, during the pre-calibration stage of the multispectral camera, a drone can fly at an altitude of 100 meters above the ground and simultaneously control the first, second, third, and fourth multispectral cameras to continuously capture images of the ground, corresponding to the first, second, third, and fourth spectral image sequences. The spectral image sequence includes multiple consecutively captured calibration spectral images; the first calibration spectral image is the first calibration image, and the second, third, and fourth calibration spectral images are all second calibration images.

[0043] S220. Determine the homography parameter at infinity based on the first calibration image and the second calibration image.

[0044] For example, the infinity homography parameter between the first and second spectroscopic cameras can be determined based on multiple frames of the first calibration spectroscopic image in the first spectroscopic image sequence and multiple frames of the second calibration spectroscopic image in the second spectroscopic image sequence. Similarly, the process for determining the infinity homography parameter between the first and third spectroscopic cameras, as well as between the first and fourth spectroscopic cameras, is the same as described above.

[0045] In one embodiment, Figure 4 This is a flowchart illustrating the determination of homography parameters at infinity, provided in an embodiment of this application. For example... Figure 4 As shown, the steps for determining the homography parameter at infinity specifically include S2201-S2203:

[0046] S2201. Extract the first feature point from the first calibration image and the second feature point from the second calibration image.

[0047] This embodiment is described using the determination of the homography parameter at infinity between the first and second spectral cameras as an example. First feature points and second feature points are extracted from the first and second calibration spectral images respectively based on the SIFI (Scale Invariant Feature Transform) feature extraction algorithm.

[0048] S2202. Match the first feature point with the second feature point to determine the feature matching pair.

[0049] For example, a first calibration spectral image and a second calibration spectral image captured simultaneously in a first spectral image sequence and a second spectral image sequence are determined. A first feature point in the first calibration spectral image is matched with a second feature point in the simultaneously captured second calibration spectral image to obtain multiple sets of feature matching pairs.

[0050] S2203. Determine the homography parameter at infinity based on feature matching pairs.

[0051] For example, the first calibration spectral image can be viewed as an image obtained by converting the simultaneously captured second calibration spectral image through infinity homography parameter transformation. A cost function for infinity homography parameter calibration can be constructed based on feature matching pairs. Solving for the optimal solution of this cost function yields the infinity homography parameter. It should be noted that the more feature matching pairs there are, the smaller the solution space of the cost function for calibrating the infinity homography parameter, and the smaller the error of the calculated infinity homography parameter. Therefore, multiple frames of calibration spectral images in a multispectral camera's spectral image sequence can provide sufficient feature matching pairs.

[0052] In another embodiment, feature points are extracted from the first and second calibration images using a deep learning model. These feature points are then matched to obtain feature matching pairs. A cost function for calibrating the homography parameter at infinity is constructed using these feature matching pairs, and the optimal solution of this cost function is calculated to obtain the homography parameter at infinity. Since the convolutional neural network in the deep learning model can extract more complex image features, the feature points extracted based on the deep learning model are more accurate. In another embodiment, the homography parameter at infinity is learned through a deep learning model to align the first and second calibration images. That is, the first and second calibration images are input into the deep learning model to obtain the homography parameter at infinity output by the deep learning model.

[0053] S230. Based on the homography parameter at infinity, a regularized cost function is constructed.

[0054] For example, the expression for the regularization cost function during the calibration of the first and second spectroscopic cameras is shown below:

[0055]

[0056] Among them, f λ (K1,R 2,1 K1 is the regularization cost function; λ is the regularization parameter, which determines the degree of constraint of the regularization term on the calibration cost function, and is generally taken as an empirical value of 0.1; K2 is the intrinsic parameter of the first spectroscopic camera, K1 is the intrinsic parameter of the second spectroscopic camera, and R is the regularization parameter of the second spectroscopic camera. 2,1 The external rotation parameter between the first and second spectroscopic cameras. is the infinity homography parameter between the first and second spectroscopic cameras.

[0057] Similarly, we can obtain the regularization cost functions for the calibration of the first and third spectral cameras, as well as the regularization cost functions for the calibration of the first and fourth spectral cameras.

[0058] S240. Acquire the third calibration image and the fourth calibration image. The third calibration image is obtained by the first camera taking a picture of the preset calibration board, and the fourth calibration image is obtained by the second camera taking a picture of the preset calibration board.

[0059] This embodiment describes the construction of a calibration cost function based on Zhang Zhengyou's calibration algorithm as an example. During the pre-calibration stage of the multispectral camera, a calibration board is set within the depth ranging range of the multispectral camera, and simultaneously, the first, second, third, and fourth spectral cameras are controlled to capture images of the checkerboard pattern, resulting in the fifth, sixth, seventh, and eighth calibration spectral images containing the checkerboard pattern. The fifth calibration spectral image is the third calibration image, and the sixth, seventh, and eighth calibration spectral images are the fourth calibration images.

[0060] S250. Based on the third and fourth calibration images, construct the calibration cost function.

[0061] For example, calibration cost functions for the first and second spectral cameras are constructed based on the fifth and sixth calibration spectral images. Similarly, the construction process for the calibration cost functions for the first and third spectral cameras, as well as the calibration cost functions for the first and fourth spectral cameras, is the same.

[0062] This embodiment uses the construction of calibration cost functions for the first and second spectral cameras as an example for description. Figure 5 This is a flowchart illustrating the construction of the calibration cost function provided in an embodiment of this application. For example... Figure 5 As shown, the steps for constructing the calibration cost function specifically include S2501-S2502:

[0063] S2501. Construct the mapping relationship between the calibration plate feature points in the third calibration image and the same calibration plate feature points in the fourth calibration image.

[0064] In this context, the feature points of the calibration board are the intersections of the black and white grids in the image. For example, feature point extraction from the fifth and sixth calibration spectral images yields the intersections of the black and white grids in both images. Matching these intersections with the corresponding positions on the checkerboard pattern in the two images identifies the exact locations of the black and white grid intersections within the checkerboard.

[0065] Based on Zhang Zhengyou's calibration algorithm, a mapping relationship was constructed between the black-and-white grid intersection points of the fifth calibrated spectral image after distortion correction and the black-and-white grid intersection points at the same positions in the checkerboard pattern of the sixth calibrated spectral image after distortion correction. This mapping relationship includes the intrinsic parameters of the first spectral camera, the intrinsic parameters of the second spectral camera, and the first extrinsic parameter.

[0066] S2502. Construct a calibration cost function based on the mapping relationship and the feature points of each calibration board in the third and fourth calibration images.

[0067] For example, the expressions for the calibration cost functions of the first and second spectroscopic cameras are as follows:

[0068]

[0069] Among them, f d (K1,R 2,1 K2) is the calibration cost function, M is the total number of black and white intersections on the chessboard, (u 1,j ,v 1,j (u) represents the pixel coordinates of the j-th black-and-white intersection point on the checkerboard pattern in the fifth calibrated spectral image after distortion correction. 2,j ,v 2,j (j) represents the pixel coordinates of the j-th black-and-white intersection point on the checkerboard pattern in the sixth calibrated spectral image after distortion correction. T represents the mapping relationship between the black-and-white grid intersection points of the fifth calibrated spectral image after distortion correction and the black-and-white grid intersection points at the same positions in the checkerboard pattern of the sixth calibrated spectral image after distortion correction. 2,1 The extrinsic translation parameter is the extrinsic parameter between the first and second spectroscopic cameras.

[0070] S260. Generate a cost function based on the calibration cost function and the regularization cost function.

[0071] For example, the pre-calibrated cost functions of the first and second spectroscopic cameras are expressed as follows:

[0072]

[0073] Among them, f 2,1 (K1,R 2,1 K2) is the cost function pre-calibrated for the first and second spectral cameras.

[0074] S270. Based on the preset algorithm, iteratively solve the optimal solution of the cost function to obtain the first intrinsic parameter, the second intrinsic parameter, and the extrinsic parameter.

[0075] For example, the optimal solution of the pre-calibrated cost function of the first and second spectroscopic cameras is calculated to obtain the intrinsic parameters of the first and second spectroscopic cameras, as well as the extrinsic rotation parameters between the first and second spectroscopic cameras. Similarly, the determination process for the intrinsic parameters of the third spectroscopic camera, the extrinsic rotation parameters of the first and third spectroscopic cameras, and the intrinsic parameters of the fourth spectroscopic camera, as well as the extrinsic rotation parameters of the first and fourth spectroscopic cameras, is the same as above.

[0076] In this embodiment, the optimal solution of the cost function can be calculated iteratively based on the Newton-Gauss algorithm, or the optimal solution of the cost function can be calculated using the Levenberg-Marquardt algorithm.

[0077] After the multispectral camera is pre-calibrated, the calibrated intrinsic and extrinsic parameters can be applied to real-time image registration in actual operations. This embodiment describes the registration process of the first spectral image and the second spectral image as an example. Figure 6 This is a flowchart illustrating image registration based on intrinsic and extrinsic parameters provided in an embodiment of this application. For example... Figure 6 As shown, the steps for image registration based on intrinsic and extrinsic parameters specifically include S1201-S1202:

[0078] S1201. Determine the calibration homography parameters between the first camera and the second camera based on the first internal reference parameter, the external reference parameter, and the second internal reference parameter.

[0079] For example, based on the expression of the homography parameter, the calibration homography parameter between the first and second spectroscopic cameras can be determined by the intrinsic parameter of the first spectroscopic camera, the intrinsic parameter of the second spectroscopic camera, and the extrinsic rotation parameter between the first and second spectroscopic cameras.

[0080] S1202. Based on the calibration homography parameter, the second image is converted to obtain the second image registered with the first image.

[0081] For example, by using the calibration homography parameters between the first and second spectral cameras, the second spectral image is converted to the pixel space of the first spectral image to achieve registration of the first and second spectral images.

[0082] In summary, the multi-camera image registration method provided in this application adds a regularization term to the calibration cost function to constrain its solution space, preventing the intrinsic and extrinsic parameter model calculated based on the calibration cost function from overfitting close-up shooting scenes. A regularized cost function is constructed using the infinity homography parameter calibrated at a long distance. This regularized cost function constrains the solution space of the calibration cost function to tend towards the infinity homography parameter, ensuring that the intrinsic and extrinsic parameter model calculated by the calibration cost function meets the physical model requirements of long-distance shooting scenes in actual operations, thus improving the generalization ability of the intrinsic and extrinsic parameter model. Therefore, the cost function used in the pre-calibration of multi-camera intrinsic and extrinsic parameters includes a calibration cost function constructed based on a calibration board and a regularization function constructed based on the infinity homography parameter. This ensures that the calibrated intrinsic and extrinsic parameter model simultaneously satisfies the geometric constraints of the calibration board and the spatial constraints of the long-distance shooting scene, reducing the calibration error of the multi-camera intrinsic and extrinsic parameters and thereby improving the accuracy of real-time image registration, thus solving the problem that the real-time image registration error in the prior art does not meet the requirements.

[0083] Based on the above embodiments, Figure 7 This is a schematic diagram of the structure of a multi-camera image registration device provided in an embodiment of this application. (Reference) Figure 7 The multi-camera image registration device provided in this embodiment specifically includes: an image acquisition module 31 and an image registration module 32.

[0084] The image acquisition module is configured to acquire a first image captured by a first camera and a second image captured by a second camera.

[0085] The image registration module is configured to register the first image and the second image based on the pre-calibrated first intrinsic parameter of the first camera, the second intrinsic parameter of the second camera, and the extrinsic parameter between the first camera and the second camera. The cost function used when pre-calibrating the first intrinsic parameter, the second intrinsic parameter, and the extrinsic parameter includes a regularized cost function, which is constructed based on the infinity homography parameter between the first camera and the second camera.

[0086] Based on the above embodiments, the multi-camera image registration device further includes: a first calibration image acquisition module, configured to acquire a first calibration image and a second calibration image, wherein the first calibration image is captured by a first camera at a preset height, and the second calibration image is captured by a second camera at a preset height; an infinity homography parameter determination module, configured to determine an infinity homography parameter based on the first calibration image and the second calibration image; and a regularization cost function construction module, configured to construct a regularization cost function based on the infinity homography parameter.

[0087] Based on the above embodiments, the infinity homography parameter determination module includes: a feature point extraction unit configured to extract a first feature point in a first calibration image and a second feature point in a second calibration image; a feature matching unit configured to match the first feature point with the second feature point to determine a feature matching pair; and an infinity homography parameter determination unit configured to determine the infinity homography parameter based on the feature matching pair.

[0088] Based on the above embodiments, the multi-camera image registration device further includes: a second calibration image acquisition module, configured to acquire a third calibration image and a fourth calibration image, wherein the third calibration image is obtained by the first camera capturing a preset calibration board, and the fourth calibration image is obtained by the second camera capturing a preset calibration board; a calibration cost function construction module, configured to construct a calibration cost function based on the third calibration image and the fourth calibration image; and a cost function generation module, configured to generate a cost function based on the calibration cost function and the regularization cost function.

[0089] Based on the above embodiments, the calibration cost function construction module includes: a mapping relationship construction unit, configured to construct a mapping relationship between calibration board feature points in the third calibration image and the same calibration board feature points in the fourth calibration image; and a calibration cost function construction unit, configured to construct a calibration cost function based on the mapping relationship, each calibration board feature point in the third calibration image and the fourth calibration image.

[0090] Based on the above embodiments, the multi-camera image registration device further includes: a cost function calculation module, which is configured to iteratively calculate the optimal solution of the cost function based on a preset algorithm to obtain a first intrinsic parameter, a second intrinsic parameter, and an extrinsic parameter.

[0091] Based on the above embodiments, the image registration module includes: a calibration homography parameter unit, configured to determine the calibration homography parameters between the first camera and the second camera based on the first intrinsic parameter, the extrinsic parameter, and the second intrinsic parameter; and an image registration unit, configured to convert the second image based on the calibration homography parameters to obtain a second image registered with the first image.

[0092] The multi-camera image registration apparatus provided in this application, by adding a regularization term to the calibration cost function, constrains the solution space of the calibration cost function and prevents the intrinsic and extrinsic parameter model calculated based on the calibration cost function from overfitting the close-range shooting scene. A regularized cost function is constructed using the infinity homography parameter calibrated at a long distance. This regularized cost function constrains the solution space of the calibration cost function to tend towards the infinity homography parameter, ensuring that the intrinsic and extrinsic parameter model calculated by the calibration cost function meets the physical model requirements of the long-range shooting scene in actual operation, thus improving the generalization ability of the intrinsic and extrinsic parameter model. Therefore, the cost function used in the pre-calibration of multi-camera intrinsic and extrinsic parameters includes a calibration cost function constructed based on a calibration board and a regularization function constructed based on the infinity homography parameter. This ensures that the calibrated intrinsic and extrinsic parameter model simultaneously satisfies the geometric constraints of the calibration board and the spatial constraints of the long-range shooting scene, reducing the calibration error of the multi-camera intrinsic and extrinsic parameters, thereby improving the accuracy of real-time image registration and solving the problem of insufficient real-time image registration error in the prior art.

[0093] The multi-camera image registration apparatus provided in this application embodiment can be used to execute the multi-camera image registration method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0094] Figure 8 This is a schematic diagram of the structure of a multi-camera image registration device provided in an embodiment of this application, with reference to... Figure 8 The multi-camera image registration device includes a processor 41, a memory 42, a communication device 43, an input device 44, and an output device 45. The number of processors 41 and the number of memories 42 in the multi-camera image registration device can be one or more. The processor 41, memory 42, communication device 43, input device 44, and output device 45 of the multi-camera image registration device can be connected via a bus or other means.

[0095] The memory 42, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the multi-camera image registration method in any embodiment of this application (e.g., image acquisition module 31 and image registration module 32 in a multi-camera image registration device). The memory 42 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory 42 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0096] The communication device 43 is used for data transmission.

[0097] The processor 41 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 42, thereby realizing the above-mentioned multi-camera image registration method.

[0098] Input device 44 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 45 may include display devices such as a display screen.

[0099] The multi-camera image registration device provided above can be used to execute the multi-camera image registration method provided in the above embodiments, and has corresponding functions and beneficial effects.

[0100] This application embodiment also provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to perform a multi-camera image registration method. The multi-camera image registration method includes: acquiring a first image captured by a first camera and a second image captured by a second camera; registering the first image and the second image based on pre-calibrated first intrinsic parameters of the first camera, second intrinsic parameters of the second camera, and extrinsic parameters between the first camera and the second camera; wherein the cost function used in pre-calibrating the first intrinsic parameters, second intrinsic parameters, and extrinsic parameters includes a regularized cost function, which is constructed based on the infinity homography parameter between the first camera and the second camera.

[0101] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROM, floppy disk, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. Storage medium may also include other types of memory or combinations thereof. Furthermore, storage medium may reside in a first computer system in which the program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media residing in different locations (e.g., in different computer systems connected via a network). Storage medium may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0102] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the multi-camera image registration method described above, but can also perform related operations in the multi-camera image registration method provided in any embodiment of this application.

[0103] The multi-camera image registration apparatus, storage medium, and multi-camera image registration device provided in the above embodiments can execute the multi-camera image registration method provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the multi-camera image registration method provided in any embodiment of this application.

[0104] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application. The scope of this application is determined by the scope of the claims.

Claims

1. A multi-camera image registration method, characterized by, include: Acquire a first image captured by a first camera and a second image captured by a second camera; the first camera and the second camera are spectral cameras; Based on pre-calibrated first intrinsic parameters of a first camera, second intrinsic parameters of a second camera, and extrinsic parameters between the first and second cameras, the first image and the second image are registered. The cost function used in pre-calibrating the first intrinsic parameters, second intrinsic parameters, and extrinsic parameters includes a calibration cost function and a regularization cost function. The calibration cost function is constructed based on the calibration of intrinsic and extrinsic parameters using a calibration board. The construction process of the regularization cost function includes: acquiring a first calibration image and a second calibration image, where the first calibration image is captured by the first camera at a preset height, and the second calibration image is captured by the second camera at the preset height, where the preset height is greater than the ranging depth of the spectral camera; determining the infinity homography parameter based on the first and second calibration images; and constructing the regularization cost function based on the infinity homography parameter.

2. The multi-camera image registration method of claim 1, wherein, Determining the homography parameter at infinity based on the first calibration image and the second calibration image includes: Extract the first feature point from the first calibration image and the second feature point from the second calibration image; The first feature point is matched with the second feature point to determine the feature matching pair; The infinity homography parameter is determined based on the feature matching pair.

3. The multi-camera image registration method according to any one of claims 1-2, wherein, Before registering the first image and the second image, the method further includes: A third calibration image and a fourth calibration image are acquired. The third calibration image is obtained by the first camera capturing a preset calibration board, and the fourth calibration image is obtained by the second camera capturing a preset calibration board. Based on the third and fourth calibration images, a calibration cost function is constructed; The cost function is generated based on the calibration cost function and the regularization cost function.

4. The multi-camera image registration method of claim 3, wherein, The step of constructing the calibration cost function based on the third calibration image and the fourth calibration image includes: Construct a mapping relationship between the calibration board feature points in the third calibration image and the same calibration board feature points in the fourth calibration image; The calibration cost function is constructed based on the mapping relationship, the feature points of each calibration board in the third calibration image and the fourth calibration image. 5.The multi-camera image registration method of claim 1, wherein, Before registering the first image and the second image, the method further includes: The optimal solution of the cost function is obtained by iteratively solving the preset algorithm, and the first intrinsic parameter, the second intrinsic parameter, and the extrinsic parameter are obtained. 6.The multi-camera image registration method of claim 1, wherein, The registration of the first image and the second image based on the pre-calibrated first intrinsic parameters of the first camera, the second intrinsic parameters of the second camera, and the extrinsic parameters between the first and second cameras includes: Based on the first intrinsic parameter, the extrinsic parameter, and the second intrinsic parameter, determine the calibration homography parameter between the first camera and the second camera; The second image is converted based on the calibration homography parameter to obtain a second image registered with the first image.

7. A multi-camera image registration apparatus, characterized by, include: The image acquisition module is configured to acquire a first image captured by a first camera and a second image captured by a second camera; the first camera and the second camera are spectral cameras. The image registration module is configured to register the first image and the second image based on pre-calibrated first intrinsic parameters of the first camera, second intrinsic parameters of the second camera, and extrinsic parameters between the first and second cameras. The cost function used in pre-calibrating the first intrinsic parameters, second intrinsic parameters, and extrinsic parameters includes a calibration cost function and a regularization cost function. The calibration cost function is constructed based on the calibration of the intrinsic and extrinsic parameters using a calibration board. The construction process of the regularization cost function includes: acquiring a first calibration image and a second calibration image, wherein the first calibration image is captured by the first camera at a preset height, and the second calibration image is captured by the second camera at the preset height, where the preset height is greater than the ranging depth of the spectral camera; determining the infinity homography parameter based on the first calibration image and the second calibration image; and constructing the regularization cost function based on the infinity homography parameter.

8. A multi-camera image registration device, characterized by include: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the multi-camera image registration method as described in any one of claims 1-6.

9. A storage medium containing computer-executable instructions, wherein: The computer-executable instructions, when executed by a computer processor, are used to perform the multi-camera image registration method as described in any one of claims 1-6.

Images