Calibration device, method and storage medium for multi-view heterogeneous cameras

Abstract

The application relates to a multi-view heterogeneous camera calibration device, method and storage medium, which are characterized in that a first calibration board, a second calibration board, an optical collimation platform, a plane mirror, a first six-axis adjusting platform, a second six-axis adjusting platform, an optical prism, an optical collimation mirror, a first locking structure and a second locking structure are arranged; the first calibration board is installed on the optical collimation platform through the first locking structure; the first six-axis adjusting platform is installed on the optical collimation platform, and the second calibration board is installed on the first six-axis adjusting platform through the second locking structure; the second six-axis adjusting platform is installed on the optical collimation platform, and the optical collimation mirror is installed on the second six-axis adjusting platform; the plane mirror is arranged in parallel with the first calibration board; the optical prism is arranged in parallel with the second calibration board; and the outgoing light of the optical collimation mirror passes through the optical prism to reach the first calibration board, so that the accuracy of the multi-view heterogeneous camera calibration result is effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular to a calibration device, method, storage medium, and assembly method of a calibration device for a multi-view heterogeneous camera. Background Technology

[0002] With the development of science and technology, multi-view cameras are widely used in video surveillance and eye-tracking technologies. A multi-view camera consists of multiple cameras rigidly connected in structure. Before using a multi-view camera, calibration is required to establish the connection between the cameras and obtain the extrinsic parameter transformation matrix between them. If the multiple cameras share a common viewpoint and the depth of field differences are small, then calibration can be easily performed using a single calibration board to obtain the extrinsic parameter transformation matrix between the cameras. However, if the cameras do not share a common viewpoint, this method is not applicable.

[0003] In existing technologies, a third-party camera is introduced as an intermediary, requiring that the third-party camera and the cameras in the multi-camera system that do not share a common viewpoint have a common viewpoint. The extrinsic parameter transformation matrix between the cameras in the multi-camera system that do not share a common viewpoint is then solved using the common viewpoint.

[0004] However, when the depth of field of cameras without a common viewpoint in a multi-camera system differs greatly, and the field of view coverage of a third-party camera differs too much from that of cameras without a common viewpoint in a multi-camera system, there is a problem of low calibration accuracy. In terms of related technologies, there is a problem of low calibration accuracy in multi-camera heterogeneous camera calibration, and no effective solution has been proposed yet. Summary of the Invention

[0005] Therefore, it is necessary to provide a calibration device, method, storage medium, and assembly method for a multi-view heterogeneous camera to address the aforementioned technical problems, so as to solve the problem of low accuracy of calibration results in the calibration of multi-view heterogeneous cameras in related technologies.

[0006] In the first aspect, embodiments of this application provide a calibration device for a multi-view heterogeneous camera, the device comprising a first calibration plate, a second calibration plate, an optical collimation platform, a plane mirror, a first six-axis adjustment platform, a second six-axis adjustment platform, an optical prism, an optical collimation mirror, a first locking structure, and a second locking structure;

[0007] The first calibration plate is mounted on the optical collimation platform via the first locking structure.

[0008] The first six-axis adjustment platform is mounted on the optical collimation platform, and the second calibration plate is mounted on the first six-axis adjustment platform through the second locking structure. The first six-axis adjustment platform is used to adjust the orientation of the second locking structure and the second calibration plate.

[0009] The second six-axis adjustment platform is mounted on the optical collimating platform, and the optical collimating lens is mounted on the second six-axis adjustment platform. The second six-axis adjustment platform is used to adjust the direction of the outgoing light from the optical collimating lens.

[0010] The plane mirror is arranged parallel to the first calibration plate; the optical prism is arranged parallel to the second calibration plate; the light emitted from the optical collimator passes through the optical prism and reaches the first calibration plate.

[0011] In some embodiments, the calibration device for the multi-view heterogeneous camera further includes a third calibration plate, which is mounted on the first six-axis adjustment platform via the second locking structure.

[0012] In some embodiments, the device further includes a locator;

[0013] The second locking structure includes an extension structure with a plurality of positioning holes for mounting the positioner.

[0014] In some embodiments, the locator is a crosshair laser pointer.

[0015] Secondly, this embodiment provides an assembly method for a calibration device of a multi-view heterogeneous camera, used to assemble the calibration device of the multi-view heterogeneous camera described in the first aspect above. The method includes:

[0016] Adjust the fixing direction of the first locking structure on the optical collimation platform so that the first calibration plate is perpendicular to the optical collimation platform;

[0017] Adjust the first six-axis adjustment platform so that the second calibration plate is perpendicular to the optical collimation platform;

[0018] Adjust the second six-axis adjustment platform and translate the plane mirror so that the outgoing light from the optical collimating lens is reflected by the plane mirror and the outgoing light path coincides with the incoming light path;

[0019] Adjust the first six-axis adjustment platform so that the light emitted from the optical collimating lens is projected onto the plane mirror through the optical prism. After being reflected by the plane mirror, the emitted light path coincides with the incident light path.

[0020] In some embodiments, the method further includes:

[0021] Install the locator on one of the positioning holes on the extension structure of the second locking structure, and translate the plane mirror so that the light emitted by the locator is projected onto the plane mirror;

[0022] Adjust the screws on the positioning hole so that the light path of the emitted light from the positioner coincides with the light path reflected by the plane mirror.

[0023] Thirdly, this embodiment provides a calibration method for a multi-view heterogeneous camera, wherein the multi-view heterogeneous camera includes at least two cameras without a common field of view. The calibration method for the multi-view heterogeneous camera is implemented based on the assembly method of the calibration device for the multi-view heterogeneous camera described in the second aspect above. The method includes:

[0024] Acquire calibration board images captured by cameras without a common field of view; wherein each camera without a common field of view captures a separate image of a corresponding planar calibration board, and calibration points are set on the calibration board;

[0025] Determine the image coordinates of the calibration points on the calibration plate image;

[0026] Based on the camera intrinsic parameters of each camera without a common field of view, the image coordinates of the calibration points on each calibration plate image, and the world coordinates of each calibration point, the initial extrinsic parameter transformation matrix between each pair of cameras without a common field of view is obtained.

[0027] In some embodiments, obtaining the initial extrinsic transformation matrix between each pair of cameras without a common field of view, based on the camera intrinsic parameters of each camera, the image coordinates of calibration points on each calibration board image, and the world coordinates of each calibration point, includes the following steps:

[0028] Establish a world coordinate system with one of the calibration points on the calibration plate as the origin, and obtain the world coordinates of each calibration point;

[0029] Based on the camera intrinsic parameters of the camera without a common field of view, the image coordinates of the calibration points on each calibration plate image, and the world coordinates of each calibration point, the initial extrinsic parameter transformation matrix between each pair of cameras without a common field of view is obtained.

[0030] In some embodiments, the method further includes:

[0031] A preset number of times is set, instructing the multi-view heterogeneous camera to perform the preset number of pose changes. Each time the multi-view heterogeneous camera performs a pose change, it acquires a calibration board image captured by the camera without a common field of view.

[0032] Based on the image coordinates of the calibration points on all the obtained calibration board images and the world coordinates of each calibration point, linear constraints are constructed.

[0033] The linear constraints are iteratively calculated using optimization methods to obtain the optimal extrinsic transformation matrix between each pair of cameras without a common field of view.

[0034] In some embodiments, the optimization method includes the Levenberg-Marquard algorithm or Newton's iteration method.

[0035] Fourthly, this embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the third aspect above.

[0036] The aforementioned calibration device, method, storage medium, and assembly method of the calibration device for a multi-view heterogeneous camera involve setting up a first calibration plate, a second calibration plate, an optical collimating platform, a plane mirror, a first six-axis adjustment platform, a second six-axis adjustment platform, an optical prism, an optical collimating lens, a first locking structure, and a second locking structure. The first calibration plate is mounted on the optical collimating platform via the first locking structure. The first six-axis adjustment platform is mounted on the optical collimating platform, and the second calibration plate is mounted on the first six-axis adjustment platform via the second locking structure. The first six-axis adjustment platform is used to adjust the direction of the second locking structure and the second calibration plate. The second six-axis adjustment platform is mounted on the optical collimating platform, and the optical collimating lens is mounted on the second six-axis adjustment platform. The second six-axis adjustment platform is used to adjust the direction of the emitted light from the optical collimating lens. The plane mirror is parallel to the first calibration plate. The optical prism is parallel to the second calibration plate, and the emitted light from the optical collimating lens passes through the optical prism to reach the first calibration plate. The aforementioned multi-view heterogeneous camera calibration device, by setting up independent first and second calibration plates, and then utilizing a plane mirror, a first six-axis adjustment platform, a second six-axis adjustment platform, an optical prism, an optical collimating lens, a first locking structure, and a second locking structure, achieves fine adjustment of the relative spatial position relationship between the first and second calibration plates. Furthermore, based on the size of the positioning holes of the optical collimating platform and the size of each checkerboard grid on the first and second calibration plates, the relative spatial coordinates of each calibration point on the first and second calibration plates can be accurately obtained. This allows for the accurate acquisition of the extrinsic parameter transformation matrix between pairs of cameras without a common viewpoint, effectively improving the accuracy of the multi-view heterogeneous camera calibration results. Attached Figure Description

[0037] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0038] Figure 1 This is a schematic diagram of the structure of a calibration device for a multi-view heterogeneous camera provided according to an embodiment of this application;

[0039] Figure 2 This is a schematic diagram of the structure of a planar calibration plate provided according to an embodiment of this application;

[0040] Figure 3 This is a schematic diagram of the structure of the second locking structure provided according to an embodiment of this application;

[0041] Figure 4 This is a flowchart of an assembly method for a calibration device for a multi-view heterogeneous camera provided according to an embodiment of this application;

[0042] Figure 5a This is a schematic diagram of the optical path effect of the collimating lens provided in the embodiments of this application. Figure 1 ;

[0043] Figure 5b This is a schematic diagram of the optical path effect of the collimating lens provided in the embodiments of this application. Figure 2 ;

[0044] Figure 6 This is a flowchart of a calibration method for a multi-view heterogeneous camera provided according to an embodiment of this application;

[0045] Figure 7 This is a schematic diagram of the structure of a computer device provided according to an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0047] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0048] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0049] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0050] Figure 1 A calibration device 10 for a multi-view heterogeneous camera is provided as an embodiment of this application. For example... Figure 1 As shown, the calibration device 10 for the multi-view heterogeneous camera includes a first calibration plate 101, a second calibration plate 102, an optical collimation platform 103, a plane mirror 104, a first six-axis adjustment platform 105, a second six-axis adjustment platform 106, an optical prism 107, an optical collimation mirror 108, a first locking structure 109, and a second locking structure 110.

[0051] The sizes of the first calibration plate 101 and the second calibration plate 102 can be adjusted according to actual needs to meet the field of view, object distance, and depth of field requirements of the camera to be calibrated. Figure 2As shown, in this embodiment, both the first calibration plate 101 and the second calibration plate 102 are planar calibration plates. Further, the planar calibration plate has a plurality of black and white checkerboard grids, where the cross intersections of the grids are calibration points. The size of the checkerboard grids can also be adjusted according to actual needs. The first calibration plate 101 is mounted on the optical collimation platform 103 via the first locking structure 109; the first six-axis adjustment platform 105 is mounted on the optical collimation platform 103, and the second calibration plate 102 is mounted on the first six-axis adjustment platform 105 via the second locking structure 110. The first six-axis adjustment platform 105 is used to adjust the orientation of the second locking structure 110 and the second calibration plate 102. Both the first locking structure 109 and the second locking structure 110 are commonly used installation devices in engineering, used for fixing objects in place. The second six-axis adjustment platform 106 is mounted on the optical collimation platform 103, and the optical collimating lens 108 is mounted on the second six-axis adjustment platform 106. The second six-axis adjustment platform 106 is used to adjust the direction of the emitted light from the optical collimating lens 108. Both the first six-axis adjustment platform 105 and the second six-axis adjustment platform 106 are precision orientation adjustment instruments, capable of directional adjustment in six dimensions, including the first translation direction (X), the second translation direction (Y), the third translation direction (Z), and the first rotation direction (pitch), the second rotation direction (yaw), and the third rotation direction (roll). The plane mirror 104 can be any type of mirror used to reflect light, and is used to reflect light projected onto the plane mirror 104. The plane mirror 104 is arranged parallel to the first calibration plate 101. The optical prism 107 is arranged parallel to the second calibration plate 102. Preferably, the optical prism 107 is placed horizontally on the second locking structure 110, so that the optical prism 107 is parallel to the second locking structure 110, that is, the optical prism 107 is parallel to the second calibration plate 102, effectively saving the additional installation cost of the optical prism 107. The optical collimating lens 108 has its own light source. The light emitted from the light source in the optical collimating lens 108 is emitted parallel to the light-emitting surface of the optical collimating lens 108, and the emitted light from the optical collimating lens 108 passes through the optical prism 107 to reach the first calibration plate 101.

[0052] Since the calibration device 10 of the multi-view heterogeneous camera includes a first calibration plate 101 and a second calibration plate 102, when calibrating cameras without a common viewpoint, each camera without a common viewpoint is required to correspond to a calibration plate. The calibration device 10 of the multi-view heterogeneous camera can calibrate two cameras without a common viewpoint at one time. If there are multiple cameras without a common viewpoint in the multi-view heterogeneous camera, the calibration device 10 of the multi-view heterogeneous camera can be used to select two cameras without a common viewpoint for calibration at one time. After multiple calibrations, the extrinsic parameter transformation matrix between each pair of cameras without a common viewpoint can be obtained.

[0053] In existing technologies, a third-party camera is introduced as an intermediary. This requires that the third-party camera and the cameras in the multi-camera system that do not share a common viewpoint have a common viewpoint. The extrinsic parameter transformation matrix between these cameras is then calculated using this common viewpoint. However, when the depth-of-field differences between the cameras without a common viewpoint are significant, and the field-of-view coverage differences between the third-party camera and these cameras are also large, the accuracy of the calibration results is low. Based on this, this application sets up independent first calibration plates 101 and second calibration plates 102, and then utilizes a plane mirror 104, a first six-axis adjustment platform 105, a second six-axis adjustment platform 106, an optical prism 107, an optical collimating lens 108, a first locking structure 109, and a second locking structure 110 to achieve fine adjustment of the relative positional relationship between the first calibration plates 101 and second calibration plates 102. Furthermore, based on the dimensions of the positioning holes on the optical collimating platform 103 and the dimensions of each checkerboard grid on the first and second calibration plates 101 and 102, the relative spatial coordinates of each calibration point on the first and second calibration plates 101 and 102 can be accurately obtained. By accurately obtaining the relative spatial coordinates of each calibration point on the first and second calibration plates 101 and 102, cameras without a common viewpoint can be accurately calibrated, thereby accurately obtaining the extrinsic parameter transformation matrix between pairs of cameras without a common viewpoint, effectively improving the accuracy of the calibration results for multi-view heterogeneous cameras.

[0054] See also Figure 1 In one embodiment, the calibration device 10 for the multi-view heterogeneous camera further includes a third calibration plate 111, which is mounted on the first six-axis adjustment 105 platform via a second locking structure 110.

[0055] Taking an existing helmet-mounted eye-tracking device as an example, this device includes one world camera and two eye-tracking cameras. The world camera faces the direction in which the user observes the world, while the eye-tracking cameras face the left and right eyeballs respectively. There is no common field of view between the world camera and the eye-tracking cameras. Although the two eye-tracking cameras are identical and have the same orientation, they also lack a common field of view. By setting a third calibration plate 111 on the calibration device 10 of the multi-view heterogeneous camera, which fits the structure of the helmet-mounted eye-tracking device, the calibration of one world camera and two eye-tracking cameras can be performed simultaneously, obtaining the extrinsic parameter transformation matrices between each pair of the world camera and the two eye-tracking cameras.

[0056] Furthermore, in one embodiment, the calibration device 10 for the multi-view heterogeneous camera of this application also includes a locator. See also Figure 3 The second locking structure 110 includes an extension structure 112, on which a plurality of positioning holes 113 are provided, which are used to install the positioner.

[0057] In multi-camera heterogeneous systems, cameras without a common field of view may have significant differences in field of view, object distance, and depth of field. Taking the aforementioned helmet-mounted eye-tracking device as an example, the eye-tracking camera has a small field of view, an object distance typically within 30mm, and a small depth of field (typically 19mm–32mm), while the world camera has a large field of view, an object distance typically beyond 50cm, and a large depth of field (typically 40cm–120cm). To meet the requirements for field of view, object distance, and depth of field of cameras without a common field of view, the first calibration plate 101 and the second calibration plate 102 are usually significantly different in size. Due to the large difference in structural dimensions between the first calibration plate 101 and the second calibration plate 102, it is difficult to visually determine their relative spatial relationship. However, due to the presence of the extension structure 112, magnifying the plane containing the second calibration plate 102 allows for a clearer observation of the positional relationship between the first calibration plate 101 and the second calibration plate 102. In addition, the spacing between each positioning hole 113 is the same. The positioning holes 113 can be used to install the locator. The locator can be installed on the positioning holes 113 using screws. Without the need for instrument measurement, the locator can be used to accurately locate the correspondence between the points on the plane where the second calibration plate 102 is located and the points on the plane where the first calibration plate 101 is located.

[0058] In one implementation, the locator is a crosshair laser pointer. Specifically, since the light emitted by the crosshair laser pointer will form a crosshair when it hits any plane, the position of the crosshair can be used to accurately determine the projection point of the light emitted by the crosshair laser pointer.

[0059] Figure 4 This is a flowchart illustrating an assembly method for a calibration device for a multi-view heterogeneous camera according to an embodiment of this application. It is used to assemble the aforementioned calibration device for a multi-view heterogeneous camera. Figure 4 The process includes the following steps:

[0060] Step S210: Adjust the fixing direction of the first locking structure on the optical collimation platform so that the first calibration plate is perpendicular to the optical collimation platform.

[0061] Specifically, during the process of fixing the first locking structure to the optical collimation platform, the measurement of the level instrument can be used to ensure that the first calibration plate is perpendicular to the optical collimation platform.

[0062] Step S220: Adjust the first six-axis adjustment platform so that the second calibration plate is perpendicular to the optical collimation platform.

[0063] Specifically, the first six-axis adjustment platform can be adjusted in six directions. By adjusting the first six-axis adjustment platform and combining it with the measurement of the level, the second calibration plate can be ensured to be perpendicular to the optical collimation platform.

[0064] Step S230: Adjust the second six-axis adjustment platform and translate the plane mirror so that the outgoing light from the optical collimating lens is reflected by the plane mirror and the outgoing light path coincides with the incoming light path.

[0065] Specifically, the plane mirror is set parallel to the first calibration plate, and the plane mirror can be translated along the plane of the first calibration plate. In one implementation, the plane mirror can be translated while remaining flush with the first calibration plate. The exit target surface of the optical collimating lens has a crosshair. The light emitted from the optical collimating lens passes through the crosshair and is projected onto the plane mirror. After reflection by the plane mirror, it is reflected back onto the exit target surface of the optical collimating lens. If the crosshair in the exit light path coincides with the crosshair in the incident light path, then the exit light path and the incident light path are guaranteed to coincide.

[0066] Figure 5a and 5b This is a schematic diagram of the optical path effect of the collimating lens provided in the embodiment of this application. Figure 5a This is the effect of the outgoing and incoming light paths not coinciding in an optical collimating lens. Figure 5b This demonstrates the effect of the outgoing and incoming light paths coinciding with those of the optical collimator. When the outgoing light from the optical collimator is reflected by the plane mirror, the outgoing and incoming light paths coincide. This ensures that the direction of the light emitted by the optical collimator is perpendicular to the plane mirror, meaning the direction of the light emitted by the optical collimator is perpendicular to the first calibration plate.

[0067] Step S240: Adjust the first six-axis adjustment platform so that the light emitted from the optical collimator is projected onto the plane mirror through the optical prism. After being reflected by the plane mirror, the emitted light path coincides with the incident light path.

[0068] Specifically, when the light emitted from the optical collimating lens is projected onto the plane mirror through the optical prism, after reflection by the plane mirror, the outgoing light path coincides with the incoming light path, thus ensuring that the direction of the light emitted by the optical collimating lens is simultaneously perpendicular to both the plane mirror and the optical prism. Since the optical prism is parallel to the second calibration plate, the direction of the light emitted by the optical collimating lens is simultaneously perpendicular to both the plane mirror and the second calibration plate, meaning the direction of the light emitted by the optical collimating lens is simultaneously perpendicular to both the first and second calibration plates. Because the direction of the light emitted by the optical collimating lens is simultaneously perpendicular to both the first and second calibration plates, according to the mathematical relationship between a line and a plane, it can be concluded that the first and second calibration plates are parallel.

[0069] Through the above steps S210 to S240, the first calibration plate and the first calibration plate can be precisely adjusted to be perpendicular to the optical collimation platform at the same time, and the first calibration plate and the second calibration plate are parallel to each other.

[0070] Furthermore, in one embodiment, the assembly method of the calibration device for the multi-view heterogeneous camera further includes the following steps:

[0071] Step S250: Install the locator on one of the positioning holes on the extension structure of the second locking structure, and translate the plane mirror so that the light emitted by the locator is projected onto the plane mirror.

[0072] Step S260: Adjust the screws on the positioning hole so that the light path of the emitted light from the positioner coincides with the light path reflected by the plane mirror.

[0073] Specifically, the direction of the locator's emitted light can be adjusted by rotating the screws on the positioning holes a certain number of times. When the light path of the locator's emitted light coincides with the light path reflected by the plane mirror, the direction of the locator's emitted light is perpendicular to the plane mirror. This confirms that the direction of the locator's emitted light is simultaneously perpendicular to both the first and second calibration plates. At this point, it is possible to accurately determine, from an optical perspective, which point on the first calibration plate corresponds to the current positioning hole. Based on this correspondence, the dimensions of each checkerboard grid on the first and second calibration plates, and the dimensions between each pair of positioning holes, the relative spatial coordinates of each calibration point on the first and second calibration plates can be precisely obtained. After obtaining the spatial coordinates of each calibration point on the first calibration plate, the coordinates of each calibration point on the second calibration plate can be obtained based on the relative spatial coordinates.

[0074] The assembly method of the calibration device of the multi-view heterogeneous camera is to adjust the calibration device 10 of the multi-view heterogeneous camera. Specifically, it can be adjusted by an automated robotic arm or by manual adjustment, and no specific limitation is made here.

[0075] In one embodiment, the multi-view heterogeneous camera includes at least two cameras without a common field of view. The calibration method for the multi-view heterogeneous camera is implemented based on the assembly method of the calibration device for the multi-view heterogeneous camera described above. Figure 6 As shown, the calibration method for multi-view heterogeneous cameras includes the following steps:

[0076] Step S310: Obtain the calibration board image captured by the camera without a common field of view; wherein, each camera without a common field of view captures a corresponding planar calibration board separately, and calibration points are set on the calibration board;

[0077] Step S320: Determine the image coordinates of the calibration points on the calibration plate image;

[0078] Step S330: Based on the camera intrinsic parameters of each camera without a common field of view, the image coordinates of the calibration points on each calibration plate image, and the world coordinates of each calibration point, obtain the initial extrinsic parameter transformation matrix between each pair of cameras without a common field of view.

[0079] Specifically, after assembling the calibration device for the multi-view heterogeneous camera using the assembly method described above, it can be ensured that the first calibration plate and the second calibration plate are simultaneously perpendicular to the optical collimation platform, and that the first calibration plate and the second calibration plate are parallel to each other. By placing cameras without a common field of view within the field of view of their respective calibration plates, images of the calibration plates captured by the cameras without a common field of view can be obtained. Preferably, in order to accurately obtain the coordinates of each calibration point on the calibration plate image and to use multiple sets of calibration points as references, the multi-view heterogeneous camera is adjusted before acquiring the calibration plate image captured by the cameras without a common field of view, ensuring that the calibration plate image captured by the cameras without a common field of view is clear and that the calibration plate occupies more than 70% of the calibration plate image. Camera intrinsic parameters refer to the transformation matrix that converts coordinates in the image coordinate system to coordinates in the camera coordinate system. The camera intrinsic parameters of each camera without a common field of view can be obtained using existing camera intrinsic parameter calibration methods. Assume that among the multi-view heterogeneous cameras, there are cameras A1 and B without a common field of view, and the camera intrinsic parameters obtained according to existing camera intrinsic parameter calibration methods are K. A1 and K B According to the camera's intrinsic parameters K A1 and K B By combining the image coordinates of the calibration points on the calibration plate image, the coordinates of each calibration point in the camera coordinate system of each camera without a common field of view can be obtained.

[0080] Camera extrinsic parameters refer to the transformation matrix that converts coordinates in the camera coordinate system to coordinates in the world coordinate system. Based on the coordinates of each calibration point in the camera coordinate system of each camera without a common field of view, and the world coordinates of each calibration point, the camera extrinsic parameters of each camera without a common field of view can be obtained. and Where A1 and B are coordinates in the camera coordinate system, and W is the coordinate in the world coordinate system. In one implementation, a world coordinate system is established at any point in space. The world coordinates of each calibration point on the first and second calibration plates can be obtained through the spacing between the positioning holes on the optical collimation platform, the size of the checkerboard pattern on the first and second calibration plates, and additional measurements. Preferably, since the relative spatial coordinates of each calibration point on the second calibration plate and each calibration point on the first calibration plate have been obtained through the assembly method of the multi-view heterogeneous camera calibration device described above, the world coordinates of each calibration point on the second calibration plate can be obtained based on the relative spatial coordinates after obtaining the world coordinates of each calibration point on the first calibration plate. This allows for the construction of a high-precision relative coordinate system between the first and second calibration plates. exist and Given a given set of conditions, the initial extrinsic parameter transformation matrix between cameras A1 and B can be obtained using mathematical methods.

[0081] Furthermore, if a multi-view heterogeneous camera system includes more than two cameras without a common viewpoint, such as A2 and B, then cameras A2 and B can be calibrated again through steps S310 to S330 above to obtain the initial extrinsic parameter transformation matrix between A2 and B. After obtaining and Afterwards, according to This allows us to obtain the initial extrinsic transformation matrix between A1 and A2. Therefore, regardless of how many cameras without a common viewpoint are included in a multi-view heterogeneous camera system, the initial extrinsic parameter transformation matrix between each pair of cameras without a common viewpoint can be accurately obtained through the above steps S310 to S330.

[0082] As one implementation method, taking the helmet-mounted eye-tracking device mentioned above as an example, the multi-view heterogeneous camera can make full use of the third calibration plate to simultaneously acquire the calibration plate images captured by the world camera and the two eye-tracking cameras. Thus, the helmet-mounted eye-tracking device can be calibrated once through the above steps S310 to S330 to obtain the initial extrinsic parameter transformation matrix between the world camera and the two eye-tracking cameras.

[0083] Furthermore, step S330 above, based on the camera intrinsic parameters of each camera without a common field of view, the image coordinates of the calibration points on each calibration board image, and the world coordinates of each calibration point, obtains the initial extrinsic parameter transformation matrix between each pair of cameras without a common field of view, including the following steps:

[0084] Step S331: Establish a world coordinate system with one of the calibration points on the calibration plate as the origin, and obtain the world coordinates of each calibration point;

[0085] Step S332: Based on the camera intrinsic parameters of the cameras without a common field of view, the image coordinates of the calibration points on each calibration plate image, and the world coordinates of each calibration point, obtain the initial extrinsic parameter transformation matrix between each pair of cameras without a common field of view.

[0086] Specifically, a world coordinate system is established with the calibration point on one of the calibration plates as the origin. Without additional measurements, the world coordinates of the calibration points on each calibration plate can be accurately obtained based on the spacing between the positioning holes on the optical collimation platform and the size of the checkerboard pattern on the first and second calibration plates. Because the world coordinates of each calibration plate on the first and second calibration plates are more accurate, the initial extrinsic parameter transformation matrix between pairs of cameras without a common field of view can be solved more accurately.

[0087] As one implementation method, the calibration method for multi-view heterogeneous cameras further includes the following steps:

[0088] Step S340: Set a preset number of times, instruct the multi-view heterogeneous camera to perform a preset number of pose changes, and acquire a calibration board image taken by a camera without a common field of view for each pose change performed by the multi-view heterogeneous camera.

[0089] Step S350: Based on the image coordinates of the calibration points on all the obtained calibration board images and the world coordinates of each calibration point, construct linear constraint conditions;

[0090] Step S360: The linear constraint conditions are iteratively calculated using the optimization method to obtain the optimal extrinsic transformation matrix between each pair of cameras without a common field of view.

[0091] Specifically, for scenarios where high precision is not required, the initial extrinsic transformation matrix obtained through steps S310 to S330 is sufficient to meet practical needs. To achieve higher precision, pose transformations of the multi-view heterogeneous cameras are required to acquire calibration board images in multiple poses. Based on the image coordinates of the calibration points in the calibration board images under multiple poses and the world coordinates of each calibration point, linear constraints are constructed. Optimization methods are used to iteratively calculate these linear constraints, thereby obtaining the optimal extrinsic transformation matrix between each pair of cameras without a common field of view. The number of pose transformations can be preset according to actual needs. For example, if the preset number is N, and the cameras without a common field of view are A1 and B, then the N sets of acquired calibration board image pairs are {Img}. A1 1mg B} i Let i represent the different pose index, with a value range of i∈[1,N]. Let Img be the image acquired by camera B in the i-th pose. B The image coordinates of the k-th marker point are D represents the first calibration board, k takes values ​​in the range [1, Q], and Q represents the number of marker points on the second calibration board. The image Img acquired by camera A1 in the i-th pose... A1 The image coordinates of the j-th marker point are: C1 represents the second calibration board, j takes values ​​in the range [1, M], and M represents the number of calibration points on the first calibration board; the following linear constraints are constructed:

[0092]

[0093] In the formula, λ1, λ2, and λ3 are penalty coefficients for each term, with initial fixed values, such as 17, 1, and 10 respectively. λ1 and λ2 balance the minimum resolution of cameras A1 and B, respectively; λ3 is a preset value based on experience. ΔP C1,j ΔP represents the absolute difference between the theoretical and actual 3D coordinates of the j-th calibration point on the second calibration plate, with an initial value of 0;D,k This represents the absolute difference between the theoretical and actual three-dimensional coordinates of the k-th calibration point on the first calibration plate, initially set to 0. for The coordinates of the back-projected image. for The back-projected image coordinates are given, and the projection pixel coordinates and back-projected pixel coordinates satisfy the camera intrinsic parameter transformation formula. An optimization method is used to iterate over the linear constraints. During iteration, the sum of the errors between the projection and back-projected pixel coordinates from the previous iteration is compared with a set threshold. If the iteration result is greater than the threshold, the penalty coefficients λ1 and λ3 are appropriately increased, generally with λ3 being increased first. Iteration stops only when the sum of the errors between the projection and back-projected pixel coordinates is less than the threshold. The final result of the iteration, params, is the optimal value, from which the optimal extrinsic parameter transformation matrix between cameras A1 and B without a common field of view can be obtained.

[0094] In one embodiment, the optimization method includes the Levenberg-Marquard algorithm or Newton's iteration method.

[0095] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores a set of preset configuration information. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the aforementioned calibration method for multi-view heterogeneous cameras.

[0096] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, memory, a network interface, a display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a calibration method for a multi-view heterogeneous camera. The display screen may be a liquid crystal display (LCD) or an e-ink display. The input devices may be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0097] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0098] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0099] Acquire calibration board images taken by cameras without a common field of view; wherein each camera without a common field of view takes a separate image of a corresponding planar calibration board, and calibration points are set on the calibration board;

[0100] Determine the image coordinates of the calibration points on the calibration plate image;

[0101] Based on the camera intrinsic parameters of each camera without a common field of view, the image coordinates of the calibration points on each calibration plate image, and the world coordinates of each calibration point, the initial extrinsic parameter transformation matrix between each pair of cameras without a common field of view is obtained.

[0102] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0103] Establish a world coordinate system with one of the calibration points on the calibration plate as the origin, and obtain the world coordinates of each calibration point;

[0104] Based on the camera intrinsic parameters of cameras without a common field of view, the image coordinates of calibration points on each calibration plate image, and the world coordinates of each calibration point, the initial extrinsic parameter transformation matrix between each pair of cameras without a common field of view is obtained.

[0105] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0106] Set a preset number of times to instruct the multi-view heterogeneous camera to perform a preset number of pose changes. Each time the multi-view heterogeneous camera performs a pose change, it acquires a calibration board image taken by a camera without a common field of view.

[0107] Based on the image coordinates of the calibration points on all the obtained calibration board images and the world coordinates of each calibration point, linear constraints are constructed.

[0108] The optimal extrinsic transformation matrix between each pair of cameras without a common field of view is obtained by iteratively calculating the linear constraints using optimization methods.

[0109] In one embodiment, the optimization method includes the Levenberg-Marquard algorithm or Newton's iteration method.

[0110] The aforementioned storage medium, after finely adjusting the relative spatial relationship between the first and second calibration boards and accurately acquiring the relative spatial coordinates of each positioning point on the first and second calibration boards, obtains the initial extrinsic parameter transformation matrix between each pair of cameras without a common field of view by utilizing the camera intrinsic parameters of the cameras without a common field of view, the image coordinates of the calibration points on each calibration board image, and the world coordinates of each calibration point. Because the relative spatial coordinates of each calibration point on the first and second calibration boards are accurate, the accuracy of the calibration results for multi-view heterogeneous cameras is effectively improved.

[0111] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0112] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0113] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0114] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A device for calibrating a multi-view heterogeneous camera, characterized in that, The device comprises a first calibration board, a second calibration board, an optical collimation platform, a plane mirror, a first six-axis adjustment platform, a second six-axis adjustment platform, an optical prism, an optical collimation mirror, a first locking structure and a second locking structure; The first calibration board is installed on the optical collimation platform through the first locking structure; The first six-axis adjustment platform is installed on the optical collimation platform, and the second calibration board is installed on the first six-axis adjustment platform through the second locking structure, and the first six-axis adjustment platform is used for adjusting the direction of the second locking structure and the second calibration board; The second six-axis adjustment platform is installed on the optical collimation platform, and the optical collimation mirror is installed on the second six-axis adjustment platform, and the second six-axis adjustment platform is used for adjusting the direction of the outgoing light of the optical collimation mirror; The plane mirror is arranged in parallel with the first calibration board, the optical prism is arranged in parallel with the second calibration board, and the outgoing light of the optical collimation mirror passes through the optical prism to reach the first calibration board; The device further comprises a positioner; The second locking structure comprises an extension structure, and a plurality of positioning holes are arranged on the extension structure, and the positioner is installed on one of the positioning holes.

2. The apparatus for calibrating multi-view heterogeneous cameras of claim 1, wherein, The calibration device of the multi-view heterogeneous camera further comprises a third calibration board, which is installed on the first six-axis adjustment platform through the second locking structure.

3. The apparatus for calibrating multi-view heterogeneous cameras of claim 1, wherein, The positioner is a cross star laser pen.

4. A method for assembling a multi-view heterogeneous camera calibration device, for assembling the multi-view heterogeneous camera calibration device according to any one of claims 1 to 3, characterized in that, The method comprises: adjusting the fixed direction of the first locking structure on the optical collimation platform so that the first calibration board is perpendicular to the optical collimation platform; adjusting the first six-axis adjustment platform so that the second calibration board is perpendicular to the optical collimation platform; adjusting the second six-axis adjustment platform and translating the plane mirror so that the outgoing light of the optical collimation mirror coincides with the incident light path after being reflected by the plane mirror; adjusting the first six-axis adjustment platform so that the outgoing light of the optical collimation mirror passes through the optical prism and is projected onto the plane mirror, and the outgoing light path coincides with the incident light path after being reflected by the plane mirror.

5. The assembling method of the multi-view heterogeneous camera calibration device according to claim 4, characterized in that, The method further comprises: installing the positioner on one of the positioning holes on the extension structure of the second locking structure, and translating the plane mirror so that the outgoing light of the positioner is projected onto the plane mirror; adjusting the screw on the positioning hole so that the outgoing light path of the positioner coincides with the light path reflected by the plane mirror.

6. A method for calibrating a multi-view heterogeneous camera comprising at least two cameras without common field of view, the method for calibrating a multi-view heterogeneous camera being implemented based on the method for assembling the device for calibrating a multi-view heterogeneous camera according to claim 4 or 5, characterized in that, The method comprises: acquiring calibration board images captured by cameras without a common field of view; wherein each of the cameras without a common field of view captures a corresponding plane calibration board, and the calibration board is provided with calibration points; determining the image coordinates of the calibration points on the calibration board images; obtaining initial extrinsic conversion matrices between the cameras without a common field of view according to the camera intrinsic parameters of each of the cameras without a common field of view, the image coordinates of the calibration points on each of the calibration board images, and the world coordinates of the calibration points.

7. The method of claim 6, wherein, The method comprises the following steps: establishing a world coordinate system with a calibration point on one of the calibration boards as the origin, and obtaining the world coordinates of each calibration point; obtaining the initial extrinsic transformation matrix between each pair of cameras without a common field of view according to the camera intrinsic parameters of each camera without a common field of view, the image coordinates of the calibration points on each calibration board image, and the world coordinates of each calibration point.

8. The method of claim 6, wherein, The method further comprises: setting a preset number of times, instructing the multi-view heterogeneous camera to change the pose for the preset number of times, and obtaining calibration board images captured by the cameras without a common field of view each time the pose is changed; constructing a linear constraint condition according to the image coordinates of the calibration points on all the calibration board images and the world coordinates of each calibration point; iteratively calculating the linear constraint condition by using an optimization method, thereby obtaining the optimal extrinsic transformation matrix between each pair of cameras without a common field of view.

9. The method of claim 8, wherein, The optimization method comprises a Levenberg-Marquard algorithm or a Newton iteration method.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method of any one of claims 6 to 9.

Images