An automatic calibration method of a multi-view camera, an electronic device and a storage medium

Abstract

The application is suitable for the field of camera technology, and provides an automatic calibration method of a multi-view camera, an electronic device and a computer readable storage medium. The method comprises the following steps: controlling a mechanical arm provided with the multi-view camera to move according to a first track, and controlling the multi-view camera to collect images of a first calibration board; determining internal parameters of each camera according to a plurality of first calibration board images; controlling the mechanical arm to move according to a second track, and controlling the multi-view camera to collect images of a second calibration board; determining first coordinate system conversion parameters between a camera coordinate system of the camera and a coordinate system of the second calibration board according to a plurality of second calibration board images and the internal parameters of the camera; and determining second coordinate system conversion parameters between the camera coordinate systems of each camera according to the first coordinate system conversion parameters. Through the above method, automatic calibration of the multi-view camera with a very small common view area can be realized.

Description

Technical Field

[0001] This application belongs to the field of camera technology, and in particular relates to an automatic calibration method for multi-view cameras, electronic devices, and computer-readable storage media. Background Technology

[0002] The robotic arm-based automated calibration platform is a calibration system that combines precise motion control of the robotic arm with image acquisition from multiple cameras. This approach uses a robotic arm to move a calibration board along a preset path to different positions, while multiple cameras capture images of the calibration board at each position, thus collecting a dataset for calibration. However, this approach relies on the robotic arm for only the data acquisition portion; subsequent manual calibration or post-processing of the calibration results is still required, necessitating human intervention in the calibration process and failing to achieve complete automation. This not only reduces calibration efficiency but may also introduce errors due to human intervention, affecting calibration accuracy.

[0003] The Kalibr algorithm is a toolkit specifically designed for multi-sensor calibration, particularly suitable for the intrinsic and extrinsic parameter calibration of multi-camera systems. It utilizes images from a large number of AprilTag plates for parameter estimation. Its working principle involves the user first printing out a series of different AprilTag patterns and arranging them on a plane to form a calibration plate. The multi-camera system moves around this calibration plate, capturing images from different angles. The Kalibr algorithm analyzes these images, identifies the positions of the AprilTags at each viewpoint, and calculates the transformation relationship of each camera relative to the global coordinate system and its intrinsic parameters. However, its drawback lies in its requirement for certain camera positions; a certain shared field of view is needed between the cameras to meet calibration requirements. This limits its application on certain devices; for example, when the camera layout cannot guarantee a sufficient shared field of view, the algorithm cannot effectively perform calibration. Summary of the Invention

[0004] In view of this, this application provides an automatic calibration method, electronic device and computer-readable storage medium for multi-view cameras, which can realize automatic calibration of multi-view cameras with a very small common viewing area.

[0005] In a first aspect, this application provides an automatic calibration method for a multi-view camera, the method comprising: The robotic arm equipped with the multi-view camera is controlled to move along a first trajectory, so that the robotic arm passes through multiple first position points on the first trajectory, and when the robotic arm reaches each of the multiple first position points, the multi-view camera is controlled to acquire images of the first calibration plate. Based on the multiple first calibration board images acquired by the multi-view camera, determine the internal parameters of each camera in the multi-view camera; The robotic arm is controlled to move along a second trajectory, so that the robotic arm passes through multiple second position points on the second trajectory, and when the robotic arm reaches each of the multiple second position points, the multi-view camera is controlled to acquire images of the second calibration board. Based on the multiple second calibration board images acquired by the multi-view camera and the internal parameters of each camera in the multi-view camera, determine the first coordinate system transformation parameters between the camera coordinate system of each camera in the multi-view camera and the coordinate system of the second calibration board; Based on the first coordinate system transformation parameters corresponding to each camera in the multi-camera system, the second coordinate system transformation parameters between the camera coordinate systems of each camera in the multi-camera system are determined.

[0006] In some embodiments, as the robotic arm moves along the second trajectory, each of the multi-camera systems observes the second calibration plate at at least one of the second position points.

[0007] In some embodiments, determining the second coordinate system transformation parameters between the camera coordinate systems of the multi-view camera based on the first coordinate system transformation parameters corresponding to each camera in the multi-view camera includes: Based on the first coordinate system transformation parameters corresponding to each camera in the multi-camera system, the coordinate system of each camera is transformed to the coordinate system of the second calibration plate, and the second coordinate system transformation parameters between any two camera coordinate systems are determined.

[0008] In some embodiments, as the robotic arm moves along the first trajectory, each of the multi-view cameras observes the first calibration plate from multiple different poses.

[0009] In some embodiments, the position of the first calibration plate is fixed during the period when the robotic arm moves along the first trajectory; the position of the second calibration plate is fixed during the period when the robotic arm moves along the second trajectory.

[0010] In some implementations, determining the internal parameters of each camera in the multi-view camera system based on multiple first calibration board images acquired by the multi-view camera includes: Based on the multiple first calibration board images acquired by the multi-view camera, the internal parameters of each camera in the multi-view camera are calculated using the Zhang Zhengyou calibration algorithm.

[0011] In some embodiments, determining the first coordinate system transformation parameters between the camera coordinate system of each camera in the multi-view camera and the coordinate system of the second calibration board based on the multiple second calibration board images acquired by the multi-view camera and the internal parameters of each camera in the multi-view camera includes: For each of the multi-view cameras, perform the following steps: a. For each image of the second calibration board captured by the camera, detect the markers in the second calibration board image; b. For each detected marker, calculate a first transformation from the camera coordinate system to the marker coordinate system based on the internal parameters; c. Using a predefined fixed transformation from the marker coordinate system to the coordinate system of the second calibration plate, the first transformation is converted into a second transformation from the camera coordinate system to the coordinate system of the second calibration plate; d. If multiple markers are detected in the second calibration board image, the second transformations of the multiple markers are fused to obtain the fusion calculation result; Repeat steps a, b, c, and d, and iteratively optimize the previous fusion calculation result based on the fusion calculation result obtained in each repetition to obtain the first coordinate system transformation parameters.

[0012] In some embodiments, the coordinate system of a preset mark among a plurality of marks on the second calibration plate is used as the coordinate system of the second calibration plate.

[0013] Secondly, this application provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; The memory stores program instructions that, when executed by the processor, cause the electronic device to perform the method provided in the first aspect above.

[0014] Thirdly, this application provides a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to perform the method provided in the first aspect above.

[0015] Fourthly, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the method provided in the first aspect above.

[0016] In this application, a robotic arm equipped with multiple cameras is controlled to move along a first trajectory, and the multiple cameras are controlled to acquire images of a first calibration board. Based on multiple images of the first calibration board, the internal parameters of each camera are determined. The robotic arm is then controlled to move along a second trajectory, and the multiple cameras are controlled to acquire images of a second calibration board. Based on multiple images of the second calibration board and the internal parameters of the cameras, a first coordinate system transformation parameter is determined between the camera coordinate system of the camera and the coordinate system of the second calibration board. Based on the first coordinate system transformation parameter, a second coordinate system transformation parameter is determined between the camera coordinate systems of each camera. Traditional calibration methods suffer a significant drop in accuracy when the common field of view of the cameras is small. First, this solution uses a robotic arm to precisely control the camera movement and acquire calibration board images at different angles and positions, obtaining sufficient data for calibration even when the common field of view is small. Second, by first determining the camera's internal parameters using the first calibration board images, and then using these internal parameters and the second calibration board images to calculate the extrinsic parameters (second coordinate system transformation parameters), this step-by-step calibration method makes the calculation of internal and external parameters more accurate. Furthermore, this solution automates the entire calibration process. The robotic arm automatically acquires images of the calibration board according to a preset trajectory, eliminating the need for manual movement of the calibration board or adjustment of the camera position, greatly reducing manual intervention time and shortening the calibration time.

[0017] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the structure of a multi-view camera provided in an embodiment of this application.

[0020] Figure 2 This is a schematic diagram of the structure of the automatic calibration system for multi-view cameras provided in the embodiments of this application.

[0021] Figure 3 This is a flowchart illustrating the automatic calibration method for multi-view cameras provided in an embodiment of this application.

[0022] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0024] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0025] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0026] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0027] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0028] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0029] First, the technical terms used in this application will be explained.

[0030] Multi-camera system: A multi-camera system consisting of multiple cameras in different poses that can be synchronized in time.

[0031] Camera internal parameters: The internal parameters of camera imaging mainly include focal length, optical center, distortion coefficient, etc.

[0032] Camera extrinsic parameters: the positional transformation relationship between multiple cameras, i.e., the transformation matrix between different camera coordinate systems (such as the position and orientation of the camera).

[0033] Calibration board: A labeling board used to provide positional markings to the camera, including checkerboard patterns, AprilTag boards, etc. The AprilTag board is a marking system for visual positioning and tracking, similar to a QR code or AR tag.

[0034] Multi-camera calibration is a key technology in computer vision, with the core objective of accurately determining the intrinsic and extrinsic parameters of one or more cameras. This technology has significant applications in numerous fields, including but not limited to 3D reconstruction, augmented reality, robot navigation, and autonomous driving. Early research in multi-camera calibration primarily focused on monocular cameras, calculating their intrinsic and extrinsic parameters using objects with known geometric shapes (such as a checkerboard pattern). Among these, the Zhang Zhengyou calibration method, a widely used planar template calibration method, requires only a printed checkerboard pattern for calibration, greatly simplifying the process.

[0035] Feature extraction and matching are fundamental steps in multi-view camera calibration, primarily tasked with identifying and matching feature points of the same scene from different viewpoints. Common feature extraction techniques used in this process include Scale Invariant Feature Transform (SIFT), Speed-Up Robust Feature Transform (SURF), and Fast Binary Descriptor (ORB). To obtain optimal calibration results, a method that minimizes reprojection error is typically employed, solved using nonlinear optimization algorithms such as the Levenberg-Marquardt algorithm. By iteratively adjusting parameter values ​​until the best fit is achieved, the accuracy and reliability of the calibration results are ensured.

[0036] Figure 1 A schematic diagram of the structure of a multi-view camera provided in an embodiment of this application is shown. The multi-view camera system includes a bracket and cameras A, B, C, and D fixed thereon. These cameras are mounted at different positions on the bracket, and the fields of view of each camera are staggered. Furthermore, each camera has a small field of view (FOV), resulting in a correspondingly small shared field of view between any two adjacent cameras.

[0037] Figure 2A schematic diagram of the automatic calibration system for a multi-view camera provided in an embodiment of this application is shown. The system includes a multi-view camera, a robotic arm operating table, and a calibration plate support. Please refer to... Figure 1 , 2 The multi-camera system includes cameras A, B, C, and D, which are fixedly mounted at different positions on a support frame. The support frame is mounted on a robotic arm, and a control panel for the user to issue commands to the robotic arm. The user sends commands through the control panel to control the robotic arm to automatically move the multi-camera system precisely. During this process, the multi-camera system acquires images of both the checkerboard and AprilTag boards. The checkerboard and AprilTag boards are placed on their respective supports to ensure clear images are obtained during acquisition. The supports for both the checkerboard and AprilTag boards are located near the control panel, but their positions differ.

[0038] The acquired image data will be used in the automatic calibration process. First, the intrinsic parameters of each camera are calculated using the checkerboard image data. Then, by acquiring image data from the AprilTag board and combining it with the previously calculated intrinsic parameters, the coordinate transformation parameters between the camera coordinate system and the AprilTag board coordinate system are further calculated, thereby determining the relative position and orientation between the various camera coordinate systems.

[0039] Through the automated image acquisition and calibration calculation process described above, accurate calibration of multi-view camera systems is achieved without manual intervention, thus improving the efficiency and accuracy of calibration.

[0040] Figure 3 A flowchart illustrating an automatic calibration method for a multi-view camera according to an embodiment of this application is shown below in detail: Step 301: Control the robotic arm equipped with the multi-view camera to move along the first trajectory, so that the robotic arm passes through multiple first position points on the first trajectory, and when the robotic arm reaches each of the multiple first position points, control the multi-view camera to acquire images of the first calibration plate.

[0041] Step 302: Determine the internal parameters of each camera in the multi-view camera based on the multiple first calibration board images acquired by the multi-view camera.

[0042] Step 303: Control the robotic arm to move along the second trajectory, so that the robotic arm passes through multiple second position points on the second trajectory, and when the robotic arm reaches each of the multiple second position points, control the multi-view camera to acquire images of the second calibration board.

[0043] Step 304: Based on the multiple second calibration board images acquired by the multi-view camera and the internal parameters of each camera in the multi-view camera, determine the first coordinate system transformation parameter between the camera coordinate system of each camera in the multi-view camera and the coordinate system of the second calibration board.

[0044] Step 305: Determine the second coordinate system transformation parameters between the camera coordinate systems of each camera in the multi-view camera based on the first coordinate system transformation parameters corresponding to each camera in the multi-view camera.

[0045] In this embodiment, a plurality of first position points are provided on the first trajectory. These position points are preset to ensure that the multi-view camera can capture images of the first calibration plate from different angles and positions. When the robotic arm reaches each first position point, the multi-view camera is triggered to acquire images of the first calibration plate, obtaining an image of the first calibration plate. By smoothly connecting these first position points, a first trajectory is obtained, and the robotic arm moves smoothly along the first trajectory. Using the plurality of first calibration plate images acquired in step 301, the internal parameters of each camera in the multi-view camera are calculated using a calibration algorithm (such as the Zhang Zhengyou calibration method). Exemplarily, the multi-view camera in this embodiment is as follows: Figure 1 As shown, based on multiple first calibration board images acquired by camera A in a multi-view camera system, the internal parameters of camera A can be calculated. Similarly, the internal parameters of the corresponding cameras can be calculated based on multiple first calibration board images acquired by other cameras in the multi-view camera system. These internal parameters may include focal length, optical center, distortion coefficients, etc.

[0046] After the robotic arm completes its movement along the first trajectory, it can then move along a second trajectory. This second trajectory includes multiple second position points. At each second position point, a multi-view camera is triggered to acquire an image of the second calibration board (e.g., the AprilTag board). By smoothly connecting these second position points, the second trajectory is obtained, and the robotic arm moves smoothly along it. Based on the acquired images of the multiple second calibration boards and known internal parameters, a first coordinate system transformation parameter is determined between the camera coordinate system of each camera and the coordinate system of the second calibration board. This first coordinate system transformation parameter describes the relative position and orientation between the camera coordinate system and the calibration board coordinate system. Specifically, image feature extraction and matching techniques (e.g., SIFT, SURF) and optimization algorithms (e.g., Levenberg-Marquardt) can be used to calculate these transformation parameters from the image data. For example, based on multiple images of the second calibration board acquired by camera A, the first coordinate system transformation parameter between the camera coordinate system of camera A and the coordinate system of the second calibration board can be calculated. Similarly, the first coordinate system transformation parameter of the corresponding camera can be calculated based on the second calibration board images acquired by other cameras in the multi-view camera system. Using the first coordinate system transformation parameters corresponding to each camera obtained in step 304, the second coordinate system transformation parameter between the coordinate systems of each camera in the multi-view camera system can be calculated, i.e., the external parameters of the multi-view camera system. The second coordinate system transformation parameter describes the relative position and orientation between different camera coordinate systems, thereby achieving accurate calibration of the multi-view camera system.

[0047] In some embodiments, as the robotic arm moves along the second trajectory, each camera in the multi-view camera observes the second calibration plate at at least one of the second position points. Specifically, during the execution of the automatic calibration procedure, the robotic arm will move precisely along a pre-set second trajectory. This trajectory includes multiple specific second position points to ensure that each camera in the multi-view camera mounted at the end of the robotic arm can observe the second calibration plate placed in a fixed position at at least one of the second position points.

[0048] Exemplary examples include multi-view cameras in this application embodiment, such as... Figure 1As shown, the robotic arm moves along a second trajectory, sequentially passing through positions 1, 2, 3, 4, and 5. At each position, the corresponding camera captures an image of the second calibration plate. Specifically, camera A observes the second calibration plate from two different second position points (position 1 and position 3) on the second trajectory. At position 1, camera A captures an image of the second calibration plate from the left at a certain angle. At position 3, camera A captures an image of the second calibration plate from the right at another angle. Camera B observes the second calibration plate from only one second position point (position 2). At position 2, camera B captures an image of the second calibration plate from the front. Camera C observes the second calibration plate at position 4, while camera D observes the second calibration plate at position 5. Cameras C and D capture images of the second calibration plate from different angles and positions. This design ensures that each camera can observe the second calibration plate from at least one position point, even with a small field of view and a small shared viewing area.

[0049] In some embodiments, determining the second coordinate system transformation parameters between the camera coordinate systems of the multi-view camera based on the first coordinate system transformation parameters corresponding to each camera in the multi-view camera system includes: uniformly transforming the coordinate systems of each camera to the coordinate system of the second calibration plate based on the first coordinate system transformation parameters corresponding to each camera in the multi-view camera system, and determining the second coordinate system transformation parameters between any two camera coordinate systems. Specifically, after obtaining the transformation parameters (first coordinate system transformation parameters) of each camera's coordinate system relative to the second calibration plate, the relative position and direction between any two camera coordinate systems, i.e., the second coordinate system transformation parameters, can be determined by comparing these parameters. For example, given the coordinate system transformation parameters of camera A and camera B relative to the second calibration plate, the transformation parameters (second coordinate system transformation parameters) of camera A's coordinate system relative to camera B's coordinate system can be calculated through mathematical operations (such as combinations of coordinate transformations).

[0050] In some embodiments, when the robotic arm moves along the first trajectory, each of the multi-view cameras observes the first calibration plate from multiple different poses.

[0051] For example, suppose a multi-camera system has four cameras: Camera A, Camera B, Camera C, and Camera D. These cameras are mounted on the end effector of a robotic arm that can move along a first trajectory. The first trajectory includes multiple first position points, such as position points P1, P2, P3, and P4. Camera A observes the first calibration plate at positions P1 and P3 and acquires images. At position P1, Camera A might capture an image of the first calibration plate from the left at a certain angle. At position P3, Camera A might capture an image of the first calibration plate from the right at another angle. Camera B observes the first calibration plate at positions P2 and P4 and acquires images. At position P2, Camera B might capture an image of the first calibration plate from above. At position P4, Camera B might capture an image of the first calibration plate from below. Camera C might observe the first calibration plate at positions P1 and P4 and acquire images. Camera D might observe the first calibration plate at positions P2 and P3 and acquire images. In this way, each camera can observe the first calibration plate from at least two different first position points, thus ensuring the comprehensiveness and diversity of data acquisition. Multi-angle observation helps improve the accuracy of calibration because camera parameters can be corrected and optimized from different angles.

[0052] In some embodiments, the position of the first calibration plate is fixed during the movement of the robotic arm along the first trajectory; the position of the second calibration plate is fixed during the movement of the robotic arm along the second trajectory. Specifically, the position of the first calibration plate (e.g., a checkerboard pattern) is fixed during the movement of the robotic arm along the first trajectory. A multi-camera system is mounted at the end of the robotic arm, which moves along a preset first trajectory, passing through multiple first position points on the trajectory. At each first position point, the robotic arm stops so that each camera in the multi-camera system can acquire an image of the fixed-position first calibration plate. Similarly, the position of the second calibration plate (e.g., an AprilTag pattern) is also fixed during the movement of the robotic arm along the second trajectory. The robotic arm moves along the second trajectory, passing through multiple second position points on the trajectory. At each second position point, the robotic arm stops so that the multi-camera system can acquire an image of the fixed-position second calibration plate.

[0053] In some embodiments, determining the internal parameters of each camera in the multi-view camera system based on multiple first calibration board images acquired by the multi-view camera system includes: calculating the internal parameters of each camera in the multi-view camera system using the Zhang Zhengyou calibration algorithm based on the multiple first calibration board images acquired by the multi-view camera system. Specifically, at multiple first position points on the first trajectory, the robotic arm stops moving, allowing each camera in the multi-view camera system to acquire images of the fixed-position first calibration board (checkerboard) from different angles. The Zhang Zhengyou calibration algorithm step may include: extracting corner features from the checkerboard images acquired by each camera; matching these corner features with the actual geometric model of the checkerboard; and iteratively solving for the camera's internal parameters using a nonlinear optimization algorithm (such as the Levenberg-Marquardt algorithm) by minimizing the reprojection error.

[0054] In some embodiments, determining the first coordinate system transformation parameter between the camera coordinate system of each camera in the multi-view camera and the coordinate system of the second calibration board based on multiple second calibration board images acquired by the multi-view camera and the internal parameters of each camera in the multi-view camera includes: for each camera in the multi-view camera, performing the following steps: a) for each second calibration board image acquired by the camera, detecting a mark in the second calibration board image; b) for each detected mark, calculating a first transformation from the camera coordinate system to the mark coordinate system based on the internal parameters; c) using a predefined fixed transformation from the mark coordinate system to the coordinate system of the second calibration board, converting the first transformation into a second transformation from the camera coordinate system to the coordinate system of the second calibration board; d) if multiple marks are detected in the second calibration board image, fusing the second transformations of the multiple marks to obtain a fusion calculation result; repeating steps a, b, c, and d, and iteratively optimizing the previous fusion calculation result based on the fusion calculation result obtained in each repeated execution to obtain the first coordinate system transformation parameter.

[0055] Specifically, at each second position point, each camera in the multi-view camera system acquires an image of a fixed-position second calibration board (e.g., an AprilTag board). For each second calibration board image acquired by each camera, markers (e.g., AprilTag markers) are detected in the image. The AprilTag board contains multiple AprilTag markers. For each detected marker, a first transformation from the camera coordinate system to the marker coordinate system of the marker is calculated based on the camera's intrinsic parameters (determined in step 302). Using a predefined fixed transformation from the marker coordinate system to the second calibration board coordinate system, the first transformation is converted into a second transformation from the camera coordinate system to the second calibration board coordinate system. During camera calibration, the position and orientation of the calibration board (or marker) in the world coordinate system are known. The predefined fixed transformation refers to the transformation relationship from the marker coordinate system of the calibration board itself to the overall coordinate system of the calibration board, which is determined during the manufacturing process of the calibration board. This transformation relationship is fixed and determined by the physical characteristics of the calibration board. Optionally, the marker coordinate system of a preset marker among the multiple markers of the second calibration board can be used as the coordinate system of the second calibration board. On the second calibration board (such as the AprilTag board), there are multiple markers, each with a unique pattern. Each preset marker has an associated marker coordinate system, which can have its origin at the center of the marker. A preset marker can be selected from the multiple markers, and its coordinate system is known.

[0056] In step d, if multiple markers are detected, their transformations are fused, and the calibration accuracy is improved through iterative optimization. The fusion method could be to take a weighted average of these transformations to obtain the fused calculation result.

[0057] Repeat steps a, b, c, and d above, and iteratively optimize the previous fusion calculation result based on the fusion calculation result obtained in each repetition, finally obtaining the first coordinate system transformation parameters.

[0058] For example, assuming a multi-camera system contains four cameras, the specific steps a, b, c, and d above can be as follows: First, set a roughly reasonable initial pose for the four cameras, i.e., initial extrinsic parameters. These initial values ​​will serve as the starting point for algorithm optimization. On the AprilTag calibration board, there are multiple tags, for example, 36 tags from 0 to 35. The algorithm sets a tag coordinate system for each detected tag. The coordinate system of one tag (e.g., tag-20) is selected as the global coordinate system, i.e., the coordinate system of the AprilTag calibration board. This global coordinate system will serve as a reference. Detect whether each image captures the AprilTag board. If detected, calculate the coordinate positions of all captured tag coordinate systems to obtain the camera coordinate system position. Based on the detected tag coordinate systems and the predefined fixed transformation from the tag coordinate system to the global coordinate system, calculate the transformation from the camera coordinate system to the AprilTag board coordinate system. If multiple tags are detected in the AprilTag board image, fuse the transformations of these multiple tags to obtain the fused calculation result. Repeat the above detection and calculation steps, and iteratively optimize the previous fusion calculation result based on the fusion calculation result obtained from each repetition, finally obtaining the first coordinate system transformation parameters. After obtaining the coordinate system transformation of each camera to the AprilTag board, the coordinate system transformation between the coordinate systems of the multi-camera system can be obtained, which is the final four-camera extrinsic parameters.

[0059] In some embodiments, the robotic arm needs to rotate and move back and forth along a preset trajectory (second trajectory) to ensure that each camera can capture an image of the AprilTag. This movement should be smooth to avoid blurring or distortion during image acquisition. Optionally, each camera can capture images of the AprilTag at different robotic arm positions, allowing each camera to capture images of the AprilTag in various locations, such as the AprilTag appearing in the upper left, lower, upper right, etc., to collect rich data for calculating extrinsic parameters.

[0060] As can be seen from the above, in this application, a robotic arm equipped with multiple cameras is controlled to move along a first trajectory, and the multiple cameras are controlled to acquire images of a first calibration board; based on multiple images of the first calibration board, the internal parameters of each camera are determined; the robotic arm is controlled to move along a second trajectory, and the multiple cameras are controlled to acquire images of a second calibration board; based on multiple images of the second calibration board and the internal parameters of the cameras, the first coordinate system transformation parameters between the camera coordinate system of the camera and the coordinate system of the second calibration board are determined; based on the first coordinate system transformation parameters, the second coordinate system transformation parameters between the camera coordinate systems of each camera are determined. Traditional calibration methods suffer a significant drop in accuracy when the common field of view of the cameras is small. First, this solution precisely controls the camera movement through a robotic arm, acquiring calibration board images at different angles and positions, thus obtaining sufficient data for calibration even when the common field of view is small. Second, by first determining the camera's internal parameters using the first calibration board images, and then using these internal parameters and the second calibration board images to calculate the extrinsic parameters (second coordinate system transformation parameters), this step-by-step calibration method makes the calculation of internal and external parameters more accurate. Furthermore, this solution automates the entire calibration process. The robotic arm automatically acquires images of the calibration board according to a preset trajectory, eliminating the need for manual movement of the calibration board or adjustment of the camera position, greatly reducing manual intervention time and shortening the calibration time.

[0061] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0062] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 4 of this embodiment includes: at least one processor 40 ( Figure 4 (Only one is shown in the image), memory 41, and computer program 42 stored in the memory 41 and executable on at least one processor 40, wherein when the processor 40 executes the computer program 42, the electronic device performs the steps of the method described above.

[0063] The aforementioned electronic device 4 may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will understand that... Figure 4 This is merely an example of electronic device 4 and does not constitute a limitation on electronic device 4. It may include more or fewer components than shown, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0064] The processor 40 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0065] In some embodiments, the aforementioned memory 41 may be an internal storage unit of the electronic device 4, such as a hard disk or memory of the electronic device 4. In other embodiments, the aforementioned memory 41 may be an external storage device of the electronic device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 4. Furthermore, the aforementioned memory 41 may include both internal storage units and external storage devices of the electronic device 4. The aforementioned memory 41 is used to store operating systems, applications, bootloaders, data, and other programs, such as the program code of the aforementioned computer programs. The aforementioned memory 41 may also be used to temporarily store data that has been output or will be output.

[0066] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0067] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0068] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the various method embodiments described above.

[0069] This application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the steps described in the various method embodiments above.

[0070] This application provides a chip system including a processor, which is used to call and run a computer program from a memory, causing an electronic device equipped with the chip system to perform the steps in the various method embodiments described above.

[0071] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to an electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0072] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0073] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0074] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0075] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0076] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An automatic calibration method for a multi-view camera, characterized in that, The method includes: The robotic arm equipped with the multi-view camera is controlled to move along a first trajectory, so that the robotic arm passes through multiple first position points on the first trajectory, and when the robotic arm reaches each of the multiple first position points, the multi-view camera is controlled to acquire images of the first calibration plate. Based on the multiple first calibration board images acquired by the multi-view camera, determine the internal parameters of each camera in the multi-view camera; The robotic arm is controlled to move along a second trajectory, so that the robotic arm passes through multiple second position points on the second trajectory, and when the robotic arm reaches each of the multiple second position points, the multi-view camera is controlled to acquire images of the second calibration board. Based on the multiple second calibration board images acquired by the multi-view camera and the internal parameters of each camera in the multi-view camera, determine the first coordinate system transformation parameters between the camera coordinate system of each camera in the multi-view camera and the coordinate system of the second calibration board; Based on the first coordinate system transformation parameters corresponding to each camera in the multi-view camera, a second coordinate system transformation parameter is determined between the camera coordinate systems of each camera in the multi-view camera. When the robotic arm moves according to the second trajectory, each camera in the multi-view camera observes the second calibration board at at least one second position point. The second calibration board is an AprilTag board. The determination of the first coordinate system transformation parameter between the camera coordinate system of each camera in the multi-view camera and the coordinate system of the second calibration board, based on multiple second calibration board images acquired by the multi-view camera and the internal parameters of each camera in the multi-view camera, includes: For each of the multi-view cameras, perform the following steps: a. For each image of the second calibration board captured by the camera, detect the markers in the second calibration board image; b. For each detected marker, calculate a first transformation from the camera coordinate system to the marker coordinate system based on the internal parameters; c. Using a predefined fixed transformation from the marker coordinate system to the coordinate system of the second calibration plate, the first transformation is converted into a second transformation from the camera coordinate system to the coordinate system of the second calibration plate; d. If multiple markers are detected in the second calibration board image, the second transformations of the multiple markers are fused to obtain the fusion calculation result; Repeat steps a, b, c, and d, and iteratively optimize the previous fusion calculation result based on the fusion calculation result obtained in each repetition to obtain the first coordinate system transformation parameters.

2. The method as described in claim 1, characterized in that, The step of determining the second coordinate system transformation parameters between the camera coordinate systems of the various cameras in the multi-camera system based on the first coordinate system transformation parameters corresponding to each camera in the multi-camera system includes: Based on the first coordinate system transformation parameters corresponding to each camera in the multi-camera system, the coordinate system of each camera is transformed to the coordinate system of the second calibration plate, and the second coordinate system transformation parameters between any two camera coordinate systems are determined.

3. The method as described in claim 1, characterized in that, When the robotic arm moves along the first trajectory, each camera in the multi-view camera observes the first calibration plate from multiple different poses.

4. The method as described in claim 1, characterized in that, During the period when the robotic arm moves along the first trajectory, the position of the first calibration plate is fixed; during the period when the robotic arm moves along the second trajectory, the position of the second calibration plate is fixed.

5. The method as described in claim 1, characterized in that, The step of determining the internal parameters of each camera in the multi-view camera system based on multiple first calibration board images acquired by the multi-view camera includes: Based on the multiple first calibration board images acquired by the multi-view camera, the internal parameters of each camera in the multi-view camera are calculated using the Zhang Zhengyou calibration algorithm.

6. The method as described in claim 1, characterized in that, The coordinate system of the preset mark among the multiple marks on the second calibration plate is used as the coordinate system of the second calibration plate.

7. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is coupled to the processor; The memory stores program instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed on a computer or processor, causes the computer or processor to perform the method as described in any one of claims 1 to 6.

Images