A camera calibration method, control device, and calibration system

Abstract

This application provides a camera calibration method, control device, and calibration system, relating to the field of camera calibration, which can accurately calibrate cameras in various scenarios. The method includes: obtaining the pixel coordinates of the first corner point in the initial calibration image captured by each camera in its initial state; determining the pose transformation matrix between each camera and its corresponding calibration plate in the initial state based on the internal parameters of each camera, the pixel coordinates of the first corner point corresponding to each camera, and the three-dimensional coordinates of the first corner point on the calibration plate; obtaining the pixel coordinates of the second corner point in the rotated calibration image captured by each camera after the rotating platform has rotated; determining the pose transformation matrix between each camera and its corresponding calibration plate after the rotating platform has rotated based on the internal parameters of each camera, the pixel coordinates of the second corner point corresponding to each camera, and the three-dimensional coordinates of the second corner point on the calibration plate; and determining the pose transformation matrix between any two cameras based on the pose transformation matrices of each camera and its corresponding calibration plate before and after the rotating platform has rotated.

Description

Technical Field

[0001] The application relates to the field of camera calibration, and in particular to a camera calibration method, control device, and calibration system. Background Technology

[0002] With the increasing demands of users, single cameras with smaller fields of view are no longer sufficient to meet the perception requirements of machine vision applications (such as 3D reconstruction, obstacle avoidance, navigation, and simultaneous localization and mapping (SLAM)) in various electronic devices (such as virtual reality (VR) / augmented reality (AR), robots, drones, and autonomous vehicles). Therefore, current machine vision applications require the use of multi-view perception suites composed of multiple cameras with larger fields of view. To ensure the proper functioning of multi-view perception suites in application scenarios, the intrinsic and extrinsic parameters of each camera within the suite must be calibrated beforehand.

[0003] Traditional calibration schemes for camera intrinsics and extrinsic parameters in multi-view sensing kits mostly assume the existence of a shared field of view between cameras. If two cameras in the kit do not share a field of view, their extrinsic parameters cannot be calibrated. Furthermore, existing techniques for calibrating camera intrinsics and extrinsic parameters in multi-view sensing kits without a shared field of view impose significant limitations on the camera's placement or offer low calibration accuracy. Therefore, a more versatile and accurate method for calibrating cameras in multi-view sensing kits is urgently needed. Summary of the Invention

[0004] This application provides a camera calibration method, control device, and calibration system that can accurately calibrate cameras in various scenarios.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, embodiments of this application provide a camera calibration method applied to a calibration system. The calibration system includes a rotating platform, an electronic device equipped with a multi-view sensing kit, multiple calibration plates, and a control device established with communication connections to both the rotating platform and the electronic device. The electronic device is fixedly mounted on the rotating platform. The multi-view sensing kit includes multiple cameras, and a calibration plate is fixedly mounted in the shooting area of ​​each camera. The method includes:

[0007] The control device acquires the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image captured by each camera in the multi-view perception kit in the initial state; the initial state is the state in which the initial setting of the calibration system is completed;

[0008] The control device determines the pose transformation matrix between each camera and its corresponding calibration board in the initial state based on the internal parameters of each camera, the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image of each camera, and the three-dimensional coordinates of the first corner point on the corresponding calibration board.

[0009] The control device acquires the pixel coordinates of the second corner point of the calibration pattern in the rotation calibration image captured by each camera when the rotating platform rotates by a preset angle;

[0010] Based on the internal parameters of each camera, the pixel coordinates of the second corner point in the rotation calibration image of each camera, and the three-dimensional coordinates of the second corner point on the corresponding calibration plate, the control device determines the pose transformation matrix between each camera and the corresponding calibration plate when the rotating platform rotates at a preset angle.

[0011] The control device determines the pose transformation matrix between any two cameras in the multi-view perception kit based on the pose transformation matrix between each camera and its corresponding calibration board in the initial state, and the pose transformation matrix between each camera and its corresponding calibration board when the rotating platform rotates by a preset angle.

[0012] Based on the technical solution provided in this application, by introducing rotation angle information through a rotating platform, the external parameters (i.e., pose transformation matrix) of each camera in the multi-view perception kit can be determined by combining the pure visual information in the images captured by each camera. For pure visual information, the stability and uniqueness of its feature points are relatively good; therefore, this calibration scheme can make the final camera external parameters more accurate. Furthermore, as long as there are enough feature points in the pure visual information in different scenarios, the calibration objective can be successfully achieved. This makes the calibration scheme more adaptable. Simultaneously, since this calibration scheme does not determine the external parameters (i.e., pose transformation matrix between different cameras) based on the differences in images of the same object captured by different cameras, the calibration scheme provided in this application can successfully obtain the external parameters between different cameras in the multi-view perception kit regardless of whether there is a shared viewing area between different cameras in the multi-view perception kit. Furthermore, the calibration scheme provided in this application does not impose strong constraints on the position of the calibration board; it only needs to be adapted to the layout of the cameras in the multi-view perception kit, and the style of the calibration board is not specifically limited. In summary, the technical solution provided in this application can accurately calibrate the external parameters of cameras in various scenarios (such as production lines or repair points) regardless of whether there is a shared field of view between different cameras in the multi-view perception kit. Compared with existing camera calibration solutions for multi-view perception kits, it has better adaptability and stability.

[0013] In one possible design of the first aspect, the control device acquires the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image captured by each camera in the multi-view sensing kit in the initial state, including: the control device sending a first instruction to the electronic device; the first instruction instructing the electronic device to control each camera in the multi-view sensing kit to take pictures and send the captured initial calibration image to the control device; the control device receiving the initial calibration images corresponding to each camera from the electronic device and determining the pixel coordinates of the first corner point of the calibration pattern in each initial calibration image.

[0014] Based on the above technical solution, the control device can control the camera in the electronic device through instructions, thereby obtaining the initial calibration images captured by each camera in the multi-view perception kit of the electronic device and the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image, thus providing a data basis for the execution of subsequent solutions.

[0015] In one possible design of the first aspect, the control device determines the pose transformation matrix between each camera and its corresponding calibration board in the initial state based on the internal parameters of each camera, the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image corresponding to each camera, and the three-dimensional coordinates of the first corner point on the corresponding calibration board. This includes: the control device converting the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image corresponding to each camera into two-dimensional coordinates in the image coordinate system based on the internal parameters of each camera; and the control device determining the pose transformation matrix between each camera and its corresponding calibration board in the initial state based on the internal parameters of each camera, the two-dimensional coordinates of the first corner point of the calibration pattern in the initial calibration image corresponding to each camera in the image coordinate system, and the three-dimensional coordinates of the first corner point on the corresponding calibration board.

[0016] Based on the above technical solution, the pose transformation matrix between each camera and its corresponding calibration board in the multi-view perception kit in the initial state can be calculated using a specific algorithm, providing a data foundation for the execution of subsequent solutions.

[0017] In one possible design of the first aspect, the control device acquires the pixel coordinates of the second corner point of the calibration pattern in the rotation calibration image captured by each camera when the rotating platform rotates by a preset angle, including: the control device sending a second instruction to the rotating platform; the second instruction instructing the rotating platform to rotate by a preset angle; the control device receiving a first response from the rotating platform and, in response to the first response, sending a third instruction to the electronic device; the first response instructing the rotating platform to complete its rotation; the third instruction instructing the electronic device to control each camera in the multi-view perception kit to capture images and send the captured rotation calibration images to the control device; the control device receiving the rotation calibration images corresponding to each camera from the electronic device and determining the pixel coordinates of the second corner point of the calibration pattern in each rotation calibration image.

[0018] Based on the above technical solution, the control device can control the rotating platform and electronic equipment through commands, thereby obtaining the rotation calibration images captured by each camera in the multi-view perception kit of the electronic equipment when the rotating platform rotates at a preset angle, as well as the pixel coordinates of the second corner point of the calibration pattern in the rotation calibration image, thus providing a data basis for the execution of subsequent solutions.

[0019] In one possible design of the first aspect, the control device determines the pose transformation matrix between any two cameras in the multi-view perception kit based on the pose transformation matrix between each camera and its corresponding calibration plate in the initial state, and the pose transformation matrix between each camera and its corresponding calibration plate when the rotating platform rotates by a preset angle. This includes: the control device calculating the pose transformation matrix between the ground and each calibration plate based on the pose transformation matrix between each camera and its corresponding calibration plate in the initial state and the pose transformation matrix between each camera and its corresponding calibration plate when the rotating platform rotates by a preset angle; the control device determining the pose transformation matrix between any two calibration plates based on the pose transformation matrix between the ground and each calibration plate; and the control device determining the pose transformation matrix between any two cameras based on the pose transformation matrix between any two calibration plates and the pose transformation matrix between each camera and its corresponding calibration plate in the initial state.

[0020] In this calibration system, the pose transformation matrix between any two cameras can be obtained by converting the pose transformation matrix between the calibration boards corresponding to those two cameras into the pose transformation relationship between each of those two cameras and its corresponding calibration board. Similarly, the pose transformation matrix between the calibration boards corresponding to any two cameras can be obtained by converting the pose transformation matrices between these two calibration boards and the ground. Furthermore, since the relative positions between the calibration boards and the ground are fixed throughout the calibration system, the pose transformation matrix between the calibration boards and the ground is fixed; that is, the pose transformation matrix between the calibration boards and the ground remains unchanged regardless of whether the rotating platform is rotated before or after rotation. Simultaneously, the relative positions between the rotating platform and each camera are also fixed, meaning the pose transformation matrix between the rotating platform and the camera is the same regardless of whether the platform is rotated before or after rotation. Moreover, the pose transformation matrix between the rotating platform and the camera, whether before or after rotation, can be obtained by converting the pose transformation matrix between the rotating platform and the ground, the pose transformation matrix between the ground and the calibration board, and the pose transformation matrix between the calibration board and the corresponding camera. Furthermore, since the rotation angle is known for each rotation after the initial state, the difference between the pose transformation matrix between the rotating platform and the ground after each rotation and the pose transformation matrix between the rotating platform and the ground before rotation lies only in the transformation of the rotation angle mapping in the rotation matrix. Based on the above, the pose transformation matrix between the rotating platform and the ground before and after rotation can be considered known. To obtain the pose transformation matrix between the ground and each calibration plate, it is necessary to obtain the pose transformation matrix between each camera and its corresponding calibration plate before and after rotation.

[0021] Therefore, after obtaining the pose transformation matrix between each camera and its corresponding calibration board before and after rotation, the pose transformation matrix between the ground and each calibration board, the pose transformation matrix between any two calibration boards, and the pose transformation matrix between any two cameras can be obtained sequentially according to the above logic. Thus, based on the above scheme, the pose transformation matrix between any two cameras can be successfully obtained, that is, the external parameters between any two cameras in the multi-view perception kit can be obtained.

[0022] In one possible design of the first aspect, the control device calculates the pose transformation matrix between the ground and each calibration plate based on the pose transformation matrix between each camera and its corresponding calibration plate in the initial state, and the pose transformation matrix between each camera and its corresponding calibration plate when the rotating platform rotates by a preset angle. This includes:

[0023] Based on the pose transformation matrix between each camera and its corresponding calibration plate in the initial state, and the pose transformation matrix between each camera and its corresponding calibration plate after the rotating platform has rotated by a preset angle, the control device calculates the pose transformation matrix between the ground and each calibration plate using the following formula:

[0024]

[0025] in, This represents the pose transformation relationship between the rotating platform and the ground in the initial state. Let n be the pose transformation matrix between the ground and the nth calibration plate. Let be the pose transformation matrix between the nth calibration board and the nth camera, where the calibration board corresponding to the nth camera is the nth calibration board. The pose transformation matrix between the rotating platform and the ground when the rotating platform rotates by a preset angle. Given a preset angle for rotating the platform, the pose transformation matrix between the nth calibration plate and the nth camera is defined; the nth calibration plate is one of the calibration plates corresponding to multiple cameras, and the nth camera is any one of the multiple cameras.

[0026] Based on the above technical solution, the pose transformation matrix between the ground and each calibration plate can be calculated based on the transformation relationship between the pose transformation matrices, providing data support for the subsequent calculation of the pose transformation matrix between any two cameras.

[0027] In one possible design of the first aspect, before the control device determines the pose transformation matrix between each camera and its corresponding calibration board in the initial state based on the internal parameters of each camera, the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image corresponding to each camera, and the three-dimensional coordinates of the first corner point on the corresponding calibration board, the method further includes: the control device determining the internal parameters of each camera in the multi-view perception kit.

[0028] Based on the above technical solution, since the internal parameters of the camera are required when calculating the pose transformation matrix between the camera and the corresponding calibration board, the control device needs to obtain the internal parameters of each camera in advance before calculating the pose transformation matrix, so as to provide strong data support for subsequent calculations.

[0029] In a second aspect, this application provides a control device, the electronic device including a memory and one or more processors; the memory is coupled to the processors; wherein the memory stores computer program code, the computer program code including computer commands, which, when executed by the processor, cause the control device to perform a camera calibration method as provided in the first aspect and any of its possible design embodiments.

[0030] Thirdly, this application provides a computer-readable storage medium including computer commands that, when executed on a control device, cause the control device to perform a camera calibration method as provided in the first aspect and any of its possible design embodiments.

[0031] Fourthly, this application provides a computer program product that, when run on a control device, causes the control device to perform a camera calibration method as provided in the first aspect and any of its possible design embodiments.

[0032] Understandably, the beneficial effects that the technical solutions provided in the second to fourth aspects described above can be achieved can be referred to the beneficial effects of the first aspect and any of its possible design methods, which will not be repeated here. Attached Figure Description

[0033] Figure 1 A schematic diagram of various calibration plates provided in the embodiments of this application;

[0034] Figure 2 This is a schematic diagram illustrating the relationship between an image coordinate system and a pixel coordinate system, provided in an embodiment of this application.

[0035] Figure 3 This is a schematic diagram illustrating the relationship between an image coordinate system and a camera coordinate system, provided in an embodiment of this application.

[0036] Figure 4A schematic diagram of a three-dimensional calibration plate provided in an embodiment of this application;

[0037] Figure 5 This is a schematic diagram of the structure of a calibration system provided in an embodiment of this application;

[0038] Figure 6 A schematic diagram illustrating the principle of a calibration method provided in an embodiment of this application;

[0039] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0040] Figure 8 This is a schematic diagram of the hardware structure of a control device provided in an embodiment of this application;

[0041] Figure 9 A schematic flowchart illustrating a calibration method provided in an embodiment of this application;

[0042] Figure 10 A schematic diagram of multiple target images provided in embodiments of this application;

[0043] Figure 11 This is a schematic diagram of various coordinate systems provided in the calibration system of this application embodiment;

[0044] Figure 12 A flowchart illustrating another calibration method provided in an embodiment of this application;

[0045] Figure 13 A flowchart illustrating another calibration method provided in an embodiment of this application;

[0046] Figure 14 This is a schematic diagram of the structure of a control device provided in an embodiment of this application. Detailed Implementation

[0047] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that “ / ” means “or,” for example, A / B can mean A or B; “and / or” in the text is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can mean: A alone, A and B simultaneously, and B alone.

[0048] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0049] The terms "first" and "second" in the following embodiments of this application are for descriptive purposes only and should not be construed as implying relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0050] First, the terms used in this application are explained as follows:

[0051] (1) Calibration plate: A calibration plate is a typical calibration object used for visual measurements. By photographing the calibration plate with a camera, specific calibration algorithms can be used to obtain the camera's intrinsic and extrinsic parameters. The calibration plate may include calibration patterns; depending on the different calibration patterns, there can be multiple types of calibration plates, for example... Figure 1 The calibration pattern shown in (a) is a checkerboard calibration board. Figure 1 The calibration pattern shown in (b) is a dot calibration plate with round dots, and Figure 1 The calibration pattern shown in (c) is a cross-shaped calibration plate.

[0052] (2) Corner Points: Corner points are points that highlight the features of an image. They often possess certain significant characteristics, making them easy to identify and locate in an image. In practice, corner detection is commonly used to extract points of interest (POIs) in an image. These POIs can be corner points, isolated points with the greatest or least intensity in certain attributes, the endpoints of line segments, or points with the greatest local curvature on a curve. It is important to note that there is no explicit mathematical definition for corner points; therefore, in practical applications, specific methods must be used to select appropriate corner points based on specific needs.

[0053] For example, such as Figure 1 In the checkerboard marking board shown in (a), the corner points can be the points where black and white squares intersect, or the vertices of each black / white square. For example, as... Figure 1 In the dot calibration plate shown in (b), the corner points can be the centers of each dot. For example, as... Figure 1 In the crosshair calibration plate shown in (c), the corner points can be the intersection points of all the straight lines in the chessboard grid.

[0054] (3) Calibration: In the fields of machine vision and image processing, calibration is typically used to determine the internal and external parameters of a camera or other sensor in order to accurately map pixels in an image to their actual positions in three-dimensional space. The calibration process involves photographing a calibration board with known geometry and calculating the camera's internal and external parameters using algorithms. Calibration is one of the important steps in establishing a high-precision vision system and is of great significance for achieving tasks such as accurate measurement, positioning, and reconstruction.

[0055] (4) Camera intrinsic parameters: Camera intrinsic parameters refer to the internal parameters of a camera, specifically used to describe the basic configuration and characteristics of the camera's optical system. For example, the intrinsic parameters of a camera may include focal length, principal point coordinates, pixel pitch, and distortion coefficients.

[0056] The principal point, also known as the intersection of the principal optical axis and the image plane, is usually the center point of the image. The coordinates of the principal point can be the coordinates of the principal point in the pixel coordinate system.

[0057] Pixel pitch specifically refers to the physical distance between adjacent pixels on an image sensor.

[0058] Distortion coefficients are used to describe image distortion caused by imperfections in the camera lens (such as lens shape and assembly errors). For example, distortion coefficients may include radial distortion coefficients and tangential distortion coefficients; radial distortion coefficients are mainly used to characterize distortion caused by different magnifications at different parts of the camera lens, while tangential distortion coefficients are mainly used to characterize distortion caused by the camera lens not being mounted perpendicular to the imaging plane.

[0059] In some embodiments, camera intrinsics may also include the aspect ratio of pixels, sensor size, pixel size, etc.

[0060] (5) Camera extrinsic parameters: Camera extrinsic parameters specifically refer to the external parameters of the camera, which can be parameters describing the camera's position and orientation in three-dimensional space. In some embodiments, camera extrinsic parameters can refer to the rotation and translation transformations of the camera relative to a certain coordinate system (e.g., the world coordinate system or the camera coordinate system of another camera). The rotation transformation can be represented by a rotation matrix, and the translation transformation can be represented by a translation vector. The rotation matrix is ​​usually denoted by R, and is a 3×3 matrix. Each column of the rotation matrix is ​​a unit vector, corresponding to the orientation of the x-axis, y-axis, and z-axis in the camera coordinate system within that coordinate system. The translation vector is usually denoted by t, and is a 3×1 vector that describes the position of the origin of the camera coordinate system within that coordinate system.

[0061] In some embodiments, the rotation matrix and translation vector can form the pose transformation matrix T. Specifically, the relationship between the pose transformation matrix, the rotation matrix, and the translation vector is as follows: (1)

[0062]

[0063] Here, 0 is the zero vector of 3×1, and the "1" in the lower right corner of the pose transformation matrix is ​​the scaling factor.

[0064] (6) Pixel coordinate system: The pixel coordinate system is a two-dimensional planar coordinate system. Based on the pixel coordinate system, the number of columns and rows of each pixel in an image (or captured image) can be represented. For example, refer to... Figure 2 As shown, the origin O of the pixel coordinate system s This can be the top left corner of the image, or O on the horizontal axis. s u-axis and ordinate axis O s The v-axis points directly to the right and directly downwards. For a given pixel (or pixel point) in the image, its x-coordinate in the pixel coordinate system represents its column number, and its y-coordinate represents its row number. For example, a pixel with coordinates (5,6) in the image indicates that it is the pixel in the 5th column and 6th row of the image.

[0065] (7) Image Coordinate System: The image coordinate system is a two-dimensional planar coordinate system. Based on the image coordinate system, the actual position and size of the image in physical space can be represented. (Refer to...) Figure 2 As shown, the origin O of the image coordinate system t This can be the intersection of the camera's optical axis and the imaging plane, typically the center point of the image. The horizontal axis O of the image coordinate system... t x t The x-axis and the horizontal axis of the pixel coordinate system O s The u-axis is parallel, and the vertical axis O of the image coordinate system is... t y t The vertical axis O of the pixel coordinate system s v t The axes are parallel.

[0066] In some embodiments, if it is necessary to convert coordinates in the pixel coordinate system to coordinates in the image coordinate system, the conversion can be performed based on camera intrinsic parameters.

[0067] (8) Camera Coordinate System: The camera coordinate system is a three-dimensional rectangular coordinate system. Based on the camera coordinate system, the relative positional relationship (or pose relationship) between the camera and objects in three-dimensional space can be represented. (See reference...) Figure 3 As shown, the camera coordinate system has its origin O at the optical center of the camera (the center of the lens). c With the optical axis (the straight line between the optical center and the principal point) as O c z c A three-dimensional Cartesian coordinate system is established using the axes. In the camera coordinate system, O... c x c The axis and the horizontal axis O in the image coordinate system t xt Axis parallel, O c y c The axis and the horizontal axis O in the image coordinate system t y t The axes are parallel.

[0068] (9) World Coordinate System: A reference coordinate system is selected in three-dimensional space to describe the position of the camera and the object. This coordinate system can be called the world coordinate system. The world coordinate system is a three-dimensional rectangular coordinate system, and the coordinate points on it can be used to describe the position of any point in space. In some embodiments, the world coordinate system can be freely defined according to actual usage requirements. For example, the origin of the world coordinate system can be a point on the calibration plate, and the three mutually perpendicular coordinate axes can be determined according to the specific position of the calibration plate.

[0069] During the calibration of camera intrinsic and extrinsic parameters, the relationship between the camera coordinate system and the world coordinate system can be described by the rotation transformation matrix R and the translation vector t (or by the pose transformation matrix). Therefore, the homogeneous coordinates of a point P in space in the world coordinate system and the camera coordinate system are (x1, y1, z1) and (x2, y2, z2), respectively, satisfying the following formula (2):

[0070]

[0071] (10) PNP (perspective-n-point) algorithm: The PNP algorithm is a pose estimation algorithm in computer vision used to solve the correspondence between 3D spatial points and 2D image points. This algorithm can obtain the camera pose and the pose transformation relationship (or spatial relationship or relative position relationship) between the target object and the camera by knowing the coordinates of the target object (e.g., calibration board) in 3D space and the corresponding pixel coordinates in the image of the target object taken by the camera.

[0072] To determine the intrinsic and extrinsic parameters of cameras in a multi-camera perception suite, which consists of multiple cameras, the following approaches exist in related technologies:

[0073] The first option uses, for example... Figure 4 The stereo calibration board shown is used to calibrate the camera's intrinsic and extrinsic parameters. However, this approach requires high manufacturing precision for the stereo calibration board, is costly, and has certain requirements regarding the field of view and layout of each camera in the multi-view sensing kit, making it less versatile.

[0074] The second approach involves adding an inertial measurement unit (IMU) to the calibration scheme. When constructing the calibration system, a fixed relative positional relationship is established between the cameras and the IMU in the multi-view sensing kit. Then, the relative poses between multiple cameras can be indirectly obtained using the IMU as a medium. However, in this calibration scheme, the IMU itself has low measurement accuracy, and the overall scheme relies on the IMU's six degrees of freedom (6DOF) motion, typically requiring a robotic arm or handheld operation, making it unsuitable for multi-view sensing kits on medium to large-sized devices.

[0075] The third approach is traditional stereo calibration using binocular cameras. This approach requires the two cameras to share a common field of view. However, this approach cannot be used to calibrate two cameras in a multi-view perception kit that do not share a common field of view.

[0076] It is evident that existing calibration schemes all have certain limitations and cannot be applied to the accurate calibration of camera intrinsic and extrinsic parameters in all multi-view perception kits.

[0077] To address the aforementioned problems, this application provides a camera calibration scheme that can calibrate the intrinsic and extrinsic parameters of cameras in a multi-view sensing kit. This scheme can be applied to applications such as... Figure 5 The calibration system shown. (Refer to...) Figure 5 As shown, the calibration system includes a rotating platform, an electronic device equipped with a multi-view sensing kit, and multiple calibration boards. Figure 5 (Only calibration board 1 and calibration board 2 from multiple calibration boards are used as examples) and control devices. The multi-view perception kit includes multiple cameras ( Figure 5 (This example uses only cameras 1 and 2, which do not share a common field of view among multiple cameras, as examples. Each calibration plate corresponds to one camera. For example, calibration plate 1 corresponds to camera 1, and calibration plate 2 corresponds to camera 2.)

[0078] In some embodiments, the rotating platform can rotate in the horizontal direction with a rotation accuracy of 0.001°.

[0079] In some embodiments, the electronic device in this application may consist only of a multi-view sensing kit, i.e., the electronic device may be the multi-view sensing kit itself.

[0080] Reference Figure 5 As shown, the electronic device is fixedly mounted on a rotating platform, which can rotate around a central axis perpendicular to the ground. After the electronic device is fixed, a corresponding calibration plate can be placed within the shooting area of ​​each camera in the multi-view sensing kit on the electronic device, so that each camera can capture an image of a calibration plate. Figure 5As shown, calibration plate 1 can exist in the shooting area of ​​camera 1, and calibration plate 2 can exist in the shooting area of ​​camera 2. In this way, in the initial state, each camera in the multi-view perception kit can capture the complete calibration pattern on its corresponding calibration plate.

[0081] Furthermore, the electronic devices and rotating platform can all establish wired or wireless communication connections with the control device. In some embodiments, each camera in the multi-view sensing kit and the rotating platform can also establish wired or wireless communication connections with the control device.

[0082] In some embodiments, the control device may also be the electronic device itself. In this case, the rotating platform may establish a wired or wireless communication connection with the electronic device to facilitate the control device to control the rotating platform.

[0083] Based on the above calibration system, the camera calibration scheme provided in this application refers to... Figure 6 As shown, the internal parameters of each camera in the multi-view perception kit can first be determined using any intrinsic parameter calibration method. Then, the control device can introduce rotation angle information through the rotating platform. Thus, the pose relationship between the calibration plate and the ground is fixed, the pose relationship between the camera and the rotating platform is fixed, and the pose relationship between the rotating platform and the ground is determined only by the rotation angle. Based on this, the pose relationships between each calibration plate and the corresponding camera before and after the rotating platform rotation can be solved (specifically represented by a pose transformation matrix). Based on the pose relationships between each calibration plate and the corresponding camera before and after the rotating platform rotation, and the pose relationships between the ground and the rotating platform before and after rotation, the pose relationship between the ground and each calibration plate can be calculated based on the fixed pose relationship between the camera and the rotating platform and the matrix transformation principle. Furthermore, the pose relationships between each calibration plate can be derived. Finally, using the pose relationships between each calibration plate as intermediate quantities for pose transformation between different cameras, the pose relationships between each camera can be calculated as external parameters between different cameras in the multi-view perception kit.

[0084] The specific execution of the calibration scheme provided in this application can be completed by the control equipment in the calibration system.

[0085] As can be seen, in this calibration scheme, by introducing rotation angle information through a rotating platform, the internal and external parameters of each camera in the multi-view perception kit can be determined by combining the pure visual information from the images captured by each camera. Pure visual information contains feature points with good stability and uniqueness; therefore, this calibration scheme can make the final internal and external parameters of the cameras more accurate. Furthermore, as long as there are enough feature points in the pure visual information in different scenarios, the calibration objective can be successfully achieved. This makes the calibration scheme more adaptable. Simultaneously, since this calibration scheme does not determine the external parameters between cameras based on the differences in images of the same object captured by different cameras, the calibration scheme provided in this application can successfully obtain the external parameters between different cameras in the multi-view perception kit regardless of whether there is a shared viewing area between the different cameras. Furthermore, the calibration scheme provided in this application does not impose strong constraints on the position of the calibration board; it only needs to be adapted to the layout of the cameras in the multi-view perception kit, and there are no specific limitations on the style of the calibration board. In summary, the technical solution provided in this application can accurately calibrate the internal and external parameters of cameras in various scenarios (such as production lines or repair points) regardless of whether there is a shared viewing area between different cameras in the multi-view perception kit. Compared with existing camera calibration solutions for multi-view perception kits, it has higher adaptability.

[0086] The technical solutions provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0087] The technical solution provided in this application can be applied to, for example... Figure 5 The calibration system shown can be described in detail in the foregoing embodiments for its specific components. Figure 5 The relevant statements will not be repeated here.

[0088] For example, the electronic devices used in the above-mentioned calibration system can be mobile phones, tablets, handheld computers, personal computers (PCs), ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices, and / or smart city devices, etc., equipped with multi-view perception kits. This application embodiment does not impose any special restrictions on the specific type of electronic device.

[0089] For example, taking a mobile phone as an electronic device, Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown.

[0090] Reference Figure 7 As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a display screen 193, a subscriber identification module (SIM) card interface 194, and a camera 195, etc. The sensor module 180 may include pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0091] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0092] A controller can be the nerve center and command center of an electronic device. Based on command opcodes and timing signals, the controller generates operation control signals to complete the control of command retrieval and execution.

[0093] The processor 110 may also include a memory for storing commands and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store commands or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the command or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0094] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0095] The charging management module 140 is used to receive charging input from a power supply device (such as a charger, laptop power supply, etc.). The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device.

[0096] While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141. Specifically, the battery 142 can be composed of multiple batteries connected in series. The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110.

[0097] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 193, camera 195, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery voltage, current, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110.

[0098] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0099] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM). The RAM can be directly read and written by the processor 110 and can be used to store executable programs (e.g., machine commands) of the operating system or other running programs, as well as user and application data. The NVM can also store executable programs and user and application data, and can be pre-loaded into the RAM for direct read and write operations by the processor 110.

[0100] A touch sensor, also known as a "touch device," can be located on the display screen 193. The touch sensor and the display screen 193 together form a touchscreen, also called a "touchscreen." The touch sensor detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 193. In other embodiments, the touch sensor may also be located on the surface of the electronic device, in a different position than the display screen 193.

[0101] An ambient light sensor is used to detect ambient light intensity. A pressure sensor is used to sense pressure signals and can convert these signals into electrical signals. In some embodiments, the pressure sensor may be located on the display screen 193. There are many types of pressure sensors, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors.

[0102] In some embodiments, an electronic device may include one or N cameras 195, where N is a positive integer greater than 1. In this application embodiment, the type of camera 195 can be distinguished based on hardware configuration and physical location. For example, a camera located on the side of the electronic device's display screen 193 can be called a front-facing camera, and a camera located on the side of the electronic device's back cover can be called a rear-facing camera; another example is that a camera with a short focal length and a wide field of view can be called a wide-angle camera, while a camera with a long focal length and a narrow field of view can be called a regular camera. Here, focal length and field of view are relative concepts and are not specifically limited by parameters. Therefore, wide-angle cameras and regular cameras are also relative concepts, and can be specifically distinguished based on physical parameters such as focal length and field of view.

[0103] In this embodiment of the application, the combination of multiple cameras included in the electronic device can be called a multi-view perception kit, and each camera can be referred to as a camera.

[0104] The electronic device implements display functions through a GPU, a display screen 193, and an application processor. The GPU is a microprocessor for image editing, connected to the display screen 193 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program commands to generate or modify display information.

[0105] Electronic devices can achieve shooting functions through ISP, camera 195, video codec, GPU, display 193, and application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program commands to generate or modify display information.

[0106] The Information Service Provider (ISP) is used to process data fed back from the camera 195. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise and brightness. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be integrated into the camera 195. The camera 195 is used to capture still images or videos.

[0107] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when an electronic device is selecting a frequency, a DSP can perform a Fourier transform on the frequency energy.

[0108] Video codecs are used to compress or decompress digital video. Electronic devices can support one or more video codecs. This allows the electronic device to play or record video in various encoded formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0109] Display screen 193 is used to display images, videos, etc. Display screen 193 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include one or N displays 193, where N is a positive integer greater than 1.

[0110] In this embodiment of the application, the display screen 193 can be used to display pages required by the electronic device (e.g., a page displaying captured images, etc.), and to display images captured by any one or more cameras 195 in the interface.

[0111] The wireless communication function of electronic devices can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem, and baseband processor.

[0112] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in an electronic device can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0113] The mobile communication module 150 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for use in electronic devices. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 can be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 can be housed in the same device.

[0114] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 193. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0115] The wireless communication module 160 can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0116] In some embodiments, antenna 1 of the electronic device is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling the electronic device to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TDSCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0117] The SIM card interface 194 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 194 to make contact with and detach from the electronic device. The electronic device can support one or more SIM card interfaces. The SIM card interface 194 supports Nano SIM cards, Micro SIM cards, and other SIM cards. Multiple cards can be inserted into the same SIM card interface 194 simultaneously. The SIM card interface 194 is also compatible with external memory cards. The electronic device interacts with the network through the SIM card to achieve functions such as calls and data communication. One SIM card corresponds to one user number.

[0118] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0119] Of course, it is understandable that the above... Figure 7 The illustration shown is merely an example when the electronic device is in the form of a mobile phone. If the electronic device is in the form of a tablet, handheld computer, PC, PDA, wearable device (such as a smartwatch, smart bracelet), or other similar device, the structure of the electronic device may include more advanced features. Figure 7 The fewer structures shown can also include more than Figure 7 The structures shown are not limited here.

[0120] For example, the control device provided in this application may be a single server, a server cluster consisting of multiple servers, or a cloud computing service center. This application does not impose any specific limitations on this.

[0121] For example, taking the control device as a server, Figure 8 A schematic diagram of a server structure is shown. (Refer to...) Figure 8 As shown, the server includes one or more processors 801, a communication line 802, and at least one communication interface. Figure 8 (This is merely an example illustration of a communication interface 803 and a processor 801; optionally, a memory 804 may also be included.)

[0122] The processor 801 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program corresponding to the scheme of this application.

[0123] Communication line 802 may include a communication bus for communication between different components.

[0124] The communication interface 803 can be a transceiver module used to communicate with other devices (such as electronic devices equipped with multi-view sensing kits, rotating platforms, etc.) or communication networks, such as Ethernet, RAN, wireless local area networks (WLAN), etc. For example, the transceiver module can be a transceiver or a similar device. Optionally, the communication interface 803 can also be a transceiver circuit located within the processor 801, used to implement the processor's signal input and signal output.

[0125] The memory 804 can be a device with storage functionality. For example, it can be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; electrically erasable programmable read-only memory (EEPROM); compact disc read-only memory (CD-ROM) or other optical disc storage; optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.); magnetic disk storage media or other magnetic storage devices; or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited to these. The memory can exist independently and be connected to the processor via communication line 802. The memory can also be integrated with the processor.

[0126] The memory 804 stores computer execution instructions for implementing the scheme of this application, and the processor 801 controls the execution. The processor 801 executes the computer execution instructions stored in the memory 804, thereby implementing the camera calibration method provided in the embodiments of this application.

[0127] Optionally, the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0128] In a specific implementation, as one example, the processor 801 may include one or more CPUs, for example... Figure 8 CPU0 and CPU1 in the CPU.

[0129] In a specific implementation, as one example, the server may include multiple processors, for example... Figure 8The processors 801 and 807 are described herein. Each of these processors may be a single-core processor or a multi-core processor. The processors herein may include, but are not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, and other computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing.

[0130] In a specific implementation, as one embodiment, the server may further include an output device 805 and an input device 806. The output device 805 communicates with the processor 801 and can display information in various ways. For example, the output device 805 may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 806 communicates with the processor 801 and can receive user input in various ways. For example, the input device 806 may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0131] The server described above can be a general-purpose device or a dedicated device. For example, the server can be a desktop computer, laptop, network server, PDA (personal digital assistant), mobile phone, tablet computer, wireless terminal device, embedded device, the aforementioned terminal device, the aforementioned network device, or something else with... Figure 8 Devices with similar structures. This application does not limit the type of server to specific embodiments.

[0132] The technical solutions provided in the embodiments of this application can all be implemented in electronic devices and control devices with the above-described hardware architecture, or in a calibration system composed of electronic devices and control devices with the above-described hardware architecture.

[0133] Based on the above Figure 5 The calibration system shown below, in conjunction with Figure 9 As shown, the processing flow of the camera calibration method provided in the embodiments of this application will be introduced. Figure 9 This is a schematic flowchart illustrating a camera calibration method provided in an embodiment of this application. This calibration method can be executed by a control device within a calibration system. (Refer to...) Figure 9As shown, the camera calibration method may include S901-S908:

[0134] S901, The control device determines the internal parameters of each camera in the multi-view sensing kit on the electronic device.

[0135] For example, the camera's internal parameters may include: focal length, principal point coordinates, pixel pitch, and distortion coefficients. The definitions of these internal parameters can be found in the glossary of terms in the foregoing embodiments, and will not be repeated here.

[0136] In one possible implementation, if the Zhang Zhengyou calibration method is used to calibrate the camera's internal parameters, then S901 may specifically include the following steps:

[0137] S1. The control device controls the rotating platform to rotate multiple times, and after each rotation, controls the target camera to capture images of the target calibration plate to obtain multiple target images; the target images include some or all of the target calibration patterns.

[0138] The target camera is one of the multiple cameras included in the multi-view perception kit, the target calibration board is the calibration board corresponding to the target camera, and the target calibration pattern is the calibration pattern in the target calibration board. For example, the target calibration image can be a checkerboard image, an origin image, a cross image, etc., depending on the target calibration board.

[0139] In this embodiment, the control device controls the rotation of the rotating platform by sending multiple rotation commands to the platform, each command instructing the platform to rotate by a certain angle. After each rotation command is sent, and once the platform has completed its rotation, a shooting command can be sent to the target camera. The shooting command instructs the target camera to capture a target image and return it to the control device.

[0140] In some embodiments, after the control device sends a rotation command to the rotating platform, it can determine that the rotating platform has completed its rotation if it receives a rotation completion response from the rotating platform. Of course, the control device can also use any other possible method to determine whether the rotating platform has completed its rotation, and this application does not impose any specific limitations on this.

[0141] To facilitate the subsequent determination of intrinsic parameters by obtaining sufficient corner information from the target image, the target calibration pattern within the image needs to be as complete as possible. To achieve this completeness, the rotation angle of the rotating platform needs to be very small each time, ensuring that the target calibration pattern on the calibration plate can be completely captured by the target camera. Taking three rotations of the rotating platform as an example, the three target images captured by the target camera can be sequentially... Figure 10 As shown in (a)-(c).

[0142] Each time the target camera captures an image of the target, it can send that image to the control device. The target camera can also send all captured images to the control device only after the control device has ceased controlling the rotation of the rotating platform. In this case, the shooting command sent by the control device to the target camera can be used only to instruct the target camera to capture target images. When the control device no longer needs to control the rotation of the rotating platform, it can send an image acquisition request to the target camera, causing the target camera to send all target images to the control device. This image acquisition request can then be used to instruct the target camera to send all target images to the control device.

[0143] In some embodiments, the control device can first instruct the target camera to activate the recording function, and then control the rotating platform to rotate multiple times. When the rotating platform rotation is not required, a recording request command can be sent to the target camera to instruct the target camera to send the recorded target video during the rotation process to the control device. This recording request command can be used to instruct the target camera to send the recorded video to the control device. After acquiring the target video, the control device can use any possible image recognition algorithm to obtain multiple target images in the target video. The positions of the calibration plates included in different target images are different.

[0144] S2. The control device determines the pixel coordinates of multiple corner points in the target calibration pattern in each target image.

[0145] Among them, corner points can be feature points with significant characteristics in the target calibration pattern.

[0146] S2 may specifically include the following steps:

[0147] (1) First, the control device can use any feasible algorithm / method to determine all corner points in the target calibration pattern in the target image.

[0148] In one possible implementation, the corner points of the target calibration pattern in the target image can be determined using the following method:

[0149] When the target calibration image is a checkerboard pattern, determining the corner points of the target calibration pattern includes: setting a sliding window in the target calibration pattern; performing an offset operation on the sliding window; and determining the feature value group of each pixel within the sliding window. The feature value group includes a first feature value and a second feature value. The first and second feature values ​​characterize the grayscale changes of the pixel to which the feature value group belongs in two mutually perpendicular directions after the offset operation. The direction of grayscale change corresponding to the first feature value is the direction of the fastest grayscale change, and the direction of grayscale change corresponding to the second feature value is the direction of the smallest grayscale change. If the first target feature value is greater than a first preset threshold and the second target feature value is less than a second preset threshold, the pixel to which the feature value group belongs is determined as an edge point (i.e., a point on the edge of the target calibration pattern). The first preset threshold is greater than the second preset threshold, the first target feature value is the larger of the first and second feature values, and the second target feature value is the smaller of the first and second feature values. If both the first and second feature values ​​are less than the second preset threshold, and the difference between the first and second feature values ​​is less than the third preset threshold, the pixel belonging to the feature value group is determined as an intra-cell point (i.e., a point within a small cell in the target calibration pattern). If both the first and second feature values ​​are greater than the first preset threshold, and the difference between the first and second feature values ​​is less than the third preset threshold, the pixel belonging to the feature value group is determined as a corner point.

[0150] When the target calibration image is a crosshair, determining the corner points of the target calibration pattern includes: setting a sliding window in the target image; performing an offset operation on the sliding window and determining the feature value group of each pixel within the sliding window; the feature value group includes a first feature value and a second feature value. If the first target feature value is greater than a first preset threshold and the second target feature value is less than a second preset threshold, the pixel to which the feature value group belongs is determined as an edge point (i.e., a point on the edge of the target calibration pattern). If both the first and second feature values ​​are greater than the first preset threshold, and the difference between the first and second feature values ​​is less than a third preset threshold, the pixel to which the feature value group belongs is determined as a corner point.

[0151] It should be noted that the above method for determining corner points requires converting the target image to grayscale first.

[0152] When the target image is an array of dots, extracting the corner points of the target image includes taking the center of the dots in the target image as the corner point of the target image.

[0153] Of course, the above-described method for determining the corner points of the target calibration pattern in the target image is only an example. In practice, any other feasible method can be used, and this application does not impose any specific restrictions on it.

[0154] (2) Furthermore, too many corner points can lead to excessive computation in subsequent intrinsic parameter calculations, and there is a high probability that too many corner points will result in unreliable, duplicate, or invalid intersections. Therefore, after identifying all corner points in the target image and deleting duplicate corner points, any feasible filtering conditions can be used to filter out corner points with low reliability. For example, if the distance from a corner point to the target calibration pattern is less than a certain set value, then that corner point is filtered out. Another example is that if the distance between two corner points is less than a preset threshold, then one of the corner points in that chain is deleted.

[0155] Of course, in practice, the method of filtering corner points can be any other possible implementation, and this application does not impose any specific restrictions on it.

[0156] (3) After determining all the corner points in the target calibration pattern, a pixel coordinate system can be established on the target image to obtain the pixel coordinates of each corner point.

[0157] In addition, the actions of establishing a pixel coordinate system and determining the pixel coordinates of corner points can also be performed synchronously during execution (1), that is, a pixel coordinate system is established at the same time as all corner points are determined in order to obtain the pixel coordinates of the corner points.

[0158] S3. The control device calculates the internal parameters of the target camera based on the pixel coordinates of multiple corner points of the target calibration object in all target images, and the three-dimensional coordinates of each corner point in the world coordinate system of the target calibration plate.

[0159] The world coordinate system of the target calibration board can be customized by the user according to requirements. Once defined, the world coordinate system will not change when the calibration scheme provided in this application is executed. After the world coordinate system is determined, the corresponding three-dimensional coordinates of each point on all calibration boards can be determined. In some embodiments, the world coordinate system can also be called the calibration board coordinate system.

[0160] When executing S3, the control device can specifically calculate the internal parameters of the target camera based on the pixel coordinates of multiple corner points of the target calibration object in all target images, and the three-dimensional coordinates of each corner point in the world coordinate system of the target calibration plate, using a minimum reprojection error algorithm.

[0161] Specifically, after obtaining the pixel coordinates and 3D coordinates of the corner points, the first step is to construct and solve for the homography matrix. This homography matrix represents the mapping relationship from the world coordinate system to the pixel coordinate system. Next, the rotation matrix and translation vector can be decomposed from the homography matrix. These matrix and vector represent the magnitude of the rotation and translation transformations required when transforming from the world coordinate system to the pixel coordinate system. Then, based on the rotation matrix and translation vector, a remainder equation for the intrinsic parameter matrix of the target camera can be established. Solving these constraint equations yields the intrinsic parameter matrix of the target camera. This matrix can include internal parameters such as focal length, principal point coordinates, and pixel pitch. Based on the obtained intrinsic parameter matrix, algorithms such as reprojection error can be used to progressively optimize and obtain the distortion coefficients. Finally, by minimizing the reprojection error, the intrinsic parameter matrix and distortion coefficients can be continuously adjusted to obtain more accurate intrinsic parameters of the target camera.

[0162] After performing steps S1-S3 as described above for each camera in the multi-view sensing kit, the internal parameters of each camera in the multi-view sensing kit can be obtained. Of course, after performing S1, multiple target images corresponding to each camera in the multi-view sensing kit can be obtained. Subsequently, steps S2 and S3 can be performed for each camera.

[0163] Based on the technical solutions corresponding to S1-S3 above, the control device can control the rotation of the rotating platform and control the camera to take pictures, thereby quickly obtaining multiple target images from each camera. Based on these target images, the intrinsic parameters of each camera in the multi-view perception kit can be accurately determined.

[0164] It should be noted that the methods for determining camera intrinsic parameters disclosed in S1-S3 above are merely examples, and any other feasible methods can be used in practice. This application does not impose any specific restrictions on them.

[0165] Furthermore, S901 may not be the first step executed in the camera calibration scheme provided in this application, or it may be executed at any other possible time, as long as it is executed before the camera's internal parameters are needed later (e.g., step S903).

[0166] The calibration scheme provided in this application requires determining not only the internal parameters of each camera in the multi-view perception kit, but also the external parameters (specifically, the pose transformation matrix) between any two cameras. In this calibration system, the pose transformation matrix between any two cameras can be obtained by converting the pose transformation matrix between the calibration boards corresponding to those two cameras into the pose transformation relationship between each of the two cameras and its corresponding calibration board. Furthermore, the pose transformation matrix between the calibration boards corresponding to any two cameras can be obtained by converting the pose transformation matrix between these two calibration boards and the ground.

[0167] Furthermore, since the relative position between the calibration plate and the ground is fixed throughout the calibration system, the pose transformation matrix between the calibration plate and the ground is also fixed. In other words, the pose transformation matrix between the calibration plate and the ground remains unchanged regardless of whether the rotating platform is before or after rotation. Simultaneously, the relative position between the rotating platform and each camera is also fixed; that is, the pose transformation matrix between the rotating platform and the cameras is the same regardless of whether the rotating platform is before or after rotation.

[0168] Furthermore, regardless of whether the rotation is before or after, the pose transformation matrix between the rotating platform and the camera can be derived from the pose transformation matrix between the rotating platform and the ground, the pose transformation matrix between the ground and the calibration plate, and the pose transformation matrix between the calibration plate and the corresponding camera. Specifically, if the three-dimensional coordinate system (specifically a three-dimensional Cartesian coordinate system) between the rotating platform and the ground is set consistently in the initial state, then in the initial state, the rotation matrix in the pose transformation matrix between the rotating platform and the ground is an identity matrix, and the translation vector is 0.

[0169] Furthermore, after the initial state, the rotation angle is known for each rotation. Therefore, the difference between the pose transformation matrix between the rotating platform and the ground after each rotation and the pose transformation matrix between the rotating platform and the ground before the rotation is only the transformation of the rotation angle in the rotation matrix.

[0170] For example, taking a rotating platform that has rotated only by a preset angle in the horizontal direction, the positional relationship between the rotating platform and the ground in the initial three-dimensional Cartesian coordinate system, and between the rotating platform and the ground in the three-dimensional Cartesian coordinate system after the rotating platform has rotated by the preset angle, can be referred to... Figure 11As shown in the diagram. Base-Y, Base-X, and Base-Z are the axes of the ground coordinate system (the ground's three-dimensional Cartesian coordinate system). tt-Y, tt-X, and tt-Z are the axes of the rotating platform's coordinate system (the rotating platform's three-dimensional Cartesian coordinate system) in its initial state. tt'-Y, tt'-X, and tt'-Z are the axes of the rotating platform's coordinate system after rotating by a preset angle. It can be seen that before rotation, the rotating platform and the ground share a common three-dimensional Cartesian coordinate system, and their pose transformation matrices can consist of only an identity matrix. Horizontal rotation only changes two axes in the rotating platform's three-dimensional Cartesian coordinate system. Based on the rotation angle, the transformation matrix between the rotating platform's three-dimensional Cartesian coordinate system and the ground's three-dimensional Cartesian coordinate system after rotation can be easily obtained, thus yielding the pose transformation matrix between the rotating platform and the ground after rotation.

[0171] Furthermore, the world coordinate system or calibration plate coordinate system in which the calibration plate is located is unaffected by rotation. For the camera coordinate system, the positional relationship between its vertical axis and Base-Y and tt-Y is unaffected by rotation; the other two axes are affected by rotation in the same way as the rotation platform coordinate system. Figure 11 (The coordinate system of the rotated calibration plate is not shown in the image).

[0172] Based on the above, the pose transformation matrices between the rotating platform and the ground before and after the platform rotation can be considered known. To obtain the pose transformation matrix between the ground and each calibration plate, it is necessary to obtain the pose transformation matrices between each camera and its corresponding calibration plate before and after rotation.

[0173] Having obtained the pose transformation matrix between each camera and its corresponding calibration board before and after rotation, the pose transformation matrix between the ground and each calibration board, the pose transformation matrix between any two calibration boards, and the pose transformation matrix between any two cameras can be obtained sequentially according to the above logic. Using the initial state as before rotation is a less computationally intensive approach. Based on this, after S901, it is necessary to first determine the pose transformation matrix between each camera in the multi-view perception kit and its corresponding calibration board in the initial state, i.e., execute S902 and S903.

[0174] S902. In the initial state, the control device controls each camera in the multi-view perception kit to capture images, obtains the initial calibration images corresponding to each camera, and determines the pixel coordinates of the first corner point of the calibration pattern in each initial calibration image.

[0175] The initial state refers to the preliminary setup of the calibration system. In this initial state, the entire calibration board corresponding to each camera in the multi-view sensing kit exists within its shooting area. Controlling the camera to capture an image at this time will result in an initial calibration image containing the complete calibration pattern from the corresponding calibration board for that camera. Alternatively, the control device can control each camera in the multi-view sensing kit to capture images of its corresponding calibration board.

[0176] In some implementations, if during the execution of S901, in the initial state, each camera in the multi-view perception kit has captured a target image of the corresponding calibration board, then S902 can directly use the target images of each camera in the initial state as the initial calibration images of each camera. For example, the target image of the first camera in the initial state under multi-view perception conditions can be used as the initial calibration image of the first camera; and the target image of the second camera in the initial state under multi-view perception conditions can be used as the initial calibration image of the second camera.

[0177] The specific implementation of determining the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image can be referred to the relevant description after S2 in the aforementioned embodiments, and will not be repeated here. Furthermore, if during the execution of S901, in the initial state, each camera in the multi-view perception kit has captured a target image of the corresponding calibration board and returned it to the control device, and the control device has determined the pixel coordinates of the corner point of the target calibration pattern in the target image, then in S902, the pixel coordinates of the corner point of the target calibration pattern in the target images of each camera in the initial state can be used as the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image of each camera. At this time, the corner point of the target calibration pattern is the same as the first corner point of the calibration pattern in the initial calibration image.

[0178] In some embodiments, the control device's control of the electronic device can be accomplished through instructions; based on this, combined with... Figure 9 , refer to Figure 12 As shown, S902 may specifically include S1001-S1003:

[0179] S1001, The control device sends the first instruction to the electronic device.

[0180] The first instruction is used to instruct the electronic device to control each camera in the multi-view perception kit to take pictures and send the captured initial calibration images to the control device.

[0181] S1002. The electronic device receives the first instruction, responds to the first instruction, controls each camera in the multi-view perception kit to capture an initial calibration image, and sends the initial calibration images captured by each camera to the control device.

[0182] S1003. The control device receives the initial calibration images captured by each camera of the electronic device and determines the pixel coordinates of the first corner point of the calibration pattern in each initial calibration image.

[0183] In S1003, the specific implementation of the control device determining the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image can be referred to the relevant description after S2 in the aforementioned embodiment, and will not be repeated here.

[0184] Based on the technical solutions corresponding to S1001-S1003 above, the control device can control the camera in the electronic device through instructions, thereby obtaining the initial calibration images captured by each camera in the multi-view perception kit of the electronic device and the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image, thus providing a data basis for the execution of subsequent solutions.

[0185] S903, the control device determines the pose transformation matrix between each camera in the multi-view sensing kit and the corresponding calibration board in the initial state based on the internal parameters of each camera in the multi-view sensing kit, the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image of each camera, and the three-dimensional coordinates of the first corner point on the corresponding calibration board.

[0186] The calibration board corresponding to the first corner point is the calibration board to which the initial calibration image belongs, and also the calibration board corresponding to the camera to which the initial calibration image belongs. In other words, the first corner point of the calibration pattern in the initial calibration image corresponding to a certain camera is the same as the corner point on the calibration board corresponding to that camera. The three-dimensional coordinates of the first corner point on the corresponding calibration board refer to the coordinates of the first corner point in the world coordinate system (or calibration board coordinate system) where the calibration board is located.

[0187] Specifically, S903 refers to determining the pose transformation matrix between any camera and its corresponding calibration board in the initial state for any camera in the multi-view perception kit, based on the internal parameters of any camera, the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image of any camera, and the three-dimensional coordinates of the first corner point on the corresponding calibration board.

[0188] Taking the first camera among the multiple cameras included in the multi-view perception kit as an example, in one possible implementation, S902 can specifically determine the pose transformation matrix between the first camera and the first calibration board in the initial state based on the internal parameters of the first camera, the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image of the first camera, and the three-dimensional coordinates of the first corner point on the first calibration board, using the PNP algorithm. Here, the first calibration board is the first calibration board corresponding to the first camera, and also the calibration board corresponding to the first corner point. The first corner point of the calibration pattern in the initial calibration image is the same as the corner point of the calibration pattern on the first calibration board.

[0189] Determining the pose transformation matrix between the first camera and the first calibration board in the initial state using the PNP algorithm may specifically include the following steps:

[0190] (1) Based on the internal parameters of the first camera, the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image of the first camera are converted into two-dimensional coordinates in the image coordinate system.

[0191] Specifically, the pixel coordinates can be corrected using distortion coefficients to eliminate the influence of distortion on the coordinates. For example, taking the distortion coefficients as including radial distortion coefficients and tangential distortion coefficients, the elimination of distortion effects can be accomplished according to the following formulas (3)-(6):

[0192] x j =x(1+k1r) 2 +k2r 4 +k3r 6 (3)

[0193] y j =y(1+k1r 2 +k2r 4 +k3r 6 (4)

[0194] Where (x, y) are the pixel coordinates after radial distortion correction, (x j x j ) represents the pixel coordinates without radial distortion correction, k1, k2, and k3 are the radial distortion coefficients, and r 2 =x 2 +y 2 .

[0195] x q =x+[2p1xy+p2(r) 2 +2x 2 (5)

[0196] y q =y+[p1(r 2 +2y 2 )+2p2xy];(6)

[0197] Where (x, y) are the pixel coordinates after tangential distortion is eliminated, (x q x q ) represents the pixel coordinates without tangential distortion correction, and p1 and p2 are the tangential distortion coefficients.

[0198] Then, using focal length, pixel spacing, and principal point coordinates, the distortion-free pixel coordinates are converted into two-dimensional coordinates in the image coordinate system. The conversion process can be performed using the following formulas (7) and (8):

[0199]

[0200] Where (x, y) are the two-dimensional coordinates in the transformed image coordinate system, (u, v) are the pixel coordinates in the pixel coordinate system, and (cx, cy) are the principal point coordinates. f is the focal length, and dx is the pixel spacing in the horizontal direction. dy is the pixel spacing in the vertical direction, and K is the intrinsic parameter matrix.

[0201] (2) Using the PNP algorithm, the three-dimensional coordinates of the first corner point on the corresponding calibration plate and the two-dimensional coordinates of the first corner point of the calibration pattern in the initial calibration image in the image coordinate system are processed to obtain the pose transformation matrix between the first camera and the first calibration plate in the initial state.

[0202] The PNP algorithm can be any possible PNP algorithm, such as EPnP or P3P.

[0203] After performing steps (1) and (2) above on each camera in the multi-view perception kit, the pose transformation matrix between each camera in the multi-view perception kit and the corresponding calibration board in the initial state can be obtained.

[0204] After obtaining the material transformation matrix between each camera and the corresponding calibration plate in the initial state, we can further obtain the material transformation matrix between each camera and the corresponding calibration plate after rotation, i.e., execute S904 and S905.

[0205] S904: After the control device controls the rotating platform to rotate by a preset angle from its initial state, it controls each camera in the multi-view sensing kit to capture images, obtaining the rotational calibration images corresponding to each camera, and determining the pixel coordinates of the second corner point of the calibration pattern in the rotational calibration image. In this application, the purpose of rotating the platform is to obtain the corresponding calibration board image captured by each camera after rotation. Therefore, in S904, the preset angle should not be too large, for example, it can be 5° or other angles that ensure that most or all of the calibration pattern of the calibration board can be captured by the corresponding camera after rotation. Furthermore, the control device can specifically control the rotating platform to rotate in one direction by a preset angle from its initial state.

[0206] Based on this, controlling the shooting of each camera in the multi-view sensing kit in S904 can be done by controlling each camera in the multi-view sensing kit to shoot the calibration board corresponding to each camera.

[0207] In some embodiments, if during the execution of S901, each camera in the multi-view perception kit captures a target image of the corresponding calibration board while the rotating platform rotates by a small angle (e.g., a preset angle), then S904 can directly use the target image as the rotation calibration image corresponding to the corresponding camera in S904.

[0208] The pixel coordinates of the second corner point of the calibration pattern in the rotation calibration image can be determined by referring to the relevant description after S2 in the aforementioned embodiments, and will not be repeated here. Furthermore, if during the execution of S901, each camera in the multi-view perception kit captures a target image of the corresponding calibration board while the rotating platform rotates by a small angle (e.g., a preset angle) and determines the pixel coordinates of the corner point of the target calibration pattern in the target image, then in S904, the pixel coordinates of the corner point of the target calibration pattern in the target image can be used as the pixel coordinates of the second corner point of the calibration pattern in the rotation calibration image corresponding to the corresponding camera.

[0209] In some embodiments, the control device controls the rotating platform and electronic equipment by sending corresponding commands. Based on this, combined with Figure 9 , refer to Figure 12 As shown, S904 can specifically include S1004-S1009:

[0210] S1004. The control equipment sends a second command to the rotating platform.

[0211] The second instruction is used to instruct the rotating platform to rotate by a preset angle from its initial state. Specifically, it can instruct the rotating platform to rotate by a preset angle in one direction from its initial state.

[0212] S1005. The rotating platform receives the second instruction and, in response to the second instruction, rotates by a preset angle based on the initial state.

[0213] The subsequent control actions of the electronic equipment by the control device need to be performed after the rotating platform has completed its rotation. Therefore, after the rotating platform has completed its rotation, it needs to send relevant information to the control device to notify it that the rotation has been completed. Based on this, S1006 is executed after S1005.

[0214] S1006. When the rotating platform completes the rotation of the preset angle from the initial state, it sends the first response to the control device.

[0215] The first response is used to indicate that the rotating platform has completed its rotation.

[0216] S1007. The control device receives the first response and, in response to the first response, sends a third instruction to the electronic device.

[0217] The third instruction is used to instruct the electronic device to control each camera in the multi-view perception kit to capture images, obtain the rotation calibration images corresponding to each camera, and send the rotation calibration images corresponding to each camera to the control device.

[0218] S1008. The electronic device receives a third instruction, responds to the third instruction, controls each camera in the multi-view perception kit to capture images, and sends the rotation calibration images captured by each camera to the control device.

[0219] S1009. The control device receives rotation calibration images corresponding to each camera from the electronic device and determines the pixel coordinates of the second corner point of the calibration pattern in each rotation calibration image.

[0220] In S1009, the specific implementation of the control device determining the pixel coordinates of the second corner point of the calibration pattern in the rotation calibration image can be referred to the relevant description after S2 in the aforementioned embodiment, and will not be repeated here.

[0221] Based on the technical solutions corresponding to S1004-S1009 above, the control device can control the rotating platform and electronic equipment through instructions, thereby obtaining the rotation calibration images captured by each camera in the multi-view perception kit of the electronic equipment when the rotating platform rotates at a preset angle, as well as the pixel coordinates of the second corner point of the calibration pattern in the rotation calibration images, thus providing a data basis for the execution of subsequent solutions.

[0222] S905, the control device determines the pose transformation matrix between each camera in the multi-view perception kit and its corresponding calibration plate when the rotating platform rotates at a preset angle, based on the internal parameters of each camera in the multi-view perception kit, the pixel coordinates of the second corner point in the rotation calibration image of each camera, and the three-dimensional coordinates of the second corner point on the corresponding calibration plate.

[0223] The calibration board corresponding to the second corner point is the calibration board to which the calibration image belongs in the rotated calibration image, and also the calibration board corresponding to the camera to which the rotated calibration image belongs. In other words, the second corner point of the calibration pattern in the rotated calibration image of a certain camera is the same as the corner point on the calibration board corresponding to that camera.

[0224] Specifically, S905 refers to the pose transformation matrix between any camera and its corresponding calibration plate, which is determined based on the internal parameters of any camera, the pixel coordinates of the second corner point of the calibration pattern in the rotation calibration image of any camera, and the three-dimensional coordinates of the second corner point on the corresponding calibration plate, when the rotation platform rotates by a preset angle.

[0225] The specific implementation of S905 can be found by referring to the relevant description obtained after S903 in the aforementioned embodiments, and will not be repeated here.

[0226] After obtaining the pose transformation matrix between each camera and its corresponding calibration plate before and after the rotation of the rotating platform, the control device can further calculate the pose transformation matrix between the ground and each calibration plate, based on the characteristic that the relative positions between the rotating platform and the camera do not change before and after rotation. That is, S906 is executed.

[0227] S906. The control device calculates the pose transformation matrix between the ground and each calibration plate based on the pose transformation matrix between each camera and its corresponding calibration plate in the initial state and the pose transformation matrix between each camera and its corresponding calibration plate when the rotating platform rotates by a preset angle.

[0228] The following section uses the example of a multi-view perception kit including a first camera, with the calibration board corresponding to the first camera being the first calibration board, to introduce the specific implementation of the S906:

[0229] First, since the relative position between the rotating platform and the first camera does not change before and after rotation, the following formula (9) holds:

[0230]

[0231] in, Let the initial state be the pose transformation matrix between the rotating platform and the first camera. The pose transformation matrix between the rotating platform and the first camera when the rotating platform rotates by a preset angle.

[0232] Secondly, based on the transformation relationship of the pose transformation matrix, the following formula (10) can be obtained:

[0233]

[0234] in, This represents the pose transformation relationship between the rotating platform and the ground in the initial state. Let be the pose transformation matrix between the ground and the first calibration plate. Let be the pose transformation matrix between the first calibration board and the first camera. The pose transformation matrix between the rotating platform and the ground when the rotating platform rotates by a preset angle. The pose transformation matrix between the first calibration plate and the first camera when the rotating platform rotates by a preset angle.

[0235] In order to solve the problem in formula (10) You can multiply both sides of the formula on the left. The inverse matrix, multiplied by the right side of both sides of the formula. The inverse matrix is ​​obtained, thus yielding the following formula (11).

[0236]

[0237] In formula (11), Where I is a 3×3 identity matrix; Where R is the rotation matrix derived from a preset angle. Furthermore, and This has already been obtained in step S905. Therefore, by solving formula (11), the pose transformation matrix between the ground and the first calibration plate can be obtained.

[0238] Similarly, the pose transformation matrix of the ground and any calibration plate can also be calculated according to the following formula (12):

[0239]

[0240] in, Let n be the pose transformation matrix between the ground and the nth calibration plate. Let n be the pose transformation matrix between the nth calibration board and the nth camera, where the nth camera corresponds to the nth calibration board. Given a preset angle rotation of the rotating platform, the pose transformation matrix between the nth calibration plate and the nth camera is obtained. After obtaining the pose transformation matrices between the ground and all calibration plates, the pose transformation matrix between any two calibration plates can be further obtained. Then, by combining the pose transformation matrices between each camera and its corresponding calibration plate in the initial state, the pose transformation matrix between any two cameras can be obtained as an extrinsic parameter. That is, S907 and S908 are executed after S906.

[0241] S907. The control equipment determines the pose transformation matrix between any two calibration plates based on the pose transformation matrix between the ground and all calibration plates.

[0242] The following section uses a multi-view perception kit, which includes a first camera and a second camera, with the calibration board corresponding to the first camera being the first calibration board and the calibration board corresponding to the second camera being the second calibration board, as an example to introduce the specific implementation of the S907:

[0243] Based on the transformation principle of the pose transformation matrix, the following formula (13) can be obtained:

[0244]

[0245] in, Let be the pose transformation matrix between the first calibration board and the second calibration board. Let be the pose transformation matrix between the first calibration board and the ground. This is the pose transformation matrix between the ground and the second calibration plate. Furthermore,

[0246] In formula (13), It has already been calculated in S906. Then it can be based on the calculated Therefore, it can be successfully derived.

[0247] Similarly, for any two calibration boards chartn and chartm, the following formula (14) can be used to calculate:

[0248]

[0249] in Let n be the pose transformation matrix between the nth calibration plate and the mth calibration plate. Let n be the pose transformation matrix between the nth calibration plate and the ground. Let be the pose transformation matrix between the ground and the m-th calibration plate. Furthermore,

[0250] After obtaining the pose transformation matrix between any two calibration boards and the pose transformation matrix between each camera and its corresponding calibration board, the pose transformation matrix between any two cameras can be derived based on these data according to the matrix transformation principle. That is, S907 is followed by S908.

[0251] S908: The control device determines the pose transformation matrix between any two cameras based on the pose transformation matrix between any two calibration boards and the pose transformation matrix between each camera and its corresponding calibration board in the initial state.

[0252] The following example, using a multi-view perception kit that includes a first camera and a second camera, with the calibration board corresponding to the first camera being the first calibration board and the calibration board corresponding to the second camera being the second calibration board, will be used to introduce the specific implementation of the S908:

[0253] Based on the transformation principle of the pose transformation matrix, the following formula (15) can be obtained:

[0254]

[0255] in, Let be the pose transformation matrix between the first camera and the second camera. Let this be the pose transformation matrix between the first camera and the first calibration board in the initial state. This represents the pose transformation matrix between the second calibration board and the second camera in the initial state. Furthermore,

[0256] In formula (13), It has already been calculated in S906. Then it can be based on the calculated Therefore, it can be successfully derived.

[0257] Similarly, the pose transformation matrix between any two cameras can be calculated using the following formula (16):

[0258]

[0259] in, Let n be the pose transformation matrix between the nth camera and the mth camera. Let n be the pose transformation matrix between the nth camera and the nth calibration board corresponding to the nth camera. Let be the pose transformation matrix between the m-th calibration board and the m-th camera corresponding to the m-th calibration board.

[0260] Furthermore, since the relative positional relationship between different cameras in the multi-view perception kit is also fixed, in some embodiments, S908 can also determine the pose transformation matrix between any two cameras based on the pose transformation matrix between any two calibration plates and the pose transformation matrix between each camera and the calibration plate corresponding to each camera when the rotating platform rotates by a preset angle.

[0261] At this point, the pose transformation matrix between any two cameras can also be calculated using the following formula (17):

[0262]

[0263] in, Given a preset angle of rotation of the rotating platform, the pose transformation matrix between the nth camera and the nth calibration plate corresponding to the nth camera is given. The pose transformation matrix between the m-th calibration plate and the m-th camera corresponding to the m-th calibration plate, given a preset angle rotation of the rotating platform. Furthermore,

[0264] Based on the technical solution provided in this application, by introducing rotation angle information through a rotating platform, the internal and external parameters of each camera in the multi-view perception kit can be determined by combining the pure visual information in the images captured by each camera. For pure visual information, the stability and uniqueness of the feature points it contains are relatively good; therefore, this calibration scheme can make the final internal and external parameters of the camera more accurate. Furthermore, as long as there are enough feature points in the pure visual information in different scenarios, the calibration objective can be successfully achieved. This makes the calibration scheme more adaptable. Simultaneously, since this calibration scheme does not determine the external parameters (i.e., the pose transformation matrix between different cameras) based on the differences in images of the same object captured by different cameras, the calibration scheme provided in this application can successfully obtain the external parameters between different cameras in the multi-view perception kit regardless of whether there is a shared viewing area between the different cameras in the multi-view perception kit. Furthermore, the calibration scheme provided in this application does not impose strong constraints on the position of the calibration board; it only needs to be adapted to the layout of the cameras in the multi-view perception kit, and the style of the calibration board is not specifically limited. In summary, the technical solution provided in this application can accurately calibrate the internal and external parameters of cameras in various scenarios (such as production lines or repair points) regardless of whether there is a shared viewing area between different cameras in the multi-view perception kit. Compared with existing camera calibration solutions for multi-view perception kits, it has higher adaptability.

[0265] To facilitate understanding, the following will be combined with... Figure 13 The flowchart of the camera calibration method provided in the embodiments of this application is illustrated.

[0266] like Figure 13 As shown, the process of the camera calibration method provided in this embodiment may include S1301-S1305:

[0267] S1301, The control device acquires the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image captured by each camera in the multi-view perception kit in the initial state; the initial state is the state in which the initial setting of the calibration system is completed.

[0268] In this embodiment, the specific implementation of S1301 can be referred to the relevant descriptions of S902 and S1001-S1003 in the foregoing embodiments, and will not be repeated here.

[0269] S1302. The control device determines the pose transformation matrix between each camera and its corresponding calibration board in the initial state based on the internal parameters of each camera, the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image of each camera, and the three-dimensional coordinates of the first corner point on the corresponding calibration board.

[0270] In some embodiments, S1301 may specifically include: the control device converting the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image corresponding to each camera into two-dimensional coordinates in the image coordinate system based on the internal parameters of each camera; the control device determining the pose transformation matrix between each camera and the corresponding calibration plate in the initial state based on the internal parameters of each camera, the two-dimensional coordinates of the first corner point of the calibration pattern in the initial calibration image corresponding to each camera in the image coordinate system, and the three-dimensional coordinates of the first corner point on the corresponding calibration plate.

[0271] In this embodiment, the specific implementation of the steps in S1301 can be referred to the relevant descriptions after S903 in the aforementioned embodiments, and will not be repeated here.

[0272] S1303, The control device acquires the pixel coordinates of the second corner point of the calibration pattern in the rotation calibration image captured by each camera when the rotating platform rotates at a preset angle.

[0273] In some embodiments, the specific implementation of S1301 can refer to the relevant descriptions of S904 and S1004-S1009 in the foregoing embodiments, which will not be repeated here.

[0274] S1304. The control device determines the pose transformation matrix between each camera and its corresponding calibration plate when the rotating platform rotates at a preset angle, based on the internal parameters of each camera, the pixel coordinates of the second corner point in the rotation calibration image of each camera, and the three-dimensional coordinates of the second corner point on the corresponding calibration plate.

[0275] In some embodiments, the specific implementation of S1301 can refer to the relevant description of S905 in the foregoing embodiments, and will not be repeated here.

[0276] S1305. The control device determines the pose transformation matrix between any two cameras in the multi-view perception kit based on the pose transformation matrix between each camera and its corresponding calibration board in the initial state, and the pose transformation matrix between each camera and its corresponding calibration board when the rotating platform rotates by a preset angle.

[0277] In some embodiments, S1301 may specifically include: the control device calculating the pose transformation matrix between the ground and each calibration plate based on the pose transformation matrix between each camera and its corresponding calibration plate in the initial state and the pose transformation matrix between each camera and its corresponding calibration plate when the rotating platform rotates by a preset angle; the control device determining the pose transformation matrix between any two calibration plates based on the pose transformation matrix between the ground and each calibration plate; and the control device determining the pose transformation matrix between any two cameras based on the pose transformation matrix between any two calibration plates and the pose transformation matrix between each camera and its corresponding calibration plate in the initial state.

[0278] In this embodiment, the specific implementation of the steps in S1301 can be referred to the relevant descriptions of S906-S908 in the foregoing embodiments, and will not be repeated here.

[0279] Furthermore, in this embodiment of the application, before S1302, the camera calibration method further includes: the control device determining the internal parameters of each camera in the multi-view perception kit.

[0280] Based on the technical solution provided in this application, by introducing rotation angle information through a rotating platform, the external parameters (i.e., pose transformation matrix) of each camera in the multi-view perception kit can be determined by combining the pure visual information in the images captured by each camera. For pure visual information, the stability and uniqueness of its feature points are relatively good; therefore, this calibration scheme can make the final camera external parameters more accurate. Furthermore, as long as there are enough feature points in the pure visual information in different scenarios, the calibration objective can be successfully achieved. This makes the calibration scheme more adaptable. Simultaneously, since this calibration scheme does not determine the external parameters (i.e., pose transformation matrix between different cameras) based on the differences in images of the same object captured by different cameras, the calibration scheme provided in this application can successfully obtain the external parameters between different cameras in the multi-view perception kit regardless of whether there is a shared viewing area between different cameras in the multi-view perception kit. Furthermore, the calibration scheme provided in this application does not impose strong constraints on the position of the calibration board; it only needs to be adapted to the layout of the cameras in the multi-view perception kit, and the style of the calibration board is not specifically limited. In summary, the technical solution provided in this application can accurately calibrate the external parameters of cameras in various scenarios (such as production lines or repair points) regardless of whether there is a shared field of view between different cameras in the multi-view perception kit. Compared with existing camera calibration solutions for multi-view perception kits, it has better adaptability and stability.

[0281] It is understood that, in order to achieve the aforementioned functions, the control device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware-driven or software-driven manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0282] This application embodiment can divide the control device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0283] When dividing each function into modules according to its corresponding function, refer to Figure 14 As shown in the illustration, this application also provides a control device, which includes multiple sound-generating devices. The control device may include an acquisition module 1401 and a processing module 1402.

[0284] The acquisition module 1401 is used to acquire the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image captured by each camera in the multi-view perception kit in the initial state; the initial state is the state where the initial setup of the calibration system is completed. The processing module 1402 is used to determine the pose transformation matrix between each camera and its corresponding calibration plate in the initial state, based on the internal parameters of each camera, the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image captured by the acquisition module 1401, and the three-dimensional coordinates of the first corner point on the corresponding calibration plate. The acquisition module 1401 is also used to acquire the pixel coordinates of the second corner point of the calibration pattern in the rotated calibration image captured by each camera when the rotating platform rotates by a preset angle. The processing module 1402 is also used to determine the pose transformation matrix between each camera and its corresponding calibration plate when the rotating platform rotates by a preset angle, based on the internal parameters of each camera, the pixel coordinates of the second corner point in the rotated calibration image captured by the acquisition module 1401, and the three-dimensional coordinates of the second corner point on the corresponding calibration plate. The processing module 1402 is also used to determine the pose transformation matrix between any two cameras in the multi-view perception kit based on the pose transformation matrix between each camera and its corresponding calibration plate in the initial state, and the pose transformation matrix between each camera and its corresponding calibration plate when the rotating platform rotates by a preset angle.

[0285] Furthermore, the cooperation between the acquisition module 1401 and the processing module 1402 can implement any step of the camera calibration method provided in the foregoing embodiments, which will not be repeated here. Regarding the electronic device in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the camera calibration method in the foregoing embodiments, and will not be elaborated here. Its related beneficial effects can also be referred to the related beneficial effects of the foregoing camera calibration method, which will not be repeated here.

[0286] This application also provides a control device, which includes a memory and one or more processors; the memory is coupled to the processors; wherein the memory stores computer program code, which includes computer instructions, and when the computer instructions are executed by the processor, the control device performs the camera calibration method provided in the foregoing embodiments. The specific structure of this control device can be referred to... Figure 8 The structure of the control device shown is illustrated.

[0287] This application also provides a computer-readable storage medium including computer instructions that, when executed on a control device, cause the control device to perform the camera calibration method provided in the foregoing embodiments.

[0288] This application also provides a computer program product containing executable instructions that, when run on a control device, cause the control device to perform the camera calibration method provided in the foregoing embodiments.

[0289] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0290] In the several embodiments provided in this application, it should be understood that the disclosed apparatus / device and method can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0291] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0292] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0293] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0294] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A camera calibration method, characterized in that, The method is applied to a calibration system, which includes a rotating platform, an electronic device equipped with a multi-view sensing kit, multiple calibration plates, and a control device that establishes communication connections with both the rotating platform and the electronic device. The electronic device is fixedly mounted on the rotating platform, and the multi-view sensing kit includes multiple cameras, with one calibration plate fixedly mounted in the shooting area of ​​each camera. The control device acquires the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image captured by each camera in the multi-view perception kit in the initial state; the initial state is the state in which the calibration system has completed its initial setup. The control device determines the pose transformation matrix between each camera and its corresponding calibration board in the initial state based on the internal parameters of each camera, the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image corresponding to each camera, and the three-dimensional coordinates of the first corner point on the corresponding calibration board. The control device acquires the pixel coordinates of the second corner point of the calibration pattern in the rotation calibration image captured by each camera when the rotating platform rotates by a preset angle. The control device determines the pose transformation matrix between each camera and its corresponding calibration plate when the rotating platform rotates by a preset angle, based on the internal parameters of each camera, the pixel coordinates of the second corner point in the rotation calibration image corresponding to each camera, and the three-dimensional coordinates of the second corner point on the corresponding calibration plate. The control device determines the pose transformation matrix between any two cameras in the multi-view perception kit based on the pose transformation matrix between each camera and its corresponding calibration plate in the initial state, and the pose transformation matrix between each camera and its corresponding calibration plate when the rotating platform rotates by a preset angle.

2. The method according to claim 1, characterized in that, The control device acquires the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image captured by each camera in the multi-view perception kit in its initial state, including: The control device sends a first instruction to the electronic device; the first instruction instructs the electronic device to control each camera in the multi-view perception kit to take pictures and send the captured initial calibration images to the control device. The control device receives initial calibration images corresponding to each camera from the electronic device and determines the pixel coordinates of the first corner point of the calibration pattern in each initial calibration image.

3. The method according to claim 1 or 2, characterized in that, The control device determines the pose transformation matrix between each camera and its corresponding calibration board in the initial state based on the internal parameters of each camera, the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image corresponding to each camera, and the three-dimensional coordinates of the first corner point on the corresponding calibration board. This includes: The control device converts the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image corresponding to each camera into two-dimensional coordinates in the image coordinate system based on the internal parameters of each camera. The control device determines the pose transformation matrix between each camera and its corresponding calibration board in the initial state based on the internal parameters of each camera, the two-dimensional coordinates of the first corner point of the calibration pattern in the image coordinate system of the initial calibration image corresponding to each camera, and the three-dimensional coordinates of the first corner point on the corresponding calibration board.

4. The method according to claim 1, characterized in that, The control device acquires the pixel coordinates of the second corner point of the calibration pattern in the rotation calibration image captured by each camera when the rotating platform rotates by a preset angle, including: The control device sends a second command to the rotating platform; the second command is used to instruct the rotating platform to rotate by a preset angle. The control device receives a first response from the rotating platform and, in response to the first response, sends a third instruction to the electronic device; the first response is used to indicate that the rotating platform has completed its rotation; the third instruction is used to instruct the electronic device to control each camera in the multi-view perception kit to take pictures and send the captured rotation calibration images to the control device. The control device receives rotational calibration images corresponding to each camera from the electronic device and determines the pixel coordinates of the second corner point of the calibration pattern in each rotational calibration image.

5. The method according to claim 1, characterized in that, The control device determines the pose transformation matrix between any two cameras in the multi-view perception kit based on the pose transformation matrix between each camera and its corresponding calibration plate in the initial state, and the pose transformation matrix between each camera and its corresponding calibration plate when the rotating platform rotates by a preset angle. This includes: The control device calculates the pose transformation matrix between the ground and each of the calibration plates based on the pose transformation matrix between each camera and the corresponding calibration plate in the initial state and the pose transformation matrix between each camera and the corresponding calibration plate when the rotating platform rotates by a preset angle. The control device determines the pose transformation matrix between any two calibration plates based on the pose transformation matrix between the ground and each of the calibration plates; The control device determines the pose transformation matrix between any two cameras based on the pose transformation matrix between any two calibration boards and the pose transformation matrix between each camera and its corresponding calibration board in the initial state.

6. The method according to claim 5, characterized in that, The control device calculates the pose transformation matrix between the ground and each of the calibration plates based on the pose transformation matrix between each camera and its corresponding calibration plate in the initial state, and the pose transformation matrix between each camera and its corresponding calibration plate when the rotating platform rotates by a preset angle. This includes: The control device calculates the pose transformation matrix between the ground and each of the calibration plates based on the pose transformation matrix between each camera and its corresponding calibration plate in the initial state, and the pose transformation matrix between each camera and its corresponding calibration plate when the rotating platform rotates by a preset angle, using the following formula: ； in, This refers to the pose transformation relationship between the rotating platform and the ground in the initial state. Let be the pose transformation matrix between the ground and the nth calibration plate. Let n be the pose transformation matrix between the nth calibration board and the nth camera, where the calibration board corresponding to the nth camera is the nth calibration board. The pose transformation matrix between the rotating platform and the ground when the rotating platform rotates by a preset angle. The pose transformation matrix between the nth calibration plate and the nth camera when the rotating platform is rotated by a preset angle; the nth calibration plate is one of the calibration plates corresponding to the plurality of cameras, and the nth camera is any one of the plurality of cameras.

7. The method according to claim 1, characterized in that, Before the control device determines the pose transformation matrix between each camera and its corresponding calibration board in the initial state based on the internal parameters of each camera, the pixel coordinates of the first corner point of the calibration pattern in the initial calibration image corresponding to each camera, and the three-dimensional coordinates of the first corner point on the corresponding calibration board, the method further includes: The control device determines the internal parameters of each camera in the multi-view sensing suite.

8. A control device, characterized in that, include: A memory and one or more processors; the memory is coupled to the processors; wherein the memory stores computer program code, the computer program code including computer instructions, which, when executed by the processor, cause the control device to perform the camera calibration method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on a control device, cause the control device to perform the camera calibration method as described in any one of claims 1-7.

10. A calibration system, characterized in that, The device includes a rotating platform, an electronic device equipped with a multi-view sensing kit, multiple calibration plates, and a control device that establishes a communication connection with both the rotating platform and the electronic device; the electronic device is fixedly mounted on the rotating platform, and the multi-view sensing kit includes multiple cameras, with one calibration plate fixedly mounted in the shooting area of ​​each camera; The control device is configured to perform the camera calibration method as described in any one of claims 1-7.

Images