Calibration method and device of inertial sensor in panoramic camera, computer equipment

Abstract

The present disclosure relates to a kind of calibration method, device, computer equipment of inertial sensor in panoramic camera.The method comprises: determining calibration space, calibration space is equipped with the calibration mark in same horizontal plane and in multiple directions, adjust the position of panoramic camera, make the optical center of panoramic camera always with calibration mark in same horizontal plane;Based on the first calibration image in calibration space and the calibration mark in each direction of first calibration image obtained by panoramic camera shooting, calculate first relative attitude parameter;Control panoramic camera rotates around the direction of pre-determined reference axis, based on the corresponding calibration mark in two frames of second calibration image obtained after rotating and shooting, construct the target optimization function for solving second relative attitude parameter;Second relative attitude parameter is determined based on target optimization function, and the inertial sensor of panoramic camera is calibrated using first relative attitude parameter and second relative attitude parameter.The present method can guarantee calibration accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of camera calibration technology, and in particular to a calibration method, apparatus, and computer equipment for an inertial sensor in a panoramic camera. Background Technology

[0002] With the development of panoramic imaging technology, camera calibration plays a crucial role in the production process. At the time of manufacture, each panoramic camera has an inertial measurement unit (IMU) with an inherent installation orientation deviation. This leads to significant errors in the relative orientation between the IMU and the optical center, affecting subsequent image stabilization and other functions that rely on IMU data. The lack of Y-axis parameters significantly reduces stabilization effectiveness, resulting in image distortion or vertical shaking.

[0003] Traditional hand-eye calibration methods have the following limitations:

[0004] 1. Calibration process is time-consuming: Three rotating axes need to be calibrated separately, and each axis needs to be rotated multiple times and the calibration plate needs to be photographed, which has become a bottleneck restricting production efficiency;

[0005] 2. Low operational tolerance and significant errors: It is necessary to ensure that the camera base is absolutely stable during the shooting process, but industrial robotic arms inevitably have micro-vibrations (even a vibration of only 1 micrometer at the end of the robotic arm transmitted to the camera may introduce a shift of several pixels in the image).

[0006] 3. IMU measurement noise interference: The inherent measurement noise of consumer-grade IMUs can significantly affect calibration accuracy. In hand-eye calibration, the acquisition of rotational data along each axis is highly dependent on the IMU measurement accuracy, leading to the accumulation of systematic errors. Summary of the Invention

[0007] Therefore, it is necessary to provide a calibration method, apparatus, computer equipment, and storage medium for the inertial sensor in a panoramic camera to address the aforementioned technical problems.

[0008] In a first aspect, this disclosure provides a method for calibrating an inertial sensor in a panoramic camera. The lens calibration of the panoramic camera has been completed, and the method includes:

[0009] A calibration space is defined, in which calibration marks are located on the same horizontal plane and in multiple directions. The position of the panoramic camera is adjusted so that the optical center of the panoramic camera is always located on the same horizontal plane as the calibration marks.

[0010] The first relative attitude parameters are calculated based on the first calibration image in the calibration space captured by the panoramic camera and the calibration marks in each direction of the first calibration image.

[0011] The panoramic camera is controlled to rotate around a predetermined reference axis. Based on the corresponding calibration marks in the two frames of the second calibration images obtained after the rotation, an objective optimization function for solving the second relative attitude parameters is constructed.

[0012] Based on the objective optimization function, a second relative attitude parameter is determined, and the inertial sensor of the panoramic camera is calibrated using the first and second relative attitude parameters.

[0013] In one embodiment, the relative attitude parameters include: relative pitch angle and relative roll angle; the first relative attitude parameters are calculated based on the first calibration image in the calibration space obtained by the panoramic camera and the calibration marks in each direction of the first calibration image, including:

[0014] The first calibration image is projected as an equidistant cylindrical projection image to determine the inertial sensor information that matches the first calibration image;

[0015] Based on the inertial sensor information, the three-axis acceleration of the panoramic camera is determined, and based on the three-axis acceleration, the first pitch angle and the first roll angle of the panoramic camera in a stationary state are determined.

[0016] The first calibration image is projected as an equidistant cylindrical projection image. Based on the calibration marks in each direction of the equidistant cylindrical projection image, the image height of the equidistant cylindrical projection image, and the first pitch angle and the first roll angle, the relative pitch angle and the relative roll angle are calculated.

[0017] In one embodiment, the calculation of the relative pitch angle and relative roll angle based on the calibration marks in each direction of the equidistant cylindrical projection image, the image height of the equidistant cylindrical projection image, and the first pitch angle and the first roll angle includes:

[0018] Determine the mean ordinate of the calibration mark in each direction in the equidistant cylindrical projection;

[0019] In response to the mean ordinate of the calibration mark in each direction and the image height of the equidistant cylindrical projection image, it is determined that the horizontal plane of the inertial sensor is not fully calibrated. Based on the predetermined initial pitch angle and initial roll angle, the mean ordinate of the calibration mark in each direction, and the preset scaling factor, the initial pitch angle and initial roll angle are updated to obtain the second pitch angle and the second roll angle.

[0020] The first calibration image is adjusted based on the second pitch angle and the second roll angle. The adjusted first calibration image is then reprojected as an equidistant cylindrical projection image until the horizontal plane calibration of the inertial sensor is determined to be complete.

[0021] Based on the first pitch angle, the first roll angle, the second pitch angle, and the second roll angle, the relative pitch angle and the relative roll angle are calculated.

[0022] In one embodiment, the relative attitude parameters further include: a relative attitude quaternion, wherein the first relative attitude parameters are calculated based on the first calibration image in the calibration space captured by the panoramic camera and the calibration marks in each direction of the first calibration image, including:

[0023] The initial pose quaternion of the panoramic camera is determined based on the coordinate axes of the camera coordinate system of the panoramic camera.

[0024] Determine the mean ordinate of the calibration mark in each direction in the equidistant cylindrical projection;

[0025] In response to the mean ordinate of the calibration mark in each direction and the image height of the equidistant cylindrical projection image, it is determined that the horizontal plane of the inertial sensor is not fully calibrated. Based on the predetermined initial pitch angle and initial roll angle, the mean ordinate of the calibration mark in each direction, and the preset scaling factor, the initial pitch angle and initial roll angle are updated to obtain the second pitch angle and the second roll angle.

[0026] Adjust the initial attitude quaternion based on the second pitch angle and the second roll angle;

[0027] The first calibration image is adjusted using the adjusted initial attitude quaternion, and the adjusted first calibration image is reprojected as an equidistant cylindrical projection image until the horizontal plane calibration of the inertial sensor is determined to be complete.

[0028] In response to the completion of the horizontal plane calibration of the inertial sensor, the relative attitude quaternion is obtained based on the adjusted initial attitude quaternion.

[0029] In one embodiment, the plurality of directions includes: a forward direction, a left direction, and a right direction, and the step of determining that the horizontal plane of the inertial sensor has not been calibrated includes:

[0030] Calculate the absolute values ​​of the differences between the mean ordinate of the front direction calibration mark, the mean ordinate of the left direction calibration mark, and the mean ordinate of the right direction calibration mark and half of the image height, respectively, to obtain the first difference, the second difference, and the third difference;

[0031] If any one of the first difference, the second difference, and the third difference is greater than a preset difference threshold, it is determined that the horizontal plane calibration of the inertial sensor is not complete.

[0032] In one embodiment, the second relative attitude parameter includes: a relative yaw angle. The objective optimization function for solving the second relative attitude parameter, based on the corresponding calibration marks in the two frames of the second calibration images obtained after rotation, includes:

[0033] Based on the relative pitch angle, relative roll angle and the preset initial yaw angle, an attitude transformation matrix is ​​constructed;

[0034] Based on the inertial sensor data collected by the panoramic camera during the rotation process, the attitude transfer relationship between the two frames of the second calibration image is estimated.

[0035] Based on the attitude transfer relationship and the attitude transformation matrix, image correction is performed on the second frame of the second calibration image in the two frames of second calibration images;

[0036] Based on the displacement difference between the corresponding calibration marks in the previous second calibration image in the next frame after image correction and between the two frames of second calibration images, an objective optimization function for solving the relative yaw angle is constructed.

[0037] In one embodiment, the second relative attitude parameter includes: a variable representing the imaginary unit axis in the relative attitude quaternion; and a target optimization function for solving the second relative attitude parameter is constructed based on the corresponding calibration marks in at least two frames of the second calibration images obtained after rotation.

[0038] Based on the relative attitude quaternion, image correction is performed on the second frame of the second calibration image in the two frames of the second calibration image. The initial value of the variable of the imaginary unit axis in the relative attitude quaternion is preset.

[0039] Based on the displacement difference between the corresponding calibration marks in the previous second calibration image and the two frames of second calibration images after image correction, an objective optimization function is constructed to solve the variable of the imaginary unit axis in the relative attitude quaternion.

[0040] Secondly, this disclosure also provides a calibration device for an inertial sensor in a panoramic camera. The lens calibration of the panoramic camera has been completed, and the device includes:

[0041] A camera position adjustment module is used to determine a calibration space, in which calibration marks are provided on the same horizontal plane and located in multiple directions. The position of the panoramic camera is adjusted so that the optical center of the panoramic camera is always on the same horizontal plane as the calibration marks.

[0042] The first calculation module is used to calculate the first relative attitude parameters based on the first calibration image in the calibration space obtained by the panoramic camera and the calibration mark in each direction of the first calibration image.

[0043] The optimization function construction module is used to control the panoramic camera to rotate around a predetermined reference axis direction, and construct an objective optimization function to solve the second relative attitude parameters based on the corresponding calibration marks in the two frames of the second calibration images obtained after the rotation.

[0044] The calibration module is used to determine the second relative attitude parameters based on the target optimization function, and to calibrate the inertial sensor of the panoramic camera using the first and second relative attitude parameters.

[0045] Thirdly, this disclosure also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in any of the above method embodiments.

[0046] Fourthly, this disclosure also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps in any of the above method embodiments.

[0047] Fifthly, this disclosure also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0048] In the above embodiments, a calibration space is constructed using multi-directional, coplanar calibration marks. This provides a uniform and reliable reference for subsequent attitude parameter calculations, ensuring the validity of the image data. It also allows the calibration marks to be arranged on multiple planes or large arc surfaces, with coplanar optical centers initially achieved through mechanical positioning, eliminating the need for complex tooling and reducing the difficulty and operational threshold of calibration space construction. Furthermore, it avoids initial data deviations caused by unreasonable mark distribution or non-coplanar optical centers. Based on the positional information of the multi-directional calibration marks in the first calibration image, and combined with IMU data, the first relative attitude parameters are calculated collaboratively. This accurately captures the initial relative attitude between the IMU and the camera coordinate system, laying a solid foundation for subsequent completion of the vertical axis parameters. Moreover, the multi-directional marks effectively offset the influence of errors from a single mark. Based on the corresponding calibration marks in the two frames of the second calibration image obtained after rotation, a target optimization function for solving the second relative attitude parameters is constructed. The optimization function is built around the displacement difference of the marks in the two frames, with a clear objective that aligns with actual calibration requirements, enabling targeted solutions for the second relative attitude parameters. The optimal second relative attitude parameters are obtained by solving the optimization function. After being integrated with the first relative attitude parameters, they fully cover the three rotational dimensions of pitch, roll, and yaw, completely completing the spatial pose relationship between the IMU and the camera coordinate system. This effectively eliminates the problem of missing Y-axis parameters caused by IMU mounting deviation. The final calibration results can directly support the precise image stabilization of the panoramic camera, avoiding image twisting or vertical shaking. Moreover, the entire process does not require complex operation by professional personnel and can be easily integrated into the production line for batch calibration. This not only ensures calibration accuracy (parameter repeatability up to 0.3 pixels) but also breaks through the production efficiency bottleneck of traditional technologies. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram illustrating the application environment of the calibration method for the inertial sensor in a panoramic camera in one embodiment;

[0051] Figure 2 This is a flowchart illustrating a calibration method for an inertial sensor in a panoramic camera, as shown in one embodiment.

[0052] Figure 3 This is a flowchart illustrating step S204 in one embodiment;

[0053] Figure 4This is a schematic block diagram of the calibration device for the inertial sensor in a panoramic camera in one embodiment;

[0054] Figure 5 This is a schematic diagram of the internal structure of a computer device in one embodiment;

[0055] Figure 6 This is a schematic diagram of the internal structure of a computer device in one embodiment. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this disclosure.

[0057] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0058] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0059] This disclosure provides a calibration method for an inertial sensor in a panoramic camera, which can be applied to applications such as... Figure 1In the application environment shown, terminal 102 communicates with panoramic camera 104 via a network. Lens calibration of panoramic camera 104 has been completed. A calibration space can be determined, containing calibration marks located on the same horizontal plane and in multiple directions. Terminal 102 can adjust the position of panoramic camera 104 so that the optical center of panoramic camera 104 is always located on the same horizontal plane as the calibration marks. Based on a first calibration image in the calibration space captured by panoramic camera 104 and the calibration marks in each direction of the first calibration image, terminal 102 calculates a first relative attitude parameter. Terminal 102 controls panoramic camera 104 to rotate around a predetermined reference axis direction, and based on the corresponding calibration marks in two frames of second calibration images captured after rotation, constructs a target optimization function to solve for the second relative attitude parameter. Based on the target optimization function, terminal 102 determines the second relative attitude parameter and uses the first and second relative attitude parameters to calibrate the inertial sensor of the panoramic camera. Terminal 102 can be, but is not limited to, various personal computers, laptops, etc.

[0060] In one embodiment, such as Figure 2 As shown, a calibration method for an inertial sensor in a panoramic camera is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps:

[0061] S202, Determine the calibration space, wherein calibration marks are provided in the calibration space and are located on the same horizontal plane and in multiple directions. Adjust the position of the panoramic camera so that the optical center of the panoramic camera is always located on the same horizontal plane as the calibration marks.

[0062] The calibration space is typically a specific physical area designed specifically for panoramic camera calibration. It must meet the following conditions: unobstructed views, stable lighting, and the ability to accurately place calibration marks. Calibration marks are usually reference markers (such as dots, checkerboard corners, etc.) placed within the calibration space. The core requirement is that they are arranged horizontally in at least three directions, and all marks are on the same horizontal plane. The optical center of the panoramic camera is usually the core reference point of the panoramic camera imaging system. It is the key benchmark for determining whether the camera and calibration marks are coplanar, directly affecting the accuracy of the mark's projected position in the image.

[0063] Specifically, first, designate a flat, non-reflective, and unobstructed physical area as the calibration space, ensuring uniform lighting within the area (to avoid shadows interfering with marker recognition) and sufficient space to arrange calibration markers and place camera clamping equipment (such as a turntable). Select at least three different directions (e.g., left, right, and front) and fix the markers at the same horizontal height (calibrated using a level). The plane on which the markers are located can be multiple independent planes or a large arc surface, ensuring that the center line connecting all the markers forms a horizontal line. Clamp the camera using an industrial turntable or mechanical fixture, and use mechanical positioning accuracy to assist in adjustments (e.g., fine-tuning the fixture height and angle) so that the optical center of the camera is on the same horizontal plane as the center of all calibration markers (preliminary positioning can be achieved through prior mechanical calibration). After ensuring the camera is stationary, acquire the image at the current moment (for identifying marker positions) and IMU data (for calculating IMU pitch and roll angles). At this point, the markers should be roughly distributed around the equatorial horizontal line in the camera's ERP unfolded image, laying the foundation for subsequent numerical iteration optimization (correcting camera pitch and roll angles).

[0064] S204, based on the first calibration image in the calibration space captured by the panoramic camera and the calibration mark in each direction of the first calibration image, the first relative attitude parameters are calculated.

[0065] The first calibration image is an image taken by the panoramic camera in the calibration space during the horizontal calibration stage (the camera optical center and the calibration mark are coplanar, and the image is acquired when the camera is stationary). Its core purpose is to extract the position information of the calibration marks in each direction, providing image data support for calculating the pose of the camera coordinate system.

[0066] The first relative attitude parameter is used to characterize the initial relative attitude relationship between the IMU coordinate system and the camera coordinate system. It can include three specific parameters: relative pitch angle (the difference between the pitch angle of the IMU and the camera), relative roll angle (the difference between the roll angle of the IMU and the camera), and relative attitude quaternion (the relative attitude of the two in quaternion form). All three of these parameters lack relevant data on rotation around the vertical axis and need to be supplemented by subsequent vertical axis calibration.

[0067] Specifically, the panoramic camera takes a static picture within the calibration space to acquire the first calibration image, while simultaneously saving the acceleration data of the camera's built-in IMU; the pixel coordinates of calibration markers in multiple directions such as left, right, and front are identified and extracted from the first calibration image, and the average y-coordinate of the markers in each direction is calculated; the pitch angle and roll angle (or the corresponding attitude quaternion) of the IMU are calculated based on the IMU acceleration data; based on the average y-coordinate of the markers in the image, the pitch angle and roll angle (or the corresponding attitude quaternion) of the camera are solved through numerical iterative optimization; the first relative attitude parameters (including relative pitch angle, relative roll angle, and relative attitude quaternion) are obtained through the conversion logic of IMU attitude parameters to camera attitude parameters.

[0068] S206, control the panoramic camera to rotate around a predetermined reference axis direction, and construct an objective optimization function to solve the second relative attitude parameters based on the corresponding calibration marks in the two frames of the second calibration images obtained after the rotation.

[0069] The pre-determined reference axis direction typically refers to the camera rotation axis preset during the vertical axis calibration stage. This can be the X-axis (horizontal axis) and Z-axis (fixed direction axis) of the camera coordinate system OXcYcZc, serving as the core rotation reference for acquiring the displacement difference of the marker points and solving for the second relative attitude parameters. The second calibration image is an image captured by the panoramic camera during the "vertical axis calibration" stage, rotating around the X / Z axis. At least two frames are acquired for each axis to extract the displacement difference of the calibration markers, supporting the construction of the target optimization function. The second relative attitude parameters are the core output of the vertical axis calibration, used to complete the full relative attitude relationship between the IMU and the camera coordinate system. These include two types of parameters: 1. Relative yaw angle (attitude angle representation); 2. Imaginary unit axis variable in the attitude quaternion (i.e., j-axis coefficient, quaternion representation). Both correspond to key data about rotation around the vertical axis and are essential parameters for accurate image stabilization. The target optimization function is a function constructed with the corresponding calibration markers in the two frames of the second calibration image as the target, used to iteratively solve for the optimal second relative attitude parameters.

[0070] Specifically, the predetermined reference axis directions are defined as the X-axis and Z-axis of the camera coordinate system. A panoramic camera is held in place by a turntable (keeping the camera's optical center coplanar with the calibration mark). The turntable is controlled to rotate the camera around the X-axis and Z-axis respectively, capturing one frame before and after each rotation, for a total of at least two frames of second calibration images. IMU data is acquired simultaneously. Based on the first relative attitude parameters obtained from the previous horizontal plane calibration, the last frame of the at least two second calibration images is corrected to initially align with the first frame of the at least two second calibration images. From the corrected images, the pixel coordinates of the same calibration mark are extracted, and the displacement difference between the two is calculated. With minimizing the mark displacement difference as the optimization objective, a target optimization function is constructed. The solution to this function is the second relative attitude parameter (relative yaw angle or attitude quaternion j-axis coefficient).

[0071] S208, based on the target optimization function, determine the second relative attitude parameters, and use the first relative attitude parameters and the second relative attitude parameters to calibrate the inertial sensor of the panoramic camera.

[0072] The inertial measurement unit (IMU) is a built-in measurement device in the panoramic camera that collects acceleration and angular velocity data. Its relative spatial relationship with the camera lens directly affects the image stabilization effect. Inertial sensor calibration integrates the first and second relative attitude parameters to complete the full relative pose relationship between the IMU and the camera coordinate system, eliminating errors caused by IMU mounting deviations and ensuring that the IMU data can accurately support the image stabilization process.

[0073] Specifically, numerical optimization algorithms such as the Nelder-Mead simplex method are used to iteratively solve the objective optimization function constructed in the vertical axis calibration stage to obtain the optimal second relative attitude parameters (relative yaw angle or j-axis precision coefficient of the attitude quaternion). The first relative attitude parameters and the optimal second relative attitude parameters are then integrated: if the attitude angle representation is used, the relative pitch angle, relative roll angle, and optimal relative yaw angle are combined; if the quaternion representation is used, the optimal j-axis coefficient is substituted into the initial relative attitude quaternion to obtain the complete unitized relative attitude quaternion. The parameters combined above fully characterize the spatial pose transformation relationship from the IMU coordinate system to the camera coordinate system (including all data in the three rotational dimensions). This complete pose transformation relationship is written into the panoramic camera's system configuration to complete the calibration of the inertial sensor (IMU). During subsequent camera operation, this calibration result can be used to accurately fuse IMU data and image data, achieving efficient image stabilization (avoiding image twisting and vertical shaking).

[0074] In the aforementioned calibration method for the inertial sensor in the panoramic camera, a calibration space is constructed using multi-directional, coplanar calibration marks. This provides a uniform and reliable reference for subsequent attitude parameter calculations, ensuring the validity of the image data. It also allows the calibration marks to be arranged on multiple planes or large arc surfaces, with coplanar optical centers initially achieved through mechanical positioning, eliminating the need for complex tooling and reducing the difficulty and operational threshold of setting up the calibration space. Simultaneously, it avoids initial data deviations caused by unreasonable mark distribution or non-coplanar optical centers. Based on the positional information of the multi-directional calibration marks in the first calibration image, and combined with IMU data, the first relative attitude parameters are calculated collaboratively. This accurately captures the initial relative attitude between the IMU and the camera coordinate system, laying a solid foundation for subsequent completion of the vertical axis parameters. Furthermore, the multi-directional markings effectively offset the influence of errors from a single mark. Based on the corresponding calibration marks in the two frames of the second calibration image obtained after rotation, a target optimization function for solving the second relative attitude parameters is constructed. The optimization function is built around the displacement difference of the marks in the two frames, with a clear objective that aligns with actual calibration requirements, enabling targeted solutions for the second relative attitude parameters. The optimal second relative attitude parameters are obtained by solving the optimization function. After being integrated with the first relative attitude parameters, they fully cover the three rotational dimensions of pitch, roll, and yaw, completely completing the spatial pose relationship between the IMU and the camera coordinate system. This effectively eliminates the problem of missing Y-axis parameters caused by IMU mounting deviation. The final calibration results can directly support the precise image stabilization of the panoramic camera, avoiding image twisting or vertical shaking. Moreover, the entire process does not require complex operation by professional personnel and can be easily integrated into the production line for batch calibration. This ensures calibration accuracy (parameter repeatability up to 0.3 pixels) and breaks through the production efficiency bottleneck of traditional technologies.

[0075] In one embodiment, such as Figure 3 As shown, the relative attitude parameters include: relative pitch angle and relative roll angle; the first relative attitude parameters are calculated based on the first calibration image in the calibration space obtained by the panoramic camera and the calibration marks in each direction of the first calibration image, including:

[0076] S302, determine the inertial sensor information that matches the first calibration image.

[0077] Specifically, after the panoramic camera completes the coplanar adjustment of the optical center and the calibration mark in the calibration space, it remains stationary and starts the synchronous triggering mechanism for image acquisition and inertial sensor (IMU) data acquisition (ensuring that the acquisition timestamps of the two are completely consistent); while the camera captures and saves the first calibration image, it simultaneously records the raw data of the built-in IMU, including acceleration data (core data) of the three axes and the acquisition timestamp;

[0078] Filter stable data under static conditions (removing sudden interference data at the moment of acquisition) to verify whether the resultant vector of the three-axis acceleration is close to the gravitational acceleration; extract the verified IMU acceleration data and bind it with the corresponding first calibration image through timestamp to form an image-IMU data matching pair. The bound IMU acceleration data is the inertial sensor information that matches the first calibration image.

[0079] S304, Based on the inertial sensor information, determine the three-axis acceleration of the panoramic camera, and based on the three-axis acceleration, determine the first pitch angle and the first roll angle of the panoramic camera in a stationary state.

[0080] Among them, the three-axis acceleration refers to the acceleration values ​​measured by the inertial measurement unit (IMU) along the X, Y, and Z coordinate axes. In a stationary state, the resultant vector of these three accelerations is equal in magnitude and opposite in direction to the gravitational acceleration, and is the core basis for calculating attitude angles. The first pitch angle (IMU pitch angle) is the pitch angle (angle of rotation around the X-axis) of the IMU coordinate system relative to the world coordinate system, calculated based on the IMU's three-axis acceleration, and is one of the key parameters characterizing the IMU's attitude. The first roll angle (IMU roll angle) is the roll angle (angle of rotation around the Z-axis) of the IMU coordinate system relative to the world coordinate system, calculated based on the IMU's three-axis acceleration, and together with the first pitch angle, constitutes the basic attitude information of the IMU.

[0081] Specifically, raw IMU triaxial acceleration data in a stationary state is extracted from the inertial sensor information matched and bound to the first calibration image, denoted as Ax (X-axis acceleration), Ay (Y-axis acceleration), and Az (Z-axis acceleration). The magnitude of the resultant vector of the three is calculated, and it is confirmed that its deviation from the local gravitational acceleration (approximately 9.8 m / s²) is within the allowable range, ensuring that the data is stable and valid in a stationary state. Utilizing the characteristic that the resultant vector of the triaxial accelerations in a stationary state is the gravitational acceleration, the first pitch angle and the first roll angle are calculated through formula derivation. The core logic is: by using the numerical relationship of Ax, Ay, and Az, the tilt state of the IMU in the plane perpendicular to the direction of gravity is inferred, and the corresponding angle values ​​are finally output (e.g., based on the arctangent function, the derivation formulas are: pitch = arctan(Ax / √(Ay²+Az²)), roll = arctan(Ay / √(Ax²+Az²)). The specific formulas need to be adjusted according to the IMU coordinate system definition.

[0082] S306, the first calibration image is projected as an equidistant cylindrical projection image, and the relative pitch angle and relative roll angle are calculated based on the calibration marks in each direction of the equidistant cylindrical projection image, the image height of the equidistant cylindrical projection image, and the first pitch angle and the first roll angle.

[0083] Among them, the equidistant cylindrical projection image (ERP image) is a projection of the first calibration image (spherical projection image) captured by the panoramic camera, unfolded into a planar image. It is used to visually present the position of the calibration marks in each direction. Its equatorial horizon corresponds to the ideal horizontal position of the calibration marks and is the basis for subsequent mark coordinate analysis. The image height is usually the vertical pixel size of the equidistant cylindrical projection image (denoted as H), and its center point (H / 2) corresponds to the equatorial horizon, which is the core reference benchmark for determining whether the calibration marks are horizontal.

[0084] Specifically, the first calibration image is projected into an equidistant cylindrical projection image (ERP image), and the image height H of the image is defined, with its center point H / 2 corresponding to the equatorial horizontal line. The pixel coordinates of calibration marks in multiple directions such as left, right, and front are extracted from the ERP image, and the average y-coordinates (y_left, y_right, y_front) of the marks in each direction are calculated. The camera pitch angle and camera roll angle are initialized to 0, and the convergence condition is that the deviation of the average y-coordinate of the marks from H / 2 is ≤1px. Numerical iterative optimization is performed: if the deviation does not meet the requirement, the camera pitch angle and camera roll angle are updated, the unit sphere is rotated synchronously, and the ERP image is reprojected. The iteration is repeated until convergence. After convergence, the final camera pitch angle and camera roll angle are obtained. The relative pitch angle and relative roll angle are calculated by subtracting them from the first pitch angle (IMU pitch angle) and the first roll angle (IMU roll angle), respectively.

[0085] In this embodiment, by utilizing the characteristic that the resultant vector of the three-axis acceleration of the IMU at rest is the acceleration due to gravity, the attitude angle can be calculated using existing IMU data without the need for additional complex equipment, simplifying the calibration process. The logic of deriving the attitude angle from the acceleration values ​​is direct and accurate, effectively avoiding measurement noise interference from civilian-grade IMUs, and can stably output the IMU's own basic attitude information (first pitch angle, first roll angle), providing a precise benchmark for subsequent relative attitude parameter calculations. Converting the panoramic image into an ERP image makes the positional relationship of the multi-directional calibration marks intuitive, and using the image height center point (H / 2) as the horizontal benchmark simplifies the judgment of mark position deviation. Through numerical iterative optimization (combined with an error threshold of 1px and a magnification factor of 0.0236° / px), the attitude deviation between the camera and the IMU is accurately corrected, and the final relative pitch angle and relative roll angle can accurately represent the initial relative attitude of the two. The entire process does not require multiple camera rotations or complex fixtures, making the operation simple and the accuracy controllable, laying a solid foundation for subsequent vertical axis calibration parameter completion, while also meeting the efficiency requirements of industrial batch calibration.

[0086] In one embodiment, calculating the relative pitch angle and relative roll angle based on the calibration marks in each direction of the equidistant cylindrical projection image, the image height of the equidistant cylindrical projection image, and the first pitch angle and the first roll angle includes:

[0087] S402, Project the first calibration image into an equidistant cylindrical projection image, and determine the mean value of the ordinate of the calibration mark in each direction in the equidistant cylindrical projection.

[0088] S404, in response to determining that the horizontal plane of the inertial sensor is not fully calibrated based on the mean of the ordinate of the calibration mark in each direction and the image height of the equidistant cylindrical projection image, and based on the predetermined initial pitch angle and initial roll angle, the mean of the ordinate of the calibration mark in each direction and the preset scaling factor, the initial pitch angle and initial roll angle are updated to obtain the second pitch angle and the second roll angle.

[0089] S406, adjust the first calibration image based on the second pitch angle and the second roll angle, and reproject the adjusted first calibration image into an equidistant cylindrical projection image until the horizontal plane calibration of the inertial sensor is determined to be complete.

[0090] S408, based on the first pitch angle, the first roll angle, the second pitch angle, and the second roll angle, the relative pitch angle and the relative roll angle are calculated.

[0091] Specifically, the first calibration image is projected into an equidistant cylindrical projection image (ERP image). All calibration markers in multiple directions (left, right, front, etc.) are identified and extracted from the ERP image. The pixel ordinate (y-coordinate) of each marker is recorded, and the mean ordinate values ​​(y_left, y_right, y_front) for each direction are calculated. Using the image height H of the ERP image as a reference, the absolute values ​​of the differences between the mean ordinate values ​​in each direction and H / 2 (the equatorial horizontal line) are calculated, i.e., |y_left-H / 2|, |y_right-H / 2|, |y_front-H / 2|. If the absolute value of any of these differences is greater than 1px (ε), it is determined that "the inertial sensor horizontal plane calibration is not complete," and the attitude angle update process is initiated. The attitude angle is updated based on preset rules:

[0092] Second pitch angle = initial pitch angle + scaling factor β × (y_front - (y_right + y_left) / 2) (If the front marker is too low, the pitch angle increases to correct the marker position).

[0093] Second roll angle = initial roll angle + scaling factor α × (y_left - y_right) (when the left marker is too low, the roll angle increases to correct the marker position).

[0094] The updated second pitch angle and second roll angle will be used as the "initial pitch angle and initial roll angle" for the next iteration. Repeat the above steps until the deviation of the average ordinate of all directional markers from H / 2 is ≤1px, and the horizontal plane calibration is completed.

[0095] Once it is confirmed that the horizontal plane calibration iteration has converged (the deviation between the mean of the ordinates of all calibration marks and the height center point H / 2 of the ERP image is ≤1px), the second pitch angle and the second roll angle are the final values ​​of the camera attitude angles.

[0096] Extract the determined first pitch angle (IMU pitch angle) and first roll angle (IMU roll angle), as well as the second pitch angle (camera final pitch angle) and second roll angle (camera final roll angle) after iterative convergence.

[0097] Relative attitude parameters are calculated using the logic of IMU attitude angle minus camera attitude angle:

[0098] Relative pitch angle = First pitch angle - Second pitch angle;

[0099] Relative roll angle = first roll angle - second roll angle.

[0100] The calculation results are the core output of the horizontal plane calibration stage (a key component of the first relative attitude parameters), laying the foundation for subsequent vertical axis calibration to complete the yaw-related parameters.

[0101] In this embodiment, the closed-loop iterative process of "adjusting the image - reprojecting - verifying the deviation" can gradually correct the camera attitude angle deviation, ensuring that the deviation of the final average value of all directional markers from H / 2 is ≤1px, achieving a high level of calibration convergence accuracy. The iterative process requires no manual intervention and has a high degree of automation, which not only avoids calibration errors caused by incomplete adjustments in a single instance, but also adapts to the differences in IMU mounting deviations of different cameras, making the calibration results more universal. At the same time, no additional hardware costs are required, and accuracy can be improved solely through software iteration. The first pitch angle and the first roll angle are the IMU's precise basic attitude (based on three-axis acceleration calculation), while the second pitch angle and the second roll angle are the camera's iterative convergence attitude (based on marker bias optimization). The calculation logic of the difference between the two is direct and fits the core requirement of representing the relative attitude of the IMU and the camera. The resulting relative pitch angle and relative roll angle can accurately offset the basic attitude deviation caused by the IMU mounting deviation, fully covering the relative attitude dimensions in the horizontal plane. This lays a solid and accurate foundation for subsequent vertical axis calibration to complete the yaw parameters. At the same time, no additional error is introduced in the parameter calculation process, ensuring the reliability of the first relative attitude parameters.

[0102] In one embodiment, the relative attitude parameters further include: a relative attitude quaternion, the first relative attitude parameters being calculated based on the first calibration image in the calibration space captured by the panoramic camera and the calibration marks in each direction of the first calibration image, including:

[0103] S502, based on the coordinate axes of the panoramic camera's camera coordinate system, determines the initial pose quaternion of the panoramic camera.

[0104] S504, determine the mean value of the ordinate of the calibration mark in each direction in the equidistant cylindrical projection.

[0105] S506, in response to determining that the horizontal plane of the inertial sensor is not fully calibrated based on the average ordinate of the calibration marks in each direction and the image height of the equidistant cylindrical projection image, and based on the predetermined initial pitch angle and initial roll angle, the average ordinate of the calibration marks in each direction and the preset scaling factor, the initial pitch angle and initial roll angle are updated to obtain the second pitch angle and the second roll angle.

[0106] The camera coordinate system axes refer to the 3D coordinate system OXYZ of the panoramic camera, where X is the horizontal axis, Y is the vertical axis of symmetry, and Z is the fixed direction axis. This serves as the spatial reference for constructing the attitude quaternion. The imaginary units i, j, and k of the quaternion correspond one-to-one with X, Y, and Z, respectively. The initial attitude quaternion is typically constructed based on the camera coordinate system axes and represents the initial relative attitude from the IMU coordinate system to the camera coordinate system. It takes the form: ,in (correspond The axis is a preset initial value, and the other coefficients can be initialized in conjunction with the initial attitude angle. It is the quaternion form of the starting reference for horizontal plane calibration. The mean ordinate of the calibration mark in each direction in the equidistant cylindrical projection: In the ERP image after the panoramic image is unfolded, for the preset directions such as left, right, and front, calculate the ordinate of all calibration mark pixels in that direction. The average value of the coordinates is denoted as . , , This is the core data for determining whether the camera's attitude is horizontal. The preset scaling factor is a fixed coefficient that converts the pixel deviation marked in the ERP image into an attitude angle adjustment amount, denoted as . It is used to balance the iteration convergence speed and calibration accuracy.

[0107] Specifically, in the panoramic camera coordinate system , , Using the axis as a reference, the imaginary unit axis of the quaternion is... , , respectively with , , Axis binding; based on the initial mounting pose of the IMU and camera, in the preset quaternion Axis factor The initial values, combined with the initial pitch angle and initial roll angle, are all... Initialize the real part of the quaternion under the given conditions. and , Axis factor , The initial attitude quaternion representing the initial relative attitude between the IMU and the camera is obtained. The first calibration image is projected as an equidistant cylindrical projection image (ERP image), and all calibration marks in the ERP image in preset directions such as left, right, and front are extracted using an image recognition algorithm; the pixel ordinate of each mark is read respectively. (Coordinates), calculate the arithmetic mean of the ordinates of all markers in each direction to obtain the average ordinates of the markers in each direction: y_left, y_right, y_front. Using the image height of the ERP image... Using this as a benchmark, calculate the mean of the ordinates in each direction and the ideal horizontal position. The absolute value of the difference, i.e.

[0108]

[0109] If the absolute value of any difference is greater than 1px, the inertial sensor's horizontal plane calibration is considered incomplete. Second pitch angle = initial pitch angle + scaling factor Second roll angle = initial roll angle + scaling factor The updated second pitch angle and second roll angle will be used as the initial pitch angle and initial roll angle for the next iteration and will continue to participate in the calibration optimization.

[0110] S508, adjust the initial attitude quaternion based on the second pitch angle and the second roll angle.

[0111] S510, adjust the first calibration image using the adjusted initial attitude quaternion, and reproject the adjusted first calibration image into an equidistant cylindrical projection image until the horizontal plane calibration of the inertial sensor is determined to be complete.

[0112] S512, in response to the completion of the horizontal plane calibration of the inertial sensor, obtains the relative attitude quaternion based on the adjusted initial attitude quaternion.

[0113] Among them, the relative attitude quaternion is the final quaternion representing the relative attitude between the IMU and the camera coordinate system in the horizontal plane, directly determined by the adjusted initial attitude quaternion after the horizontal plane calibration converges. It takes the form: This belongs to the first relative attitude parameters, which only cover pitch and roll dimensions. Further vertical axis calibration is needed to complete the yaw-related parameters. Axis coefficient.

[0114] Specifically, the second pitch angle obtained through iteration (denoted as...) ), second roll angle (denoted as Convert to the corresponding rotation quaternion The conversion logic is based on "around" Axis pitch, rotation The rotation sequence of "axis tumbling" is calculated using the following formula:

[0115]

[0116] The initial attitude quaternion With rotation quaternions Perform quaternion multiplication (following the rules of quaternion multiplication to ensure the continuity of attitude transformation) to obtain the adjusted initial attitude quaternion. .

[0117] Based on the adjusted initial attitude quaternion The first calibration image is subjected to spatial attitude rotation correction to eliminate image tilt corresponding to the current pitch and roll angle deviations. The corrected first calibration image is then reprojected as an equidistant cylindrical projection (ERP) image. The mean ordinate of the calibration marks in each direction is repeatedly calculated, and its deviation from H / 2 is verified to be ≤1px. For example, for any point P[x,y,z]ᵀ on a unit sphere, its rotated position is... ,in *yes The conjugate of the values. If the deviation does not meet the requirements, the current second pitch angle and second roll angle are used as the initial pitch angle and initial roll angle for the next iteration, and the process of "updating attitude angles → adjusting quaternions → correcting projected images" is repeated until the calibration completion conditions are met. When the horizontal plane calibration of the inertial sensor is determined to be complete, the adjusted initial attitude quaternion is the relative attitude quaternion in the horizontal plane. This quaternion accurately represents the relative attitude relationship between the IMU and the camera coordinate system in the pitch and roll dimensions, and can be used as the basic parameters for subsequent vertical axis calibration.

[0118] In this embodiment, an initial attitude quaternion is constructed based on the camera coordinate system axes, directly transforming the coordinate system spatial relationship into quantified attitude parameters. Compared to Euler angles, this effectively avoids the gimbal lock problem, ensuring the continuity and integrity of the attitude representation. Simultaneously, the quaternion form is compatible with subsequent attitude rotation calculations, eliminating the need for complex coordinate transformations and reducing computational complexity. This provides a precise and efficient attitude quantification benchmark for horizontal plane calibration. Through a closed-loop iterative process of "attitude angle update → quaternion adjustment → image correction → projection verification," bidirectional verification and optimization of attitude parameters and image data are achieved. This gradually reduces the horizontal plane attitude deviation between the camera and the IMU until all directional marker deviations meet a preset threshold (≤1px), ensuring high accuracy of the calibration results.

[0119] In one embodiment, the plurality of directions includes: forward, left, and right directions, and determining that the horizontal plane of the inertial sensor has not been calibrated includes:

[0120] Calculate the absolute values ​​of the differences between the mean ordinate of the front direction calibration mark, the mean ordinate of the left direction calibration mark, and the mean ordinate of the right direction calibration mark and half of the image height, respectively, to obtain the first difference, the second difference, and the third difference;

[0121] If any one of the first difference, the second difference, and the third difference is greater than a preset difference threshold, it is determined that the horizontal plane calibration of the inertial sensor is not complete.

[0122] Specifically, from the completed equidistant cylindrical projection (ERP) image, all calibration marks in the front, left, and right directions are extracted, and the mean ordinate of each mark in each direction is calculated. Simultaneously, the vertical pixel size of the ERP image is read to determine the image height H, and half of the image height H / 2 is calculated. The absolute values ​​of the differences are calculated using formulas for the front, left, and right directions, yielding the first, second, and third differences, which quantify the deviation of each mark from the ideal horizontal position. To determine the calibration completion status, the first, second, and third differences are compared with a preset difference threshold (1px): if any difference is greater than 1px, the inertial sensor's horizontal plane calibration is considered incomplete, and the next round of attitude angle update and image correction process needs to be initiated; if all three differences are less than or equal to 1px, the horizontal plane calibration is considered complete, and no further iteration is required.

[0123] In one embodiment, the second relative attitude parameter includes: a relative yaw angle. The objective optimization function for solving the second relative attitude parameter, based on the corresponding calibration marks in the two frames of the second calibration images obtained after rotation, includes:

[0124] S502, construct the attitude transformation matrix based on the relative pitch angle, relative roll angle and the preset initial yaw angle;

[0125] S504, based on the inertial sensor data collected by the panoramic camera during the rotation, estimates the attitude transfer relationship between two frames of the second calibration image;

[0126] S506, Based on the attitude transfer relationship and the attitude transformation matrix, perform image correction on the second frame of the second calibration image in the two frames of second calibration images;

[0127] S508, based on the displacement difference between the corresponding calibration marks in the previous frame of the second calibration image after image correction and the two frames of second calibration images, construct the objective optimization function for solving the relative yaw angle.

[0128] The attitude transformation matrix is ​​a 3×3 rotation matrix (denoted as R) constructed based on the relative pitch angle, relative roll angle, and initial yaw angle. It is used to quantify the initial spatial attitude transformation relationship from the IMU coordinate system to the camera coordinate system and is the core mathematical tool for image correction. The second calibration image, captured during the vertical axis calibration phase, includes the previous frame (t0 frame) before rotation and the next frame (t1 frame) after rotation. It is used to extract the displacement difference of the calibration markers to solve for the relative yaw angle. The displacement difference between calibration markers is the difference in pixel coordinates (x / y coordinates) of the same calibration marker between the corrected t1 frame and t0 frame, and is the core quantitative indicator for measuring image alignment accuracy.

[0129] Specifically, the relative pitch angle obtained based on the horizontal plane calibration ( ), relative roll angle ( ), combined with the pre-set initial yaw angle ( Following the rotation sequence of "roll first, then pitch, and finally yaw," a 3×3 attitude transformation matrix is ​​constructed. The core formula is as follows:

[0130]

[0131]

[0132]

[0133]

[0134] The final result This is the initial attitude transformation matrix from the IMU to the camera coordinate system. The panoramic camera is controlled to rotate around the X and Z axes, and the IMU's three-axis angular velocity and acceleration data are simultaneously acquired during the rotation process. Based on the IMU data, algorithms such as integration / Kalman filtering are used to calculate the camera attitude change from frame t0 (the previous frame) to frame t1 (the next frame), expressed as a rotation matrix. or quaternion This parameter represents the pose transition relationship between two frames. Pose transformation matrix Relationship with attitude transfer Multiplying them together yields the combined attitude transformation matrix. ;based on right The second calibration image in the next frame undergoes spatial rotation correction. The coordinates of each pixel in the image are mapped to a new position according to matrix transformation rules, offsetting the image shift / tilt caused by rotation, thus improving the corrected image. Frame and Initial alignment of frame images. Extracting... Frame (previous frame) second calibration image, after correction The pixel coordinates of the same calibration mark in the second calibration image of the next frame are denoted as... and ( To mark the sequence number, i can be 0 or 1. When i is 0, it can be two frames obtained around the X-axis; when i is 1, it can be two frames obtained by rotating around the Z-axis.

[0135] Calculate the displacement difference for each marker: , ;

[0136] With relative yaw angle ( To optimize the variables, construct the objective optimization function:

[0137]

[0138] in, To determine the total number of calibrators, the optimization objective of the function is to minimize the sum of the displacement differences of all calibrators, thereby solving for the optimal relative yaw angle.

[0139] In this embodiment, the relative pitch and roll angle parameters verified during the horizontal calibration stage are directly reused, eliminating the need for additional data acquisition and significantly simplifying the pre-process of matrix construction. The attitude transfer relationship is estimated based on the raw IMU data (angular velocity and acceleration) synchronously acquired during camera rotation, eliminating the need for external positioning equipment (such as high-precision turntable encoders) and reducing hardware costs. The high-frequency acquisition characteristics of IMU data can accurately capture subtle attitude changes between two frames, reflecting the camera's true rotation trajectory more quickly and directly compared to traditional image feature matching methods. Through a two-parameter fusion correction strategy of "attitude transfer relationship (reflecting the relative attitude change between two frames) + attitude transformation matrix (reflecting the inherent pose deviation between the IMU and the camera)," the attitude offset caused by camera rotation and the systematic error caused by IMU mounting deviation can be simultaneously offset, achieving accurate preliminary alignment between the subsequent frame and the previous frame. The corrected image eliminates tilt and offset interference, providing a clear and consistent benchmark for subsequent calculation of calibration mark displacement differences, avoiding misjudgments of displacement differences due to image misalignment, and ensuring the effectiveness of subsequent optimization functions.

[0140] In one embodiment, the second relative attitude parameter includes: a variable representing the imaginary unit axis in the relative attitude quaternion; and a target optimization function for solving the second relative attitude parameter is constructed based on the corresponding calibration marks in at least two frames of the second calibration images obtained after rotation.

[0141] S602, based on the relative attitude quaternion, perform image correction on the second frame of the second calibration image in the two frames of the second calibration image, wherein the initial value of the variable of the imaginary unit axis in the relative attitude quaternion is preset.

[0142] S604, based on the displacement difference between the corresponding calibration marks in the previous frame of the second calibration image after image correction and the two frames of second calibration images, constructs an objective optimization function to solve the variable of the imaginary unit axis in the relative attitude quaternion.

[0143] Specifically, obtain the relative attitude quaternion after the horizontal plane calibration is completed, and confirm its imaginary unit axis variable. The initial values ​​are set beforehand. The relative attitude quaternion is then converted into the corresponding 3×3 rotation matrix. (The transformation rules follow the mapping formula between quaternions and rotation matrices:) Depend on Calculated); for the second frame of the two frames of the second calibration image ( Frame) Perform image correction: Traverse Each pixel of the frame image, based on the rotation matrix Perform spatial rotation transformation on the pixel coordinates to eliminate attitude deviations between the IMU and the camera in the pitch and roll dimensions, resulting in a more accurate and corrected image. Frame and previous frame ( The second calibration image (frame) achieves initial alignment. From the corrected... Frame images and raw In the frame image, the pixel coordinates of the same calibration mark in the front, left, and right directions are extracted respectively, and denoted as . and (i is the index of the calibration mark, with the same meaning as above); calculate the displacement difference for each calibration mark: the displacement difference of a single mark is... ,in , The imaginary unit axis variable of the relative orientation quaternion. Using the total displacement difference of all calibration marks as the independent variable, and taking the minimization of the total displacement difference of all calibration marks as the optimization objective, the objective optimization function is constructed as follows:

[0144]

[0145] Where n is the total number of calibration marks, and the minimum value of this function corresponds to... The value is the optimal solution required for vertical axis calibration.

[0146] In this embodiment, relative attitude quaternions can effectively avoid gimbal lock problems compared to traditional Euler angles, and the continuity and stability of attitude representation are stronger. They can more accurately describe the spatial relative attitude of the IMU and the camera. The image correction based on them can offset the attitude deviation in pitch and roll dimensions to the greatest extent, so that the next frame image is initially aligned with the previous frame image with high quality, eliminating systematic errors for subsequent displacement difference calculations. The design of preset initial values ​​for the imaginary unit axis variable not only ensures the integrity of the quaternion structure, but also reserves optimization space for subsequent iterative solutions, taking into account both the flexibility of calibration and the convenience of initial calculation.

[0147] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0148] Based on the same inventive concept, this disclosure also provides a calibration device for an inertial sensor in a panoramic camera for implementing the calibration method for an inertial sensor in a panoramic camera as described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the calibration device for an inertial sensor in a panoramic camera provided below can be found in the limitations of the calibration method for an inertial sensor in a panoramic camera described above, and will not be repeated here.

[0149] In one embodiment, such as Figure 4 As shown, a calibration device 700 for an inertial sensor in a panoramic camera is provided, comprising: a camera position adjustment module 702, a first calculation module 704, an optimization function construction module 706, and a calibration module 708, wherein:

[0150] The camera position adjustment module 702 is used to determine the calibration space, in which calibration marks are provided on the same horizontal plane and located in multiple directions. The position of the panoramic camera is adjusted so that the optical center of the panoramic camera is always on the same horizontal plane as the calibration marks.

[0151] The first calculation module 704 is used to calculate the first relative attitude parameters based on the first calibration image in the calibration space obtained by the panoramic camera and the calibration mark in each direction of the first calibration image.

[0152] The optimization function construction module 706 is used to control the panoramic camera to rotate around a predetermined reference axis direction, and construct an objective optimization function to solve the second relative attitude parameters based on the corresponding calibration marks in the two frames of the second calibration images obtained after the rotation.

[0153] The calibration module 708 is used to determine the second relative attitude parameters based on the target optimization function, and to calibrate the inertial sensor of the panoramic camera using the first relative attitude parameters and the second relative attitude parameters.

[0154] In one embodiment of the device, the relative attitude parameters include: relative pitch angle and relative roll angle; the first calculation module 704 includes:

[0155] The information determination module is used to determine the inertial sensor information that matches the first calibration image;

[0156] An angle determination module is used to determine the three-axis acceleration of the panoramic camera based on the inertial sensor information, and to determine the first pitch angle and the first roll angle of the panoramic camera in a stationary state based on the three-axis acceleration.

[0157] An angle calculation module is used to project the first calibration image into an equidistant cylindrical projection image, and calculate the relative pitch angle and relative roll angle based on the calibration marks in each direction of the equidistant cylindrical projection image, the image height of the equidistant cylindrical projection image, and the first pitch angle and the first roll angle.

[0158] In one embodiment of the device, the angle calculation module includes:

[0159] The coordinate calculation module is used to determine the mean value of the ordinate of the calibration mark in each direction in the equidistant cylindrical projection;

[0160] An angle update module is used to respond to the average ordinate of each calibration mark and the image height of the equidistant cylindrical projection image, determine that the horizontal plane of the inertial sensor is not fully calibrated, and update the initial pitch angle and initial roll angle based on the predetermined initial pitch angle and initial roll angle, the average ordinate of each calibration mark and the preset scaling factor to obtain the second pitch angle and the second roll angle.

[0161] The reprojection module is used to adjust the first calibration image based on the second pitch angle and the second roll angle, and reproject the adjusted first calibration image into an equidistant cylindrical projection image until the horizontal plane calibration of the inertial sensor is determined to be complete.

[0162] An angle calculation submodule is used to calculate the relative pitch angle and the relative roll angle based on the first pitch angle, the first roll angle, the second pitch angle, and the second roll angle.

[0163] In one embodiment of the device, the relative attitude parameters further include: a relative attitude quaternion. An angle calculation module includes:

[0164] The initial data determination module is used to determine the initial pose quaternion of the panoramic camera based on the coordinate axes of the camera coordinate system of the panoramic camera.

[0165] The coordinate calculation module is used to determine the mean value of the ordinate of the calibration mark in each direction in the equidistant cylindrical projection;

[0166] An angle update module is used to determine that the horizontal plane of the inertial sensor is not fully calibrated based on the mean of the ordinate of the calibration mark in each direction and the image height of the equidistant cylindrical projection image. Based on the predetermined initial pitch angle and initial roll angle, the mean of the ordinate of the calibration mark in each direction, and a preset scaling factor, the module updates the initial pitch angle and initial roll angle to obtain a second pitch angle and a second roll angle.

[0167] The data adjustment module is used to adjust the initial attitude quaternion based on the second pitch angle and the second roll angle.

[0168] The reprojection module is used to adjust the first calibration image using the adjusted initial attitude quaternion, and reproject the adjusted first calibration image into an equidistant cylindrical projection image until the horizontal plane calibration of the inertial sensor is determined to be complete.

[0169] The quaternion determination module is used to determine the relative attitude quaternion based on the adjusted initial attitude quaternion in response to the completion of the horizontal plane calibration of the inertial sensor.

[0170] In one embodiment of the device, the plurality of directions includes: a forward direction, a left direction, and a right direction, and the angle update module includes:

[0171] The horizontal calibration determination module is used to calculate the absolute values ​​of the differences between the mean ordinate of the front calibration mark, the mean ordinate of the left calibration mark, and the mean ordinate of the right calibration mark and half of the image height, respectively, to obtain a first difference, a second difference, and a third difference; if any one of the first difference, the second difference, and the third difference is greater than a preset difference threshold, it is determined that the horizontal plane of the inertial sensor has not been calibrated.

[0172] In one embodiment of the device, the second relative attitude parameters include: relative yaw angle. The optimization function construction module 706 is further configured to: construct an attitude transformation matrix based on the relative pitch angle, relative roll angle, and a pre-set initial yaw angle; estimate the attitude transfer relationship between two frames of second calibration images based on inertial sensor data acquired by the panoramic camera during rotation; perform image correction on the latter frame of the two frames of second calibration images based on the attitude transfer relationship and the attitude transformation matrix; and construct an objective optimization function for solving the relative yaw angle based on the displacement difference between the corresponding calibration marks in the image-corrected latter frame of second calibration images and the former frame of the two frames of second calibration images.

[0173] In one embodiment of the device, the second relative attitude parameter includes: a variable of the imaginary unit axis in the relative attitude quaternion; the optimization function construction module 706 is further configured to perform image correction on the second frame of the two second calibration images based on the relative attitude quaternion; the initial value of the variable of the imaginary unit axis in the relative attitude quaternion is preset; and construct an objective optimization function to solve for the variable of the imaginary unit axis in the relative attitude quaternion based on the displacement difference between the corresponding calibration marks in the second frame of the second calibration image after image correction and the first frame of the two second calibration images.

[0174] The modules in the inertial sensor calibration device of the aforementioned panoramic camera can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0175] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores first and second relative attitude parameters. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a calibration method for an inertial sensor in a panoramic camera.

[0176] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a calibration method for an inertial sensor in a panoramic camera. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0177] Those skilled in the art will understand that Figure 5 or Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the computer device to which the present disclosure is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0178] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0179] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0180] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0181] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this disclosure may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this disclosure may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0182] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0183] The embodiments described above are merely illustrative of several implementations of this disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent disclosure. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these all fall within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the appended claims.

Claims

1. A calibration method for an inertial sensor in a panoramic camera, characterized in that, The lens calibration of the panoramic camera has been completed, and the method includes: A calibration space is defined, in which calibration marks are located on the same horizontal plane and in multiple directions. The position of the panoramic camera is adjusted so that the optical center of the panoramic camera is always located on the same horizontal plane as the calibration marks. Based on the first calibration image in the calibration space captured by the panoramic camera and the calibration marks in each direction of the first calibration image, the first relative attitude parameters are calculated; the first relative attitude parameters include: relative pitch angle, relative roll angle or relative attitude quaternion; The panoramic camera is controlled to rotate around a predetermined reference axis. Based on the corresponding calibration marks in the two frames of the second calibration images obtained after rotation, a target optimization function for solving the second relative attitude parameters is constructed. The construction of the target optimization function for solving the second relative attitude parameters based on the corresponding calibration marks in the two frames of the second calibration images obtained after rotation includes: correcting the latter frame of the two frames of the second calibration images based on the first relative attitude parameters, so that the latter frame is initially aligned with the former frame of the two frames of the second calibration images; extracting the pixel coordinates of the same calibration mark from the corrected second calibration images; calculating the pixel coordinate displacement difference; and constructing the target optimization function with the minimum displacement difference as the optimization objective. The second relative attitude parameters include: the relative yaw angle corresponding to the relative pitch angle and the relative roll angle, or the imaginary unit axis variable in the relative attitude quaternion; Based on the objective optimization function, a second relative attitude parameter is determined, and the inertial sensor of the panoramic camera is calibrated using the first and second relative attitude parameters.

2. The method according to claim 1, characterized in that, The first relative attitude parameters include: relative pitch angle and relative roll angle; the first relative attitude parameters are calculated based on the first calibration image in the calibration space obtained by the panoramic camera and the calibration marks in each direction of the first calibration image, including: Determine the inertial sensor information that matches the first calibration image; Based on the inertial sensor information, the three-axis acceleration of the panoramic camera is determined, and based on the three-axis acceleration, the first pitch angle and the first roll angle of the panoramic camera in a stationary state are determined. The first calibration image is projected as an equidistant cylindrical projection image. Based on the calibration marks in each direction of the equidistant cylindrical projection image, the image height of the equidistant cylindrical projection image, and the first pitch angle and the first roll angle, the relative pitch angle and the relative roll angle are calculated.

3. The method according to claim 2, characterized in that, The calculation of the relative pitch angle and relative roll angle based on the calibration marks in each direction of the equidistant cylindrical projection image, the image height of the equidistant cylindrical projection image, and the first pitch angle and the first roll angle includes: Determine the mean ordinate of the calibration mark in each direction in the equidistant cylindrical projection; In response to the mean ordinate of the calibration mark in each direction and the image height of the equidistant cylindrical projection image, it is determined that the horizontal plane of the inertial sensor is not fully calibrated. Based on the predetermined initial pitch angle and initial roll angle, the mean ordinate of the calibration mark in each direction, and the preset scaling factor, the initial pitch angle and initial roll angle are updated to obtain the second pitch angle and the second roll angle. The first calibration image is adjusted based on the second pitch angle and the second roll angle. The adjusted first calibration image is then reprojected as an equidistant cylindrical projection image until the horizontal plane calibration of the inertial sensor is determined to be complete. Based on the first pitch angle, the first roll angle, the second pitch angle, and the second roll angle, the relative pitch angle and the relative roll angle are calculated.

4. The method according to claim 1, characterized in that, The first relative attitude parameter further includes: a relative attitude quaternion. The first relative attitude parameter is calculated based on the first calibration image in the calibration space obtained by the panoramic camera and the calibration markers in each direction of the first calibration image, including: The initial pose quaternion of the panoramic camera is determined based on the coordinate axes of the camera coordinate system of the panoramic camera. The first calibration image is projected as an equidistant cylindrical projection image, and the mean value of the ordinate of the calibration mark in each direction in the equidistant cylindrical projection is determined. In response to the mean ordinate of the calibration mark in each direction and the image height of the equidistant cylindrical projection image, it is determined that the horizontal plane of the inertial sensor is not fully calibrated. Based on the predetermined initial pitch angle and initial roll angle, the mean ordinate of the calibration mark in each direction, and the preset scaling factor, the initial pitch angle and initial roll angle are updated to obtain the second pitch angle and the second roll angle. Adjust the initial attitude quaternion based on the second pitch angle and the second roll angle; The first calibration image is adjusted using the adjusted initial attitude quaternion, and the adjusted first calibration image is reprojected as an equidistant cylindrical projection image until the horizontal plane calibration of the inertial sensor is determined to be complete. In response to the completion of the horizontal plane calibration of the inertial sensor, the relative attitude quaternion is obtained based on the adjusted initial attitude quaternion.

5. The method according to claim 3 or 4, characterized in that, The multiple directions include: forward, left, and right directions. Determining that the horizontal plane calibration of the inertial sensor is not complete includes: Calculate the absolute values ​​of the differences between the mean ordinate of the front direction calibration mark, the mean ordinate of the left direction calibration mark, and the mean ordinate of the right direction calibration mark and half of the image height, respectively, to obtain the first difference, the second difference, and the third difference; If any one of the first difference, the second difference, and the third difference is greater than a preset difference threshold, it is determined that the horizontal plane calibration of the inertial sensor is not complete.

6. The method according to claim 1, characterized in that, The second relative attitude parameter includes: relative yaw angle. The objective optimization function for solving the second relative attitude parameter, based on the corresponding calibration marks in the two frames of the second calibration images obtained after rotation, includes: Based on the relative pitch angle, relative roll angle and the preset initial yaw angle, an attitude transformation matrix is ​​constructed; Based on the inertial sensor data collected by the panoramic camera during the rotation process, the attitude transfer relationship between the two frames of the second calibration image is estimated. Based on the attitude transfer relationship and the attitude transformation matrix, image correction is performed on the second frame of the second calibration image in the two frames of second calibration images; Based on the displacement difference between the corresponding calibration marks in the previous second calibration image in the next frame after image correction and between the two frames of second calibration images, an objective optimization function for solving the relative yaw angle is constructed.

7. The method according to claim 1, characterized in that, The second relative attitude parameters include: the variable of the imaginary unit axis in the relative attitude quaternion; and the objective optimization function for solving the second relative attitude parameters is constructed based on the corresponding calibration marks in at least two frames of the second calibration images obtained after rotation. Based on the relative attitude quaternion, image correction is performed on the second frame of the second calibration image in the two frames of the second calibration image. The initial value of the variable of the imaginary unit axis in the relative attitude quaternion is preset. Based on the displacement difference between the corresponding calibration marks in the previous second calibration image and the two frames of second calibration images after image correction, an objective optimization function is constructed to solve the variable of the imaginary unit axis in the relative attitude quaternion.

8. A calibration device for an inertial sensor in a panoramic camera, characterized in that, The lens calibration of the panoramic camera has been completed, and the device includes: A camera position adjustment module is used to determine a calibration space, in which calibration marks are provided on the same horizontal plane and located in multiple directions. The position of the panoramic camera is adjusted so that the optical center of the panoramic camera is always on the same horizontal plane as the calibration marks. The first calculation module is used to calculate the first relative attitude parameters based on the first calibration image in the calibration space obtained by the panoramic camera and the calibration marks in each direction of the first calibration image; the first relative attitude parameters include: relative pitch angle, relative roll angle or relative attitude quaternion; The optimization function construction module is used to control the panoramic camera to rotate around a predetermined reference axis direction, and construct an objective optimization function to solve the second relative attitude parameters based on the corresponding calibration marks in the two frames of the second calibration images obtained after the rotation. The optimization function construction module is further configured to correct the latter frame of the two-frame second calibration images based on the first relative attitude parameters, so that the latter frame is initially aligned with the former frame of the two-frame second calibration images; extract the pixel coordinates of the same calibration mark from the corrected second calibration images; calculate the pixel coordinate displacement difference; construct a target optimization function with the minimum displacement difference as the optimization objective; the second relative attitude parameters include: the relative yaw angle corresponding to the relative pitch angle and the relative roll angle, or the imaginary unit axis variable in the relative attitude quaternion; The calibration module is used to determine the second relative attitude parameters based on the target optimization function, and to calibrate the inertial sensor of the panoramic camera using the first and second relative attitude parameters.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images