Method and device for extrinsic calibration, electronic equipment and storage medium

Abstract

Provided are a method and device for calibrating camera and inertial measurement unit (IMU) extrinsic parameters on a two-wheeled vehicle, an electronic device, and a storage medium. The method comprises: performing a segmented excitation motion on the two-wheeled vehicle to synchronously collect an image sequence and inertial data by the camera and the IMU; extracting image feature points between adjacent image frames in the image sequence to determine the relative rotation of the camera between the adjacent image frames; performing pre-integration processing on the inertial data between the time instants corresponding to the adjacent image frames to determine the relative rotation of the IMU between the adjacent time instants; constructing a hand-eye calibration model based on the relative rotation of the camera and the relative rotation of the IMU to determine the rotation matrix between the camera and the IMU; and optimizing a target function based on a sliding window to minimize the equation residual based on the hand-eye calibration model to obtain the calibration extrinsic parameters between the camera and the IMU.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a method for extrinsic parameter calibration, an apparatus for extrinsic parameter calibration of a two-wheeled vehicle, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In visual-inertial fusion systems, accurately acquiring the relative pose relationship (i.e., extrinsic parameters) between the camera and the inertial measurement unit (IMU) is the foundation for achieving multi-sensor data fusion and precise positioning.

[0003] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and computer-readable storage medium for calibrating the extrinsic parameters of a camera and inertial measurement unit (IMU) on a two-wheeled vehicle, so as to achieve self-calibration of the extrinsic parameters of the camera and inertial measurement unit on a two-wheeled vehicle.

[0005] According to one aspect of this disclosure, a method for calibrating the extrinsic parameters of a camera and an inertial measurement unit (IMU) on a two-wheeled vehicle is provided, comprising: performing segmented excitation motion on the two-wheeled vehicle to synchronously acquire image sequences and inertial data via the camera and the IMU, wherein the segmented excitation motion includes motion with at least two axial degrees of freedom; extracting image feature points between adjacent image frames in the image sequence to determine the relative rotation of the camera between adjacent image frames; pre-integrating the inertial data between corresponding moments of adjacent image frames to determine the relative rotation of the IMU between adjacent moments; constructing a hand-eye calibration model based on the relative rotation of the camera and the relative rotation of the IMU to determine the rotation matrix between the camera and the IMU; and obtaining the calibration extrinsic parameters between the camera and the IMU by optimizing a sliding window-based objective function to minimize the residuals of the equations based on the hand-eye calibration model, wherein the objective function is constructed by using the rotation matrix as the extrinsic parameter to be optimized.

[0006] According to another aspect of this disclosure, an apparatus for calibrating the extrinsic parameters of a camera and an inertial measurement unit (IMU) on a two-wheeled vehicle is provided, comprising: a data acquisition unit configured to acquire image sequences and inertial data synchronously acquired by the camera and the IMU during a segmented excitation motion of the two-wheeled vehicle, wherein the segmented excitation motion includes motion with at least two axial degrees of freedom; a vision processing unit configured to extract image feature points between adjacent image frames in the image sequence to determine the relative rotation of the camera between adjacent image frames; an inertial processing unit configured to pre-integrate the inertial data between corresponding moments of adjacent image frames to determine the relative rotation of the IMU between adjacent moments; a model building unit configured to construct a hand-eye calibration model based on the relative rotation of the camera and the relative rotation of the IMU to determine the rotation matrix between the camera and the IMU; and an optimization unit configured to optimize a sliding window-based objective function to minimize the residuals of the equations based on the hand-eye calibration model to obtain the calibration extrinsic parameters between the camera and the IMU, wherein the objective function is constructed by using the rotation matrix as the extrinsic parameter to be optimized.

[0007] According to another aspect of this disclosure, an electronic circuit is also provided, including a circuit configured to perform the steps of the above-described method.

[0008] According to another aspect of this disclosure, an electronic device is also provided, the electronic device including a processor; and a memory storing a program including instructions that, when executed by the processor, cause the processor to perform the methods described above.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is also provided, the program including instructions that, when executed by a processor of an electronic device, cause the electronic device to perform the methods described above.

[0010] According to another aspect of this disclosure, a computer program product is also provided, the computer program product comprising a computer program, wherein the computer program implements the above-described method when executed by a processor.

[0011] According to another aspect of this disclosure, a two-wheeled vehicle is also provided, which includes the aforementioned electronic circuitry or electronic equipment.

[0012] According to embodiments of this disclosure, by performing specific segmented excitation motion on a two-wheeled vehicle and combining it with multi-scene vision-inertial joint optimization, not only can the dependence on specific calibration objects such as manual calibration boards be reduced, thus lowering the difficulty for users to perform calibration operations, but also the visual degradation problem caused by complex environments in natural scenes such as low texture, repetitive structures, large-angle rotation, and dynamic obstacle interference can be overcome. This significantly improves the accuracy and system stability of the self-calibration of camera and IMU extrinsic parameters used in two-wheeled vehicles, thereby enabling large-scale promotion on two-wheeled vehicles.

[0013] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0014] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0015] Figure 1 A flowchart illustrating an exemplary process for calibrating the extrinsic parameters of a camera and inertial measurement unit on a two-wheeled vehicle according to an embodiment of the present disclosure is shown.

[0016] Figure 2 A flowchart illustrating an exemplary process of a method for performing segmented excitation motion according to an embodiment of the present disclosure is shown.

[0017] Figure 3 A flowchart illustrating an exemplary process of a method 300 for optimizing the equations of a hand-eye calibration model according to an embodiment of the present disclosure is shown.

[0018] Figure 4 A flowchart illustrating an exemplary process of an optimization method 400 for low-texture regions according to an embodiment of the present disclosure is shown.

[0019] Figure 5 An exemplary block diagram of an apparatus for calibrating the extrinsic parameters of a camera and an inertial measurement unit (IMU) on a two-wheeled vehicle, according to an embodiment of the present disclosure, is shown.

[0020] Figure 6 A block diagram illustrating an example of an electronic device according to an exemplary embodiment of the present disclosure is shown. Detailed Implementation

[0021] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0022] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0023] In multi-sensor fusion positioning technology, especially in scenarios involving camera and inertial measurement unit (IMU) extrinsic parameter calibration, specific calibration objects such as manual calibration boards are typically required. This necessitates large-scale spatial rotation of the device as a whole, making calibration operations difficult and environmentally demanding, hindering large-scale application on large mobile devices like two-wheeled vehicles that cannot be easily flipped or lifted. Existing self-calibration algorithms often heavily rely on ideal static, high-texture environments, making them highly susceptible to interference in natural, everyday scenarios. This not only increases the difficulty of user data collection but can also lead to data divergence, affecting the normal operation of subsequent visual-inertial joint positioning systems. Furthermore, existing calibration algorithms lack optimization for two-wheeled vehicles and their actual operating environments. In complex environments such as parking lots and real roads, issues arise such as matching ambiguities in repetitive scenes, feature degradation in low-texture areas, tracking loss due to large-angle rapid turns, and interference from dynamic obstacles. These problems make it difficult to achieve the expected accuracy and stability of extrinsic parameter calibration, failing to meet the application requirements of two-wheeled vehicles in natural scenarios.

[0024] To address the aforementioned problems in related technologies, this disclosure provides a novel method for calibrating the extrinsic parameters of a camera and inertial measurement unit (IMU) on a two-wheeled vehicle. The self-calibration method according to embodiments of this disclosure is based on a natural scene calibration scheme without a calibration board. It acquires data through segmented excitation motion specifically designed for two-wheeled vehicles and processes the synchronously acquired image sequences and inertial data using a multi-scene adapted vision-inertial joint optimization algorithm to jointly solve for the extrinsic parameters between the camera and the IMU. This method not only eliminates the reliance on specific equipment such as manual calibration boards, lowering the operational threshold for users, but also improves the robustness of the calibration model in complex environments through algorithm optimization for various natural operating scenarios of two-wheeled vehicles, ensuring the accuracy and reliability of extrinsic parameter calibration, thereby enabling large-scale deployment on two-wheeled vehicles.

[0025] The embodiments of this disclosure are described in detail below with reference to the accompanying drawings.

[0026] Figure 1 A flowchart illustrating an exemplary process of a method 100 for calibrating extrinsic parameters of a camera and inertial measurement unit on a two-wheeled vehicle according to an embodiment of the present disclosure is shown.

[0027] In step S102, segmented excitation motion is performed on the two-wheeled vehicle to synchronously acquire image sequences and inertial data through the camera and IMU.

[0028] In step S104, image feature points between adjacent image frames in the image sequence are extracted to determine the relative rotation of the camera between adjacent image frames.

[0029] In step S106, the inertial data between the corresponding times of adjacent image frames is pre-integrated to determine the relative rotation of the IMU between adjacent times.

[0030] In step S108, the rotation matrix between the camera and the IMU is determined based on the relative rotation of the camera and the relative rotation of the IMU.

[0031] In step S110, the calibration extrinsic parameters between the camera and the IMU are obtained by optimizing the objective function based on the sliding window to minimize the residual of the equation based on the hand-eye calibration model. The objective function is constructed by using the rotation matrix as the extrinsic parameter to be optimized.

[0032] The method for calibrating the extrinsic parameters of a camera and inertial measurement unit (IMU) on a two-wheeled vehicle, provided by embodiments of this disclosure, simultaneously collects data while performing specialized segmented excitation motion on the two-wheeled vehicle, and processes the data in conjunction with vision-inertial joint optimization. This method not only eliminates the dependence on specific calibration objects such as manual calibration boards, enabling convenient calibration in natural life scenarios and reducing the difficulty of operation for users, but also improves the accuracy of self-calibration of the camera and IMU extrinsic parameters.

[0033] The steps of method 100 are described in detail below.

[0034] In step S102, segmented excitation motion is performed on the two-wheeled vehicle to synchronously acquire image sequences and inertial data through the camera and IMU.

[0035] In embodiments of this disclosure, the two-wheeled vehicle may be an electric bicycle, an electric motorcycle, a traditional bicycle, or other personal transportation vehicle or micro-mobility vehicle with two wheels. This disclosure does not impose any specific limitations on this.

[0036] In some embodiments, the camera and inertial measurement unit (IMU) are rigidly fixed to the two-wheeled vehicle. Specifically, the camera and IMU can be integrated into the same module or fixed separately as independent hardware devices on a shared rigid structural component (e.g., a dedicated mounting bracket) to ensure that the physical relative position and attitude between the camera and IMU remain stable during data acquisition and vehicle movement. In some examples, the camera and IMU can be mounted in the center of the handlebars, above the dashboard, or on the windshield at the front of the vehicle. In some examples, the camera and IMU can also be mounted at the rear of the vehicle (e.g., above the rear seat, tail box, or rear fender), so that the camera's field of view faces the environment behind the vehicle. In such examples, since the camera and IMU maintain a relatively fixed rigid connection, the extrinsic parameter data between the camera and IMU can be accurately solved and obtained by extracting the relative motion of the camera image feature points and calculating the IMU pre-integration. This disclosure does not impose specific limitations in this regard.

[0037] In vision-inertial fusion systems (e.g., VIO or SLAM systems), extrinsic parameter calibration of the camera and inertial measurement unit typically involves solving for the spatial relative pose relationship between the camera coordinate system and the IMU coordinate system. This relative relationship is usually represented by a rotation matrix that describes the angular alignment between the two coordinate systems and a translation vector that describes the spatial physical distance between the origins of the two coordinate systems.

[0038] To complete this calibration process, the system needs to simultaneously acquire visual and inertial data. Visual data is typically presented as a sequence of timestamped images (i.e., an image-video stream) captured continuously by a camera at a certain frame rate (e.g., 30 frames / second or 60 frames / second). Inertial data is typically presented as timestamped physical measurements continuously acquired by an IMU at a higher frequency (e.g., 200Hz or higher), usually including three-axis angular velocity data measured by a gyroscope and three-axis acceleration data measured by an accelerometer.

[0039] Typically, visual and inertial data need to be collected during the movement of the two-wheeled vehicle. This is because, in a stationary state, the sensor can only observe gravitational acceleration and static images, and the system lacks sufficient dynamic excitation. Without changes in motion, many degrees of freedom describing the relative translation and rotation between the camera and the IMU are mathematically difficult to solve uniquely using equations. However, when the two-wheeled vehicle undergoes certain rotational and translational motions in three-dimensional space, generating time-varying angular velocities and linear accelerations, an effective mathematical relationship can be established between the geometric constraints between visual image frames and the IMU pre-integral values, thereby solving for the rotation matrix and translation vectors included in the extrinsic parameters.

[0040] In some embodiments, the user performs segmented excitation motions of the two-wheeled vehicle. Excitation motions refer to a series of specific attitude change actions applied to the two-wheeled vehicle in order to fully activate and measure the degrees of freedom of the IMU in different spatial axes, such as pitch, yaw, and / or roll.

[0041] Figure 2 A flowchart illustrating an exemplary process of a method 200 for performing segmented excitation motion according to an embodiment of the present disclosure is shown.

[0042] Because two-wheeled vehicles are typically large and heavy, they cannot be easily flipped in the air or subjected to traditional calibration movements such as figure-eight loops, as handheld devices can. Therefore, this disclosure provides a segmented excitation motion that does not require lifting the vehicle body. The user simply keeps the vehicle's tires on the ground and performs simple push-pull and tilt movements by holding the handlebars. This user-guided segmented motion ensures that the IMU receives sufficient dynamic excitation signals in multiple axes, the visual data acquired by the camera includes sufficient scene changes, and avoids purely linear motion. The camera and IMU acquire rotational components including at least two axial degrees of freedom, thus ensuring the mathematical observability of the extrinsic parameter calibration results.

[0043] In step S202, the two-wheeled vehicle can be kept upright and stationary for a preset time in the initial state to synchronously start the acquisition of image sequences and inertial data.

[0044] In some embodiments, the initial state of the two-wheeled vehicle is that the vehicle body is upright. Uprightness means that both wheels of the two-wheeled vehicle are on the ground, and the central plane of the frame is approximately perpendicular to the ground. In this state, the IMU's gravity measurement axis is substantially parallel to the actual gravity direction, which helps the calibration system quickly determine the initial gravity direction and attitude reference.

[0045] In some embodiments, the initial state of the two-wheeled vehicle is that it is naturally and statically placed in the environment. This environment can be a common real-world scenario, such as an outdoor parking lot, street, or courtyard, or an indoor corridor or garage. Specifically, the embodiments of this disclosure do not require the artificial placement of specific calibration boards (e.g., black and white checkerboard patterns, dot arrays, etc.) in the environment; it only requires that the natural environment contains objectively existing visual features with a certain texture (e.g., wall textures, road markings, trees, building edges or outlines, etc.).

[0046] The data collection process can be initiated by the user. In some examples, the user can start data collection by clicking a corresponding button on a smart device such as a smartphone or tablet that is linked to the two-wheeler. In other examples, the data collection process can be triggered and started directly by the vehicle's built-in dashboard screen, touch panel, or physical mechanical buttons on the handlebars.

[0047] In some embodiments, after data acquisition begins, the vehicle body needs to remain stationary for a preset time (e.g., 2 seconds, 3 seconds, etc.). During this stationary waiting period, the user's smart device or the application installed on the two-wheeled vehicle begins to synchronously store the image sequence and IMU data. In some examples, specific preparatory work may include timestamp alignment of the underlying sensors and data stream caching. Furthermore, the static IMU data acquired during this stationary state can be used to evaluate the initial bias and noise level of the IMU, providing more accurate prior measurement information for subsequent optimization of the hand-eye calibration model equations.

[0048] After a preset period of stillness, the application or in-vehicle device (e.g., via voice prompts or vibrations on a mobile phone, or via a display on the vehicle's dashboard or horn) prompts the user to begin moving the two-wheeled vehicle. To ensure the observability of the system calibration, the segmented excitation motion mainly includes tilt excitation motion and steering excitation motion.

[0049] In step S204, a tilting excitation motion can be performed on the two-wheeled vehicle, including tilting the two-wheeled vehicle to the left and right respectively to a preset reference angle and then restoring it to an upright position.

[0050] Specifically, the user can stand beside the two-wheeled vehicle or straddle it, and while keeping the front and rear wheels in contact with the ground, hold the handlebars and tilt the vehicle to the left at a certain angle (e.g., a preset reference angle of approximately 15 to 20 degrees), then pull the vehicle back to its initial upright position. Then, tilt it to the right at the same or similar angle, and return to the upright position again. Performing this tilting excitation introduces a significant dynamic angular velocity excitation to the IMU's roll axis and alters the IMU's three-axis attitude, making the associated external parameter rotational components mathematically significant.

[0051] In step S206, a steering excitation motion can be performed on the two-wheeled vehicle, including moving the two-wheeled vehicle to the left front and right front respectively in a turning manner and then returning it to an upright position.

[0052] Specifically, the user can push or drive the two-wheeled vehicle, turning the handlebars while moving forward to propel the vehicle along a curved trajectory (e.g., an arc or an "S" shape) a distance to the left (e.g., 1 to 2 meters), then straighten the handlebars to return the vehicle to a straight, upright position. Similarly, the user can then move to the right along a curved trajectory with a turn and straighten the handlebars. This steering excitation process not only introduces the linear acceleration required for translation but also applies sufficient rotational excitation to the IMU's yaw and pitch axes. Simultaneously, the forward motion with steering causes a significant change in the camera's field of view, generating sufficient visual parallax.

[0053] After collecting sufficient data, the application indicates that the calibration action or data acquisition process is complete. Collecting sufficient data means that the system monitors the current data stream in real-time in the background and confirms that it meets the calibration requirements. For example, the system internally detects that the number of extracted effective image feature points is sufficient, the translational parallax change between image frames meets the standard, and the variance of the angular velocity and acceleration signals (i.e., motion excitation intensity) on the three axes of the IMU exceeds the preset observability threshold. Meeting these conditions indicates that the current dataset is sufficient to support the convergence of the subsequent optimization algorithm for the hand-eye calibration model, and there will be no problems with extrinsic parameter solution failure or excessive errors.

[0054] It should be noted that although the embodiments and accompanying drawings of this disclosure describe the order of performing the roll excitation motion first and then the steering excitation motion, in practical applications, the specific execution of the segmented excitation motion is not strictly limited. For example, the user can perform the steering excitation motion first and then the roll excitation motion; or alternate between steering and roll during slow advancement. Furthermore, the user can also repeatedly perform at least one of the steering excitation motion and the roll excitation motion as needed. The same calibration effect can be achieved as long as the sequence of excitation actions in a complete data acquisition cycle fully includes the motion components of both axes.

[0055] To ensure the mathematical observability and high accuracy of the final extrinsic parameter calibration results, the system backend can perform real-time verification or guidance during the data acquisition process based on certain acquisition constraints. Specifically, these constraints mainly include: stimulating at least two IMU axes of freedom, ensuring sufficient scene variation in the camera's field of view, and avoiding purely linear motion. The trajectory of the two-wheeled vehicle must contain a significant rotational component, because pure linear translational motion will cause mathematical degradation of the hand-eye calibration model, making it impossible to uniquely and accurately solve the relative rotation matrix between the camera and the IMU.

[0056] After acquiring image sequences and inertial data through the segmented excitation motion of the two-wheeled vehicle as described above, and ensuring that the dataset meets the above acquisition constraints, the system ends the acquisition work and begins the specific algorithm calculation for calibration.

[0057] In step S104, image feature points between adjacent image frames in the image sequence are extracted to determine the relative rotation of the camera between adjacent image frames.

[0058] By extracting and tracking image feature points in adjacent image frames, the relative rotation of the camera between adjacent image frames can be calculated. In some embodiments, the relative translation of the camera between adjacent image frames can also be calculated by extracting and tracking image feature points in adjacent image frames.

[0059] In some examples, step S104 may include: The system detects image feature points in each image frame of an image sequence; establishes a correspondence between feature points between adjacent image frames based on the image feature points to obtain matching point pairs between adjacent image frames; and calculates the relative rotation and relative translation of the camera between adjacent image frames based on the matching point pairs through epipolar geometric constraints.

[0060] Specifically, the first step is to detect image feature points for each image frame in the image sequence. In some embodiments, for each frame of an image sequence (e.g., an image video stream) captured by the camera (e.g., denoted as the k-th frame), I k The system uses feature extraction algorithms to detect pixel regions with significant features (such as corners, edges, or spots) in the image as image feature points.

[0061] This disclosure may employ feature detection algorithms such as FAST (Features from Accelerated Segment Test), ORB (Oriented FAST and Rotated BRIEF), SIFT (Scale-Invariant Feature Transform), or SURF (Speeded Up Robust Features).

[0062] After image feature point extraction is completed, feature point tracking can be performed. This involves establishing the correspondence between feature points in adjacent image frames based on the image feature points, thereby obtaining matching point pairs between adjacent image frames. For two consecutive frames in an image sequence (e.g., the k-th frame), this tracking is performed. I k and the (k+1)th frame image I k+1 The system needs to find out which feature point in the previous frame corresponds to a certain feature point in the current image frame.

[0063] In some embodiments, feature point correspondences between adjacent image frames can be established through feature matching. Specifically, the system calculates a descriptor for each extracted feature point, and determines the closest points as matching point pairs by calculating the Hamming distance or Euclidean distance between the feature point descriptors in two image frames.

[0064] In other embodiments, the correspondence between feature points between adjacent image frames can be established using optical flow. Specifically, based on the assumptions of grayscale invariance and local spatial consistency, the system can use, for example, the KLT (Kanade-Lucas-Tomasi) sparse optical flow tracking algorithm to directly track the pixel motion trajectories of feature points between consecutive image frames, thereby establishing correspondences and generating matching point pairs. Optical flow methods generally have advantages in real-time performance when handling high-speed movement of two-wheeled vehicles because they do not require the computation of complex descriptors.

[0065] Based on the matching point pairs obtained in the preceding steps, the relative rotation and translation of the camera between adjacent image frames can be calculated using epipolar geometry constraints. In stereo vision theory, epipolar geometry describes the projective geometric constraints of the same 3D scene on two 2D images from different viewpoints. In some embodiments, the system uses several sets (e.g., more than 5 sets) of matching point pairs determined in two adjacent image frames to construct a fundamental matrix or essential matrix. Then, by performing singular value decomposition on this matrix, the relative pose changes of the camera from acquiring frame k to acquiring frame k+1 are calculated, including the relative rotation of the camera (which can be denoted as R). c ) and relative translation (which can be denoted as t) c ).

[0066] In other instances, the relative rotation and translation of the camera between adjacent image frames can also be determined using other suitable matching point constraints.

[0067] In step S106, the inertial data between the corresponding times of adjacent image frames is pre-integrated to determine the relative rotation of the IMU between adjacent times.

[0068] Because the sampling frequencies of the camera and the inertial measurement unit (IMU) are typically inconsistent, with the IMU's sampling frequency (e.g., 200 Hz or higher) being much higher than the camera's frame rate (e.g., 30 Hz), the IMU typically generates dozens of high-frequency raw measurement inertial data points during the time interval between the camera acquiring two adjacent frames (i.e., the k-th frame and the (k+1)-th frame). This raw measurement inertial data can include three-axis angular velocities measured by gyroscopes and linear accelerations measured by accelerometers.

[0069] In order to align sensor data of different frequencies on the timeline, embodiments of this disclosure employ IMU pre-integration to solve for the relative rotation of the IMU.

[0070] Specifically, the system can perform continuous integration on the acquired IMU inertial data based on the raw IMU gyroscope data within the aforementioned time interval, using the IMU body coordinate system at the k-th frame as a reference. During the pre-integration calculation, the system can use the IMU hardware attitude at the k-th frame as the initial reference state (i.e., considered as an initial reference without rotation). The initial attitude can be the IMU inertial data acquired during the static waiting time in the segmented excitation acquisition process. Then, the system can process each IMU sampling point within a certain time interval sequentially along the time axis. For each IMU sampling time step, the system can multiply the angular velocity measured by the gyroscope at the current moment (in some examples, the estimated gyroscope bias can be subtracted) with that time step to calculate the three-dimensional rotation increment. In some examples, based on the kinematic equations, these continuous rotation increments can be multiplied and accumulated sequentially in time using quaternions or exponential mapping operations of the rotation matrix.

[0071] When the cumulative calculation progresses from frame k to frame (k+1), the final product represents the total rotational change of the IMU body within the corresponding time interval of these adjacent image frames, i.e., the relative rotation R. i This pre-integration process not only transforms multiple discrete high-frequency IMU measurements into a single relative motion constraint independent of the absolute pose of the two-wheeled vehicle in the global coordinate system, but also reduces the amount of repetitive integration calculations in the subsequent optimization of the hand-eye calibration model equations, thereby accurately and efficiently extracting the IMU's own rotation component within the corresponding time period.

[0072] In step S108, the rotation matrix between the camera and the IMU is determined based on the relative rotation of the camera and the relative rotation of the IMU.

[0073] In some embodiments, a hand-eye calibration model can be constructed based on the relative rotation of the camera and the relative rotation of the IMU. Using the hand-eye calibration model, the relative rotation relationship between the camera and the IMU can be calculated. Since the camera and IMU are rigidly fixed to the two-wheeled vehicle, their absolute relative attitude in space is constant. When the two-wheeled vehicle performs the segmented excitation motion as described above, the relative rotation observed by the camera in its own coordinate system (i.e., R obtained from step S104) c The relative rotation between the IMU and the IMU measured in its own coordinate system (i.e., R obtained from step S106) iIn essence, they describe the same spatial motion, differing only in a fixed coordinate system transformation matrix.

[0074] In some embodiments, the system utilizes the corresponding R within the same time period c and R i Establish hand-eye calibration constraint equations, which can be expressed as formulas. ,in Let be the rotation matrix between the camera and the IMU to be determined.

[0075] In other embodiments, the rotation matrix between the camera and the IMU can also be determined using any other suitable spatial transformation model based on the relative rotation of the camera and the relative rotation of the IMU.

[0076] As the two-wheeled vehicle generates rotational motions along different axes during segmented excitation motion, the system can collect multiple sets of corresponding motion constraints. These constraints can be solved nonlinearly using linear algebraic methods (such as singular value decomposition (SVD)) or by constructing a least-squares problem to obtain the rotation matrix R that satisfies all motion constraints.

[0077] It should be understood that the rotation matrix obtained in step S108 is a preliminary estimate of the relative attitude offset between the camera and the IMU. This rotation matrix is ​​optimized as the initial value of the final calibration extrinsic parameters by minimizing the equation residuals of the hand-eye calibration model.

[0078] In step S110, the calibration extrinsic parameters between the camera and the IMU are obtained by optimizing the objective function based on the sliding window to minimize the residual of the equation based on the hand-eye calibration model. The objective function is constructed by using the rotation matrix as the extrinsic parameter to be optimized.

[0079] In some embodiments, extrinsic parameter optimization can be performed using a sliding window based solely on the relative motion characteristics of the camera and the IMU. Specifically, the system can maintain a time window containing a certain number (e.g., the most recent 10 or 20 frames) of consecutive image frames and the corresponding IMU inertial data, i.e., a sliding window. As new data is continuously acquired, old data is removed from the window, and new data enters the window. Within this sliding window, the system can collect the relative rotation of the camera between multiple adjacent frames calculated in step S104, and the relative rotation of the IMU within the corresponding time period calculated in step S106.

[0080] In some embodiments, the IMU bias generated during the segmented excitation motion of the two-wheeled vehicle from a stationary state to the end of acquisition can be determined. Then, based on the matching point pairs and the relative rotation of the camera, the corresponding 3D map points are determined by triangulation. The rotation matrix can be used as the extrinsic parameter to be optimized, and the camera pose, IMU bias, and 3D map points can be used together as optimization parameters of the objective function. The camera pose is determined by the relative rotation of the camera.

[0081] Based on multiple sets of corresponding relative motion data within these sliding windows, the system can construct a joint optimization objective function. The core of this objective function lies in constructing the residuals of a multi-frame hand-eye calibration equation system, such as the difference between the expected motion transformed from camera motion to IMU coordinates and the actual motion measured by the IMU. In some examples, the residuals of the hand-eye calibration model equations can be minimized by nonlinearly iterating to minimize the visual reprojection error and the IMU pre-integration error, thereby optimizing the extrinsic parameters to obtain the optimized extrinsic parameters between the camera and the IMU. The visual reprojection error is determined at least based on 3D map points and camera pose.

[0082] In some embodiments, the objective function can also be constructed by using the rotation matrix and the translation vector between the camera and the IMU as extrinsic parameters to be optimized. Within this sliding window, the system can collect the relative rotation and relative translation of the camera between multiple adjacent frames calculated in step S104, and the relative rotation of the IMU within the corresponding time period calculated in step S106, as well as the relative translation of the IMU obtained through further integration. In this case, the camera pose is determined by the relative rotation and relative translation.

[0083] In some examples, the system can iteratively fine-tune the extrinsic parameters (i.e., the initially determined rotation matrix, and in some examples, the translation vector between the camera and the IMU) used as optimization variables through a nonlinear least squares optimization algorithm (such as the Gauss-Newton algorithm), so that the sum of this residual across all historical image frame pairs within the sliding window is minimized. In this way, the sliding window can integrate observation information from multiple frames, effectively smoothing out random noise introduced by a single feature extraction or single integration calculation, thereby calculating more accurate calibration extrinsic parameters than the initial calculation in step S108.

[0084] Because the relative translation calculated by a monocular camera through epipolar geometry may often lack an absolutely true physical scale, and the raw measurement inertial data of the IMU may contain IMU bias that varies slowly with temperature and time (initial bias at zero input, also known as zero bias), there is a drift error in the IMU pre-integral quantity.

[0085] To improve the calibration accuracy of extrinsic parameters in natural scenes, in some embodiments, the joint optimization objective function based on sliding windows can further incorporate 3D map points, IMU bias, and specific camera poses for joint optimization in addition to using extrinsic parameters as variables to be optimized.

[0086] Figure 3 A flowchart illustrating an exemplary process of a method 300 for optimizing the equations of a hand-eye calibration model according to an embodiment of the present disclosure is shown.

[0087] In step S302, the IMU deviation generated during the process from a stationary state to the end of data acquisition during the segmented excitation motion of the two-wheeled vehicle can be determined.

[0088] In step S304, the 3D map points corresponding to the image feature points can be determined by triangulation based on the matching point pairs, the relative rotation and relative translation of the camera.

[0089] In step S306, the rotation matrix and the translation vector between the camera and the IMU can be used as external parameters to be optimized, and the camera pose, IMU bias, and 3D map points can be used as optimization parameters of the objective function. The camera pose is determined by the relative rotation and relative translation of the camera.

[0090] In step S308, the residuals of the hand-eye calibration model equations can be minimized by nonlinear iteration based on a sliding window to minimize the visual reprojection error and the IMU pre-integration error, thereby optimizing the extrinsic parameters to obtain the optimized extrinsic parameters between the camera and the IMU. The visual reprojection error is determined at least based on the 3D map points and the camera pose.

[0091] By utilizing a sliding window optimization model that includes 3D map points, IMU bias, camera pose, and external parameters to be optimized, and by simultaneously minimizing visual reprojection error and IMU pre-integration error through nonlinear iteration, the inherent zero-bias drift of the IMU sensor over time can be effectively estimated and corrected. At the same time, the random noise caused by single image feature extraction is smoothed by physical projection constraints in multi-view 3D space, enabling visual observation and inertial physical measurement to cross-verify in the underlying mathematical model. This achieves in-depth correction of multi-source underlying errors of the system, improving the convergence stability, robustness, and final external parameter calibration accuracy of camera and IMU external parameter self-calibration in natural life scenarios.

[0092] The following describes in detail each step of method 300.

[0093] In step S302, the IMU deviation generated during the process from a stationary state to the end of data acquisition during the segmented excitation motion of the two-wheeled vehicle is determined.

[0094] Specifically, IMU bias (initial bias at zero input) refers to the inherent offset of an inertial sensor, caused by physical limitations in manufacturing processes and the characteristics of its internal electronic components, resulting in a non-zero reading of its output angular velocity (gyroscope) or linear acceleration (accelerometer) when it is absolutely stationary without any external force. Furthermore, this bias will slowly undergo random walk (drift) as the device operates for an extended period and temperature changes. If these biases are not estimated and eliminated, they will accumulate over time during pre-integration processing, leading to significant errors in extrinsic parameter calculations. Therefore, the system can use the purely static IMU data collected during the preset time the two-wheeled vehicle remains stationary in step S202 to calculate the initial gyroscope and accelerometer biases. Subsequently, before the end of the acquisition of the entire segmented excitation motion, the system treats this IMU bias as a dynamically changing internal state variable and continuously estimates and corrects it in subsequent joint optimization.

[0095] In step S304, the 3D map points corresponding to the image feature points can be determined by triangulation based on the matching point pairs, the relative rotation and relative translation of the camera.

[0096] Specifically, since the two-dimensional images captured by a monocular camera lose physical depth information, the system needs to recover the three-dimensional structure of the scene through multi-view geometry. In step S104, the system has obtained the matching point pairs between two adjacent frames (i.e., the pixel coordinates of the same spatial physical point on two-dimensional images from different viewpoints) and calculated the relative rotation and relative translation of the camera between these two frames.

[0097] In some embodiments, 3D map points corresponding to image feature points can be constructed based on triangulation. During triangulation, the system can use the optical center position of the first frame of the image sequence as the starting point and simulate emitting a virtual ray through the pixel coordinates of the feature point in that image frame into 3D space; simultaneously, using the optical center position of the second frame as the starting point, another virtual ray is emitted through its corresponding matching pixel coordinates. Ideally, these two rays will intersect at a point in 3D space, and the 3D coordinates of this intersection point are the 3D map point corresponding to the feature point. In some examples, due to pixel quantization errors and feature point extraction errors in actual calculations, the two rays may not intersect perfectly. The system can use the least squares method to solve for the midpoint of the common perpendicular of these two rays as the estimated spatial coordinates of the 3D map point.

[0098] In step S306, the rotation matrix and translation vector can be used as external parameters to be optimized, and the camera pose, IMU bias, and 3D map points can be used as optimization parameters of the objective function. The camera pose is determined by the relative rotation and relative translation of the camera.

[0099] In some embodiments, the system can combine all unknown state variables affecting the observation results into a set of optimization parameters, such as a state vector. This set can include not only the extrinsic variables to be determined, i.e., the rotation matrix and translation vector between the camera and the IMU, but also the camera pose corresponding to each image frame within the sliding window, the dynamically fluctuating IMU bias, and the spatial coordinates of each 3D map point in the environment. The camera pose can be the absolute 3D position and rotation direction of the camera in the global reference frame, and its initial value is calculated by accumulating the aforementioned relative rotation and relative translation.

[0100] In step S308, the residuals of the hand-eye calibration model equations can be minimized by nonlinear iteration based on a sliding window to minimize the visual reprojection error and the IMU pre-integration error, thereby optimizing the extrinsic parameters to obtain the optimized extrinsic parameters between the camera and the IMU. The visual reprojection error is determined at least based on the 3D map points and the camera pose.

[0101] In some embodiments, the system constructs a joint optimization objective function based on a sliding window. This objective function can be a cost function, the magnitude of which reflects the degree of discrepancy between the currently estimated optimization parameters and the actual sensor physical measurements, that is, the error between the parameters to be optimized and the actual sensor physical measurements.

[0102] In some embodiments, the objective function mainly consists of visual reprojection error and IMU pre-integration error.

[0103] Specifically, assuming the estimated coordinates of the 3D map point and the camera pose are completely accurate, based on the camera's internal optical parameters (e.g., focal length, optical center, etc.) and the pinhole camera imaging model, this 3D map point can be reprojected from 3D space back onto the 2D image plane, calculating a theoretical 2D pixel coordinate. Because the estimated value deviates from the actual situation, there will be a distance difference between this theoretical 2D pixel coordinate and the actual 2D pixel coordinate of the feature point extracted by the system in the image. The square of this distance is the camera's visual reprojection error.

[0104] For an IMU, by using the IMU pose and velocity from the previous moment, plus the pre-integrated motion after deducting IMU bias, the current IMU pose and velocity can be derived or predicted using kinematic laws. This derived pose state is then compared with the current IMU pose state to be optimized in the optimization parameter set; the difference between the two is the IMU pre-integration error.

[0105] In some embodiments, the system can employ a nonlinear iterative minimization method to solve the objective function. Since rotations in 3D space (e.g., represented by Lie groups or quaternions) and camera projection models are highly nonlinear mathematical processes, the system cannot directly obtain the optimal solution using a simple formula. Therefore, the system uses a nonlinear optimization algorithm to calculate the partial derivatives or gradients of the objective function with respect to each optimization parameter based on the current initial parameter values. Then, along the direction of fastest error descent, it performs joint updates and iterations for all optimization parameters, including extrinsic variables, camera pose, 3D map points, and IMU bias. The system can continuously repeat the iterative process of calculating error, calculating gradients, and updating parameters based on a sliding window of time or image frames.

[0106] The nonlinear iteration can end when the sum of the visual reprojection error and the IMU pre-integration error decreases and converges to a minimum value, or when the parameter update amount is less than the set threshold. At this point, each variable in the optimized parameter set has reached an optimal self-consistent state, and the extrinsic parameters that can be directly extracted are the high-precision camera and IMU calibration extrinsic parameters that have been constrained by multi-source data and have eliminated various sensor noises and drifts.

[0107] In real-world natural scenarios, the extrinsic parameter self-calibration process of two-wheeled vehicles may be affected by complex external environments. Due to the variability of the physical environment and the rapid switching of camera perspectives when the user performs segmented excitation motion, visual feature extraction may degrade or tracking errors may accumulate, thereby affecting the accuracy of extrinsic parameter calibration and the convergence stability of joint optimization. To address this, embodiments of this disclosure provide targeted optimization schemes for various typical situations that may be encountered during calibration in natural scenarios.

[0108] The embodiments disclosed herein provide an optimized solution to the problem of feature matching ambiguity in repetitive scenarios.

[0109] In scenarios such as underground parking lots, indoor corridors, or roads with regular textures, there are many similar load-bearing columns, tile joints, or identical doors, creating repetitive structural features. In such environments, mismatches are easily caused; similar feature points in different physical locations can easily generate incorrect matching point pairs between adjacent image frames. This directly leads to ambiguity and computational bias when the system uses epipolar geometry to calculate the relative rotation of the camera.

[0110] In some embodiments, scene feature recognition can be performed on each image frame in the image sequence based on a sliding window; in response to detecting a scene with repetitive structural features between consecutive image frames, at least one of the following steps is performed: establishing local map constraints on historical image frames of scenes with repetitive structural features in the sliding window through inverse depth parameterization; and increasing the weight of IMU pre-integration error in the joint optimization objective function.

[0111] In some embodiments, the scene feature recognition can be performed using a bag-of-words model. Specifically, the system can utilize the bag-of-words model's image classification and retrieval algorithm to pre-train or construct a visual dictionary before or during system operation. This dictionary clusters various feature point descriptors into standard visual words. When the camera on the two-wheeled vehicle captures image frames in real time, the system extracts all image feature points in the image frame and maps their descriptors to the visual dictionary, counting the frequency of each visual word in the current frame. Thus, each image frame can be transformed and represented as a multidimensional vector composed of word frequencies. By calculating the similarity between the multidimensional vector of the current image frame and the multidimensional vectors of historical image frames in the sliding window (e.g., calculating the dot product or cosine similarity of the vectors), the system can identify whether it has entered a scene containing a large number of repetitive structural features or returned to a previously visited location.

[0112] In response to the detection of scenes containing repetitive structural features between consecutive image frames, the system can trigger a targeted visual and inertial fusion adjustment strategy to prevent mismatches from contaminating the optimization objective function.

[0113] In terms of visual constraints, the system can establish local map constraints on historical image frames of scenes with repetitive structural features in a sliding window through, for example, inverse depth parameterization.

[0114] Specifically, triangulation typically uses 3D spatial coordinates (X, Y, Z) to represent 3D map points. However, in repetitive scenes, due to mismatches, the depth of feature points calculated by triangulation (i.e., the distance Z from the camera) may fluctuate drastically, even reaching infinity or becoming negative, causing the optimization process to diverge. Using inverse depth parameterization, a 3D point can be represented as (u, v, 1 / Z), where u and v are the pixel coordinates of the feature point on the reference image frame, and 1 / Z is the reciprocal of the depth. The numerical distribution of inverse depth is closer to a normal distribution, ideally representing points at infinity. Based on this, the system can forcibly anchor the camera pose of the current frame to an earlier accumulated, structurally stable, and unaffected historical local map within the sliding window, using inverse depth to construct a smoother and more robust visual reprojection residual, thereby suppressing ambiguity drift caused by mismatches.

[0115] Regarding the weighting of visual and inertial data fusion, the system can increase the weight of IMU pre-integration error in the joint optimization objective function.

[0116] Specifically, vision-inertial joint optimization can be viewed as a probabilistic framework for multi-sensor information fusion. In the objective function, the visual reprojection error and IMU pre-integration error each carry an information matrix (e.g., the inverse of the covariance matrix) to characterize the system's weighting of that data source. When the probability of matching ambiguity increases sharply due to the detection of repetition in the visual scene, the confidence of the visual observation data decreases significantly. At this point, the system can dynamically lower the information matrix weight of the visual error term and correspondingly increase the information matrix weight of the IMU pre-integration error term. Because the inertial physical measurements of the IMU do not depend on external lighting and texture, they are not affected by any interference from repetitive scenes. By dynamically adjusting the weights, the system will have greater confidence in the IMU's predicted trajectory during periods containing repetitive scenes, relying on the IMU's short-term high-precision integration to maintain the correct inference of the two-wheeled vehicle's pose until the image frame no longer contains repetitive scenes, at which point the visual and inertial weights can be restored to normal.

[0117] Through the synergistic effect of scene recognition, inverse depth parameterization, and dynamic adjustment of IMU weights, the embodiments provided in this disclosure can shield visual mismatch interference in repetitive environments, thereby ensuring the continuity and high accuracy of camera and IMU calibration extrinsic parameters in repetitive scenarios.

[0118] The embodiments disclosed herein provide an optimized solution to address the visual degradation problem caused by low-texture regions.

[0119] In natural scenes such as the white walls, smooth floors, or textureless corridors of underground parking lots, feature extraction algorithms may be unable to extract a sufficient number of salient image feature points (e.g., corner points), and these feature points may be unevenly distributed across the image plane (e.g., a large number concentrated in a corner at the edge of the field of view). This sparse and uneven distribution of feature points leads to a significant decrease in the strength of visual constraints, especially when solving for pose changes due to relative camera rotation and translation, which can easily cause a substantial reduction in the accuracy of rotation estimation or even divergence in the solution.

[0120] Figure 4 A flowchart illustrating an exemplary process of an optimization method 400 for low-texture regions according to an embodiment of the present disclosure is shown.

[0121] In step S402, it can be determined whether the image frame includes low-texture regions by detecting the number and distribution of image feature points in the image frame.

[0122] In some embodiments, after performing preliminary feature extraction on the current image frame, the system can count the total number of extracted feature points and compare it with a preset threshold. Simultaneously, the system can divide the two-dimensional image plane into multiple grids and count the proportion of grids containing feature points to the total number of grids. If the total number of feature points is lower than the set threshold, or if feature points are excessively concentrated in a few grids, resulting in substandard distribution uniformity, the system can determine that the current image frame contains low-texture regions, thereby triggering feature enhancement extraction and constraint optimization mechanisms for these low-texture regions.

[0123] In step S404, in response to determining that the image frame includes a low-texture region, feature extraction in step S104 can be performed on adjacent image frames based on a multi-scale pyramid to establish the correspondence between feature points between adjacent image frames and obtain matching point pairs between adjacent image frames.

[0124] In some embodiments, to extract as many usable feature points as possible in low-texture regions, the system constructs an image multi-scale pyramid with L levels and performs feature detection independently on each level. To adapt to the resolution and contrast attenuation of images at different levels, the feature point detection threshold can be adaptively adjusted. In some examples, the FAST corner detection algorithm can be used, and its detection threshold can be denoted as... And it will be adaptively adjusted according to the following formula:

[0125] in, The baseline threshold is set at the original image scale (layer 0); This is the scaling factor. In some examples, the scaling factor can be set to 1. . This represents the current level index of the image pyramid, for example... .

[0126] Using the adaptive threshold formula described above, as the image pyramid level increases, the image becomes more blurred, and the feature detection threshold is dynamically amplified or adjusted. This ensures that the system can extract a stable number of image feature points at different scales. Utilizing this multi-scale pyramid feature extraction strategy, the system can increase the cardinality of available feature points in low-texture regions, thereby improving the accuracy of matching point pairs.

[0127] In step S406, the relative rotation and relative translation of the camera between adjacent image frames can be calculated based on the matching point pairs through epipolar geometric constraints.

[0128] Specifically, the system can utilize the large number of matching point pairs extracted through the multi-scale pyramid in step S404 to reconstruct the essential matrix or fundamental matrix for epipolar geometry calculation. Because the number and spatial distribution of matching point pairs are improved, the calculated camera relative pose, especially the relative rotation, will have higher reliability, thus effectively avoiding matrix degradation or singular solutions caused by insufficient feature points.

[0129] In step S408, the line segment residuals calculated based on line segment feature extraction and the vanishing point residuals calculated based on vanishing point constraints can be added to the objective function optimized in step S110, and the point residuals constructed based on 3D map points can be used to jointly optimize the external parameter variables to be optimized.

[0130] In some embodiments, based on the detection of low-texture regions, all steps in steps S404, S406, and S408 can be performed.

[0131] In some embodiments, based on the detection of low-texture regions, steps S404 and S406 may be performed only, or step S408 may be performed only.

[0132] In low-texture areas such as walls or floors, while point features (e.g., corner points) are lacking, line features (e.g., wall edges, door frames, parking lines) are typically significantly increased. Furthermore, the vanishing points formed by parallel lines after perspective projection in 3D space can serve as constraints on the relative rotation of the camera in such cases.

[0133] Therefore, in some embodiments, the system can introduce segment residuals and vanishing point residuals in sliding window joint optimization to assist in optimization.

[0134] To address line segment residuals, in some embodiments, the system can employ line segment extraction algorithms such as LSD (Line Segment Detector) to detect straight line segments in the image. For each detected two-dimensional line segment endpoint pair... The system calculates the line segment residuals that serve as visual constraints. Its mathematical expression can be:

[0135] in, Let be the unit normal vector of the line segment in the image plane. Given the current camera pose and extrinsic parameters, this represents the theoretical projection point of a 3D line segment onto the 2D image plane. The residual reflects the distance between the theoretical projection point and the perpendicular direction of the actually observed line segment.

[0136] Regarding vanishing point residuals, since architectural scenes typically follow the Manhattan World Assumption, there are numerous parallel straight lines. The system can estimate the principal direction vector in the scene using a vanishing point detection algorithm. To strengthen the constraint on the camera rotation matrix in low-texture regions, the system can construct vanishing point residuals. Its mathematical expression is:

[0137] in, The known or estimated 3D principal direction vectors of the scene. This represents the rotation matrix from the reference world coordinate system to the current camera coordinate system, including the extrinsic parameters of the camera and IMU to be optimized, as well as the pose of the IMU itself. This represents the 3D direction vector of the corresponding parallel lines extracted from space. This residual term ensures that 3D lines within the same parallel line system still strictly adhere to the convergence geometry of the vanishing point after the camera rotates.

[0138] By fusing the line segment residual, vanishing point residual, and feature point-based residual (i.e., the aforementioned visual reprojection error) in the objective function, the embodiments provided in this disclosure can enhance the integrity of visual constraints by utilizing the geometric characteristics of line segments and vanishing points, which are still widely present in low-texture scenes and are highly sensitive to rotational attitude, rather than relying solely on visual reprojection. This enables the system to still calculate the calibration extrinsic parameters of the camera and IMU with high accuracy and stability when faced with low-texture interference where it is difficult to extract effective feature points.

[0139] The embodiments disclosed herein also provide an optimized solution to the problem of feature point tracking loss under large-angle rotation.

[0140] When performing steering-induced motions in natural scenes, two-wheeled vehicles have greater maneuverability than four-wheeled vehicles. When a two-wheeled vehicle makes a large-angle, rapid turn (e.g., a sharp turn or a large swing of the handlebars while stationary), the camera's field of view changes drastically. At this time, image feature points extracted in the previous frame may quickly move out of the camera's field of view or be occluded by vehicle / environmental objects in the current frame, causing traditional optical flow methods or short-range feature matching to fail, and the feature tracking chain to break. With the loss of visual feature tracking, the system will degenerate into relying solely on the IMU for pose inference. Furthermore, large-angle, drastic movements may cause the IMU pre-integration error to accumulate rapidly, leading to divergence or calibration failure in the extrinsic parameter joint optimization process.

[0141] To overcome this deficiency, this disclosure constructs a bidirectional tracking anti-loss mechanism based on IMU prediction and visual correction. In some embodiments, in response to determining, based on IMU pre-integration, that a motion exceeding a preset rotation angle threshold has occurred, the following operations may be performed: expanding the spatial search radius when establishing the correspondence between feature points in adjacent image frames; and re-matching the image feature points of the current image frame with the image feature points of historical image frames to reconstruct the correspondence between feature points.

[0142] Specifically, the system can monitor or detect high-frequency IMU pre-integration results in real time or within a subsequent sliding window. When the relative rotation angle between adjacent image frames derived from pre-integration exceeds a preset rotation angle threshold (e.g., the system determines that it is currently in a state of rapid rotation), the system can expand the visual spatial search radius. Under normal, stable motion conditions, to save computational power and reduce the false matching rate, the system typically searches for matching points in the current frame only within a small pixel neighborhood radius of the feature point location in the previous frame, such as 10 to 20 pixels. However, during rapid turns at large angles, the pixel displacement of feature points between adjacent frames increases rapidly, and the original search radius may not be able to cover the new locations of the feature points. Therefore, the system can dynamically and adaptively expand the spatial search radius for feature matching or optical flow tracking based on the displacement prior information derived from IMU pre-integration, for example, expanding it to 50 pixels or a large neighborhood centered on the epipolar position predicted by the IMU, thereby re-tracking image feature points that have undergone large displacements in a wider area of ​​the current image frame.

[0143] In some embodiments, in addition to expanding the search radius, re-matching and localization with historical image frames can also be performed. When the camera's perspective changes significantly, causing some feature points to have indeed moved out of the field of view or been completely occluded in the immediately preceding image frame, resulting in a physical break in the tracking chain, the system does not discard these lost feature points. The system can extract new image feature points from the current image frame and perform a wide-ranging descriptor matching attempt in a local feature library composed of multiple historical image frames (e.g., historical keyframes) held in a sliding window, which is local relocalization. Once the two-wheeled vehicle rotates back to center from a large angle or some feature points re-enter the field of view, the system can promptly find highly similar matching point pairs between the current image frame and earlier historical image frames, thereby reconnecting the broken feature point correspondences.

[0144] Through the aforementioned bidirectional tracking mechanism based on IMU pre-integration triggering dynamic path expansion and historical frame relocation, the method provided in this disclosure enables two-wheeled vehicles to maintain highly consistent visual feature tracking even when facing extreme dynamic excitation movements such as large-angle rotations. This method effectively curbs IMU large-angle integration error drift caused by visual tracking loss, while continuously providing stable and uninterrupted visual reprojection error constraints for subsequent sliding window joint optimization. This improves the robustness and calibration success rate of the extrinsic parameter self-calibration system under severe maneuvering scenarios.

[0145] The embodiments disclosed herein also provide optimization solutions for dealing with interference from dynamic obstacles.

[0146] In real-world road or everyday scenarios, there are often numerous dynamic objects, such as pedestrians, vehicles, or pets. Because the underlying mathematical model of visual-inertial joint optimization heavily relies on the assumption of a static scene—that is, that the extracted visual feature points are absolutely stationary in three-dimensional space—mistaking feature points on dynamic objects for background feature points leads to incorrect visual data associations. This induces the system to calculate incorrect camera poses, severely impacting the joint optimization process of extrinsic parameters and resulting in significant deviations in the calibration results.

[0147] To completely eliminate dynamic feature points, this disclosure provides a dual filtering mechanism based on semantic mask elimination and IMU motion consistency verification. During the extrinsic parameter self-calibration process, the input original image sequence enters the feature extraction and tracking module to obtain candidate image feature points on the one hand, and the dynamic object detection and verification process on the other hand.

[0148] In some embodiments, the presence of dynamic objects in an image frame can be determined based on image feature points extracted from the image frame; the image frame can be semantically segmented to determine the dynamic regions in the image frame corresponding to the dynamic objects; a binary mask corresponding to the dynamic regions can be generated, and image feature points included in the dynamic regions can be removed using the binary mask.

[0149] Specifically, the system can invoke a lightweight image semantic segmentation network (such as MobileNet or other lightweight architectures) in real time and in parallel to adapt to onboard computing resources. This network performs pixel-by-pixel semantic classification on the input image frames and outputs a category confidence map.

[0150] In some examples, the system can predefine a set of potential dynamic object categories (e.g., including people, cars, bicycles, cats, dogs, etc.). When the network identifies pixels in an image that belong to these potential dynamic categories, the system classifies them as dynamic regions corresponding to dynamic objects and generates a binary mask. In this binary mask, dynamic regions and static background regions can be distinguished and labeled with two different numerical values. For example, the pixel value of a dynamic region can be labeled as, for example, 255 or 1, to indicate that it needs to be removed, while the pixel value of a static background region can be labeled as 0, indicating that it should be retained.

[0151] In some embodiments, the system maps the pixel coordinates of the previously extracted candidate image feature points onto the binary mask image, directly eliminating all image feature points that fall into the dynamic region, thereby initially obtaining the remaining static feature points.

[0152] In some embodiments, the binary mask may be subjected to morphological dilation.

[0153] Specifically, considering that lightweight semantic segmentation networks may have pixel deviations during edge segmentation, or that dynamic objects may produce motion blur and shadows in the image due to rapid movement, causing some feature points belonging to dynamic objects to fall outside the mask boundaries and not be covered by the mask, the system can perform morphological dilation on the generated binary mask image before removing feature points using a binary mask. This dilation process expands the boundary of the dynamic region outward by several preset pixels, increasing the range of the dynamic object. Through this dilation process, it is possible to minimize the mislabeling of blurred feature points at the boundaries of dynamic objects as static background points, thereby further improving the accuracy of dynamic feature removal.

[0154] In some cases, relying solely on semantic segmentation networks may have limitations. For example, some objects that do not belong to the preset categories may move (e.g., leaves blown by the wind), or objects that the semantic network failed to recognize may move. Therefore, the system can further perform IMU motion consistency scoring on the remaining or complete image feature points.

[0155] In some embodiments, the epipolar positions of unremoved image feature points in the next image frame can be predicted based on IMU pre-integration; based on the epipolar positions, the number of consecutive tracking frames of image feature points, visual reprojection error, and epipolar residuals, IMU motion consistency verification is performed on image frames containing dynamic objects; based on the IMU motion consistency verification results, the image feature points that ultimately participate in the joint optimization objective function are determined.

[0156] Specifically, the IMU's immunity to external visual interference can be used to assist in verification. The system can use the short-term, high-precision relative rotation of the camera derived from the IMU's pre-integration to calculate the geometric line that a feature point in the previous image frame should project onto in the current image frame, which is the epipolar line.

[0157] Then, the system can comprehensively evaluate the motion consistency of each remaining or complete feature point using multiple metrics. First, using the epipolar residual, the shortest vertical distance from the actual tracked position of the feature point in the current image frame to the epipolar position predicted by the IMU can be calculated. If this distance is too large, it indicates that the point has undergone independent displacement in 3D space that is not caused by the camera's own motion, meaning the point is dynamic. Then, the visual reprojection error of the point in multiple historical image frames can be evaluated; points with larger error fluctuations are more likely to be dynamic and unstable. The system can also record the number of consecutive tracking frames. The longer the tracking time, the higher the probability that the point belongs to a stable, static background.

[0158] The system calculates a comprehensive consistency score based on a weighted average of the aforementioned indicators (epidural residual, visual reprojection error, and tracking time). Only when this score exceeds a set threshold, indicating that the image feature point conforms to the static geometric consistency of the IMU inference, will the image feature point be ultimately retained and enter the subsequent joint optimization.

[0159] Therefore, through this joint processing mechanism of semantic network prior filtering, mask inflation, and IMU geometric consistency verification, this disclosure can accurately remove various known or unknown dynamic obstacle interferences from the optimization data stream in complex real-world road environments. Only pure static background feature points are retained for the sliding window joint optimization solution step, thus greatly avoiding the impact of the dynamic environment on the calibration system and achieving high-precision extrinsic parameter self-calibration of two-wheeled vehicles in natural life scenarios.

[0160] It should be understood that, Figures 2 to 4 The specific processes shown are for illustrative purposes only and not for limiting purposes.

[0161] According to embodiments of this disclosure, an image processing apparatus is also provided. Figure 5 An exemplary block diagram of an apparatus 500 for calibrating the extrinsic parameters of a camera and an inertial measurement unit (IMU) on a two-wheeled vehicle, according to an embodiment of the present disclosure, is shown.

[0162] The image processing apparatus 500 may include the following units.

[0163] The data acquisition unit 510 is configured to acquire image sequences and inertial data synchronously collected by the camera and the IMU during the process of performing segmented excitation motion on the two-wheeled vehicle, wherein the segmented excitation motion includes motion with at least two axial degrees of freedom.

[0164] The visual processing unit 520 is configured to extract image feature points between adjacent image frames in an image sequence to determine the relative rotation of the camera between adjacent image frames.

[0165] The inertial processing unit 530 is configured to perform pre-integration processing on the inertial data between times corresponding to adjacent image frames to determine the relative rotation of the IMU between adjacent times.

[0166] The model building unit 540 is configured to build a hand-eye calibration model based on the relative rotation of the camera and the relative rotation of the IMU to determine the rotation matrix between the camera and the IMU.

[0167] The optimization unit 550 is configured to construct a joint optimization objective function based on a sliding window, using the rotation matrix as the extrinsic parameter to be optimized, so as to jointly solve the calibration extrinsic parameters between the camera and the IMU through the objective function.

[0168] Here, the operation of each unit of the device used to calibrate the extrinsic parameters of the camera and the inertial measurement unit (IMU) on the two-wheeled vehicle is similar to the operation of steps S102 to S110 described above, and will not be repeated here.

[0169] According to another aspect of this disclosure, an electronic circuit is also provided, including a circuit configured to perform the steps of the above-described method.

[0170] According to another aspect of this disclosure, an electronic device is also provided, comprising: a processor; and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform the methods described above.

[0171] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing a program is also provided, the program including instructions that, when executed by a processor of an electronic device, cause the electronic device to perform the method described above.

[0172] According to another aspect of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described method.

[0173] According to another aspect of this disclosure, a two-wheeled vehicle is also provided, including the aforementioned electronic equipment.

[0174] See Figure 6The block diagram shown is an example of an electronic device according to exemplary embodiments of the present disclosure. Electronic device 600 will now be described as an example of a hardware device (electronic device) that can be applied to various aspects of the present disclosure. Electronic device 600 can be any machine configured to perform processing and / or computation, and can be, but is not limited to, a workstation, server, desktop computer, laptop computer, tablet computer, personal digital assistant, robot, smartphone, in-vehicle computer, or any combination thereof. The image processing method 100 described above can be implemented wholly or at least partially by electronic device 600 or similar devices or systems.

[0175] Electronic device 600 may include elements that connect to or communicate with bus 602 (possibly via one or more interfaces). For example, electronic device 600 may include bus 602, one or more processors 604, one or more input devices 606, and one or more output devices 608. The one or more processors 604 may be any type of processor and may include, but are not limited to, one or more general-purpose processors and / or one or more dedicated processors (e.g., special-purpose chips). Input devices 606 may be any type of device capable of inputting information to electronic device 600 and may include, but are not limited to, a mouse, keyboard, touchscreen, microphone, and / or remote control. Output devices 608 may be any type of device capable of presenting information and may include, but are not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Electronic device 600 may also include a non-transitory storage device 610. The non-transitory storage device can be any storage device that is non-transitory and capable of storing data, including but not limited to disk drives, optical storage devices, solid-state storage, floppy disks, flexible disks, hard disks, magnetic tapes or any other magnetic media, optical discs or any other optical media, ROM (read-only memory), RAM (random access memory), cache memory and / or any other memory chip or cartridge, and / or any other medium from which a computer can read data, instructions, and / or code. The non-transitory storage device 610 can be detached from an interface. The non-transitory storage device 610 may have data / programs (including instructions) / code for implementing the methods and steps described above. Electronic device 600 may also include a communication device 612. The communication device 612 can be any type of device or system enabling communication with external devices and / or with a network, and may include, but is not limited to, modems, network interface cards, infrared communication devices, wireless communication devices and / or chipsets, such as Bluetooth™ devices, 802.11 devices, Wi-Fi devices, Wi-Max devices, cellular communication devices, and / or the like.

[0176] Electronic device 600 may also include working memory 614, which may be any type of working memory that can store programs (including instructions) and / or data useful for the operation of processor 604, and may include, but is not limited to, random access memory and / or read-only memory devices.

[0177] Software elements (programs) may reside in working memory 614, including but not limited to operating system 616, one or more application programs 618, drivers, and / or other data and code. Instructions for performing the above methods and steps may be included in one or more application programs 618, and the above image processing method 100 may be implemented by processor 604 reading and executing the instructions of one or more application programs 618. More specifically, in the above image processing method 100, steps S102-S110 may be implemented, for example, by processor 604 executing application programs 618 having instructions for steps S102-S110. Furthermore, other steps in the above image processing method 100 may be implemented, for example, by processor 604 executing application programs 618 having instructions for performing the corresponding steps. Executable code or source code of the instructions of the software elements (programs) may be stored in a non-transitory computer-readable storage medium (e.g., the above-described storage device 610) and may be stored in working memory 614 during execution (possibly compiled and / or installed). Executable code or source code of the instructions of the software elements (programs) may also be downloaded from a remote location.

[0178] It should also be understood that various modifications can be made depending on specific requirements. For example, custom hardware can also be used, and / or specific elements can be implemented using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. For example, some or all of the disclosed methods and apparatus can be implemented by programming hardware (e.g., programmable logic circuits including field-programmable gate arrays (FPGAs) and / or programmable logic arrays (PLAs)) using logic and algorithms according to this disclosure in assembly language or hardware programming languages ​​(such as Verilog, VHDL, C++).

[0179] It should also be understood that the aforementioned methods can be implemented using a server-client model. For example, the client can receive user input data and send it to the server. Alternatively, the client can receive user input data, perform a portion of the processing described in the aforementioned methods, and send the resulting data to the server. The server can receive data from the client, execute the aforementioned methods or a portion thereof, and return the execution result to the client. The client can receive the execution result from the server and, for example, present it to the user via an output device.

[0180] It should also be understood that the components of electronic device 600 can be distributed across a network. For example, some processing can be performed using one processor, while other processing can be performed simultaneously by another processor located far away from that processor. Other components of computing system 600 can also be distributed similarly. Thus, electronic device 600 can be interpreted as a distributed computing system that performs processing in multiple locations.

[0181] The following describes some exemplary aspects of this disclosure.

[0182] Aspect 1. A method for calibrating the extrinsic parameters of a camera and an inertial measurement unit (IMU) on a two-wheeled vehicle, comprising: The two-wheeled vehicle is subjected to segmented excitation motion to synchronously acquire image sequences and inertial data through the camera and the IMU, wherein the segmented excitation motion includes motion with at least two axial degrees of freedom; Extract image feature points between adjacent image frames in the image sequence to determine the relative rotation of the camera between adjacent image frames; The inertial data between the times corresponding to the adjacent image frames is pre-integrated to determine the relative rotation of the IMU between adjacent times. A hand-eye calibration model is constructed based on the relative rotation of the camera and the relative rotation of the IMU to determine the rotation matrix between the camera and the IMU; and The rotation matrix is ​​used as the extrinsic parameter to be optimized to construct a joint optimization objective function based on a sliding window, so as to jointly solve the calibration extrinsic parameters between the camera and the IMU through the objective function.

[0183] Aspect 2. The method as described in aspect 1, wherein performing segmented excitation motion on the two-wheeled vehicle comprises: In the initial state, the two-wheeled vehicle is kept upright and stationary for a preset time to synchronously start the acquisition of the image sequence and inertial data; Performing a tilting excitation on the two-wheeled vehicle includes tilting the two-wheeled vehicle to the left and right respectively to preset reference angles in place before restoring it to an upright position; and Performing a steering excitation motion on the two-wheeled vehicle includes moving the two-wheeled vehicle to the left front and right front respectively in an angled manner and then returning it to an upright position.

[0184] Aspect 3. The method of aspect 1, wherein extracting image feature points between adjacent image frames in the image sequence to determine the relative rotation of the camera between adjacent image frames comprises: Detect image feature points in each image frame of the image sequence; Based on the image feature points, establish the correspondence between feature points of adjacent image frames to obtain matching point pairs of adjacent image frames; and Based on the matching point pairs, the relative rotation and relative translation of the camera between adjacent image frames are calculated through epipolar geometric constraints.

[0185] Aspect 4. The method as described in aspect 3, wherein feature point correspondence between adjacent image frames is established by feature matching.

[0186] Aspect 5. The method as described in aspect 3, wherein the feature point correspondence between adjacent image frames is established by optical flow.

[0187] Aspect 6. The method as described in Aspect 3, wherein optimizing the objective function based on a sliding window to minimize the residuals of the equations based on the hand-eye calibration model to obtain the calibration extrinsic parameters between the camera and the IMU further includes: Determine the IMU deviation generated during the segmented excitation motion of the two-wheeled vehicle from a stationary state to the end of the data acquisition process; Based on the matching point pairs, the relative rotation of the camera, and the relative translation, the 3D map points corresponding to the image feature points are determined by triangulation. The rotation matrix and the translation vector between the camera and the IMU are used as extrinsic parameters to be optimized, and the camera pose, the IMU bias, and the 3D map points are used together as optimization parameters of the objective function. The camera pose is determined by the relative rotation and relative translation of the camera. Based on the sliding window, the equation residuals of the hand-eye calibration model are minimized by nonlinear iteration to minimize the visual reprojection error and the IMU pre-integration error, thereby optimizing the extrinsic variables to obtain the optimized extrinsic parameters between the camera and the IMU. The visual reprojection error is determined at least based on the 3D map points and the camera pose.

[0188] Aspect 7. The method as described in aspect 6, wherein the method further comprises: Based on the sliding window, scene feature recognition is performed on each image frame in the image sequence; In response to detecting a scenario where repeating structural features are detected between consecutive image frames, at least one of the following steps is performed: Local map constraints are established on historical image frames of scenes with repetitive structural features within the sliding window using inverse depth parameterization; and Increase the weight of the IMU pre-integration error in the joint optimization objective function.

[0189] Aspect 8. The method as described in aspect 7, wherein the scene feature recognition is performed using a bag-of-words model.

[0190] Aspect 9. The method as described in aspect 6, wherein the method further comprises: By detecting the number and distribution of image feature points in the image frame, it is determined whether the image frame includes low-texture regions.

[0191] Aspect 10. The method of aspect 9, wherein, in response to determining that the image frame includes the low-texture region, feature extraction is performed on the adjacent image frames based on a multi-scale pyramid to determine the relative rotation and relative translation of the camera between adjacent image frames.

[0192] Aspect 11. The method of aspect 9, wherein, in response to determining that the image frame includes the low-texture region, line segment residuals calculated based on line segment feature extraction and vanishing point residuals calculated based on vanishing point constraints are added to the objective function to optimize the extrinsic variables to be optimized together with point residuals constructed based on the three-dimensional map points.

[0193] Aspect 12. The method as described in aspect 1 further includes: In response to the determination, based on the IMU pre-integration, that a motion exceeding a preset rotation angle threshold has occurred, the following operations are performed: Expand the spatial search radius when establishing the correspondence between feature points between adjacent image frames; and The image feature points of the current image frame are rematched with the image feature points of the historical image frames to reconstruct the correspondence between the feature points.

[0194] Aspect 13. The method as described in aspect 1, wherein the method further comprises: Based on the image feature points extracted from the image frame, it is determined that the image frame includes dynamic objects; Perform image semantic segmentation on the image frame to determine the dynamic region in the image frame corresponding to the dynamic object; and A binary mask corresponding to the dynamic region is generated, and the image feature points included in the dynamic region are removed using the binary mask.

[0195] Aspect 14. The method of aspect 13, wherein, before removing the image feature points falling into the dynamic region by means of the binary mask, the method further includes: performing morphological dilation on the binary mask.

[0196] Aspect 15. The method of aspect 13, wherein the method further comprises: Based on the IMU pre-integration, the epipolar positions of the image feature points that were not eliminated are predicted in the next image frame. Based on the epipolar position, the number of consecutive tracking frames of image feature points, visual reprojection error, and epipolar residual, IMU motion consistency verification is performed on the image frames containing dynamic objects; and Based on the IMU motion consistency verification results, the image feature points that ultimately participate in the objective function are determined.

[0197] Aspect 16. An apparatus for calibrating the extrinsic parameters of a camera and an inertial measurement unit (IMU) on a two-wheeled vehicle, comprising: The data acquisition unit is configured to acquire image sequences and inertial data synchronously collected by the camera and the IMU during the process of performing segmented excitation motion on the two-wheeled vehicle, wherein the segmented excitation motion includes motion with at least two axial degrees of freedom; A vision processing unit is configured to extract image feature points between adjacent image frames in the image sequence to determine the relative rotation of the camera between adjacent image frames; An inertial processing unit is configured to pre-integrate the inertial data between times corresponding to the adjacent image frames to determine the relative rotation of the IMU between adjacent times. A model building unit is configured to build a hand-eye calibration model based on the relative rotation of the camera and the relative rotation of the IMU, to determine the rotation matrix between the camera and the IMU; and The optimization unit is configured to obtain the calibration extrinsic parameters between the camera and the IMU by optimizing a sliding window-based objective function to minimize the residuals of the equations based on the hand-eye calibration model, wherein the objective function is constructed by using the rotation matrix as the extrinsic parameter to be optimized.

[0198] Aspect 17. An electronic circuit, comprising: A circuit configured to perform the steps of the method according to any one of aspects 1-15.

[0199] Aspect 18. An electronic device comprising: Processor; and A memory storing a program, the program comprising instructions that, when executed by the processor, cause the processor to perform the method according to any one of aspects 1-15.

[0200] Aspect 19. A non-transitory computer-readable storage medium storing a program, the program comprising instructions that, when executed by a processor of an electronic device, cause the electronic device to perform the method according to any one of Aspects 1-15.

[0201] Aspect 20. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method described in any one of Aspects 1-15.

[0202] Aspect 21. A two-wheeled vehicle comprising the electronic circuitry according to aspect 17 or the electronic equipment according to aspect 18.

[0203] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A method for calibrating the extrinsic parameters of a camera and an inertial measurement unit (IMU) on a two-wheeled vehicle, comprising: The two-wheeled vehicle is subjected to segmented excitation motion to synchronously acquire image sequences and inertial data through the camera and the IMU, wherein the segmented excitation motion includes motion with at least two axial degrees of freedom; Extract image feature points between adjacent image frames in the image sequence to determine the relative rotation of the camera between adjacent image frames; The inertial data between the times corresponding to the adjacent image frames is pre-integrated to determine the relative rotation of the IMU between adjacent times. A hand-eye calibration model is constructed based on the relative rotation of the camera and the relative rotation of the IMU to determine the rotation matrix between the camera and the IMU; and The calibration extrinsic parameters between the camera and the IMU are obtained by optimizing the objective function based on the sliding window to minimize the residual of the equation based on the hand-eye calibration model. The objective function is constructed by using the rotation matrix as the extrinsic parameter to be optimized.

2. The method as described in claim 1, wherein, The segmented excitation motion applied to the two-wheeled vehicle includes: In the initial state, the two-wheeled vehicle is kept upright and stationary for a preset time to synchronously start the acquisition of the image sequence and inertial data; Performing a tilting excitation on the two-wheeled vehicle includes tilting the two-wheeled vehicle to the left and right respectively to preset reference angles in place before restoring it to an upright position; and Performing a steering excitation motion on the two-wheeled vehicle includes moving the two-wheeled vehicle to the left front and right front respectively in an angled manner and then returning it to an upright position.

3. The method as described in claim 1, wherein, The step of extracting image feature points between adjacent image frames in the image sequence to determine the relative rotation of the camera between adjacent image frames includes: Detect image feature points in each image frame of the image sequence; Based on the image feature points, establish the correspondence between feature points of adjacent image frames to obtain matching point pairs of adjacent image frames; and Based on the matching point pairs, the relative rotation and relative translation of the camera between adjacent image frames are calculated through epipolar geometric constraints.

4. The method of claim 3, wherein, The feature point correspondence between adjacent image frames is established by feature matching.

5. A device for calibrating the extrinsic parameters of a camera and an inertial measurement unit (IMU) on a two-wheeled vehicle, comprising: The data acquisition unit is configured to acquire image sequences and inertial data synchronously collected by the camera and the IMU during the process of performing segmented excitation motion on the two-wheeled vehicle, wherein the segmented excitation motion includes motion with at least two axial degrees of freedom; A vision processing unit is configured to extract image feature points between adjacent image frames in the image sequence to determine the relative rotation of the camera between adjacent image frames; An inertial processing unit is configured to pre-integrate the inertial data between times corresponding to the adjacent image frames to determine the relative rotation of the IMU between adjacent times. A model building unit is configured to build a hand-eye calibration model based on the relative rotation of the camera and the relative rotation of the IMU, to determine the rotation matrix between the camera and the IMU; and The optimization unit is configured to obtain the calibration extrinsic parameters between the camera and the IMU by optimizing a sliding window-based objective function to minimize the residuals of the equations based on the hand-eye calibration model, wherein the objective function is constructed by using the rotation matrix as the extrinsic parameter to be optimized.

6. An electronic circuit, comprising: A circuit configured to perform the steps of the method according to any one of claims 1-4.

7. An electronic device, comprising: processor; as well as A memory storing a program, the program comprising instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-4.

8. A non-transitory computer-readable storage medium storing a program, the program comprising instructions that, when executed by a processor of an electronic device, cause the electronic device to perform the method according to any one of claims 1-4.

9. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method of any one of claims 1-4.

10. A two-wheeled vehicle comprising the electronic circuitry of claim 6 or the electronic equipment of claim 7.

Images