Method for realizing VIO with multi-camera asynchronous acquisition
By acquiring the timestamps of images from multiple cameras, calculating the pose compensation amount, and performing iterative optimization, the problem of reduced accuracy and robustness of the VIO system caused by the asynchrony of multiple cameras is solved, achieving higher detection accuracy and efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA UNIV OF MINING & TECH (BEIJING)
- Filing Date
- 2025-01-15
- Publication Date
- 2026-07-03
AI Technical Summary
The inability of multi-camera systems to capture images simultaneously reduces the accuracy and robustness of VIO systems, especially causing problems with visual measurement in motion scenarios.
By acquiring timestamps from multiple images, the timestamp of the reference window is determined, and the pose compensation amount is calculated based on inertial measurement data and state vectors. Inertial and visual measurement residuals are constructed, and iterative optimization is performed using nonlinear optimization methods.
It improves the accuracy and robustness of the VIO system in the case of multiple cameras being out of sync, and enhances the detection accuracy and efficiency of rotating targets.
Smart Images
Figure CN119946204B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, specifically to a VIO implementation method for asynchronous acquisition by multiple cameras. Background Technology
[0002] In many scenarios, different cameras (or different webcams) on the same device cannot capture images simultaneously. This is because there is a time difference in the hardware, which cannot be uniformly controlled by software. The definition of "simultaneous shooting" is: for multiple cameras, the timestamps of the center points of the exposure times of the images captured by each camera are equal.
[0003] For example, multiple cameras on a mobile phone do not actually capture images simultaneously (there is usually a time difference on the order of milliseconds, or even up to 10 milliseconds), which inevitably causes problems for visual measurement, especially in motion scenarios. The inability to capture images simultaneously is mainly due to different exposure times (affected by light intensity). If there are two cameras on the device, triggered at the same time, the camera facing the brighter side will have a shorter exposure time, while the camera facing the darker side will have a longer exposure time, resulting in different imaging times (calculated at the midpoint of the exposure time interval).
[0004] The aforementioned "inability to capture images simultaneously" situation typically leads to the following consequences: reduced accuracy of the VIO (Visual Inertial Odometry) or VISLAM (Visual Inertial Simultaneous Localization and Mapping) system, resulting in lower pose accuracy output by the VIO system; or reduced robustness, resulting in lower VIO stability, and in severe cases, causing the VIO system to diverge and become unusable. Summary of the Invention
[0005] To at least partially overcome the problem of reduced accuracy caused by camera asynchrony in related technologies, this application provides a VIO implementation method for asynchronous acquisition by multiple cameras.
[0006] According to a first aspect of the embodiments of this application, a VIO implementation method for asynchronous acquisition by multiple cameras is provided, comprising the following steps:
[0007] Obtain the timestamps of multiple images corresponding to the current frame; where the multiple images are images captured by multiple cameras within the same frame's time window;
[0008] Determine the timestamp of the reference window corresponding to the current frame based on the timestamps of multiple images;
[0009] The pose compensation between the timestamp of each image and the reference window is calculated based on inertial measurement data and state vectors.
[0010] Construct inertial measurement residuals and visual measurement residuals for each image based on pose compensation, respectively;
[0011] The inertial measurement residuals and visual measurement residuals are iteratively optimized and solved using a nonlinear optimization method.
[0012] Furthermore, determining the timestamp of the reference window corresponding to the current frame based on the timestamps of multiple images includes the following steps:
[0013] The smallest timestamp among multiple images is determined as the timestamp of the reference window corresponding to the current frame; or...
[0014] The largest timestamp among multiple images is determined as the timestamp of the baseline window corresponding to the current frame; or...
[0015] The average of the timestamps of multiple images is used to determine the timestamp of the reference window corresponding to the current frame.
[0016] Furthermore, the state vector is obtained through the step of iteratively optimizing the inertial measurement residual and the visual measurement residual based on the nonlinear optimization method;
[0017] The state vector is ; This represents the rotational quaternion of the IMU (Inertial Measurement Unit). Indicates the three-dimensional position of the IMU. Indicates the three-dimensional velocity of the IMU. This indicates the accelerometer bias of the IMU. Indicates the gyroscope bias of the IMU;
[0018] The calculation of pose compensation between the timestamp of each image and the reference window based on inertial measurement data and state vectors includes the following steps:
[0019] Calculate the time difference between the timestamp of each image and the timestamp of the baseline window;
[0020] The pose compensation between each image and the reference window is calculated based on inertial measurement data, state vector, and time difference.
[0021] Furthermore, the pose compensation amount includes: rotation compensation amount and translation compensation amount; the inertial measurement data includes: accelerometer readings and gyroscope readings;
[0022] The calculation of pose compensation between each image and the reference window based on inertial measurement data and time difference includes the following steps:
[0023] The rotation compensation amount is calculated based on the gyroscope reading, gyroscope bias, and time difference.
[0024] The translation compensation is calculated based on the accelerometer readings, accelerometer bias, velocity, and time difference.
[0025] Furthermore, the calculation of the rotation compensation amount based on the gyroscope readings, gyroscope bias, and time difference includes the following steps:
[0026] ,in ;
[0027] in, Indicates that camera k is in The rotation compensation amount at time t, where I represents the identity matrix. Indicates in The gyroscope reading at that moment. This indicates the gyroscope bias of the IMU. Indicates that camera k is in The time difference between the current moment and the timestamp of the baseline window.
[0028] Furthermore, the calculation of the translation compensation amount based on accelerometer readings, accelerometer bias, velocity, and time difference includes the following steps:
[0029] ,in ;
[0030] in, Indicates that camera k is in Translation compensation amount at time, Indicates the three-dimensional velocity of the IMU. Indicates that camera k is in The time difference between the current time and the timestamp of the baseline window. This indicates that quaternions Equivalently converted to the corresponding 3×3 rotation matrix, Indicates in Accelerometer readings at time 10:00 This indicates the accelerometer bias of the IMU. It represents the acceleration due to gravity of the earth.
[0031] Furthermore, the construction of the inertial measurement residual and the visual measurement residual corresponding to each image based on the pose compensation amount includes the following steps:
[0032] The inertial measurement residual at time t is: ;in, for The system state vector at time t. for The system state vector at time t. for Time's up IMU pre-integration results at time step;
[0033] The visual measurement residual at time t is:
[0034] ;
[0035] in, Indicates the first position in the sliding window The actual observation value of camera k at the j-th landmark point. Represents the camera projection model. This indicates that quaternions Equivalently converted to the corresponding 3×3 rotation matrix, Represents the rotation quaternion of the IMU. Indicates that camera k is in Rotational compensation amount at time. For camera external parameters, Indicates that camera k is in Translation compensation amount at time, This represents the three-dimensional coordinates of a given landmark point.
[0036] Furthermore, the iterative optimization calculation of the inertial measurement residual and the visual measurement residual based on the nonlinear optimization method includes the following steps:
[0037] Calculate the Jacobian matrix of all inertial measurement residuals and all visual measurement residuals corresponding to all frames in the current sliding window with respect to the system state.
[0038] All inertial measurement residuals and visual measurement residuals, along with their corresponding Jacobian matrices and the system state, are fed into a nonlinear optimizer to iteratively optimize the system state; or,
[0039] In filter-based VIO / VSLAM, the system state is predicted to the next time step using inertial measurement data. Then, the visual measurement residuals and their corresponding Jacobian matrix, along with the system state, are substituted into the extended Kalman filter (EKF) to update the system state.
[0040] The system state is as follows:
[0041] .
[0042] According to a second aspect of the embodiments of this application, a computer device is provided, comprising:
[0043] Memory, used to store computer programs;
[0044] A processor is configured to execute a computer program in the memory to implement the operational steps of the method as described in any of the above embodiments.
[0045] According to a third aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the operation steps of the method as described in any of the above embodiments.
[0046] The technical solutions provided by the embodiments of this application have the following beneficial effects:
[0047] The proposed solution models the difference in timestamps between images captured by different cameras in the VIO system, enabling higher accuracy and robustness even when different cameras do not capture images simultaneously. This solution also allows the VIO system to achieve higher accuracy and efficiency in detecting rotating targets.
[0048] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0050] Figure 1 This is a flowchart illustrating a VIO implementation method for asynchronous acquisition by multiple cameras according to an exemplary embodiment.
[0051] Figure 2 This is a schematic diagram illustrating the ideal scenario of synchronizing timestamps across multiple cameras.
[0052] Figure 3 This is a diagram illustrating a situation where the timestamps of the two cameras are inconsistent. Detailed Implementation
[0053] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of methods and apparatus consistent with some aspects of this application as detailed in the appended claims.
[0054] Figure 1 This is a flowchart illustrating a VIO implementation method for asynchronous acquisition by multiple cameras according to an exemplary embodiment. The method may include the following steps:
[0055] Step S1: Obtain the timestamps of multiple images corresponding to the current frame; wherein, the multiple images are images captured by multiple cameras within the same frame's time window;
[0056] Step S2: Determine the timestamp of the reference window corresponding to the current frame based on the timestamps of multiple images;
[0057] Step S3: Calculate the pose compensation amount between the timestamp of each image and the reference window based on the inertial measurement data and state vector;
[0058] Step S4: Construct the inertial measurement residual and the visual measurement residual corresponding to each image based on the pose compensation amount, respectively;
[0059] Step S5: Iteratively optimize and solve the inertial measurement residual and visual measurement residual based on the nonlinear optimization method.
[0060] The proposed solution models the difference in timestamps between images captured by different cameras in the VIO system, enabling higher accuracy and robustness even when different cameras do not capture images simultaneously. This solution also allows the VIO system to achieve higher accuracy and efficiency in detecting rotating targets.
[0061] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0062] To further elaborate on the technical solution of this application, a general / conventional VIO system framework is first briefly introduced here. This system is configured with dual cameras and one IMU (Inertial Measurement Unit). The implementation process of a VIO system for multi-camera timestamp synchronization under ideal conditions is then described first.
[0063] According to Figure 2 The illustrated embodiment demonstrates the case where the camera timestamps are the same (both cameras are observing simultaneously). It should be noted that, although... Figure 2This description uses a stereo camera as an example, but the principle can be extended to multi-view cameras. In VIO systems, the camera image frame rate is usually low (typically 25-30Hz), but the IMU frame rate is high (typically 200-8000Hz). Therefore, in optimized VIO, the IMU is usually pre-integrated and then fused with the camera image observation results for solution.
[0064] Existing VIO systems in both academia and industry all employ a sliding window-based mechanism. For example... Figure 2 As shown, the sliding window contains multiple windows, where t0, t1, and t2 represent one window within the sliding window. Each window contains a 16-dimensional state vector, which includes: the IMU's 3D attitude (represented by 4D variables, i.e., quaternions), the IMU's 3D position (represented by 3D variables), the IMU's 3D velocity (represented by 3D variables), the IMU's acceleration bias (represented by 3D variables), and the IMU's gyroscope bias (represented by 3D variables).
[0065] The state vector is represented as: ;in Represents the rotation quaternion of the IMU. This represents the three-dimensional translation of the IMU. This represents the three-dimensional velocity of the IMU. This indicates the accelerometer bias of the IMU. This represents the gyroscope bias of the IMU. It should be noted that the state vector is obtained through iterative optimization in step S5.
[0066] like Figure 2 As shown, between t0 and t1, the IMU takes multiple observations (each observing the system's acceleration and angular velocity), which are pre-integrated into M (i.e., the IMU pre-integration result). At time t0, two images are obtained from two cameras; similarly, two camera images are obtained at times t1, t2, t3, etc. Based on the IMU and camera measurement results, VIO observation modeling can be performed, and the system state at each time point can be estimated. The state vector at time t0 is... The state vector at time t1 is And so on.
[0067] (1) Inertial measurement modeling: and The state vector at time step 1, together with the IMU pre-integral between the two, constructs the inertial residual (i represents the sliding window number), as shown in the following formula. Represents the inertial measurement residual:
[0068] (1)
[0069] (2) Visual measurement modeling: Based on the observation information (two-dimensional coordinates of the landmark in the image) of each window in the sliding window for each landmark point (or feature), visual observation modeling is performed separately, as shown in the following formulas. Represents the visual measurement residuals (i represents the sliding window number, j represents the landmark number, and k represents the camera number):
[0070] (2)
[0071] in, Indicates the first position in the sliding window The actual observation (distorted actual observation) of camera k at time i (i.e., the i-th window) for the j-th landmark point. The visual observation residual is obtained by subtracting the distorted actual observation from the distortion-free theoretical observation for that landmark point. .
[0072] This represents the camera projection model, which means that given the camera rotation matrix and translation values, as well as the three-dimensional coordinates of a landmark point, we can obtain the theoretical observation value of the landmark point in the camera without distortion (normalized two-dimensional coordinates of the image).
[0073] pass Internally contained The observation time can be obtained from the IMU pose and camera extrinsic parameters. The camera pose at any given time. The IMU pose includes the IMU's 3D rotation and 3D position, i.e. The three-dimensional rotation (or attitude) represents the transformation from the IMU coordinate system to the geodetic coordinate system, and the three-dimensional position represents the translation of the IMU in the geodetic coordinate system. The camera extrinsic parameters are the rotation and translation from the camera to the IMU, defined as follows: This parameter can be obtained through offline calibration at the system's factory settings, or it can be calibrated by the user using a calibration device; it is a known quantity. This method is a well-known technique, described below:
[0074] The camera rotates at any moment:
[0075] Camera panning at any moment:
[0076] Indicates rotating quaternions This is equivalent to a 3×3 rotation matrix. Academically, quaternions represent three-axis rotations; this application uses... This indicates that quaternions It is equivalent to the corresponding 3×3 rotation matrix.
[0077] Therefore, given the three-dimensional coordinates of the landmark point , We can obtain the theoretical observation value (normalized two-dimensional coordinates of the image) of the landmark point in the camera at the observation time without distortion.
[0078] The system state (i.e., the state to be estimated) includes the state vectors of all windows within the sliding window, and the three-dimensional coordinates (three-dimensional coordinates in the world coordinate system) of all observable landmarks within the sliding window. That is:
[0079]
[0080] Calculate the Jacobian matrix of all effective inertial measurement residuals and binocular camera visual residuals within the aforementioned sliding window with respect to the system state.
[0081] All residuals, corresponding Jacobian matrices, and system states are input into g2o, ceres, or gtsam (three commonly used nonlinear optimization solvers) or other nonlinear optimizers to iteratively optimize all system states.
[0082] Alternatively, in filter-based VIO / VSLAM, the system state can be predicted to the next time step using inertial measurement data, and then the visual measurement residuals and their corresponding Jacobian matrix, along with the system state, can be substituted into the Extended Kalman Filter (EKF) to update the system state.
[0083] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
[0084] In a conventional binocular VIO system, the image data acquired by the two cameras must be exposed at the same timestamp, meaning the centers of the exposure intervals of the two cameras need to coincide. Figure 2 The ideal situation is illustrated; the same applies to multi-camera systems, which require multiple cameras to be exposed at the same timestamp. In a multi-camera VIO system, misalignment of image exposure timestamps between multiple cameras leads to reduced accuracy.
[0085] To address the issue of camera asynchrony, this application's solution models the difference in timestamp inconsistencies between different cameras within the VIO system, enabling higher accuracy and robustness even when different cameras are not shooting simultaneously within the VIO system.
[0086] This embodiment uses stereo VIO as an example to illustrate in detail the VIO implementation method for asynchronous camera situations proposed in this invention. It is easy to understand that the method of this invention is not limited to stereo cameras, but is fully applicable to VIO systems with multi-view cameras.
[0087] like Figure 3 As shown, since the timestamps of the two corresponding images in each received binocular image frame are not equal, the timestamp of the window for the current frame is specified by the user. Step S2 determines the timestamp of the reference window corresponding to the current frame based on the timestamps of multiple images. There are several ways to do this: (1) determine the timestamp of the reference window corresponding to the current frame as the smallest timestamp among the multiple images; (2) determine the timestamp of the reference window corresponding to the current frame as the largest timestamp among the multiple images; (3) determine the timestamp of the reference window corresponding to the current frame as the result of averaging the timestamps of multiple images.
[0088] In this embodiment, the smallest timestamp in a frame of stereo images (i.e., t0 in the figure) is specified as the timestamp of the window corresponding to the current stereo frame (the largest timestamp can also be selected, or the result of averaging the timestamps of all images in the frame can be used, etc.). This embodiment uses the "smallest timestamp" method as an example. The principle of other methods is exactly the same and can be easily derived, so they will not be described in detail.
[0089] like Figure 3 As shown, for the problem that a single frame of stereo image (where the timestamps of the two images in the frame are inconsistent) has only one corresponding window (i.e., IMU pose), the IMU pose corresponding to the timestamp of another image can be represented based on the IMU measurement results and the existing IMU pose. Then, based on this IMU pose, a residual model can be constructed.
[0090] Because this invention specifies the smallest timestamp in a frame of stereo image (i.e., t0 in the figure) as the timestamp of the window corresponding to the current stereo frame, therefore based on Figure 3 All timestamps are based on the observation timestamp of camera 1 as the timestamp of the corresponding window. In each sliding window, there exists a system state vector corresponding to the IMU pose. And targeting There is no corresponding IMU pose in the sliding window.
[0091] In this embodiment, step S3 calculates the pose compensation amount between the timestamp of each image and the reference window based on inertial measurement data and state vectors. Specifically, this includes the following steps: calculating the time difference between the timestamp of each image and the timestamp of the reference window; and calculating the pose compensation amount between each image and the reference window based on the inertial measurement data, state vectors, and time difference. The pose compensation amount includes rotation compensation and translation compensation; the inertial measurement data includes accelerometer readings and gyroscope readings.
[0092] Specifically, the pose compensation between each image and the reference window is calculated based on inertial measurement data and time difference, including the following steps: calculating rotation compensation based on gyroscope readings, gyroscope bias, and time difference; and calculating translation compensation based on accelerometer readings, accelerometer bias, velocity, and time difference.
[0093] The following is for Figure 3 For example, when i=0, it represents the 0th window in the sliding window, therefore and These represent the rotation compensation and translation compensation amounts for camera k in the 0th window, respectively. express The rotation matrix of the IMU at each time step. express Translation of the IMU at any given moment.
[0094] and The calculation method is as follows:
[0095]
[0096]
[0097] IMU readings are divided into accelerometer and gyroscope readings. Defined in At time , the accelerometer reading is The gyroscope reading is .therefore , ,and It represents the acceleration due to gravity of the earth.
[0098] Next, step S4 is executed to construct the inertial measurement residual and the visual measurement residual corresponding to each image based on the pose compensation amount. The inertial measurement residual can be modeled using formula (1) described in the previous embodiments. This invention will... The corresponding IMU pose compensation method is expressed as follows The IMU pose corresponding to the observation timestamp. Therefore, it is related to... Figure 1 The implementation methods differ; in this embodiment... The visual residual modeling is as follows:
[0099] (3)
[0100] The above Indicates that camera k is in Rotational compensation amount at time. Indicates that camera k is in Translation compensation amount at any given moment.
[0101] Next, in the case of a stereo camera, calculate the Jacobian matrix of all effective inertial measurement residuals within the sliding window (Equation 1), the visual residuals of camera 1 (Equation 2), and the visual residuals of camera 2 (Equation 3) against the system state. In the case of a multi-camera system, the visual residuals of each camera need to be calculated according to Equation 3.
[0102] Finally, all residuals, their corresponding Jacobian matrices, and system states are input into g2o, ceres, or gtsam (three commonly used nonlinear optimization solvers) or other nonlinear optimizers to iteratively optimize all system states and landmark coordinates. Alternatively, in filter-based VIO / VSLAM, inertial measurement data is used to predict the system state to the next time step, and then visual measurement residuals and their corresponding Jacobian matrices, along with the system state, are input into an extended Kalman filter (EKF) to update the system state.
[0103] It should be noted that the foregoing embodiments only illustrate VIO based on nonlinear optimization iteration. The method of this application can be used not only for VIO based on nonlinear optimization iteration but also for VIO based on a filtering framework. Specifically, it can be applied to a filtering-based VIO / VISLAM solver. The filtering-based solver does not have a pre-integration process; instead, it uses a prediction process. The other steps are the same as the VIO method based on nonlinear optimization iteration, and can be referred to the foregoing embodiments.
[0104] In multi-view VIO (or multi-view VSLAM), when performing residual modeling for camera observations with inconsistent timestamps, the technical solution of this application uses the state vector corresponding to the nearest existing window on the timestamp of the sliding window and the IMU reading to construct the IMU pose under the timestamp corresponding to other time-misaligned cameras, thereby enabling accurate residual modeling for multi-view camera observations with inconsistent timestamps.
[0105] In one embodiment, a computer device is provided. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. When the computer program is executed by the processor, it implements a VIO (Video Interchange) implementation method for asynchronous acquisition by multiple cameras.
[0106] In one embodiment, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the VIO implementation method for asynchronous acquisition of multiple cameras as described in any of the above embodiments.
[0107] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.
[0108] It should be noted that in the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means at least two.
[0109] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.
[0110] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0111] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0112] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0113] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.
[0114] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0115] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.
Claims
1. A VIO implementation method for asynchronous acquisition by multiple cameras, characterized in that... This includes the following steps: Obtain the timestamps of multiple images corresponding to the current frame; where the multiple images are images captured by multiple cameras within the same frame's time window; Determine the timestamp of the reference window corresponding to the current frame based on the timestamps of multiple images; The pose compensation between the timestamp of each image and the reference window is calculated based on inertial measurement data and state vectors. Inertial measurement residuals and visual measurement residuals based on pose compensation for each image are constructed separately. The inertial measurement residual and the visual measurement residual are iteratively optimized and solved using a nonlinear optimization method. The process of constructing the inertial measurement residual and the visual measurement residual corresponding to each image based on the pose compensation amount includes the following steps: The inertial measurement residual at time t is: ;in, for The system state vector at time t, for The system state vector at time t, for Time's up IMU pre-integration results at time step; The visual measurement residual at time t is: ; in, Indicates the first position in the sliding window The actual observation value of camera k at the j-th landmark point. Represents the camera projection model. This indicates that quaternions Equivalently converted to the corresponding 3×3 rotation matrix, Represents the rotation quaternion of the IMU. Indicates that camera k is in Rotational compensation amount at time, For camera external parameters, Indicates that camera k is in Translation compensation amount at time, Represents the three-dimensional coordinates of a given landmark point. This indicates the three-dimensional position of the IMU.
2. The method according to claim 1, characterized in that... Determining the timestamp of the reference window corresponding to the current frame based on the timestamps of multiple images includes the following steps: The smallest timestamp among multiple images is determined as the timestamp of the baseline window corresponding to the current frame; or, The largest timestamp among multiple images is determined as the timestamp of the baseline window corresponding to the current frame; or, The average of the timestamps of multiple images is used to determine the timestamp of the reference window corresponding to the current frame.
3. The method according to claim 1, characterized in that... The state vector is obtained through the step of iteratively optimizing the inertial measurement residual and the visual measurement residual based on the nonlinear optimization method. The state vector is ; Represents the rotation quaternion of the IMU. Indicates the three-dimensional position of the IMU. Represents the three-dimensional velocity of the IMU. Indicates the accelerometer bias of the IMU. Indicates the gyroscope bias of the IMU; The calculation of pose compensation between the timestamp of each image and the reference window based on inertial measurement data and state vectors includes the following steps: Calculate the time difference between the timestamps of each image and the timestamp of the baseline window; The pose compensation between each image and the reference window is calculated based on inertial measurement data, state vector, and time difference.
4. The method according to claim 3, characterized in that... The pose compensation amount includes: rotation compensation amount and translation compensation amount; the inertial measurement data includes: accelerometer readings and gyroscope readings; The calculation of pose compensation between each image and the reference window based on inertial measurement data, state vector, and time difference includes the following steps: The rotational compensation is calculated based on the gyroscope readings, the IMU's gyroscope bias, and the time difference. The translation compensation is calculated based on the accelerometer readings, the IMU's accelerometer bias, the IMU's three-dimensional velocity, and the time difference.
5. The method according to claim 4, characterized in that... The calculation of rotation compensation based on gyroscope readings, gyroscope bias, and time difference includes the following steps: ,in ; in, Indicates that camera k is in The rotation compensation amount at time t, where I represents the identity matrix. Indicates in gyroscope reading at time, Indicates the gyroscope bias of the IMU. Indicates that camera k is in The time difference between the current moment and the timestamp of the baseline window.
6. The method according to claim 4, characterized in that... The calculation of translation compensation based on accelerometer readings, IMU accelerometer bias, IMU three-dimensional velocity, and time difference includes the following steps: ,in ; in, Indicates that camera k is in Translation compensation amount at time, Indicates the three-dimensional velocity of the IMU. Indicates that camera k is in The time difference between the current time and the timestamp of the reference window. This indicates that quaternions Equivalently converted to the corresponding 3×3 rotation matrix, Indicates in Accelerometer readings at time 10:00 Indicates the accelerometer bias of the IMU. It represents the acceleration due to gravity of the earth.
7. The method according to claim 1, characterized in that... The iterative optimization calculation of inertial measurement residuals and visual measurement residuals based on the nonlinear optimization method includes the following steps: Calculate the Jacobian matrix of the system state for all inertial measurement residuals and all visual measurement residuals corresponding to all frames within the current sliding window. Substitute all inertial measurement residuals and visual measurement residuals, along with their corresponding Jacobian matrices and the system state, into a nonlinear optimizer to iteratively optimize the system state; or, In filter-based VIO / VISLAM, the system state is predicted to the next time step using inertial measurement data. Then, the visual measurement residuals and their corresponding Jacobian matrix, along with the system state, are substituted into the extended Kalman filter to update the system state. The system state is as follows: 。 8. A computer device, characterized in that... ,include: Memory, used to store computer programs; A processor for executing a computer program in the memory to implement the operational steps of the method according to any one of claims 1 to 7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that... When the computer program is executed by the processor, it implements the operation steps of the method according to any one of claims 1 to 7.