VIO (Virtual Input Output) implementation method for asynchronous acquisition of multiple cameras

Abstract

The invention relates to a VIO implementation method for multi-camera asynchronous acquisition. The method comprises the following steps: acquiring timestamps of a plurality of images corresponding to a current frame; wherein the plurality of images are respectively images acquired by a plurality of cameras in a time window of the same frame; determining a timestamp of a reference window corresponding to the current frame according to the timestamps of the plurality of images; calculating the pose compensation amount between the timestamp of each image and the reference window based on the inertial measurement data and the state vector; respectively constructing an inertial measurement residual error and a visual measurement residual error corresponding to each image based on the pose compensation amount; and carrying out iterative optimization solution on the inertia measurement residual error and the vision measurement residual error based on a nonlinear optimization method. According to the scheme provided by the invention, modeling is carried out on the difference quantity of inconsistent timestamps of pictures shot by different cameras in the VIO system, so that higher precision and robustness can be realized when different cameras in the VIO system do not shoot at the same time; according to the scheme, the VIO system can detect the rotating target more precisely and more efficiently.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a VIO implementation method for asynchronous multi-camera acquisition. Background Art

[0002] In many scenarios, different cameras (or different cameras) on the same device cannot shoot at the same time; the reason is that there is a time difference in the hardware, which cannot be uniformly controlled by software. The definition of "simultaneous shooting" is: for multiple cameras, the timestamps of the exposure time center points of the pictures taken by each camera are equal, which can be expressed as simultaneous shooting.

[0003] For example, multiple cameras on a mobile phone do not actually shoot at the same time (usually there is a time difference of milliseconds, or even up to 10 milliseconds), and visual measurement is bound to be problematic, especially in motion scenes. The main reason for not being able to shoot at the same time is that the exposure time is different (affected by light intensity). If there are two cameras on the device and they are triggered at the same time, the exposure time of the camera facing the bright direction is short, and the exposure time of the camera facing the dim direction is long, so the imaging time (calculated at the midpoint of the exposure time interval) is different.

[0004] The above-mentioned "inability to shoot simultaneously" situation usually leads to the following consequences: the solution accuracy of the VIO (Visual Inertial Odometry) system or the VISLAM (Visual Inertial Simultaneous Localization and Mapping) system is reduced, which leads to lower pose accuracy output by the VIO system; or the robustness is reduced, which leads to lower VIO stability. In severe cases, the VIO system may diverge and become unusable. Summary of the invention

[0005] In order to at least overcome the problem of reduced accuracy caused by camera asynchrony in the related art to a certain extent, the present application provides a VIO implementation method for asynchronous acquisition of multiple cameras.

[0006] According to a first aspect of an embodiment of the present application, a VIO implementation method for asynchronous multi-camera acquisition is provided, comprising the following steps:

[0007] Obtaining timestamps of multiple images corresponding to the current frame; wherein the multiple images are images captured by multiple cameras within a time window of the same frame;

[0008] Determine the timestamp of the reference window corresponding to the current frame according to the timestamps of the multiple images;

[0009] Calculate the pose compensation between the timestamp of each image and the reference window based on the inertial measurement data and the state vector;

[0010] Construct inertial measurement residuals and visual measurement residuals corresponding to each image based on posture compensation respectively;

[0011] The inertial measurement residuals and visual measurement residuals are iteratively optimized and solved based on the nonlinear optimization method.

[0012] Furthermore, the determining the timestamp of the reference window corresponding to the current frame according to the timestamps of the multiple images comprises the following steps:

[0013] Determine the smallest timestamp among the multiple images as the timestamp of the reference window corresponding to the current frame; or,

[0014] Determine the largest timestamp among the multiple images as the timestamp of the reference window corresponding to the current frame; or,

[0015] The result of averaging the timestamps of the multiple images is determined as the timestamp of the reference window corresponding to the current frame.

[0016] Furthermore, the state vector is obtained by performing iterative optimization and solving of the inertial measurement residual and the visual measurement residual based on the nonlinear optimization method;

[0017] The state vector is ; Represents the rotation quaternion of the IMU (Inertial Measurement Unit), represents the three-dimensional position of the IMU, represents the IMU three-dimensional velocity, represents the accelerometer bias of the IMU, Indicates the gyroscope bias of the IMU;

[0018] The method of calculating the pose compensation between the timestamp of each image and the reference window based on the inertial measurement data and the state vector comprises the following steps:

[0019] Calculate the time difference between the timestamp of each image and the timestamp of the reference window;

[0020] The pose compensation between each image and the reference window is calculated based on the inertial measurement data, state vector and time difference.

[0021] Further, the posture compensation includes: rotation compensation and translation compensation; the inertial measurement data includes: accelerometer reading and gyroscope reading;

[0022] The method of calculating the position compensation between each image and the reference window according to the inertial measurement data and the time difference comprises the following steps:

[0023] Calculate the rotation compensation amount based on the gyroscope reading, the gyroscope bias and the time difference;

[0024] The translation compensation is calculated based on the accelerometer reading, accelerometer bias, velocity, and time difference.

[0025] Further, the calculation of the rotation compensation amount according to the gyroscope reading, the gyroscope bias and the time difference comprises the following steps:

[0026] ,in ;

[0027] in, Indicates that camera k is The rotation compensation at the moment, I represents the unit matrix, Indicated in The gyroscope reading at the time, Indicates the gyroscope bias of the IMU, Indicates that camera k is The time difference between the instant and the timestamp of the base window.

[0028] Further, the calculation of the translation compensation amount according to the accelerometer reading, the accelerometer bias, the speed and the time difference comprises the following steps:

[0029] ,in ;

[0030] in, Indicates that camera k is The translation compensation at time, represents the IMU three-dimensional velocity, Indicates that camera k is The time difference between the instant and the timestamp of the reference window, Represents the quaternion Equivalently converted to the corresponding 3×3 rotation matrix, Indicated in The accelerometer reading at time, represents the accelerometer bias of the IMU, Represents the gravitational acceleration of the Earth.

[0031] Furthermore, the construction of the inertial measurement residual and the visual measurement residual corresponding to each image based on the posture compensation amount respectively comprises the following steps:

[0032] The inertial measurement residual at time is: ;in, for The system state vector at time , for The system state vector at time , for Time has come IMU pre-integration result at the moment;

[0033] The visual measurement residual at the moment is:

[0034] ;

[0035] in, Indicates that in the sliding window The actual observation value of camera k at the jth landmark point at the moment, represents the camera projection model, Represents the quaternion Equivalently converted to the corresponding 3×3 rotation matrix, Represents the rotation quaternion of the IMU, Indicates that camera k is The rotation compensation amount at the moment, is the camera extrinsic parameter, Indicates that camera k is The translation compensation at time, Represents the three-dimensional coordinates of a given landmark point.

[0036] Furthermore, the iterative optimization and solving of the inertial measurement residual and the visual measurement residual based on the nonlinear optimization method comprises the following steps:

[0037] Calculate the Jacobian matrix of the system state for all inertial measurement residuals and all visual measurement residuals corresponding to all frames in the current sliding window;

[0038] Bring all inertial measurement residuals and visual measurement residuals and their corresponding Jacobian matrices, as well as the system state, into the nonlinear optimizer to perform iterative optimization on the system state; or,

[0039] In filter-based VIO / VSLAM, inertial measurement data is used to predict the system state to the next moment, and then the visual measurement residual and its corresponding Jacobian matrix and system state are substituted into the extended Kalman filter (EKF) to update the system state;

[0040] The system status is:

[0041] .

[0042] According to a second aspect of an embodiment of the present application, a computer device is provided, including:

[0043] Memory for storing computer programs;

[0044] A processor is used to execute the computer program in the memory to implement the operating steps of the method described in any one of the above embodiments.

[0045] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the operating steps of the method described in any one of the above embodiments are implemented.

[0046] The technical solution provided by the embodiments of the present application has the following beneficial effects:

[0047] The solution of the present application models the difference in the inconsistent timestamps of images taken by different cameras in the VIO system, so that higher accuracy and robustness can be achieved even when different cameras in the VIO system are not shooting at the same time; this solution can enable the VIO system to detect rotating targets with higher accuracy and efficiency.

[0048] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0050] Figure 1 The present invention is a flowchart of a VIO implementation method for asynchronous multi-camera acquisition according to an exemplary embodiment.

[0051] Figure 2 This is a schematic diagram of multi-camera timestamp synchronization under ideal conditions.

[0052] Figure 3 This is a schematic diagram of the situation where the timestamps of the two cameras are inconsistent. DETAILED DESCRIPTION

[0053] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of methods and devices consistent with some aspects of the present application as detailed in the appended claims.

[0054] Figure 1 1 is a flow chart of a VIO implementation method for asynchronous multi-camera acquisition according to an exemplary embodiment. The method may include the following steps:

[0055] Step S1, obtaining timestamps of multiple images corresponding to the current frame; wherein the multiple images are images captured by multiple cameras within a time window of the same frame;

[0056] Step S2, determining the timestamp of the reference window corresponding to the current frame according to the timestamps of the multiple images;

[0057] Step S3, calculating the pose compensation between the timestamp of each image and the reference window based on the inertial measurement data and the state vector;

[0058] Step S4, respectively constructing inertial measurement residuals and visual measurement residuals corresponding to each image based on posture compensation amounts;

[0059] Step S5: performing iterative optimization and solving on the inertial measurement residual and the visual measurement residual based on a nonlinear optimization method.

[0060] The solution of the present application models the difference in the inconsistent timestamps of images taken by different cameras in the VIO system, so that higher accuracy and robustness can be achieved even when different cameras in the VIO system are not shooting at the same time; this solution can make the VIO system more accurate and efficient in detecting rotating targets.

[0061] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0062] To further elaborate on the technical solution of this application, here we first briefly introduce the general / conventional VIO system framework, which is configured with two cameras and an IMU (Inertial Measurement Unit). First, we introduce the implementation process of the VIO system with multi-camera timestamp synchronization under ideal conditions.

[0063] According to Figure 2 The embodiment shown shows the case where the camera timestamps are the same (two cameras observe at the same time). Figure 2Only a binocular camera is used as an example for description, but the principle can be extended to multi-cameras. In a VIO system, the camera image frame rate is usually low (usually 25-30Hz), but the IMU frame rate is relatively high (usually 200-8000Hz). Therefore, in an optimized VIO, the IMU is usually pre-integrated and then fused with the camera image observation results.

[0064] The existing VIO systems in academia and industry all adopt the sliding window-based mechanism. Figure 2 As shown in the figure, the sliding window contains multiple windows, t0, t1, and t2 represent a window in the sliding window respectively. Each window contains a 16-dimensional state vector, which are: the 3D posture of the IMU (which is represented by 4-dimensional variables, which is a quaternion), the 3D position of the IMU (which is represented by 3-dimensional variables), the 3D velocity of the IMU (represented by 3-dimensional variables), the acceleration bias of the IMU (represented by 3-dimensional variables), and the gyroscope bias of the IMU (represented by 3-dimensional variables).

[0065] The state vector is represented as: ;in Represents the rotation quaternion of the IMU, represents the three-dimensional translation of the IMU, represents the three-dimensional velocity of the IMU, represents the accelerometer bias of the IMU, It should be noted that the state vector is obtained by iterative optimization solution in step S5.

[0066] like Figure 2 As shown in the figure, there are multiple IMU observations between t0 and t1 (acceleration and angular velocity of the system observed each time), which are pre-integrated into M (i.e., IMU pre-integration results); and at t0, two images observed by two cameras will be obtained, and at t1, t2, t3, etc., two camera images will be obtained. Based on the IMU and camera measurement results, VIO observation modeling can be performed, and the system state at each time can be estimated. The state vector at t0 is , the state vector at time t1 is , and so on.

[0067] (1) Inertial measurement modeling: and The state vector at the moment is combined with the IMU pre-integration between the two to construct the inertial residual (i represents the sliding window number). The following formula Represent the inertial measurement residual:

[0068] (1)

[0069] (2) Visual measurement modeling: Based on the observation information of each landmark point (or feature, English landmark) (the two-dimensional coordinates of the landmark point in the image) in each window of the sliding window, visual observation modeling is performed separately. The following formula Represents the visual measurement residual (i represents the sliding window number, j represents the landmark point number, and k represents the camera number):

[0070] (2)

[0071] in, Indicates that in the sliding window The actual observation value (de-distorted actual observation value) of camera k at the jth landmark point at the time (i.e., the i-th window). The visual observation residual can be obtained by subtracting the de-distorted actual observation value from the undistorted theoretical observation value for this landmark point. .

[0072] Represents the camera projection model, that is, given the camera rotation matrix and translation value, and the three-dimensional coordinates of the given landmark point, the theoretical observation value (normalized image two-dimensional coordinate) of the landmark point in the camera without distortion can be obtained;

[0073] pass Included inside The IMU pose and camera extrinsics at that moment can be used to obtain the observation time ( The camera pose at the time (time). The pose of the IMU includes the 3D rotation and 3D position of the IMU, namely ; The three-dimensional rotation (or attitude) represents the transition from the IMU coordinate system to the earth coordinate system, and the three-dimensional position represents the translation of the IMU in the earth coordinate system. The camera extrinsic parameters are the rotation and translation from the camera to the IMU, defined as This parameter can be obtained by offline calibration when the system leaves the factory, or it can be calibrated by the user himself with the help of a calibration device. It is a known quantity. This method is a well-known technology and is described as follows:

[0074] Camera rotation at time:

[0075] Camera translation at time:

[0076] Indicates that the quaternion will be rotated Equivalently converted to a 3×3 rotation matrix. Academically, quaternions represent three-axis rotations. This application uses Represents the quaternion Equivalently converted to the corresponding 3×3 rotation matrix.

[0077] Therefore, given the three-dimensional coordinates of the landmark point , The theoretical observation value (normalized image two-dimensional coordinate) of the landmark point without distortion in the camera at the observation time can be obtained.

[0078] The system state (that is, the state to be estimated) contains the state vectors of all windows in the sliding window and the three-dimensional coordinates of all landmarks that can be observed in the sliding window (three-dimensional coordinates in the world coordinate system). That is:

[0079]

[0080] All valid inertial measurement residuals and binocular camera visual residuals in the above sliding window are used to calculate the Jacobian matrix of the system state.

[0081] Bring all residuals, corresponding Jacobian matrices and system states into g2o, ceres or gtsam (three commonly used nonlinear optimization solvers) or other nonlinear optimizers to iteratively optimize and solve all system states.

[0082] Alternatively, in filter-based VIO / VSLAM, inertial measurement data is used to predict the system state to the next moment, and then the visual measurement residual and its corresponding Jacobian matrix and system state are substituted into the Extended Kalman Filter (EKF) to update the system state.

[0083] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0084] In a conventional binocular VIO system, the image data collected by the two cameras must be exposed at the same timestamp, that is, the centers of the exposure intervals of the two cameras need to coincide. Figure 2 The ideal situation shown in the figure is the same as that of multi-camera, which requires that multiple cameras must be exposed at the same timestamp. In a multi-camera VIO system, the mismatch of image exposure timestamps between multiple cameras will result in reduced accuracy.

[0085] In order to solve the problem of camera asynchrony, the solution of the present application models the difference in the inconsistent timestamps of different cameras in the VIO system, so that higher accuracy and robustness can be achieved even when different cameras in the VIO system are not shooting at the same time.

[0086] This embodiment takes binocular VIO as an example to elaborate on the VIO implementation method proposed by the present invention when the cameras are not synchronized. It is easy to understand that the method of the present invention is not limited to binocular cameras, but is fully applicable to the VIO system of multi-cameras.

[0087] like Figure 3 As shown, since the timestamps of the two images corresponding to each received binocular image are not equal, the timestamp of the window of the current frame is specified by itself. Step S2 determines the timestamp of the reference window corresponding to the current frame according to the timestamps of multiple images. There are multiple ways to do this: (1) determine the smallest timestamp among the multiple images as the timestamp of the reference window corresponding to the current frame; (2) determine the largest timestamp among the multiple images as the timestamp of the reference window corresponding to the current frame; (3) determine the result of averaging the timestamps of multiple images as the timestamp of the reference window corresponding to the current frame.

[0088] In this embodiment, the minimum timestamp in a frame of binocular image (i.e., t0 in the figure) is specified as the timestamp of the window corresponding to the current binocular frame (the maximum timestamp may also be selected, or the result of averaging the timestamps of all images in the frame, etc.). This embodiment is explained by way of "minimum timestamp", and the principles of other solutions are exactly the same and can be easily derived, so they will not be described in detail.

[0089] like Figure 3 As shown in the figure, for the problem that there is only one corresponding window (i.e., IMU pose) for a frame of binocular image (and the timestamps of the two images in the frame are inconsistent), the IMU measurement results can be used to represent the IMU pose corresponding to the timestamp of another image based on the existing IMU pose, and then the residual model can be constructed based on this IMU pose.

[0090] Since the present invention specifies the minimum timestamp in a frame of binocular image (i.e., t0 in the figure) as the timestamp of the window corresponding to the current binocular frame, Figure 3 The observation timestamp of camera 1 is used as the timestamp of the corresponding window. , there is a system state vector corresponding to the IMU posture in the sliding window , and for , there is no corresponding IMU pose in the sliding window.

[0091] In this embodiment, step S3 calculates the posture compensation between the timestamp of each image and the reference window based on the inertial measurement data and the state vector, and specifically includes the following steps: calculating the time difference between the timestamp of each image and the timestamp of the reference window; calculating the posture compensation between each image and the reference window according to the inertial measurement data, the state vector and the time difference. Among them, the posture compensation includes: rotation compensation and translation compensation; the inertial measurement data includes: accelerometer readings and gyroscope readings.

[0092] Specifically, the posture compensation between each image and the reference window is calculated according to the inertial measurement data and the time difference, including the following steps: calculating the rotation compensation according to the gyroscope reading, the gyroscope bias and the time difference; calculating the translation compensation according to the accelerometer reading, the accelerometer bias, the speed and the time difference.

[0093] The following is for Figure 3 For example, when i=0, it means the 0th window in the sliding window, so and They represent the rotation compensation and translation compensation of camera k in the 0th window, that is, express The rotation matrix of the IMU at this moment, express The translation of the IMU at time t.

[0094] and The calculation method is:

[0095]

[0096]

[0097] The IMU readings are divided into accelerometer and gyroscope readings. At this moment, the accelerometer reading is , the gyroscope reading is .therefore , ,and Represents the gravitational acceleration of the Earth.

[0098] Next, step S4 is executed to respectively construct the inertial measurement residual and the visual measurement residual corresponding to each image based on the posture compensation amount. The inertial measurement residual can be modeled by the formula (1) introduced in the above embodiment. The corresponding IMU pose increases the compensation method to show The IMU pose corresponding to the observation timestamp. Figure 1 The embodiments are different. In this embodiment, The visual residual of is modeled as follows:

[0099] (3)

[0100] Above Indicates that camera k is The rotation compensation amount at the moment, Indicates that camera k is The amount of translation compensation at the moment.

[0101] Next, in the case of a binocular camera, all valid inertial measurement residuals in the sliding window (Formula 1) and the visual residuals of camera 1 (Formula 2) and camera 2 (Formula 3) are used to calculate the Jacobian matrix of the system state. In the case of multiple cameras, the visual residuals of multiple cameras need to be calculated according to Formula 3.

[0102] Finally, all residuals, corresponding Jacobian matrices and system states are brought into g2o or ceres or gtsam (three commonly used nonlinear optimization solvers) or other nonlinear optimizers to iteratively optimize and solve all system states and landmark coordinates. Or in filter-based VIO / VSLAM, inertial measurement data is used to predict the system state to the next moment, and then the visual measurement residuals and their corresponding Jacobian matrices and system states are substituted into the extended Kalman filter (EKF) to update the system state.

[0103] It should be noted that the aforementioned embodiment only takes the VIO of nonlinear optimization iteration as an example. The method of the present application can be used not only for VIO based on nonlinear optimization iteration, but also for VIO based on filtering framework. Specifically, it can be applied to the VIO / VISLAM solver based on filtering. The solver based on filtering does not have a pre-integration process, but is replaced by a prediction process. The other steps are the same as the VIO method of nonlinear optimization iteration, and the aforementioned embodiment can be referred to.

[0104] The technical solution of the present application, in multi-camera VIO (or multi-camera VSLAM), when performing residual modeling for camera observations with inconsistent timestamps, uses the state vector and IMU reading corresponding to the existing window closest to the timestamp in the sliding window to construct the IMU pose at the timestamp corresponding to other time-unaligned cameras, thereby achieving accurate residual modeling for multi-camera observations with inconsistent timestamps.

[0105] In one embodiment, a computer device is provided. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the computer program is executed by the processor, a VIO implementation method for asynchronous acquisition of multiple cameras is implemented.

[0106] In one embodiment, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the VIO implementation method for asynchronous acquisition of multiple cameras as described in any of the above embodiments is implemented.

[0107] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0108] It should be noted that, in the description of this application, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0109] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0110] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0111] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0112] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0113] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0114] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0115] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A VIO implementation method for asynchronous multi-camera acquisition, characterized in that: The following steps are involved: Obtain timestamps of multiple images corresponding to the current frame; wherein the multiple images are images captured by multiple cameras in the time window of the same frame; determine the timestamp of the reference window corresponding to the current frame according to the timestamps of the multiple images; calculate the pose compensation between the timestamp of each image and the reference window based on the inertial measurement data and the state vector; respectively construct the inertial measurement residual and the visual measurement residual corresponding to each image based on the pose compensation; The inertial measurement residuals and visual measurement residuals are iteratively optimized and solved based on the nonlinear optimization method.

2. The method according to claim 1, characterized in that The method of determining the timestamp of the reference window corresponding to the current frame according to the timestamps of the multiple images comprises the following steps: determining the smallest timestamp among the multiple images as the timestamp of the reference window corresponding to the current frame; or The largest timestamp among the multiple images is determined as the timestamp of the reference window corresponding to the current frame; or, the result of averaging the timestamps of the multiple images is determined as the timestamp of the reference window corresponding to the current frame.

3. The method according to claim 1, characterized in that The state vector is obtained by performing iterative optimization and solving the inertial measurement residual and the visual measurement residual based on the nonlinear optimization method; the state vector is ; Represents the rotation quaternion of the IMU, represents the three-dimensional position of the IMU, represents the three-dimensional velocity of the IMU, represents the accelerometer bias of the IMU, Represents the gyroscope bias of the IMU; the calculation of the posture compensation between the timestamp of each image and the reference window based on the inertial measurement data and the state vector comprises the following steps: Calculate the time difference between the timestamp of each image and the timestamp of the reference window; The pose compensation between each image and the reference window is calculated based on the inertial measurement data, state vector and time difference.

4. The method according to claim 3, characterized in that The posture compensation includes: rotation compensation and translation compensation; the inertial measurement data includes: accelerometer reading and gyroscope reading; The method of calculating the position compensation between each image and the reference window according to the inertial measurement data and the time difference comprises the following steps: Calculate the rotation compensation amount based on the gyroscope reading, the gyroscope bias and the time difference; The translation compensation is calculated based on the accelerometer reading, accelerometer bias, velocity, and time difference.

5. The method according to claim 4, characterized in that The calculation of the rotation compensation amount according to the gyroscope reading, the gyroscope bias and the time difference comprises the following steps: ,in ; in, Indicates that camera k is The rotation compensation at the moment, I represents the unit matrix, Indicated in The gyroscope reading at the time, Indicates the gyroscope bias of the IMU, Indicates that camera k is The time difference between the instant and the timestamp of the base window.

6. The method according to claim 4, characterized in that The step of calculating the translation compensation amount according to the accelerometer reading, the accelerometer bias, the velocity and the time difference comprises the following steps: ,in ; in, Indicates that camera k is The translation compensation at time, represents the IMU three-dimensional velocity, Indicates that camera k is The time difference between the instant and the timestamp of the reference window, Represents the quaternion Equivalently converted to the corresponding 3×3 rotation matrix, Indicated in The accelerometer reading at time, represents the accelerometer bias of the IMU, Represents the gravitational acceleration of the Earth.

7. The method according to any one of claims 1 to 6, characterized in that: The method of respectively constructing the inertial measurement residual and the visual measurement residual corresponding to each image based on the posture compensation amount comprises the following steps: The inertial measurement residual at time is: ;in, for The system state vector at time , for The system state vector at time , for Time has come IMU pre-integration result at the moment; The visual measurement residual at the moment is: ;in, Indicates that in the sliding window The actual observation value of camera k at the jth landmark point at the moment, represents the camera projection model, Represents the quaternion Equivalently converted to the corresponding 3×3 rotation matrix, Represents the rotation quaternion of the IMU, Indicates that camera k is The rotation compensation amount at the moment, is the camera extrinsic parameter, Indicates that camera k is The translation compensation at time, Represents the three-dimensional coordinates of a given landmark point.

8. The method according to claim 7, characterized in that The method of iteratively optimizing and solving the inertial measurement residuals and visual measurement residuals based on the nonlinear optimization method includes the following steps: calculating the Jacobian matrix of the system state for all the inertial measurement residuals and all the visual measurement residuals corresponding to all the frames in the current sliding window; bringing all the inertial measurement residuals and visual measurement residuals and their corresponding Jacobian matrices, as well as the system state into the nonlinear optimizer, and iteratively optimizing and solving the system state; or, in the filtering-based VIO / VSLAM, using the inertial measurement data to predict the system state to the next moment, and then using the visual measurement residuals and their corresponding Jacobian matrix, and the system state to substitute into the extended Kalman filter, and updating the system state; wherein the system state is: 。 9. A computer device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program in the memory to implement the operation steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the operating steps of the method according to any one of claims 1 to 8 are implemented.

Images