A visual-inertial wheel-odometry positioning method and computer readable medium

Abstract

The application provides a vision-inertial wheel speed odometer positioning method and a computer readable medium. Feature extraction and tracking are performed on a front-view image of a vehicle; a time convolutional neural network is constructed for original IMU input, and gyro data in the IMU is denoised; the denoised gyro output and original accelerometer measurement value are used to calculate IMU pre-integration between adjacent two frames, and the denoised gyro output and a wheel speed meter are used to calculate a vehicle speed meter pre-integration between adjacent two frames; in an initialization stage, a super-determined equation set is constructed, and the gravity direction of the first frame and the optimized speed of each frame are solved; a nonlinear optimization objective function is constructed, the pre-integrations of the IMU and the wheel speed meter are updated according to the calibrated bias; the horizontal yaw direction obtained by integrating the denoised gyro angular velocity is replaced with the yaw direction in the optimized system state variable, and a final pose output result is obtained. The application improves the accuracy and stability of vision-inertial wheel speed odometer positioning, and avoids divergence of an estimated trajectory.

Description

Technical Field

[0001] This invention belongs to the field of unmanned vehicle positioning, and particularly relates to a visual inertial wheel speed odometer positioning method and a computer-readable medium. Background Technology

[0002] Achieving high-precision and robust localization of unmanned ground vehicles using low-cost multimodal sensors in challenging urban environments without GPS signals is a daunting task. In recent years, research on continuous vehicle pose estimation using cameras, IMUs, and wheel encoders has received widespread attention. Compared to traditional visual odometry (VO) or visual inertial odometry (VIO) methods, the additional wheel speed encoder can provide real-scale velocity information, which helps improve the accuracy and stability of scale estimation in traditional VIO methods, thereby enhancing the accuracy and robustness of vehicle pose estimation, especially when the vehicle is undergoing planar motion, such as constant-speed motion.

[0003] However, similar to traditional visual-inertial odometry (VIO) methods, visual-inertial-wheel odometry (VIWO) methods still cannot avoid the pose drift caused by the low-precision attitude acquired by the gyroscope of low-cost MEMS, especially when the (VIO) or (VIWO) methods encounter visual degradation scenarios, such as weak / repetitive textures, strong lighting, or motion blur. In fact, limited by the accuracy of MEMS inertial navigation, especially due to bias instability and angle random walk caused by AWGN, noise, and thermal effects, the attitude obtained by integrating the raw gyroscope output will accumulate a large long-term drift over time. This attitude drift will cause the position error obtained by integrating through accelerometer or wheel speedometer and the gyroscope bias error to increase cubically over time. In existing VIO or VIWO methods, this position error is input as part of the joint residual into the tightly coupled nonlinear optimization solver of VIO or VIWO, which will lead to a decrease in the solver's pose estimation accuracy and thus long-term pose drift, even though existing VIO or VIWO methods have applied pre-integration techniques.

[0004] Therefore, in both traditional VIO systems and current VIWO systems, the accuracy of IMU measurement data, especially the quality of gyroscope data, is a crucial factor affecting the overall orientation and position accuracy of the system over a long period. Thus, correcting and compensating for MEMSMU errors to achieve long-term attitude accuracy stability is essential for improving the overall positioning accuracy of VIO or VIWO systems. Currently, for VIO or VIWO systems, one approach to limit pose drift is to introduce a Global Positioning System (GPS). However, GPS is sometimes unstable, especially in urban canyons or parking lot environments. Another approach is to use a closed-loop detection strategy to constrain position drift, but this strategy is not suitable for intelligent vehicles operating in large urban environments. Therefore, accurate IMU calibration of VIO or VIWO systems is necessary. In recent years, there has been much research on IMU calibration or camera / IMU calibration. Most methods tend to model the IMU measurement output as a linear polynomial mathematical model of the system error, with the calculation of the corresponding coefficient parameters relying on high-precision external equipment, i.e., a 3-axis turntable. High-precision turntables are typically expensive and bulky, and their output models neglect higher-order error terms and time-varying characteristics. Online calibration methods, such as Kalman filtering, correct IMU errors by reducing inertial navigation result errors. While online calibration methods reduce reliance on external devices, they struggle to remove nonlinear and non-Gaussian random errors. To reduce long-term drift in autonomous vehicles, this invention proposes an effective combination of deep learning-denoised gyroscope data and traditional visual-inertial wheel odometers to achieve long-term pose estimation in complex driving environments. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a visual inertial wheel speed odometer positioning method and a computer-readable medium.

[0006] This invention proposes a visual inertial wheel speed odometer positioning method, characterized in that:

[0007] The vehicle pose ground truth values ​​from multiple frames are divided into multiple pose ground truth value sequences, and the pose increment of each pose ground truth value sequence is calculated.

[0008] A temporal convolutional neural network model is constructed. The predicted angular velocity of the three-axis angular velocity of the IMU in each frame is obtained by prediction. The attitude increment of each group of predicted angular velocity sequences is obtained by grouping and integral calculation. The loss function model is constructed by combining the attitude increment of the corresponding true attitude sequence of each group. The optimized temporal convolutional neural network is obtained by optimization training.

[0009] By combining the prediction of the optimized temporal convolutional neural network and the calculation of the pre-integration method, the IMU attitude pre-integration information, IMU velocity pre-integration information, IMU displacement pre-integration information, and wheel speed meter displacement pre-integration information of two adjacent frames in the test set are obtained.

[0010] An overdetermined system of equations is constructed, solved by the least squares method, and the initial pose of each frame is obtained by integration. The system is then triangulated to obtain the initial system state variables.

[0011] A nonlinear optimization objective function is jointly constructed by IMU-wheel speed measurement pre-integration, visual reprojection residual, and prior marginalization residual. The optimized system state variable estimates are obtained by performing a tightly coupled nonlinear optimization iterative solution based on a sliding window using the LM algorithm.

[0012] The specific steps of this invention are as follows:

[0013] Step 1: Acquire multiple consecutive frames of vehicle front view images, multiple frames of IMU three-axis acceleration, multiple frames of IMU three-axis angular velocity, multiple frames of wheel speed measurement velocity, and multiple frames of vehicle attitude ground truth; randomly divide the multiple frames of IMU three-axis angular velocity into training set and test set; divide the multiple frames of vehicle attitude ground truth into multiple sets of attitude ground truth sequences, and calculate the attitude increment of each set of attitude ground truth sequences.

[0014] Step 2: Use a feature extraction algorithm to extract Fast corner features from the currently acquired vehicle front view image, and use the KLT algorithm for optical flow tracing on consecutive frames;

[0015] Step 3: Construct a temporal convolutional neural network model. Input the IMU triaxial angular velocity of each frame in the training set into the temporal convolutional neural network model for prediction to obtain the predicted angular velocity of the IMU triaxial angular velocity of each frame. Divide the predicted angular velocities of the IMU triaxial angular velocities of multiple frames into multiple sets of predicted angular velocity sequences. Integrate each set of predicted angular velocity sequences to obtain the attitude increment of each set of predicted angular velocity sequences. Combine the attitude increment of each set of attitude ground value sequences to construct a loss function model. Solve the temporal convolutional neural network through the Adam optimizer to obtain the optimized temporal convolutional neural network.

[0016] Step 4: Input the IMU triaxial angular velocity of each frame in the test set into the optimized temporal convolutional neural network to obtain the IMU predicted angular velocity of each frame in the test set. Calculate the IMU predicted angular velocity of each frame in the test set and the corresponding frame's IMU triaxial acceleration using a pre-integration method to obtain the IMU attitude pre-integration information, IMU velocity pre-integration information, and IMU displacement pre-integration information for two adjacent frames in the test set. Calculate the IMU predicted angular velocity of each frame in the test set and the corresponding frame's wheel velocity using a pre-integration method to obtain the wheel velocity displacement pre-integration information for two adjacent frames in the test set.

[0017] Step 5: Construct a linear equation system by combining the IMU displacement pre-integration information, IMU attitude pre-integration information, IMU velocity pre-integration information, and wheel speedometer displacement pre-integration information from two adjacent frames in the test set; stack multiple linear equation systems within a sliding window of size n to construct an overdetermined equation system; solve for the optimized velocity and the first frame gravity vector in each frame using the least squares method; combine the optimized velocity with the denoised IMU angular velocity and calculate the initial pose of each frame by integration; further triangulate to obtain the initial three-dimensional coordinates of the feature points; and initialize the system state variables to obtain the initialized system state variables.

[0018] Step 6: Using the IMU attitude pre-integration information, IMU velocity pre-integration information, IMU displacement pre-integration information, and wheel speed meter displacement pre-integration information from two adjacent frames, initialize the system state variables, construct the IMU-wheel speed meter pre-integration joint residual term, and jointly construct the nonlinear optimization objective function with the visual reprojection residual term and the prior marginalization residual term. Use the initialized system state variables as the initial values ​​for the joint nonlinear optimization, and set the IMU angular velocity Z-axis offset to always be zero. Perform tight-coupled nonlinear optimization iterative solution based on sliding window using the LM algorithm to obtain the optimized system state variable estimates.

[0019] Step 7: Replace the yaw direction in the optimized system state variables with the horizontal yaw direction obtained by integrating the denoised angular velocity to obtain the final pose output result;

[0020] Preferably, the multiple sets of attitude truth sequences mentioned in step 1 are defined as follows:

[0021] R k =(R (k-1)L+1 R (k-1)L+2 , ..., R kL )

[0022] k∈[1, K]

[0023] Among them, R k Let R represent the true sequence of the k-th pose.(k-1)L+i Let represent the IMU attitude truth value of the i-th IMU in the k-th attitude truth value sequence, i.e. the IMU attitude truth value of the (k-1)L+i-th frame, where i∈[1,L], L represents the length of each attitude truth value sequence, and K represents the total number of truth value sequences.

[0024] The attitude increment for each set of true attitude sequences described in step 1 is defined as follows:

[0025]

[0026] Wherein, δR k This represents the attitude increment truth value corresponding to the k-th attitude truth value sequence. R is the true pose value of the k-th group (k-1)L+1 frames. KL Let be the true pose value of the k-th group of kL frames;

[0027] Preferably, the temporal convolutional neural network model described in step 3 consists of multiple stacked residual blocks and fully connected layers;

[0028] Each residual block includes a first basic block and a second basic block;

[0029] Both the first and second basic blocks are composed of dilated convolutional layers, batch normalization layers, ReLU activation layers, and dropout layers connected in sequence.

[0030] The specific process of training the temporal convolutional neural network described in step 3 is as follows:

[0031] The three-axis angular velocity of each IMU in the training set is used as input data and processed through multiple stacked residual blocks and fully connected layers to obtain the predicted angular velocity of each IMU frame.

[0032] Each predicted angular velocity sequence in step 3 is defined as follows:

[0033]

[0034] Among them, pre k This represents the k-th predicted angular velocity sequence. Let i represent the i-th predicted angular velocity in the k-th predicted angular velocity sequence, i.e., the predicted angular velocity of the IMU in the (k-1)L+i-th frame, where i∈[1,L] and L represents the length of each predicted angular velocity sequence.

[0035] Step 3 involves integrating each predicted angular velocity sequence to obtain the attitude increment for each predicted angular velocity sequence, as detailed below:

[0036]

[0037] in, For the k-th group of predicted angular velocity sequences, Let be the i-th predicted angular velocity in the k-th predicted angular velocity sequence, exp(·) is the exponential mapping, δt is the time interval between two adjacent IMU predicted angular velocities, and Π(·) is the multiplication sign;

[0038] The loss function model described in step 3 is defined as follows:

[0039]

[0040] Where Log-cosh is the logarithm of hyperbolic cosine, log(·) is the logarithmic mapping of SO(3), and δR k It is the attitude increment of the k-th true attitude sequence. The attitude increment of the k-th predicted angular velocity sequence;

[0041] Preferably, the IMU attitude pre-integration information of two adjacent frames in the test set described in step 4 is calculated as follows:

[0042]

[0043] (b j ) z =0

[0044] in, This is the IMU attitude pre-integration from frame k to frame (k+1), where j∈[1,N], and N represents the total number of IMU frames between frame k and frame (k+1). Π(·) is the chain multiplication symbol. This represents the predicted angular velocity of the j-th IMU frame between frame k and frame (k+1). δt represents the angular velocity offset of the j-th IMU frame between frame k and frame (k+1). j This indicates the time interval between two adjacent IMU frames. Indicates the Z-axis offset of the IMU angular velocity at time j; (b j ) z =0 indicates that the IMU angular velocity Z-axis offset is always set to 0 during the pre-integration process;

[0045] The IMU velocity pre-integration information of two adjacent frames in the test set mentioned in step 4 is calculated as follows:

[0046]

[0047] in, It is the IMU velocity pre-integration from frame k to frame (k+1), j∈[1,N], where N represents the total number of IMU frames between frame k and frame (k+1), and ∑(·) represents the summation symbol. This represents the IMU attitude pre-integration between the j-th IMU frame and the k-th frame, from frame k to frame (k+1). Indicates IMU attitude pre-integration The rotation matrix, This represents the acceleration in the j-th IMU frame between the k-th frame and the (k+1)-th frame. δt represents the acceleration bias of the j-th IMU frame between the k-th frame and the (k+1)-th frame, and δt represents the time interval between two adjacent IMU frames.

[0048] The IMU displacement pre-integration information of two adjacent frames in the test set mentioned in step 4 is calculated as follows:

[0049]

[0050] in, It is the IMU shift pre-integration from frame k to frame (k+1), j∈[1,N], where N represents the total number of IMU frames between frame k and frame (k+1), and ∑(·) represents the summation symbol. δt represents the IMU velocity pre-integration between the j-th IMU frame and the k-th frame (from frame k to frame (k+1)). This represents the IMU attitude pre-integration between the j-th IMU frame and the k-th frame, from frame k to frame (k+1). Indicates IMU attitude pre-integration The rotation matrix, This represents the acceleration in the j-th IMU frame between the k-th frame and the (k+1)-th frame. This represents the acceleration bias of the j-th IMU frame between the k-th frame and the (k+1)-th frame.

[0051] The pre-integral information of wheel velocity gauge displacement between two adjacent frames in the test set described in step 4 is calculated as follows:

[0052]

[0053] in, This represents the pre-integration of the wheel velocity meter displacement from frame k to frame (k+1), where j∈[1, N], N represents the total number of IMU frames between frame k and frame (k+1), and ∑(·) represents the summation symbol. This represents the IMU attitude pre-integration between the j-th IMU frame and the k-th frame, from the k-th frame to the (k+1)-th frame. Indicates IMU attitude pre-integration The rotation matrix, This represents the extrinsic parameter rotation matrix between the IMU and the wheel speedometer. δt represents the wheel speed measurement speed corresponding to the j-th IMU frame between the k-th frame and the (k+1)-th frame, and δt represents the time interval between two adjacent IMU frames.

[0054] Preferably, the linear equation system described in step 5 is specifically defined as follows:

[0055]

[0056]

[0057] in, This represents the IMU displacement pre-integration from frame k to frame (k+1). It is the IMU velocity pre-integration from frame k to frame k+1. Pre-integration of wheel velocity gauge displacement from frame k to frame k+1 This represents the speed at which the solution to be optimized is obtained in the k-th frame. This represents the displacement extrinsic parameter between the IMU and the wheel speed gauge. It is the IMU attitude pre-integration rotation matrix from frame k to frame (k+1). It is the IMU attitude pre-integration from frame k to frame (k+1). It is the IMU attitude pre-integration rotation matrix from the first frame to the kth frame. It is the IMU attitude pre-integration from the first frame to the kth frame. Let δT be the gravity vector in the first frame of the IMU coordinate system to be solved. k T represents the time interval between frame k and frame k+1. k =T k+1 -T k , T represents the speed at which the solution to be optimized is obtained in the (k+1)th frame. k T represents the time corresponding to k frames of the image. k+1 This represents the time corresponding to frame k+1 of the image.

[0058] The overdetermined system of equations mentioned in step 5 is specifically defined as follows:

[0059]

[0060] Where n is the size of the sliding window;

[0061] Step 5 involves initializing the system state variables, which are specifically defined as follows:

[0062]

[0063] Wherein, the set of system state variables obtained by initialization χ, x k This represents the vehicle state variables in the k-th frame within the sliding window, including the position of the k-th frame within the sliding window in the world coordinate system. The velocity of the k-th frame within the sliding window in the world coordinate system The direction of the k-th frame within the sliding window in the world coordinate system. Initial IMU acceleration bias within the sliding window in the world coordinate system (frame k) Initial angular velocity offset within the k-th frame of the sliding window in the world coordinate system. These two biases are prior known quantities, n is the size of the sliding window, and λ is the value of the bias. l This represents the inverse depth at which the l-th feature point within the sliding window is first observed. It is the initial extrinsic parameter between the camera and the IMU, including relative position. and direction It is a priori known quantity;

[0064] Preferably, the IMU-wheel speed meter pre-integration residual term in step 6 is specifically defined as follows:

[0065]

[0066] in, This represents the IMU-wheel velocity meter pre-integration joint residual term between the k-th frame and the (k+1)-th frame within the sliding window. The set of IMU-wheel speedometer pre-integration information between frame k and frame (k+1), and the set of system state variables obtained by initialization with χ. This represents the IMU displacement pre-integration residual between frame k and frame (k+1). This represents the IMU velocity pre-integration residual between frame k and frame (k+1). This represents the IMU attitude pre-integration residual between frame k and frame (k+1). δb represents the pre-integral residual of the wheel velocity gauge displacement between frame k and frame (k+1). a δb represents the IMU acceleration bias residual. ω This represents the IMU angular velocity offset residual. It is the attitude rotation matrix of the IMU frame corresponding to the k-th frame in the world coordinate system. It is the relative displacement of the IMU frame corresponding to the k-th frame in the world coordinate system. It is the attitude rotation matrix of the IMU frame corresponding to the (k+1)th frame in the world coordinate system. It is the relative displacement of the IMU frame corresponding to the (k+1)th frame in the world coordinate system, g w It is the gravity vector in the world coordinate system, δT 2 δT represents the square of the time interval between image frames. This represents the IMU displacement pre-integration from frame k to frame (k+1). It is the velocity of the IMU frame corresponding to the k-th frame in the world coordinate system. It is the IMU velocity pre-integration from frame k to frame k+1. It is the velocity of the IMU frame corresponding to the (k+1)th frame in the world coordinate system. It is the attitude quaternion of the IMU frame corresponding to the k-th frame in the world coordinate system. It is the inverse matrix of the attitude quaternion of the IMU frame corresponding to the k-th frame in the world coordinate system. It is the pose quaternion of the IMU frame corresponding to the (k+1)th frame in the world coordinate system. It is the IMU attitude pre-integration from frame k to frame (k+1). This represents the displacement extrinsic parameter between the IMU and the wheel speed gauge. Pre-integration of wheel velocity gauge displacement from frame k to frame k+1 These represent the initial IMU acceleration bias values ​​in frame (k+1) and frame (k), respectively. These represent the initial offset of the IMU angular velocity in the (k+1)th frame and the kth frame, respectively.

[0067] The joint nonlinear optimization objective function described in step 6 is specifically defined as follows:

[0068]

[0069] (b k ) z =0, k∈[0, n-1]

[0070] In the objective function χ represents the IMU-wheel velocity meter pre-integral joint residual term between frame k and frame (k+1), where χ represents the system state variable set and n is the size of the sliding window. Let be the IMU-wheel speed measurement pre-integral covariance matrix from frame k to frame (k+1). Let Ψ be the visual reprojection residual term, and c be the set of all feature points observed more than twice in the sliding window and their corresponding image frames. m Let m be the image frame containing the l-th feature point in the sliding window. The standard covariance matrix, This represents the reprojection of the l-th feature point in the m-th frame, where ρ is the robust kernel function, (r p -H p χ) represents the prior marginalization prior residual term, r p It is marginalized prior information, H p Represents the marginalized information matrix, (b k ) z This indicates the z-axis offset of the IMU angular velocity in the k-th frame within the sliding window. Here, we ensure that the online-corrected z-axis offset is not used to update the pre-integration information.

[0071] The present invention also provides a computer-readable medium storing a computer program executed by an electronic device, wherein when the computer program is run on the electronic device, it performs the steps of the visual inertial wheel speed odometer positioning method.

[0072] The advantage of this invention lies in constructing a temporal convolutional network model, which fully utilizes the powerful data processing capabilities of deep learning to denoise the gyroscope data, thereby obtaining a more accurate orientation estimate. This provides better orientation constraints for visual inertial wheel speed odometry, thereby improving the accuracy and robustness of visual inertial wheel speed odometry positioning and reducing long-term pose drift problems. Attached Figure Description

[0073] Figure 1 : Flowchart of the method according to an embodiment of the present invention.

[0074] Figure 2 Trajectory diagram of the positioning method in this embodiment of the invention.

[0075] Figure 3 The accuracy comparison results of the embodiments of the present invention. Detailed Implementation

[0076] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0077] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0078] This invention conducted experiments in an urban road environment to test the accuracy of vehicle positioning. The system and algorithms used were implemented on Ubuntu 16.04 using ROS and C++. The system featured an Intel(R) Core(TM) i7-6700HQ 2.60GHz CPU and 16GB of RAM, and was equipped with cameras, inertial sensors, and wheel encoders. During the tests, the vehicle traveled along the route at various speeds, recording raw gyroscope angular velocity measurements, accelerometer measurements, image datasets, and vehicle encoder readings. The visual images, wheel encoder readings, and gyroscope angular velocity and accelerometer data were then integrated to achieve reliable pose estimation for the autonomous vehicle in urban areas.

[0079] The following is combined Figure 1-3 The technical solution of the method described in this invention is a visual inertial wheel speed odometer positioning method, as detailed below:

[0080] like Figure 1 The diagram shown is a flowchart of a method according to an embodiment of the present invention.

[0081] Step 1: Acquire multiple consecutive frames of vehicle front view images, multiple frames of IMU three-axis acceleration, multiple frames of IMU three-axis angular velocity, multiple frames of wheel speed measurement velocity, and multiple frames of vehicle attitude ground truth; randomly divide the multiple frames of IMU three-axis angular velocity into training set and test set; divide the multiple frames of vehicle attitude ground truth into multiple sets of attitude ground truth sequences, and calculate the attitude increment of each set of attitude ground truth sequences.

[0082] The multiple sets of pose truth sequences mentioned in step 1 are defined as follows:

[0083] R k =(R (k-1)L+1 R (k-1)L+2 , ..., R kL )

[0084] k∈[1, K]

[0085] Among them, R k Let R represent the true sequence of the k-th pose. (k-1)L+i Let represent the IMU attitude truth value of the i-th IMU in the k-th attitude truth value sequence, i.e. the IMU attitude truth value of the (k-1)L+i-th frame, where i∈[1,L], L=32 represents the length of each attitude truth value sequence; K=5652 represents the total number of truth value sequences.

[0086] The attitude increment for each set of true attitude sequences described in step 1 is defined as follows:

[0087]

[0088] Wherein, δR kThis represents the attitude increment truth value corresponding to the k-th attitude truth value sequence. R is the true pose value of the k-th group (k-1)L+1 frames. KL Let be the true pose value of the k-th group of kL frames;

[0089] Step 2: Use a feature extraction algorithm to extract Fast corner features from the currently acquired vehicle front view image, and use the KLT algorithm for optical flow tracing on consecutive frames;

[0090] Step 3: Construct a temporal convolutional neural network model. Input the IMU triaxial angular velocity of each frame in the training set into the temporal convolutional neural network model for prediction to obtain the predicted angular velocity of the IMU triaxial angular velocity of each frame. Divide the predicted angular velocities of the IMU triaxial angular velocities of multiple frames into multiple sets of predicted angular velocity sequences. Integrate each set of predicted angular velocity sequences to obtain the attitude increment of each set of predicted angular velocity sequences. Combine the attitude increment of each set of attitude ground value sequences to construct a loss function model. Solve the temporal convolutional neural network through the Adam optimizer to obtain the optimized temporal convolutional neural network.

[0091] The temporal convolutional neural network model described in step 3 consists of multiple stacked residual blocks and fully connected layers;

[0092] Each residual block includes a first basic block and a second basic block;

[0093] Both the first and second basic blocks are composed of dilated convolutional layers, batch normalization layers, ReLU activation layers, and dropout layers connected in sequence.

[0094] The specific process of training the temporal convolutional neural network described in step 3 is as follows:

[0095] The three-axis angular velocity of each IMU in the training set is used as input data and processed through multiple stacked residual blocks and fully connected layers to obtain the predicted angular velocity of each IMU frame.

[0096] Each predicted angular velocity sequence in step 3 is defined as follows:

[0097]

[0098] Among them, pre k This represents the k-th predicted angular velocity sequence. Let i represent the i-th predicted angular velocity in the k-th predicted angular velocity sequence, i.e., the predicted angular velocity of the IMU in the (k-1)L+i-th frame, where i∈[1,L], L=32 represents the length of each predicted angular velocity sequence, and K=5652 represents the total number of true value sequences.

[0099] Step 3 involves integrating each predicted angular velocity sequence to obtain the attitude increment for each predicted angular velocity sequence, as detailed below:

[0100]

[0101] in, For the k-th group of predicted angular velocity sequences, Let be the i-th predicted angular velocity in the k-th predicted angular velocity sequence, exp(·) is the exponential mapping, δt is the time interval between two adjacent IMU predicted angular velocities, and Π(·) is the multiplication sign;

[0102] The loss function model described in step 3 is defined as follows:

[0103]

[0104] Where Log-cosh is the logarithm of hyperbolic cosine, log(·) is the logarithmic mapping of SO(3), and δR k It is the attitude increment of the k-th true attitude sequence. The attitude increment of the k-th predicted angular velocity sequence;

[0105] Step 4: Input the IMU triaxial angular velocity of each frame in the test set into the optimized temporal convolutional neural network to obtain the IMU predicted angular velocity of each frame in the test set. Calculate the IMU predicted angular velocity of each frame in the test set and the corresponding frame's IMU triaxial acceleration using a pre-integration method to obtain the IMU attitude pre-integration information, IMU velocity pre-integration information, and IMU displacement pre-integration information for two adjacent frames in the test set. Calculate the IMU predicted angular velocity of each frame in the test set and the corresponding frame's wheel velocity using a pre-integration method to obtain the wheel velocity displacement pre-integration information for two adjacent frames in the test set.

[0106] The IMU attitude pre-integration information of two adjacent frames in the test set mentioned in step 4 is calculated as follows:

[0107]

[0108] (b j ) z =0

[0109] in, This is the IMU attitude pre-integration from frame k to frame (k+1), j∈[1,N], N=10 indicates that there are 10 IMU frames between frame k and frame (k+1), and Π(·) is the chain multiplication symbol. This represents the predicted angular velocity of the j-th IMU frame between frame k and frame (k+1). δt represents the angular velocity offset of the j-th IMU frame between frame k and frame (k+1). j This indicates the time interval between two adjacent IMU frames. Indicates the Z-axis offset of the IMU angular velocity at time j; (b j ) z =0 indicates that the IMU angular velocity Z-axis offset is always set to 0 during the pre-integration process;

[0110] The IMU velocity pre-integration information of two adjacent frames in the test set mentioned in step 4 is calculated as follows:

[0111]

[0112] in, This is the IMU velocity pre-integration from frame k to frame (k+1), j∈[1,N], N=10 indicates that there are 10 IMU frames between frame k and frame (k+1), and ∑(·) represents the summation symbol. This represents the IMU attitude pre-integration between the j-th IMU frame and the k-th frame, from the k-th frame to the (k+1)-th frame. Indicates IMU attitude pre-integration The rotation matrix, This represents the acceleration in the j-th IMU frame between the k-th frame and the (k+1)-th frame. δt represents the acceleration bias of the j-th IMU frame between the k-th frame and the (k+1)-th frame, and δt represents the time interval between two adjacent IMU frames.

[0113] The IMU displacement pre-integration information of two adjacent frames in the test set mentioned in step 4 is calculated as follows:

[0114]

[0115] in, It is the IMU shift pre-integration from frame k to frame (k+1), j∈[1,N], N=10 indicates that there are 10 IMU frames between frame k and frame (k+1), and ∑(·) represents the summation symbol. δt represents the IMU velocity pre-integration between the j-th IMU frame and the k-th frame (from frame k to frame (k+1)). This represents the IMU attitude pre-integration between the j-th IMU frame and the k-th frame, from the k-th frame to the (k+1)-th frame. Indicates IMU attitude pre-integration The rotation matrix, This represents the acceleration in the j-th IMU frame between the k-th frame and the (k+1)-th frame. This represents the acceleration bias of the j-th IMU frame between the k-th frame and the (k+1)-th frame.

[0116] The pre-integral information of wheel velocity gauge displacement between two adjacent frames in the test set described in step 4 is calculated as follows:

[0117]

[0118] in, This represents the pre-integration of the wheel velocity meter displacement from frame k to frame (k+1), j∈[1,N], N=10 indicates that there are 10 IMU frames between frame k and frame (k+1), and ∑(·) represents the summation symbol. This represents the IMU attitude pre-integration between the j-th IMU frame and the k-th frame, from the k-th frame to the (k+1)-th frame. Indicates IMU attitude pre-integration The rotation matrix, This represents the extrinsic parameter rotation matrix between the IMU and the wheel speedometer. δt represents the wheel speed measurement speed corresponding to the j-th IMU frame between the k-th frame and the (k+1)-th frame, and δt represents the time interval between two adjacent IMU frames.

[0119] Step 5: Construct a system of linear equations by combining the IMU displacement pre-integration information, IMU attitude pre-integration information, IMU velocity pre-integration information, and wheel speedometer displacement pre-integration information from two adjacent frames in the test set; stack multiple linear equations within a sliding window of size n=10 to construct an overdetermined system of equations; solve for the optimized velocity and the first frame gravity vector in each frame using the least squares method; combine the optimized velocity with the denoised IMU angular velocity and calculate the initial pose of each frame by integration; further triangulate to obtain the initial three-dimensional coordinates of the feature points; and initialize the system state variables to obtain the initialized system state variables.

[0120] The linear equation system described in step 5 is specifically defined as follows:

[0121]

[0122]

[0123] in, This represents the IMU displacement pre-integration from frame k to frame (k+1). It is the IMU velocity pre-integration from frame k to frame k+1. Pre-integration of wheel velocity gauge displacement from frame k to frame k+1 This represents the speed at which the solution to be optimized is obtained in the k-th frame. This represents the displacement extrinsic parameter between the IMU and the wheel speed gauge. It is the IMU attitude pre-integration rotation matrix from frame k to frame (k+1). It is the IMU attitude pre-integration from frame k to frame (k+1). It is the IMU attitude pre-integration rotation matrix from the first frame to the kth frame. It is the IMU attitude pre-integration from the first frame to the kth frame. Let δT be the gravity vector in the first frame of the IMU coordinate system to be solved. k T represents the time interval between frame k and frame k+1. k =T k+1 -T k , T represents the speed at which the solution to be optimized is obtained in the (k+1)th frame. k T represents the time corresponding to k frames of the image. k+1 This represents the time corresponding to frame k+1 of the image.

[0124] The overdetermined system of equations mentioned in step 5 is specifically defined as follows:

[0125]

[0126] Where n = 10 is the size of the sliding window;

[0127] Step 5 involves initializing the system state variables, which are specifically defined as follows:

[0128]

[0129] Wherein, the set of system state variables obtained by initialization χ, x k This represents the vehicle state variables in the k-th frame within the sliding window, including the position of the k-th frame within the sliding window in the world coordinate system. The velocity of the k-th frame within the sliding window in the world coordinate system The direction of the k-th frame within the sliding window in the world coordinate system. Initial IMU acceleration bias within the sliding window in the world coordinate system (frame k) Initial angular velocity offset within the k-th frame of the sliding window in the world coordinate system. These two biases are prior knowns, n = 10 is the size of the sliding window, and λ l This represents the inverse depth at which the l-th feature point within the sliding window is first observed. It is the initial extrinsic parameter between the camera and the IMU, including relative position. and direction It is a priori known quantity;

[0130] Step 6: Using the IMU attitude pre-integration information, IMU velocity pre-integration information, IMU displacement pre-integration information, and wheel speedometer displacement pre-integration information from two adjacent frames, initialize the system state variables, construct the IMU-wheel speedometer pre-integration joint residual term, and jointly construct the nonlinear optimization objective function with the visual reprojection residual term and the prior marginalization residual term. Use the initialized system state variables as the initial values ​​for the joint nonlinear optimization, and set the IMU angular velocity Z-axis offset to always be zero. Perform tight-coupled nonlinear optimization iterative solution based on sliding window using the LM algorithm to obtain the optimized system state variable estimates.

[0131] The IMU-wheel speed meter pre-integration residual term mentioned in step 6 is specifically defined as follows:

[0132]

[0133] in, This represents the IMU-wheel velocity meter pre-integration joint residual term between the k-th frame and the (k+1)-th frame within the sliding window. The set of IMU-wheel speedometer pre-integration information between frame k and frame (k+1), and the set of system state variables obtained by initialization with χ. This represents the IMU displacement pre-integration residual between frame k and frame (k+1). This represents the IMU velocity pre-integration residual between frame k and frame (k+1). This represents the IMU attitude pre-integration residual between frame k and frame (k+1). δb represents the pre-integral residual of the wheel velocity gauge displacement between frame k and frame (k+1). a δb represents the IMU acceleration bias residual. ω This represents the IMU angular velocity offset residual. It is the attitude rotation matrix of the IMU frame corresponding to the k-th frame in the world coordinate system. It is the relative displacement of the IMU frame corresponding to the k-th frame in the world coordinate system. It is the attitude rotation matrix of the IMU frame corresponding to the (k+1)th frame in the world coordinate system. It is the relative displacement of the IMU frame corresponding to the (k+1)th frame in the world coordinate system, g w It is the gravity vector in the world coordinate system, δT 2 δT represents the square of the time interval between image frames. This represents the IMU displacement pre-integration from frame k to frame (k+1). It is the velocity of the IMU frame corresponding to the k-th frame in the world coordinate system. It is the IMU velocity pre-integration from frame k to frame k+1. It is the velocity of the IMU frame corresponding to the (k+1)th frame in the world coordinate system. It is the attitude quaternion of the IMU frame corresponding to the k-th frame in the world coordinate system. It is the inverse matrix of the attitude quaternion of the IMU frame corresponding to the k-th frame in the world coordinate system. It is the pose quaternion of the IMU frame corresponding to the (k+1)th frame in the world coordinate system. It is the IMU attitude pre-integration from frame k to frame (k+1). This represents the displacement extrinsic parameter between the IMU and the wheel speed gauge. Pre-integration of wheel velocity gauge displacement from frame k to frame k+1 These represent the initial IMU acceleration bias values ​​in frame (k+1) and frame (k), respectively. These represent the initial offset of the IMU angular velocity in the (k+1)th frame and the kth frame, respectively.

[0134] The joint nonlinear optimization objective function described in step 6 is specifically defined as follows:

[0135]

[0136] (b k ) z =0, k∈[0, n-1]

[0137] In the objective function χ represents the IMU-wheel velocity meter pre-integral joint residual term between frame k and frame (k+1), where χ represents the system state variable set, and n = 10 is the size of the sliding window. Let be the IMU-wheel speed measurement pre-integral covariance matrix from frame k to frame (k+1). Let Ψ be the visual reprojection residual term, and c be the set of all feature points observed more than twice in the sliding window and their corresponding image frames. m Let m be the image frame containing the l-th feature point in the sliding window. The standard covariance matrix, This represents the reprojection of the l-th feature point in the m-th frame, where ρ is the robust kernel function, (r p -H p χ) represents the prior marginalization prior residual term, r p It is marginalized prior information, H p Represents the marginalized information matrix, (b k ) z This indicates the z-axis offset of the IMU angular velocity in the k-th frame within the sliding window. Here, we ensure that the online-corrected z-axis offset is not used to update the pre-integration information.

[0138] Step 7: Replace the horizontal yaw direction in the optimized system state variables with the horizontal yaw direction obtained by integrating the denoised angular velocity to obtain the final pose output result.

[0139] Figure 2 This is a trajectory diagram obtained using the positioning method proposed in this invention. The accuracy comparison results are as follows: Figure 3 As shown, Figure 3 Typical algorithms compared include the Visual Inertial Measurement System (VINS Mono, Open-VINS) and the Visual Inertial Wheel Velocity Calculation (VIWO). Experimental results show that traditional VINS algorithms and VIWO are easily affected by IMU noise, especially gyroscope noise, over long periods, leading to rapid error accumulation and severe trajectory drift. Our method, however, uses a deep network to correct gyroscope noise. The more accurate horizontal yaw direction obtained by integrating the denoised angular velocity is further corrected and optimized to adjust the yaw direction in the state variables, thus providing a more accurate directional constraint for the visual inertial wheel velocity meter to obtain higher-precision positioning results.

[0140] A specific embodiment of the present invention also provides a computer-readable medium.

[0141] The computer-readable medium is a server workstation;

[0142] The server workstation stores the computer program executed by the electronic device. When the computer program runs on the electronic device, it causes the electronic device to execute the steps of the visual inertial wheel speed odometer positioning method of the present invention.

[0143] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0144] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A visual-inertial wheel speed odometer positioning method, characterized in that, Includes the following steps: Step 1: Acquire multiple consecutive frames of vehicle front view images, multiple frames of IMU three-axis acceleration, multiple frames of IMU three-axis angular velocity, multiple frames of wheel speed measurement velocity, and multiple frames of vehicle attitude ground truth; randomly divide the multiple frames of IMU three-axis angular velocity into training and testing sets; divide the multiple frames of vehicle attitude ground truth into multiple attitude ground truth sequences, and calculate the attitude increment of each attitude ground truth sequence; the attitude increment of each attitude ground truth sequence is defined as follows: in, For the first k The attitude increment truth value corresponding to the set of attitude truth value sequences For the first k Group The true value of the pose of a frame. For the first k Group k The true value of the frame's pose; Step 2: Use a feature extraction algorithm to extract Fast corner features from the currently acquired vehicle front view image, and use the KLT algorithm for optical flow tracing on consecutive frames; Step 3: Construct a temporal convolutional neural network model. Input the IMU triaxial angular velocity of each frame in the training set into the temporal convolutional neural network model for prediction to obtain the predicted angular velocity of the IMU triaxial angular velocity of each frame. Divide the predicted angular velocities of the IMU triaxial angular velocities of multiple frames into multiple sets of predicted angular velocity sequences. Integrate each set of predicted angular velocity sequences to obtain the attitude increment of each set of predicted angular velocity sequences. Combine the attitude increment of each set of attitude ground value sequences to construct a loss function model. Solve the temporal convolutional neural network through the Adam optimizer to obtain the optimized temporal convolutional neural network. Step 4: Input the IMU triaxial angular velocity of each frame in the test set into the optimized temporal convolutional neural network to obtain the IMU predicted angular velocity of each frame in the test set. Calculate the IMU predicted angular velocity of each frame in the test set and the corresponding frame's IMU triaxial acceleration using a pre-integration method to obtain the IMU attitude pre-integration information, IMU velocity pre-integration information, and IMU displacement pre-integration information for two adjacent frames in the test set. Calculate the IMU predicted angular velocity of each frame in the test set and the corresponding frame's wheel velocity using a pre-integration method to obtain the wheel velocity displacement pre-integration information for two adjacent frames in the test set. Step 5: Jointly construct a system of linear equations from the IMU displacement pre-integration information, IMU attitude pre-integration information, IMU velocity pre-integration information, and wheel speedometer displacement pre-integration information of two adjacent frames in the test set; in a system of equations of size... n Within the sliding window, multiple linear equations are stacked to construct an overdetermined equation system. The optimized velocity and the first frame gravity vector are solved by the least squares method. The optimized velocity is combined with the denoised IMU angular velocity and the initial pose of each frame is obtained by integration. The initial three-dimensional coordinates of the feature points are then obtained by triangulation. The system state variables are then initialized to obtain the initialized system state variables. Step 6: Using the IMU attitude pre-integration information, IMU velocity pre-integration information, IMU displacement pre-integration information, and wheel speed meter displacement pre-integration information from two adjacent frames, initialize the system state variables, construct the IMU-wheel speed meter pre-integration joint residual term, and jointly construct the nonlinear optimization objective function with the visual reprojection residual term and the prior marginalization residual term. Use the initialized system state variables as the initial values ​​for the joint nonlinear optimization, and set the IMU angular velocity Z-axis offset to always be zero. Perform tight-coupled nonlinear optimization iterative solution based on sliding window using the LM algorithm to obtain the optimized system state variable estimates. Step 7: Replace the horizontal yaw direction in the optimized system state variables with the horizontal yaw direction obtained by integrating the denoised angular velocity to obtain the final pose output result.

2. The visual inertial wheel speed odometer positioning method according to claim 1, characterized in that: The multiple sets of pose truth sequences mentioned in step 1 are defined as follows: k in, Indicates the first k Group pose truth sequence, Indicates the first k The first group of pose truth sequence i The true value of the IMU pose, i.e. the first IMU pose. The true value of the IMU pose in the frame. i ∈[ 1 , L ] ，L Indicates the length of each set of true pose values; This represents the total number of sets of truth values.

3. The visual inertial wheel speed odometer positioning method according to claim 2, characterized in that: The temporal convolutional neural network model described in step 3 consists of multiple stacked residual blocks and fully connected layers; Each residual block includes a first basic block and a second basic block; Both the first and second basic blocks are composed of dilated convolutional layers, batch normalization layers, ReLU activation layers, and dropout layers connected in sequence. The specific process of training the temporal convolutional neural network described in step 3 is as follows: The three-axis angular velocity of each IMU in the training set is used as input data and processed through multiple stacked residual blocks and fully connected layers to obtain the predicted angular velocity of each IMU frame. Each predicted angular velocity sequence in step 3 is defined as follows: in, Indicates the first k Group of predicted angular velocity sequences, Indicates the first k The first group of predicted angular velocity sequences i The predicted angular velocity, i.e. the... Predicted angular velocity of the frame IMU i ∈[ 1 , L ] ，L This indicates the length of each predicted angular velocity sequence; Step 3 involves integrating each predicted angular velocity sequence to obtain the attitude increment for each predicted angular velocity sequence, as detailed below: in, For the first k The attitude increment of the predicted angular velocity sequence. For the first k The first group of predicted angular velocity sequences i One predicted angular velocity, For exponential mapping, The time interval between two adjacent IMU-predicted angular velocities. The symbol for multiplication; The loss function model described in step 3 is defined as follows: in, Let be the logarithm of the hyperbolic cosine. yes logarithmic mapping, It is the first k The pose increment of a set of true pose sequences. For the first k The attitude increment of the predicted angular velocity sequence.

4. The visual inertial wheel speed odometer positioning method according to claim 3, characterized in that: The IMU attitude pre-integration information of two adjacent frames in the test set mentioned in step 4 is calculated as follows: in, It is the first k Frame to the k+ 1-frame IMU attitude pre-integration , express k Frame to the k+ There are a total of 1 frame One IMU frame, The multiplication symbol is used. Indicates the first k Frame to the k Between +1 frames Predicted angular velocity per IMU frame, Indicates the first k Frame to the k Between +1 frames Angular velocity offset per IMU frame Indicates the time interval between two adjacent IMU frames. This indicates the Z-axis offset of the IMU angular velocity at time j; This indicates that the IMU angular velocity Z-axis offset is always set to 0 during the pre-integration process; The IMU velocity pre-integration information of two adjacent frames in the test set mentioned in step 4 is calculated as follows: in, yes k Frame to the k+ IMU velocity pre-integration in 1 frame , Indicates the first k Frame to the k+ There are a total of 1 frame One IMU frame, The addition symbol is used to represent the summation symbol. Indicates the first k Frame to the k Between +1 frames The IMU frame and the k Inter-frame IMU attitude pre-integration Indicates IMU attitude pre-integration The rotation matrix, Indicates the first k Frame to the k Between +1 frames Acceleration per IMU frame, Indicates the first k Frame to the k Between +1 frames Acceleration bias per IMU frame Indicates the time interval between two adjacent IMU frames; The IMU displacement pre-integration information of two adjacent frames in the test set mentioned in step 4 is calculated as follows: in, It is the first k Frame to the k+ 1-frame IMU shift pre-integration, , Indicates the first k Frame to the k+ There are a total of 1 frame One IMU frame, The addition symbol is used to represent the summation symbol. Indicates the first k Frame to the k+ The j-th IMU frame between 1 and the j-th frame k Inter-frame IMU velocity pre-integration Indicates the time interval between two adjacent IMU frames. Indicates the first k Frame to the k Between +1 frames The IMU frame and the k Inter-frame IMU attitude pre-integration Indicates IMU attitude pre-integration The rotation matrix, Indicates the first k Frame to the k Between +1 frames Acceleration per IMU frame, Indicates the first k Frame to the k Between +1 frames Acceleration bias per IMU frame; The pre-integral information of wheel velocity gauge displacement between two adjacent frames in the test set described in step 4 is calculated as follows: in, Indicates the first k Frame to the k+ 1 frame wheel velocity meter displacement pre-integration, , Indicates the first k Frame to the k+ There are a total of 1 frame One IMU frame, The addition symbol is used to represent the summation symbol. Indicates the first k Frame to the k Between +1 frames The IMU frame and the k Inter-frame IMU attitude pre-integration Indicates IMU attitude pre-integration The rotation matrix, This represents the extrinsic parameter rotation matrix between the IMU and the wheel speedometer. Indicates the first k Frame to the k+ Between 1 frame j The wheel speed measurement speed corresponding to each IMU frame This indicates the time interval between two adjacent IMU frames.

5. The visual inertial wheel speed odometer positioning method according to claim 4, characterized in that: The linear equation system described in step 5 is specifically defined as follows: in, Indicates the first k Frame to the k+ 1-frame IMU shift pre-integration, yes k Frame to the k+ IMU velocity pre-integration in 1 frame No. k Frame to the k+ Wheel velocity meter displacement pre-integration in 1 frame Indicates the first k The speed of solving the frame to be optimized This represents the displacement extrinsic parameter between the IMU and the wheel speed gauge. It is the first k Frame to the k+ IMU attitude pre-integration rotation matrix for 1 frame It is the first k Frame to the k+ 1-frame IMU attitude pre-integration It is from the first frame to the... k The IMU attitude pre-integration rotation matrix of the frame. It is from the first frame to the... k IMU attitude pre-integration of frames Let be the gravity vector in the IMU coordinate system of the first frame to be solved. express k Frame image and k+ The time interval between one frame of an image, i.e. , They represent the (th) k +1) The speed of solving the frame to be optimized express k The time corresponding to the frame image express k+1 The time corresponding to the frame image; The overdetermined system of equations mentioned in step 5 is specifically defined as follows: in, n It is the size of the sliding window; Step 5 involves initializing the system state variables, which are specifically defined as follows: in, The system state variable set obtained after initialization Indicates the first [number] inside the sliding window k Frame vehicle state variables, including the first frame within the sliding window in the world coordinate system. k Frame position The velocity of the k-th frame within the sliding window in the world coordinate system. In the world coordinate system, within the sliding window, the first k Frame direction In the world coordinate system, within the sliding window, the first k Frame IMU acceleration initial bias In the world coordinate system, within the sliding window, the first... k Frame angular velocity initial offset These two biases are prior known quantities. n It is the size of the sliding window. Indicates the first [number] inside the sliding window l The inverse depth of the first observed feature point It is the initial extrinsic parameter between the camera and the IMU, including relative position. and direction , is a priori known quantity.

6. The visual inertial wheel speed odometer positioning method according to claim 5, characterized in that: The IMU-wheel speed meter pre-integration residual term mentioned in step 6 is specifically defined as follows: in, Indicates the first [number] inside the sliding window k Frame and the k+ The IMU-wheel velocity meter pre-integrated joint residual term between 1 frames, Indicates the first k Frame and the k+ The set of IMU-wheel velocity meter pre-integration information between 1 frame The system state variable set obtained after initialization Indicates the first k Frame and the k+ IMU displacement pre-integration residual between 1 frame Indicates the first k Frame and the k+ IMU velocity pre-integration residual between 1 frame Indicates the first k Frame and the k+ IMU attitude pre-integration residuals between 1 frames Indicates the first k Frame and the k+ Wheel velocity gauge displacement pre-integration residual between 1 frame This represents the IMU acceleration bias residual. This represents the IMU angular velocity offset residual. It is the first k The attitude rotation matrix of the IMU frame corresponding to the frame image in the world coordinate system. It is the first k The relative displacement of the IMU frame corresponding to the frame image in the world coordinate system. It is the first k The attitude rotation matrix in the world coordinate system for the IMU frame corresponding to the +1 frame image. It is the first k The relative displacement of the IMU frame in the world coordinate system corresponding to the +1 frame image. It is the gravity vector in the world coordinate system. This represents the square of the time interval between image frames. Indicates the time interval between image frames. Indicates the first k Frame to the k+ 1-frame IMU shift pre-integration, It is the first k The velocity of the IMU frame corresponding to the frame image in the world coordinate system. yes k Frame to the k+ IMU velocity pre-integration in 1 frame It is the first k The velocity of the IMU frame corresponding to the +1 frame image in the world coordinate system. It is the attitude quaternion of the IMU frame corresponding to the k-th frame in the world coordinate system. It is the first k The inverse matrix of the attitude quaternion of the IMU frame corresponding to the frame image in the world coordinate system. It is the first k The pose quaternion of the IMU frame corresponding to the +1 frame image in the world coordinate system. It is the first k Frame to the k+ 1-frame IMU attitude pre-integration This represents the displacement extrinsic parameter between the IMU and the wheel speed gauge. No. k Frame to the k+ Wheel velocity meter displacement pre-integration in 1 frame , They represent the first k +1 frame and k-th frame IMU acceleration initial value bias , These represent the initial offset of the IMU angular velocity in the (k+1)th frame and the kth frame, respectively.

7. The visual inertial wheel speed odometer positioning method according to claim 6, characterized in that: The joint nonlinear optimization objective function described in step 6 is specifically defined as follows: In the objective function Indicates the first k Frame and the k+ The IMU-wheel velocity meter pre-integrated joint residual term between 1 frames, Represents the set of system state variables. n It is the size of the sliding window. Let be the IMU-wheel speed measurement pre-integral covariance matrix from frame k to frame (k+1). For visual reprojection residuals, This refers to the set of all feature points observed more than twice in the sliding window and their corresponding image frames. For the first sliding window The first feature point is located at the... m Image frames, The standard covariance matrix, Indicates the first The first feature point is located at the... m Frame reprojection, For robust kernel functions, This represents the marginalization of prior residuals. It is marginalized prior information. Represents the marginalized information matrix. Indicates the first [number] inside the sliding window k The frame IMU angular velocity z-axis offset is used to ensure that the online-corrected angular velocity z-offset is not used to update the pre-integration information.

8. A computer-readable medium, characterized in that, It stores a computer program executed by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of the method as described in any one of claims 1-7.

Images