Inertial navigation method, device and equipment based on time series state learning model

Abstract

The present application relates to an inertial navigation method, device and equipment based on a time series state learning model. The method includes: constructing and training a time series state learning model; inputting the inertial time series data obtained by IMU measurement into the Kalman filter for prediction and into the IMU buffer area in the model for caching, and gravity aligning the buffered data according to the filter prediction results; performing feature encoding, time series modeling and feature decoding on the gravity aligned inertial time series data in turn through the feature encoding module, time series modeling module and feature decoding module in the model, and outputting the displacement estimation and uncertainty estimation of the moving carrier; inputting the output result of the time series state learning model into the Kalman filter, and using the Kalman filter output to obtain the navigation state quantity of the moving carrier. The use of this method can fully exploit the time series characteristics of inertial data, suppress the divergence of inertial errors, and improve the accuracy of inertial navigation positioning.

Description

Technical Field

[0001] The present application relates to the field of inertial navigation technology, and in particular to an inertial navigation method, device and equipment based on a temporal state learning model. Background Art

[0002] Pedestrian navigation plays an important supporting role in application scenarios such as emergency rescue, supermarket shopping and augmented reality. With the continuous development of micro-inertial sensor technology, micro-inertial measurement units (MIMUs) have made great progress in miniaturization, low power consumption and low cost. However, due to the influence of the inherent noise of the device, the strapdown navigation algorithm based on micro-inertial devices will diverge rapidly in a short time due to the quadratic integration of noise, which also limits its further application in the field of navigation.

[0003] More and more studies are focusing on using the motion state of the carrier to provide observation information to suppress the divergence of navigation errors. For example, there are two types of methods based on zero-velocity update (ZUPT) and pedestrian dead reckoning (PDR). Among them, the method based on zero-velocity detection uses the zero-velocity information of the foot touching the ground during the pedestrian's walking process to provide speed observation to improve the pedestrian navigation accuracy. This method requires that the inertial device must be installed on the pedestrian's foot, which is not suitable for navigation and positioning tasks in daily use scenarios such as mobile phone navigation. The pedestrian dead reckoning algorithm uses the inertial information in the pedestrian's motion and the pedestrian step length estimation model for navigation and positioning. This method relies on algorithm parameter adjustment and pedestrian motion state, and it is difficult to meet the actual navigation task requirements. In addition, the sensor noise of micro-inertial devices still has problems such as difficulty in accurate modeling and poor power-on repeatability. Therefore, using a data-driven approach to improve the micro-inertial navigation algorithm has become an important research direction at present.

[0004] In recent years, deep learning has shown potential worth exploring in time series signal processing, and has performed well in speech processing, sequence learning, and other aspects. Micro-inertial sensor data is a typical type of noisy time series data, so many scholars have also conducted relevant research on using deep learning for inertial data processing. The Inertial Odometry Network (IONet) algorithm first proposed to use the Long Short-Term Memory (LSTM) network to estimate the relative displacement in the two-dimensional polar coordinate plane to estimate the position. Later, algorithms such as the Robust IMU (Inertial Measurement Unit) Double Integration (RIDI), the Robust Neural Inertial Navigation (RoNIN), and the Tight Learned Inertial Odometry (TLIO) proposed to use deep neural network methods to estimate displacement, speed, or attitude to optimize navigation results. Compared with traditional pedestrian dead reckoning or zero-speed detection methods, this type of inertial navigation method based on deep learning has gradually become a research hotspot due to its advantages in eliminating cumulative errors and robustness. However, these methods use the neural network architecture of Convolutional Neural Network (CNN) or Recurrent Neural Network (RNN) to model inertial data, without fully considering the time series characteristics of inertial data and lacking the ability to fit time series data. Among them, the CNN-based model is limited by its local receptive field and has difficulty in capturing the complete contextual information of inertial data, thus affecting the overall accuracy of the inertial navigation algorithm. The RNN network structure has limited memory capacity for historical information, and it is difficult to establish effective connections for longer contextual information, which limits further improvement of positioning accuracy. Summary of the invention

[0005] Based on this, it is necessary to provide an inertial navigation method, device and equipment based on a timing state learning model that can fully consider the timing characteristics of inertial data, suppress the divergence of the inertial navigation algorithm, and improve the inertial navigation positioning accuracy in response to the above technical problems.

[0006] An inertial navigation method based on a time series state learning model, the method comprising:

[0007] Constructing and training a time series state learning model; wherein the time series state learning model includes an IMU buffer area, a feature encoding module, a time series modeling module and a feature decoding module connected in sequence;

[0008] Obtain the inertial time series data of the moving carrier measured by the IMU, input the inertial time series data into the Kalman filter for prediction and into the IMU buffer area for caching, and perform gravity alignment on the cached inertial time series data according to the rotation matrix predicted and output by the Kalman filter to obtain the gravity-aligned inertial time series data;

[0009] The feature encoding module is used to feature encode the gravity-aligned inertial time series data, and the inertial feature vector after feature encoding is output; the inertial feature vector after feature encoding is time-series modeled by a number of bidirectional state space model units connected in sequence in the time series modeling module, and the inertial feature vector after time series modeling is output; the inertial feature vector after time series modeling is feature decoded by the feature decoding module, and the displacement estimation and uncertainty estimation of the motion carrier are output;

[0010] The displacement estimation and uncertainty estimation output by the time series state learning model are input into the Kalman filter. The Kalman filter is used to take the displacement estimation and uncertainty estimation as observation quantities, and the measurement and update are performed by fusing them with the initially acquired inertial time series data to output the navigation state quantity of the motion carrier.

[0011] In one of the embodiments, during the training process of the time series state learning model, two types of loss functions, namely, root mean square error loss and Gaussian maximum likelihood loss, are successively used for training;

[0012] Among them, the root mean square error loss Expressed as

[0013] ;

[0014] in, represents the 3D displacement estimation output value of the time series state learning model, yes The corresponding true value label, Indicates displacement estimation output value and , yes The corresponding true value label, represents the number of displacement estimates, Indicates the length of inertial time series data;

[0015] Gaussian Maximum Likelihood Loss Defined as the negative log-likelihood of the displacement, expressed as

[0016] ;

[0017] in, Indicates Uncertainty Estimates The corresponding 3×3 covariance matrix is, for A collection of Express request The determinant of Represents a constant.

[0018] In one embodiment, feature encoding is performed on the gravity-aligned inertial time series data by a feature encoding module, and the inertial feature vector obtained by feature encoding is output, including:

[0019] The feature encoding module uses a 1D version of the EfficientNet-B0 network structure to encode the gravity-aligned inertial time series data. Perform feature encoding and output the inertial feature vector after feature encoding , expressed as

[0020] ;

[0021] in, , is the length of the inertial time series data, for The length of the feature vector of for The feature vector dimension of Represents the set of real numbers.

[0022] In one embodiment, a plurality of bidirectional state space model units connected in sequence in a time series modeling module are used to perform time series modeling on the inertia feature vector after feature encoding, and the inertia feature vector after time series modeling is output, including:

[0023] Through several bidirectional state space model units connected in sequence in the time series modeling module, the inertial feature vector after feature encoding is Perform time series modeling and output the inertial characteristic vector after time series modeling , expressed as

[0024] ;

[0025] in, Represents a bidirectional state-space model unit.

[0026] In one embodiment, the bidirectional state-space model unit includes a normalization layer, a first linear layer, a second linear layer, a forward convolution layer, a forward state-space model, a reverse convolution layer, a reverse state-space model, and a third linear layer;

[0027] The inertial feature vector after the feature encoding of the bidirectional state space model unit The specific process of timing modeling includes:

[0028] Get the inertial feature vector after feature encoding And input to the normalization layer for normalization;

[0029] After the normalized inertial feature vector is expanded by the first linear layer, the input vector is obtained. ; After the normalized inertial feature vector is expanded and mapped in turn through the second linear layer and activation function, the gate matrix is ​​obtained ;

[0030] For the input vector Perform forward processing and reverse processing respectively. In the process of forward processing and reverse processing, first convert the input vector Input into the forward convolution layer and the reverse convolution layer respectively, perform 1D convolution and Sigmoid linear weighting processing, and obtain the intermediate vector , expressed as

[0031] ;

[0032] in, is a linearly weighted activation function, Represents 1D convolution;

[0033] Then the middle vector Input the forward state space model and the reverse state space model respectively, and output the forward output and reverse output ;

[0034] Finally, the forward output and reverse output They are multiplied with the gate matrix respectively, and the sum of the two products is input into the third linear layer for dimensionality reduction, and the dimensionality reduction result is combined with the inertial feature vector after the feature encoding of the initial input Add together to get the output of the bidirectional state space model unit.

[0035] In one embodiment, the reverse state space model is obtained by flipping the forward state space model, and the network structures of the two are consistent, both including three expression forms: continuous time form, cyclic representation form and convolutional representation form.

[0036] In one embodiment, the feature decoding module performs feature decoding on the inertial feature vector after time series modeling, and outputs the displacement estimation and uncertainty estimation of the motion carrier, including:

[0037] The feature decoding module uses a fully connected layer to decode the inertial feature vector after time series modeling. conduct MLP Feature decoding, output is the displacement estimation of the motion carrier and uncertainty estimates , expressed as

[0038] .

[0039] In one embodiment, the navigation state of the moving carrier is expressed as

[0040] ;

[0041] in, is the past motion state and the past motion state only contains the posture state quantity for optimization. Indicates The rotation matrix from the inertial coordinate system to the world coordinate system at a moment, Indicates The position at a moment, , Indicates the number of variables to be optimized; is the current motion state, Represents the rotation matrix from the inertial coordinate system to the world coordinate system, Represents rotation operations in three dimensions; Respectively represent the speed and position of the moving carrier in the world coordinate system, They represent the zero bias of the micro inertial sensor gyro and accelerometer respectively.

[0042] An inertial navigation device based on a time series state learning model, the device comprising:

[0043] A model building module is used to build and train a time series state learning model; wherein the time series state learning model includes an IMU buffer area, a feature encoding module, a time series modeling module and a feature decoding module connected in sequence;

[0044] A preprocessing module is used to obtain the inertial time series data of the moving carrier measured by the IMU, input the inertial time series data into the Kalman filter for prediction and into the IMU buffer area for caching, and gravity align the cached inertial time series data according to the rotation matrix predicted and output by the Kalman filter to obtain the gravity-aligned inertial time series data;

[0045] The model prediction module is used to perform feature encoding on the gravity-aligned inertial time series data through the feature encoding module, and output the inertial feature vector after feature encoding; perform time series modeling on the inertial feature vector after feature encoding through a number of bidirectional state space model units connected in sequence in the time series modeling module, and output the inertial feature vector after time series modeling; perform feature decoding on the inertial feature vector after time series modeling through the feature decoding module, and output the displacement estimation and uncertainty estimation of the motion carrier;

[0046] The filter update module is used to input the displacement estimation and uncertainty estimation output by the time series state learning model into the Kalman filter, use the Kalman filter to take the displacement estimation and uncertainty estimation as observation quantities, and perform measurement update by fusing them with the initially acquired inertial time series data, and output the navigation state quantity of the motion carrier.

[0047] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0048] Constructing and training a time series state learning model; wherein the time series state learning model includes an IMU buffer area, a feature encoding module, a time series modeling module and a feature decoding module connected in sequence;

[0049] Obtain the inertial time series data of the moving carrier measured by the IMU, input the inertial time series data into the Kalman filter for prediction and into the IMU buffer area for caching, and perform gravity alignment on the cached inertial time series data according to the rotation matrix predicted and output by the Kalman filter to obtain the gravity-aligned inertial time series data;

[0050] The feature encoding module is used to feature encode the gravity-aligned inertial time series data, and the inertial feature vector after feature encoding is output; the inertial feature vector after feature encoding is time-series modeled by a number of bidirectional state space model units connected in sequence in the time series modeling module, and the inertial feature vector after time series modeling is output; the inertial feature vector after time series modeling is feature decoded by the feature decoding module, and the displacement estimation and uncertainty estimation of the motion carrier are output;

[0051] The displacement estimation and uncertainty estimation output by the time series state learning model are input into the Kalman filter. The Kalman filter is used to take the displacement estimation and uncertainty estimation as observation quantities, and the measurement and update are performed by fusing them with the initially acquired inertial time series data to output the navigation state quantity of the motion carrier.

[0052] The above-mentioned inertial navigation method, device and equipment based on the time series state learning model, by constructing and training the time series state learning model, sequentially performs caching, feature encoding, time series modeling and feature decoding of inertial time series data, and the model can be used to fully explore the time series characteristics of non-causal inertial data, thereby improving the positioning accuracy of inertial navigation. In addition, after obtaining the displacement estimate and uncertainty estimate output by the time series state learning model, the model output result is further input into the Kalman filter, and the inertial time series data and the model output result are fused through the Kalman filter to suppress the inertial error divergence of the time series state learning model, reduce error drift, further improve the accuracy of inertial navigation positioning, and achieve accurate inertial navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A flowchart of an inertial navigation method based on a time series state learning model in one embodiment

[0054] Figure 2 A schematic diagram of a framework of an inertial navigation method based on a temporal state learning model in one embodiment;

[0055] Figure 3 A schematic diagram of the architecture of a bidirectional state space model unit in one embodiment;

[0056] Figure 4 Schematic diagram of three expression forms of state space model representation in one embodiment;

[0057] Figure 5 It is a schematic diagram of positioning result comparison in an embodiment; wherein, Figure 5 (a) is a schematic diagram of the positioning results of the first group of experiments; Figure 5 (b) is a schematic diagram of the positioning results of the second group of experiments; Figure 5 (c) is a schematic diagram of the positioning results of the third group of experiments; Figure 5 (d) is a schematic diagram of the positioning results of the fourth group of experiments; Figure 5 (e) is a schematic diagram of the positioning results of the fifth group of experiments; Figure 5 (f) is a schematic diagram of the positioning results of the sixth group of experiments;

[0058] Figure 6 Schematic diagram of the cumulative distribution function of positioning error in one embodiment; wherein, Figure 6 (a) is a schematic diagram of the absolute trajectory error cumulative distribution function; Figure 6 (b) is a schematic diagram of the cumulative distribution function of the drift rate;

[0059] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] In one embodiment, Figure 1 and Figure 2 As shown, an inertial navigation method based on a temporal state learning model is provided, comprising the following steps:

[0062] Step S1, constructing and training a time series state learning model; wherein the time series state learning model includes an IMU (inertial measurement unit) buffer, a feature encoding module, a time series modeling module and a feature decoding module connected in sequence.

[0063] Among them, the IMU buffer area is also called the inertial data buffer area.

[0064] Step S2, obtaining the inertial time series data of the motion carrier measured by the IMU, inputting the inertial time series data into the Kalman filter for prediction and into the IMU buffer area for caching, and gravity aligning the cached inertial time series data according to the rotation matrix predicted and output by the Kalman filter to obtain the gravity-aligned inertial time series data.

[0065] Step S3, feature encoding is performed on the gravity aligned inertial time series data through the feature encoding module, and the inertial feature vector after feature encoding is output; time series modeling is performed on the inertial feature vector after feature encoding through a plurality of bidirectional state space model units connected in sequence in the time series modeling module, and the inertial feature vector after time series modeling is output; feature decoding is performed on the inertial feature vector after time series modeling through the feature decoding module, and the displacement estimation and uncertainty estimation of the motion carrier are output.

[0066] In step S4, the displacement estimate and uncertainty estimate output by the time series state learning model are input into the Kalman filter, and the displacement estimate and uncertainty estimate are used as observation quantities by the Kalman filter. The displacement estimate and uncertainty estimate are updated by fusing them with the initially acquired inertial time series data, and the navigation state quantity of the motion carrier is output.

[0067] Furthermore, if Figure 2 As shown, the extended Kalman filter can also be used to preprocess the inertial time series data and use the displacement estimate and uncertainty estimate output by the time series state learning model as the observed quantity, and then perform measurement update by fusing with the initially acquired inertial time series data to output the navigation state quantity of the motion carrier.

[0068] In one of the embodiments, during the training process of the time series state learning model, two types of loss functions, namely, root mean square error loss and Gaussian maximum likelihood loss, are successively used for training;

[0069] Among them, the root mean square error loss Expressed as

[0070] ;

[0071] in, represents the 3D displacement estimation output value of the time series state learning model, yes The corresponding true value label, Indicates displacement estimation output value and , yes The corresponding true value label, represents the number of displacement estimates, Indicates the length of inertial time series data;

[0072] Gaussian Maximum Likelihood Loss Defined as the negative log-likelihood of the displacement, expressed as

[0073] ;

[0074] in, Indicates Uncertainty Estimates The corresponding 3×3 covariance matrix is, for A collection of Express request The determinant of represents a constant. Furthermore, There are 6 degrees of freedom, and there are various covariance parameters for neural network uncertainty estimation. In order to increase numerical stability, the covariance matrix can be simplified to a diagonal covariance parameterized by 3 coefficients, thereby eliminating singular values ​​near the zero point of the loss function, recorded as:

[0075] ;

[0076] in, represents the first parameter of the parameterized covariance matrix, represents the second parameter of the parameterized covariance matrix, represents the third parameter of the parameterized covariance matrix. Specifically, the Adam optimizer is used for optimization during training, and the learning rate is 1e-4. In the first ten rounds of training, the loss function is set to , and then set it to Until the training is finished.

[0077] In one embodiment, feature encoding is performed on the gravity-aligned inertial time series data by a feature encoding module, and the inertial feature vector obtained by feature encoding is output, including:

[0078] The feature encoding module uses a 1D version of the EfficientNet-B0 network structure to encode the gravity-aligned inertial time series data. Perform feature encoding and output the inertial feature vector after feature encoding , expressed as

[0079] ;

[0080] in, , is the length of the inertial time series data, for The length of the feature vector of for The feature vector dimension of Represents the set of real numbers.

[0081] In one embodiment, a plurality of bidirectional state space model units connected in sequence in a time series modeling module are used to perform time series modeling on the inertia feature vector after feature encoding, and the inertia feature vector after time series modeling is output, including:

[0082] Through several bidirectional state space model units connected in sequence in the time series modeling module, the inertial feature vector after feature encoding is Perform time series modeling and output the inertial characteristic vector after time series modeling , expressed as

[0083] ;

[0084] in, Represents a bidirectional state-space model unit.

[0085] Furthermore, since the forward propagation process of the learnable state space model can only capture historical information, it is more suitable for sequence learning with causal relationships. Inertial time series data is non-causal time series data, so this application uses a bidirectional state space model unit structure to model it. Figure 3As shown in FIG. 1 , the bidirectional state space model unit includes a normalization layer, a first linear layer, a second linear layer, a forward convolution layer, a forward state space model, a reverse convolution layer, a reverse state space model, and a third linear layer. The bidirectional state space model unit can be stacked according to task requirements to improve the fitting ability of the inertial navigation algorithm. Among them, the bidirectional state space model unit encodes the inertial feature vector after feature encoding. The specific process of timing modeling includes:

[0086] Get the inertial feature vector after feature encoding And input to the normalization layer for normalization;

[0087] After the normalized inertial feature vector is expanded by the first linear layer, the input vector is obtained. ; After the normalized inertial feature vector is expanded and mapped in turn through the second linear layer and activation function, the gate matrix is ​​obtained ;Specifically, dimensional inertial eigenvector After dimension expansion, it is expanded to E dimension, that is, ;

[0088] For the input vector Perform forward processing and reverse processing respectively. In the process of forward processing and reverse processing, first convert the input vector Input into the forward convolution layer and the reverse convolution layer respectively, perform 1D convolution and Sigmoid linear weighting processing to obtain the intermediate vector , expressed as

[0089] ;

[0090] in, is a linearly weighted activation function, expressed as ; Represents 1D convolution;

[0091] Then the middle vector Input the forward state space model and the reverse state space model respectively, and output the forward output and reverse output Specifically, the intermediate vectors are transformed by linear layers MLP (multi-layer perceptron) in the forward state space model and the reverse state space model. Perform mapping processing to obtain the input matrix , the input matrix and The matrices are represented as

[0092] ;

[0093] in, yes Related optimizable parameters, represents the discretization step size parameter; then, after discretization and inference through the state space model in the form of a loop representation, the forward output of the forward state space model and the reverse state space model output are obtained and reverse output .

[0094] Finally, the forward output and reverse output They are multiplied with the gate matrix respectively, and the sum of the two products is input into the third linear layer for dimensionality reduction, and the dimensionality reduction result is combined with the inertial feature vector after the feature encoding of the initial input Add together to get the output of the bidirectional state space model unit.

[0095] Furthermore, the reverse state space model is obtained by flipping the forward state space model. The network structures of the two are consistent, including three forms of expression: continuous time form, cyclic representation form and convolutional representation form, such as Figure 4 Specifically including:

[0096] The state-space model in continuous time form encodes the inertial feature vector after the features extracted from the neural network are extracted. Mapped to N-dimensional variables in the implicit state space , which is then projected to the output variable , the calculation process can be defined by the following formula:

[0097] ;

[0098] in, Represent the state vector, output vector and input vector respectively, where Respectively The dimension of . , , , They represent the state matrix, input matrix, output matrix, and feedforward matrix respectively. When there is no direct feedforward connection in the system, The matrix is ​​a zero matrix. Therefore, the state equation in the above formula can be simplified to:

[0099] ;

[0100] After zero-order hold discretization, the above equation can be expressed as a discrete system, expressed as

[0101] ;

[0102] in,

[0103] ;

[0104] Indicates the length of the walk. and Denote the discretized state matrix and input matrix respectively. After discretization, the state space model can be inferred in the form of a linear recursive loop representation.

[0105] However, the learnable state space model in this cyclic representation cannot be parallelized and cannot fully utilize the computing platform resources during training. Further expansion can obtain a state space model in the form of convolution, which can be expressed as:

[0106] .

[0107] Therefore, the multiplier can be viewed as a convolution kernel , and parallel training is performed in the form of convolution.

[0108] In one embodiment, the feature decoding module performs feature decoding on the inertial feature vector after time series modeling, and outputs the displacement estimation and uncertainty estimation of the motion carrier, including:

[0109] The feature decoding module uses a fully connected layer to decode the inertial feature vector after time series modeling. conduct MLP Feature decoding, output is the displacement estimation of the motion carrier and uncertainty estimates , expressed as

[0110] .

[0111] In one embodiment, in order to further reduce the drift error, the present application uses a random clone multi-state constrained Kalman filter to fuse the output of the time series state learning model with the initially acquired inertial time series data. Since the filter is an observation of the displacement observation of the moving carrier associated with the past state, the state cloning method is used to expand the state dimension, thereby increasing the system convergence. The filter maintains a window of k pose vectors to construct relative observations. The navigation state quantity of the moving carrier is expressed as

[0112] ;

[0113] in, is the past motion state and the past motion state only contains the posture state quantity for optimization. Indicates The rotation matrix from the inertial coordinate system to the world coordinate system at a moment, Indicates The position at a moment, , Indicates the number of variables to be optimized; is the current motion state, Represents the rotation matrix from the inertial coordinate system to the world coordinate system, Represents rotation operations in three dimensions; Respectively represent the speed and position of the moving carrier in the world coordinate system, They represent the zero bias of the micro inertial sensor gyro and accelerometer respectively.

[0114] Specifically, the inertial time series data obtained by the initial IMU measurement includes accelerometer measurements and gyroscope measurements , its measurement model can be expressed as:

[0115] ;

[0116] in, and are the measurements of the accelerometer and gyroscope, and are the true values ​​of the accelerometer and gyroscope, and is the Gaussian white noise of the accelerometer and gyroscope, Represents the gyro bias. Due to the limited accuracy of micro-inertial sensors and the limited range of pedestrian navigation, the effects of the earth's rotation and gravity changes can often be ignored. Therefore, the system motion equation can be expressed as:

[0117] ;

[0118] in, Represents an exponential mapping from Lie algebras to Lie groups. and is the Gaussian white noise in the IMU model and Discrete representation of . Then the error state linear recursive equation can be expressed as:

[0119] ;

[0120] in, represents the system noise, Indicates The system status at a moment, Indicates The state recursion matrix at each moment is: Indicates The system status at a moment, Indicates In order to realize the displacement measurement prediction, it is necessary to expand the system state by replication, and replicate the state quantity associated with the displacement observation into the equation, that is:

[0121] ;

[0122] in, and is the historical association state posture correlation coefficient matrix, represents the recursive matrix of the system state after dimension expansion, represents the 6m-dimensional unit matrix, represents the system noise matrix after dimensional expansion. Then the covariance recursion process can be recorded as:

[0123] ;

[0124] in, express The moment covariance matrix, represents the sensor noise covariance matrix.

[0125] Furthermore, in order to avoid the unobservable influence of the heading of the inertial navigation system, the observed quantity is the three-dimensional displacement in the gravity-aligned local coordinate system. states are associated, namely:

[0126] ;

[0127] in, represents the observed quantity, is the state output by the temporal state learning model and The three-dimensional displacement between yes The heading angle rotation matrix at time, is the measured zero-mean Gaussian white noise, whose covariance is estimated by the uncertainty of the output of the time series state learning model Given, express The observation matrix after linearization of the system is recorded as , then the update process can be recorded as:

[0128] ;

[0129] in, represents the Kalman filter gain, represents the system state covariance matrix, Represents the navigation state of the motion carrier, It means that the addition on the SO(3) group is used when updating the attitude variables, and the ordinary addition is used for the other state variables. After the measurement update, the historical associated state needs to be marginalized.

[0130] Furthermore, the present application uses a public pedestrian navigation inertial dataset to verify an inertial navigation method based on a temporal state learning model provided in the present application. The dataset uses Bosch BMI055 IMU to collect inertial data at 200Hz. The complete dataset contains more than 400 sequences of pedestrian motion data with a total duration of more than 60 hours, including actions of 5 pedestrians walking, standing still, tidying up the kitchen, playing billiards, and going up and down stairs. The data acquisition device uses a human wearable device and uses the MSCKF visual inertial odometer to provide the true position value. 80% of the data in the dataset is randomly selected for training, 10% of the data is used for verification, and 10% of the data is used for testing. In order to be suitable for the input of the temporal state learning model, the data is divided into overlapping fixed 1s time window data, that is, the input dimension of the temporal state learning model is In order to enable the time series state learning model to effectively learn the pedestrian motion model under the gravity-aligned load system, the IMU measurement data is converted to the gravity-aligned coordinate system of the first frame of the window using the attitude truth in the dataset preprocessing stage. The true value label is obtained by calculating the difference between the positions at two moments.

[0131] Six sets of pedestrian navigation experiments were conducted, each lasting about 3-7 minutes. The experimental results were compared with the two most advanced deep learning-based pedestrian navigation methods, RoNIN (Robust Neural Inertial Navigation in the Wild) and TLIO (Tightly Learned Inertial Positioning). Figure 5 and Figure 6 The RoNIN method is Figure 5 and Figure 6 Comparison in 2. The training and testing of the network are based on the Nvidia RTX3090 computing platform. The displacement estimation part in the RoNIN algorithm is obtained by Resnet (deep residual network) as in the TLIO algorithm, and the attitude is provided by the mobile phone's built-in heading reference system (Automatic Heading Reference System, AHRS). The test evaluation criteria are mainly to evaluate the positioning results, and the evaluation results include the absolute trajectory error (ATE) and the drift rate (DR). They are respectively expressed as

[0132] ;

[0133] ;

[0134] in, is the position estimate of the IMU algorithm, is the position truth label value, is the total length of the trajectory.

[0135] The localization results of some sequences in the six pedestrian navigation experiments in the test dataset are as follows: Figure 5 The results show that the inertial navigation system based on deep learning method can achieve navigation and positioning in different motion scenarios. Figure 6 It can be seen from the cumulative distribution function of positioning error that by comparing the inertial navigation method based on the time series state learning model proposed in this application with the TLIO and RoNIN algorithms in terms of ATE and drift rate, the result curve of the method proposed in this application is steeper and the positioning accuracy is higher. In addition, the filter part of the method proposed in this application is replaced by only using the attitude truth value and the displacement estimate output by the time series state learning model for dead reckoning. Figure 6 It can be seen from the solid line and the dotted line in that the Kalman filter in the method proposed in this application effectively improves the positioning accuracy.

[0136] The results shown in Table 1 can be obtained by taking the average of the absolute trajectory error and displacement drift rate of the experiment. Through quantitative analysis of the values, it can be seen that the average absolute trajectory error of the method proposed in this application is 0.92m, and the displacement drift rate is 0.74%, which are 32.35% and 41.27% higher than TLIO, respectively.

[0137] Table 1 Summary of positioning errors of test datasets

[0138]

[0139] In summary, through experimental verification, the positioning accuracy of the odometer model after time series modeling by the method proposed in this application is further improved. Compared with TLIO, the absolute trajectory error and displacement drift rate are increased by 32.35% and 41.27% respectively. This effectively verifies that this method can improve the displacement estimation accuracy of the deep learning network for inertial data, effectively suppress the divergence of inertial errors, and achieve accurate pedestrian navigation.

[0140] In one embodiment, an inertial navigation device based on a temporal state learning model is provided, comprising:

[0141] A model building module is used to build and train a time series state learning model; wherein the time series state learning model includes an IMU buffer area, a feature encoding module, a time series modeling module and a feature decoding module connected in sequence;

[0142] A preprocessing module is used to obtain the inertial time series data of the moving carrier measured by the IMU, input the inertial time series data into the Kalman filter for prediction and into the IMU buffer area for caching, and gravity align the cached inertial time series data according to the rotation matrix predicted and output by the Kalman filter to obtain the gravity-aligned inertial time series data;

[0143] The model prediction module is used to perform feature encoding on the gravity-aligned inertial time series data through the feature encoding module, and output the inertial feature vector after feature encoding; perform time series modeling on the inertial feature vector after feature encoding through a number of bidirectional state space model units connected in sequence in the time series modeling module, and output the inertial feature vector after time series modeling; perform feature decoding on the inertial feature vector after time series modeling through the feature decoding module, and output the displacement estimation and uncertainty estimation of the motion carrier;

[0144] The filter update module is used to input the displacement estimation and uncertainty estimation output by the time series state learning model into the Kalman filter, use the Kalman filter to take the displacement estimation and uncertainty estimation as observation quantities, and perform measurement update by fusing them with the initially acquired inertial time series data, and output the navigation state quantity of the motion carrier.

[0145] For the specific limitations of the inertial navigation device based on the temporal state learning model, please refer to the limitations of the inertial navigation method based on the temporal state learning model mentioned above, which will not be repeated here. The various modules in the above-mentioned inertial navigation device based on the temporal state learning model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0146] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an inertial navigation method based on a time series state learning model is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0147] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0148] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0149] Constructing and training a time series state learning model; wherein the time series state learning model includes an IMU buffer area, a feature encoding module, a time series modeling module and a feature decoding module connected in sequence;

[0150] Obtain the inertial time series data of the moving carrier measured by the IMU, input the inertial time series data into the Kalman filter for prediction and into the IMU buffer area for caching, and perform gravity alignment on the cached inertial time series data according to the rotation matrix predicted and output by the Kalman filter to obtain the gravity-aligned inertial time series data;

[0151] The feature encoding module is used to feature encode the gravity-aligned inertial time series data, and the inertial feature vector after feature encoding is output; the inertial feature vector after feature encoding is time-series modeled by a number of bidirectional state space model units connected in sequence in the time series modeling module, and the inertial feature vector after time series modeling is output; the inertial feature vector after time series modeling is feature decoded by the feature decoding module, and the displacement estimation and uncertainty estimation of the motion carrier are output;

[0152] The displacement estimation and uncertainty estimation output by the time series state learning model are input into the Kalman filter. The Kalman filter is used to take the displacement estimation and uncertainty estimation as observation quantities, and the measurement and update are performed by fusing them with the initially acquired inertial time series data to output the navigation state quantity of the motion carrier.

[0153] The technical features of the embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An inertial navigation method based on a temporal state learning model, characterized in that: The method comprises: Constructing and training a time series state learning model; wherein the time series state learning model includes an IMU buffer area, a feature encoding module, a time series modeling module and a feature decoding module connected in sequence; Acquire the inertial time series data of the moving carrier measured by the IMU, input the inertial time series data into a Kalman filter for prediction and into an IMU buffer area for caching, and gravity align the cached inertial time series data according to the rotation matrix predicted and output by the Kalman filter to obtain the gravity-aligned inertial time series data; The feature encoding module performs feature encoding on the gravity-aligned inertial time series data, and outputs an inertial feature vector after feature encoding; a plurality of sequentially connected bidirectional state space model units in the time series modeling module perform time series modeling on the inertial feature vector after feature encoding, and outputs an inertial feature vector after time series modeling; the feature decoding module performs feature decoding on the inertial feature vector after time series modeling, and outputs a displacement estimate and an uncertainty estimate of the motion carrier; Inputting the displacement estimate and uncertainty estimate output by the time series state learning model into the Kalman filter, using the Kalman filter to take the displacement estimate and uncertainty estimate as observation quantities, and fusing them with the initially acquired inertial time series data for measurement update, and outputting the navigation state quantity of the moving carrier; Among them, performing time series modeling on the inertial feature vector after the feature encoding through a plurality of bidirectional state space model units connected in sequence in the time series modeling module, and outputting the inertial feature vector after time series modeling, includes: The inertial feature vector after feature encoding is obtained by using a plurality of bidirectional state space model units connected in sequence in the time series modeling module. Perform time series modeling and output the inertial characteristic vector after time series modeling , expressed as ； in, SSM represents a bidirectional state-space model unit; The bidirectional state space model unit includes a normalization layer, a first linear layer, a second linear layer, a forward convolution layer, a forward state space model, a reverse convolution layer, a reverse state space model and a third linear layer; The inertial feature vector after the bidirectional state space model unit encodes the feature The specific process of timing modeling includes: Get the inertial feature vector after feature encoding And input to the normalization layer for normalization; After the normalized inertial feature vector is expanded by the first linear layer, the input vector is obtained. ; After the normalized inertial feature vector is expanded and mapped in turn through the second linear layer and activation function, the gate matrix is ​​obtained ; For the input vector Perform forward processing and reverse processing respectively. In the process of forward processing and reverse processing, first convert the input vector The vectors are input into the forward convolution layer and the reverse convolution layer respectively, and 1D convolution and Sigmoid linear weighting are performed to obtain the intermediate vectors. , expressed as ； in, is a linearly weighted activation function, Represents 1D convolution; Then the intermediate vector Input the forward state space model and the reverse state space model respectively, and output the forward output and reverse output ; Finally, the forward output and reverse output They are multiplied with the gate matrix respectively, and the sum of the two products is input into the third linear layer for dimensionality reduction, and the dimensionality reduction result is combined with the inertial feature vector after the feature encoding of the initial input Add together to get the output of the bidirectional state space model unit.

2. The method according to claim 1, characterized in that In the training process of the time series state learning model, two types of loss functions, namely, root mean square error loss and Gaussian maximum likelihood loss, are successively used for training; Among them, the root mean square error loss Expressed as ； in, represents the 3D displacement estimation output value of the temporal state learning model, yes The corresponding true value label, Indicates j displacement estimation output value and , yes The corresponding true value label, n represents the number of displacement estimates, L Indicates the length of inertial time series data; Gaussian Maximum Likelihood Loss Defined as the negative log-likelihood of the displacement, expressed as ； in, Indicates j Uncertainty Estimates The corresponding 3×3 covariance matrix is, for A collection of Express request The determinant of Represents a constant.

3. The method according to claim 1, characterized in that The feature encoding module performs feature encoding on the gravity aligned inertial time series data, and outputs the feature-encoded inertial feature vector, including: The feature encoding module uses a 1-dimensional version of the EfficientNet-B0 network structure to encode the gravity-aligned inertial time series data. Perform feature encoding and output the inertial feature vector after feature encoding , expressed as ； in, , , L is the length of the inertial time series data, M for The length of the feature vector of D for The feature vector dimension of Represents the set of real numbers.

4. The method according to claim 1, characterized in that The reverse state space model is obtained by flipping the forward state space model. The network structures of the two are consistent, and both include three expression forms: continuous time form, cyclic representation form and convolution representation form.

5. The method according to claim 1, characterized in that The feature decoding module performs feature decoding on the inertial feature vector after the time series modeling, and outputs the displacement estimation and uncertainty estimation of the motion carrier, including: The feature decoding module uses a fully connected layer to decode the inertial feature vector after time series modeling. conduct MLP Feature decoding, output is the displacement estimation of the motion carrier and uncertainty estimates , expressed as 。 6. The method according to claim 1, characterized in that The navigation state quantity of the moving carrier is expressed as ； in, is the past motion state, and only the posture state quantity is optimized in the past motion state. Indicates j The rotation matrix from the inertial coordinate system to the world coordinate system at a moment, Indicates j The position at a moment, , Indicates the number of variables to be optimized; is the current motion state, Represents the rotation matrix from the inertial coordinate system to the world coordinate system, Represents rotation operations in three dimensions; Respectively represent the speed and position of the moving carrier in the world coordinate system, They represent the zero bias of the micro inertial sensor gyro and accelerometer respectively.

7. An inertial navigation device based on a temporal state learning model, characterized in that: The device comprises: A model building module, used to build and train a time series state learning model; wherein the time series state learning model includes an IMU buffer area, a feature encoding module, a time series modeling module and a feature decoding module connected in sequence; A preprocessing module is used to obtain the inertial time series data of the moving carrier measured by the IMU, input the inertial time series data into the Kalman filter for prediction and into the IMU buffer area for caching, and gravity align the cached inertial time series data according to the rotation matrix predicted and output by the Kalman filter to obtain the gravity-aligned inertial time series data; A model prediction module is used to perform feature encoding on the gravity-aligned inertial time series data through the feature encoding module, and output an inertial feature vector after feature encoding; perform time series modeling on the inertial feature vector after feature encoding through a plurality of bidirectional state space model units connected in sequence in the time series modeling module, and output an inertial feature vector after time series modeling; perform feature decoding on the inertial feature vector after time series modeling through the feature decoding module, and output a displacement estimate and an uncertainty estimate of the motion carrier; A filter update module, used to input the displacement estimate and uncertainty estimate output by the time series state learning model into the Kalman filter, use the Kalman filter to take the displacement estimate and uncertainty estimate as observation quantities, and perform measurement update by fusing them with the initially acquired inertial time series data, and output the navigation state quantity of the motion carrier; Among them, performing time series modeling on the inertial feature vector after the feature encoding through a plurality of bidirectional state space model units connected in sequence in the time series modeling module, and outputting the inertial feature vector after time series modeling, includes: The inertial feature vector after feature encoding is obtained by using a plurality of bidirectional state space model units connected in sequence in the time series modeling module. Perform time series modeling and output the inertial characteristic vector after time series modeling , expressed as ； in, SSM represents a bidirectional state-space model unit; The bidirectional state space model unit includes a normalization layer, a first linear layer, a second linear layer, a forward convolution layer, a forward state space model, a reverse convolution layer, a reverse state space model and a third linear layer; The inertial feature vector after the bidirectional state space model unit encodes the feature The specific process of timing modeling includes: Get the inertial feature vector after feature encoding And input to the normalization layer for normalization; After the normalized inertial feature vector is expanded by the first linear layer, the input vector is obtained. ; After the normalized inertial feature vector is expanded and mapped in turn through the second linear layer and activation function, the gate matrix is ​​obtained ; For the input vector Perform forward processing and reverse processing respectively. In the process of forward processing and reverse processing, first convert the input vector The vectors are input into the forward convolution layer and the reverse convolution layer respectively, and 1D convolution and Sigmoid linear weighting are performed to obtain the intermediate vectors. , expressed as ； in, is a linearly weighted activation function, Represents 1D convolution; Then the intermediate vector Input the forward state space model and the reverse state space model respectively, and output the forward output and reverse output ; Finally, the forward output and reverse output They are multiplied with the gate matrix respectively, and the sum of the two products is input into the third linear layer for dimensionality reduction, and the dimensionality reduction result is combined with the inertial feature vector after the feature encoding of the initial input Add together to get the output of the bidirectional state space model unit.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images