A speed estimation method and device based on convolutional neural network

Abstract

The present invention discloses a speed estimation method and device based on a convolutional neural network. The method comprises: pre-processing data collected by an inertial measurement unit (IMU) provided on a mobile carrier; inputting the training data set into a speed estimation neural network to be trained to obtain a trained target speed estimation neural network; inputting the data to be estimated detected by the IMU into the target speed estimation neural network to obtain a speed estimation result of the mobile carrier. The embodiment of the present invention provides a method for estimating carrier speed from a single inertial measurement unit data by adopting a convolutional neural network, and constructing an inertial navigation algorithm error constraint using the estimated speed, thereby reducing the error accumulation problem; based on the powerful learning ability of deep learning, the solution has stronger speed estimation robustness under different equipment loads, equipment states, and IMU differences, and can be used as an alternative to hardware devices such as speedometers and wheel speedometers.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of indoor and outdoor navigation and positioning technology, and in particular to a speed estimation method and device based on a convolutional neural network. Background Art

[0002] When existing mobile devices such as mobile phones and tablets use IMU components to achieve velocity estimation in Global Navigation Satellite System (GNSS)-denied environments, current solutions generally rely on traditional motion models for velocity estimation, which are difficult to adapt to changes in device posture. Therefore, it is impossible to use velocity to effectively constrain algorithms such as inertial navigation or pedestrian positioning.

[0003] At present, vehicles, robots and other carriers need to rely on hardware methods such as wheel speed meters for speed estimation to assist inertial navigation algorithms; at the same time, the implementation of traditional solutions depends on the accurate estimation of the external parameters of the IMU and the vehicle coordinate system.

[0004] In addition, traditional speed estimation schemes using motion models require modeling the sensor noise level in advance. This method cannot estimate the differences in equipment and does not consider the impact of real-time factors such as vehicle load and road conditions on speed estimation. Summary of the Invention

[0005] The present invention provides a convolutional neural network-based speed estimation method and device, which can estimate the carrier speed from a single inertial measurement unit data. The estimated speed is used to establish error constraints for the inertial navigation algorithm, thereby reducing the problem of error accumulation. The present invention has stronger speed estimation robustness under different equipment loads, equipment states, and IMU differences, and can serve as an alternative to hardware devices such as speedometers and wheel speedometers.

[0006] In a first aspect, an embodiment of the present invention provides a speed estimation method based on a convolutional neural network, comprising:

[0007] S1. Preprocessing data collected by an inertial measurement unit (IMU) mounted on a mobile carrier to obtain a training data set.

[0008] S2. Inputting the training data set obtained in S1 into the speed estimation neural network to be trained to obtain a trained target speed estimation neural network;

[0009] S3. Input the data to be estimated detected by the IMU into the target speed estimation neural network obtained in S2 to obtain a speed estimation result of the mobile carrier.

[0010] Optionally, after S3, the following steps are further included:

[0011] S4. Smoothing the speed estimation result in S3 based on the acceleration constraint to obtain an optimized speed estimation result.

[0012] Optionally, after S4, the step further includes:

[0013] S5: Based on the velocity estimation result after optimization in S4, an observation equation is constructed to constrain the error accumulation of the inertial navigation system.

[0014] The observation equation is specifically:

[0015]

[0016] Where δz represents the velocity observation residual; Indicates the speed calculated by the inertial system; n v represents the observation noise; δx represents the system residual; The orientation rotation matrix representing the installation angle calculation of the IMU coordinate system relative to the vehicle coordinate system; Represents the direction rotation matrix between the navigation coordinate system and the IMU coordinate system; v n represents the velocity in the carrier coordinate system; × represents the antisymmetric matrix; I represents the unit matrix; The lever arm vector representing the center of the IMU coordinate system and the center of the vehicle coordinate system; Represents the gyroscope observation in the carrier coordinate system.

[0017] Optionally, the S1 includes:

[0018] S11. According to the sampling frequency of the IMU, the acceleration, gyroscope measurement values ​​and velocity reference values ​​are arranged into the following format according to a certain window size:

[0019] (Acce,Gyro,v) n×6

[0020] Where Acce represents acceleration data, Gyro represents gyroscope data, and v represents reference velocity; n represents the window length, and 6 represents the 6-dimensional measurement value composed of 3-axis acceleration and 3-axis gyroscope;

[0021] S12. Add different types of random noise to the acceleration data and gyroscope data in the window obtained in S11.

[0022] Optionally, the S3 includes:

[0023] The speed estimation neural network model to be trained is trained by minimizing the loss function and back propagation to obtain a trained target speed estimation neural network.

[0024] In a second aspect, an embodiment of the present invention further provides a speed estimation device based on a convolutional neural network, comprising:

[0025] A preprocessing module is used to preprocess the data collected by the inertial measurement unit (IMU) installed on the mobile carrier to obtain a training data set;

[0026] A training module, configured to input the training data set into a speed estimation neural network to be trained, so as to obtain a trained target speed estimation neural network;

[0027] The estimation module is used to input the data to be estimated detected by the IMU into the target speed estimation neural network to obtain the speed estimation result of the mobile carrier.

[0028] Optionally, also include:

[0029] The optimization module is used to smooth the velocity of the fixed window based on the acceleration constraint to obtain an optimized velocity estimation result.

[0030] Optionally, also include:

[0031] The velocity constraint module is used to construct the observation equation based on the velocity estimation result of the target velocity estimation neural network to constrain the error accumulation of the inertial navigation system.

[0032] Optionally, the preprocessing module is specifically used to:

[0033] S11. According to the sampling frequency of the IMU, the acceleration, gyroscope measurement values ​​and velocity reference values ​​are arranged into the following format according to a certain window size:

[0034] (Acce,Gyro,v) n×6

[0035] Where Acce represents acceleration data, Gyro represents gyroscope data, and v represents reference velocity; n represents the window length, and 6 represents the 6-dimensional measurement value composed of 3-axis acceleration and 3-axis gyroscope;

[0036] S12. Add different types of random noise to the acceleration data and gyroscope data in the window obtained in S11.

[0037] Optionally, the estimation module is specifically configured to:

[0038] The speed estimation neural network model to be trained is trained by minimizing the loss function and back propagation to obtain a trained target speed estimation neural network.

[0039] The present invention preprocesses data collected by an inertial measurement unit (IMU) mounted on a mobile carrier to obtain a training data set. The training then trains a speed estimation neural network based on the training data set to obtain a trained target speed estimation neural network. The data to be estimated, as detected by the IMU, is then input into the target speed estimation neural network to obtain a speed estimation result for the mobile carrier. The present invention provides a method for estimating carrier speed from data from a single inertial measurement unit (IMU) by employing a convolutional neural network. Based on the powerful learning capabilities of deep learning, this method offers greater robustness in speed estimation under varying device loads, device states, and IMU variability than traditional methods that rely on motion models. This method can serve as an alternative to hardware devices such as speedometers and wheel speedometers.

[0040] Beneficial effects of the present invention:

[0041] 1. When executing algorithms such as Inertial Navigation Systems (INS) or PDR on mobile devices such as tablets and mobile phones, there is a lack of speed constraints. Traditional motion models are difficult to adapt to multimodal speed estimation. This paper proposes a speed estimation method based on convolutional neural networks. This method provides multimodal speed constraints for algorithms such as INS or PDR, thereby providing additional observations for these algorithms.

[0042] 2. To address the problem of relying on hardware (wheel speedometers or GNSS, etc.) to achieve integrity constraints when navigating and positioning application platforms such as vehicles and robots, the present invention proposes a convolutional neural network speed estimation model that does not rely on additional hardware. This saves the cost of wheel speedometers and avoids the impact of inaccurate IMU extrinsic parameter estimates on constraints. The implementation of this method can effectively take into account the real-time characteristics of the data, reduce the impact of real-time factors such as vehicle load and road conditions on speed estimation, and realize a robust speed estimation solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flow chart of a speed estimation method based on a convolutional neural network provided by an embodiment of the present invention;

[0044] Figure 2 A structural diagram of a speed estimation neural network provided by an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of speed estimation and reference speed provided by an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of the optimized speed and the reference speed provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0048] Example

[0049] Figure 1 This is a flow chart of a convolutional neural network-based speed estimation method according to an embodiment of the present invention. This embodiment is applicable to situations where the speed of a mobile carrier is estimated. Typically, the mobile carrier in this embodiment may include platforms such as smartphones, tablets, cars, and robots.

[0050] Specifically, the method includes the following steps:

[0051] S1. Preprocess the data collected by the inertial measurement unit (IMU) installed on the mobile carrier to obtain a training data set.

[0052] Specifically, the above S1 includes the following steps:

[0053] S11: Based on the IMU operating frequency, the acceleration, gyroscope measurement values, and velocity reference values ​​are organized into the following format according to a certain window size:

[0054] (Acce,Gyro,v) n×6 (1)

[0055] Acce represents acceleration data, Gyro represents gyroscope data, and v represents the reference velocity. n represents the window length. Generally, 1 second of data can be selected to construct a window. 6 represents the 6-dimensional measurement value consisting of 3-axis acceleration and 3-axis gyroscope.

[0056] Specifically, the reference speed of mobile devices such as mobile phones and tablets can be measured or calculated at a specified frequency by a mobile measurement robot or motion capture system. Since the relative posture of vehicles or robots is relatively fixed, the reference speed can be measured or calculated at a specified frequency by a high-precision GNSS / inertial navigation combined navigation system or a measurement robot.

[0057] S12: adding different types of random noise to the acceleration data and gyroscope data in the window obtained in S11.

[0058] During the subsequent model training process, the velocity estimation neural network (virtual odometer network) connects the normalized acceleration and gyroscope observations to six channels as network inputs. Due to their own characteristics, inertial devices contain a lot of noise, which to a certain extent reduces the final regression accuracy of the virtual odometer network. Therefore, this embodiment adds random noise to the observation data to enhance the network's ability to detect and identify noise during the training phase, thereby improving the network's adaptability to noise. Different variances are added to different channel inputs, that is, different types of random noise are added to the acceleration data and gyroscope data respectively.

[0059] For example, in this embodiment, Gaussian noise is randomly added to the data. The specific formula is as follows:

[0060]

[0061]

[0062] Among them, σ a and σ g The Gaussian noise that the accelerometer and gyroscope obey can be obtained using Allan variance analysis or from the IMU parameter data sheet provided by the IMU provider.

[0063] S2. Input the training data set obtained in S1 into the speed estimation neural network to be trained to obtain a trained target speed estimation neural network.

[0064] For example, see Figure 2 The virtual odometry network in this embodiment is based on a deep residual model. Its main features include convolutional layers, activation functions, identity shortcuts (IS), and attention modules. The identity shortcuts are used to learn the mapping between the input and output features of the module.

[0065] Normally, IMU observations are often affected by various interference features, such as noise-related features and road condition-related features, which results in reduced accuracy of the virtual odometer. The role of the identity jump connection and attention module is to guide the network to learn features that are strongly related to vehicle motion, while eliminating interference features. Deep learning often uses the ReLU activation function to set features that are less than a certain threshold interval to 0, thereby deleting some features / information. Compared with the ReLU activation function, the soft threshold can more flexibly set the data feature interval that the network pays attention to. Therefore, this application further introduces an attention-based soft threshold module to output a set of sample-based thresholds for adaptive selection of data features, and realizes speed estimation by introducing an attention mechanism to mine motion features that are strongly related to the carrier.

[0066] Specifically, in the model training of this embodiment, DO (Dropout) of 0.2-0.5 is used to avoid overfitting in training and improve the generalization ability of the network.

[0067] The attention mechanism is introduced to learn a set of coefficients through a set of convolutional layers, giving different weights to different channels:

[0068] ξ i ′=average(|ξ i,H,1 |) (4)

[0069] η i =Θ(w i ·ξ i ′+b i ) (5)

[0070] Where Θ represents a score function that is used to evaluate the importance of each channel given a channel vector; H represents the window size; η i represents the channel weight estimated by the attention mechanism; ξ represents the high-dimensional feature vector; w and b represent the weight and bias learned by the network. This implements a weight enhancement strategy for strongly correlated features of the carrier, improving the generalization ability and speed regression accuracy of the virtual odometry network.

[0071] Then, according to the reference speed and network output, the loss function is set to train the model by minimizing the loss function and backpropagation.

[0072] The above loss function is formula (6) or formula (7):

[0073]

[0074] L(v|g(Acce n×3 ,Gyro n×3 ))=[mean(v i＝1,2,..,n )-mean(g i＝1,2,..,n (Acce n×3 ,Gyro n×3 ))] 2 (7)

[0075] Where g i (·) represents the neural network regression function, g i The result of (·) is the speed estimated by the network, v i represents the reference velocity of the ith sample.

[0076] Figure 2Six network structures for mobile carrier velocity estimation are presented, including SdoNet-1, SdoNet-2, SdoNet-3, SdoNet-4, SdoNet-5, and SdoNet-6. For example, for scenarios with relatively intense activity, such as tablets and mobile phones, it is generally recommended to use the deeper network structures SdoNet4, SdoNet5, and SdoNet6. For scenarios such as vehicles and robots, where the relative relationship between the IMU and the carrier is relatively certain and the motion pattern is simple, it is recommended to use the shallower network structures SdoNet1, SdoNet2, and SdoNet3.

[0077] The network structure described here is only used to explain the present invention, not to limit the present invention. The length of the window data and the network structure can be adjusted according to the actual application. Taking 6@100 as an example, the symbol represents the number of channels as 6 and the sequence length as 100; CL@39 represents the convolution layer with a convolution kernel size of 39; MP@2 represents the maximum pooling layer with a pooling size (PoolingSize) of 2; FC@n represents a fully connected layer with a length of n; DO@0.5 represents the random inactivation of 50% of the neural network nodes. The network input is 6@100, and the structure of each convolutional neural network is as follows:

[0078]

[0079]

[0080] S3. Input the data to be estimated detected by the IMU into the target speed estimation neural network obtained in S2 to obtain a speed estimation result of the mobile carrier.

[0081] The embodiment of the present invention can implement hardware-free virtual odometer estimation and integrity constraint by inputting IMU data into a trained model without the need for additional hardware such as a wheel speed meter.

[0082] S4. Constructing an in-window speed estimation constraint for the speed estimation result obtained in S3 based on the acceleration constraint, and smoothing the estimated speed to obtain an optimized speed estimation result.

[0083] like Figure 3As shown, the gray curve represents the estimated speed directly output by the target speed estimation neural network, and the black curve represents the reference speed. It can be seen that the estimated speed fluctuates around the reference speed. Therefore, when estimating the speed, this embodiment increases the speed output frequency per unit time by setting a larger window overlap. Then, a window N of a certain size is selected, such as 0.5-second window data, and an acceleration residual is constructed to constrain the speed (here it is assumed that the acceleration of the carrier is 0 in a very short time). Equation 8 is the acceleration residual, and Equation 9 is the constructed optimization problem. Solving this optimization problem results in a smooth speed estimate:

[0084]

[0085]

[0086] Where δ represents the velocity residual, which can also be called the loss function; λ represents the velocity smoothing coefficient; and Δt represents the time interval between two observations.

[0087] The Levenberg-Marquardt (LM) algorithm is a nonlinear optimization method between the Newton method and the gradient method. This method is insensitive to over-parameterization and significantly reduces the possibility of the loss function being trapped in a local minimum. Therefore, using the LM algorithm to solve this problem, the second-order Taylor function of the loss function is as follows:

[0088]

[0089] Where χ represents the variable to be optimized, J and μ represent the Jacobi matrix and damping term respectively; χ0 represents the Taylor expansion at this location; δ χ Represents the residual. The k-th iteration can be expressed as:

[0090]

[0091] The trust region algorithm can control the state of χ and find the optimal estimate near the observed value to avoid the optimal intermediate value with large deviation. Therefore, the upper and lower bounds of the control trust region radius of the optimization value are set:

[0092]

[0093] Where, and v i Represent the speeds before and after optimization respectively; ξ v represents the speed optimization threshold. Furthermore, the speed data contains many outliers, which may be caused by hardware characteristics, dynamic environments, and so on. These outliers may cause the algorithm to fall into local optimality or even have difficulty converging. The "CauchLoss" residual block is used to process these outliers and improve the robustness of the model. Its expression is:

[0094]

[0095] Where b is an adjustable parameter; a=||f i || 2 ; By applying the loss function to all residuals, the weight of outliers is reduced. Then, large outliers are down-weighted and do not overly affect the optimization problem by performing the loss function on all residuals.

[0096] The optimized results are as follows Figure 4 As shown, it can be seen that the optimized speed is closer to the reference speed and the speed fluctuation is greatly reduced.

[0097] Step 5: Using the optimized velocity estimation results in S4, the following observation equation can be constructed to constrain the error accumulation of the inertial navigation system.

[0098]

[0099] Where δz represents the velocity observation residual; Indicates the speed calculated by the inertial system; n v represents the observation noise; δx represents the system residual; The orientation rotation matrix representing the installation angle calculation of the IMU coordinate system relative to the vehicle coordinate system; Represents the direction rotation matrix between the navigation coordinate system and the IMU coordinate system; v n represents the velocity in the carrier coordinate system; × represents the antisymmetric matrix; I represents the unit matrix; The lever arm vector representing the center of the IMU coordinate system and the center of the vehicle coordinate system; Represents the gyroscope observation in the vehicle coordinate system. By applying the estimated forward velocity to build a constraint model, the accumulation of inertial navigation errors can be reduced.

[0100] This embodiment further provides a speed estimation device based on a convolutional neural network, comprising:

[0101] A preprocessing module is used to preprocess the data collected by the inertial measurement unit (IMU) installed on the mobile carrier to obtain a training data set;

[0102] A training module, configured to input the training data set into a speed estimation neural network to be trained, so as to obtain a trained target speed estimation neural network;

[0103] The estimation module is used to input the data to be estimated detected by the IMU into the target speed estimation neural network to obtain the speed estimation result of the mobile carrier.

[0104] Optionally, the device further includes: an optimization module, configured to smooth the velocity of the fixed window based on the acceleration constraint to obtain an optimized velocity estimation result.

[0105] The velocity constraint module is used to construct the observation equation based on the velocity estimation result of the target velocity estimation neural network to constrain the error accumulation of the inertial navigation system.

[0106] The observation equation is specifically:

[0107]

[0108] Where δz represents the velocity observation residual; Indicates the speed calculated by the inertial system; n v represents the observation noise; δx represents the system residual; The orientation rotation matrix representing the installation angle calculation of the IMU coordinate system relative to the vehicle coordinate system; Represents the direction rotation matrix between the navigation coordinate system and the IMU coordinate system; v n represents the velocity in the carrier coordinate system; × represents the antisymmetric matrix; I represents the unit matrix; The lever arm vector representing the center of the IMU coordinate system and the center of the vehicle coordinate system; Represents the gyroscope observation in the carrier coordinate system.

[0109] Specifically, the speed estimation neural network includes a convolutional layer, an activation function, and an attention module. The preprocessing module is specifically used to:

[0110] S11. According to the sampling frequency of the IMU, the acceleration, gyroscope measurement values ​​and velocity reference values ​​are arranged into the following format according to a certain window size:

[0111] (Acce,Gyro,v) n×6

[0112] Where Acce represents acceleration data, Gyro represents gyroscope data, and v represents reference velocity; n represents the window length, and 6 represents the 6-dimensional measurement value composed of 3-axis acceleration and 3-axis gyroscope;

[0113] S12. Add different types of random noise to the acceleration data and gyroscope data in the window obtained in S11.

[0114] The estimation module is specifically used for:

[0115] The speed estimation neural network model to be trained is trained by minimizing the loss function and back propagation to obtain a trained target speed estimation neural network.

[0116] A speed estimation device based on a convolutional neural network provided in an embodiment of the present invention can execute a speed estimation method based on a convolutional neural network provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method, which will not be described in detail.

[0117] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will appreciate that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and alternatives are possible for those skilled in the art without departing from the scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A speed estimation method based on convolutional neural network, characterized in that: include: S1. Preprocessing data collected by an inertial measurement unit (IMU) mounted on a mobile carrier to obtain a training data set. S2. Inputting the training data set into the speed estimation neural network to be trained to obtain a trained target speed estimation neural network; S3, inputting the data to be estimated detected by the IMU into the target speed estimation neural network to obtain a speed estimation result of the mobile carrier; S4. Smoothing the speed estimation result based on the acceleration constraint to obtain an optimized speed estimation result; S5: Based on the optimized velocity estimation results, the observation equation is constructed to constrain the error accumulation of the inertial navigation system; The observation equation is: ； Where, represents the velocity observation residual; Indicates the velocity calculated by the inertial system; represents the observation noise; represents the system residual; The orientation rotation matrix representing the installation angle calculation of the IMU coordinate system relative to the vehicle coordinate system; Represents the direction rotation matrix between the navigation coordinate system and the IMU coordinate system; represents the velocity in the carrier coordinate system; represents an antisymmetric matrix; The lever arm vector representing the center of the IMU coordinate system and the center of the vehicle coordinate system; Represents the gyroscope observation in the carrier coordinate system.

2. The method according to claim 1, characterized in that The speed estimation neural network includes a convolutional layer, an activation function, and an attention module.

3. The method according to claim 1, characterized in that Said S1 comprises: S11. According to the sampling frequency of the IMU, the acceleration, gyroscope measurement values ​​and velocity reference values ​​are arranged into the following format according to a certain window size: ； in, Access represents acceleration data, Gyro Represents gyroscope data, v Indicates the reference speed; n Indicates the window length, 6 represents the 6-dimensional measurement value composed of 3-axis acceleration and 3-axis gyroscope; S12. Add different types of random noise to the acceleration data and gyroscope data in the window respectively.

4. The method according to claim 1, wherein The S3 includes: The speed estimation neural network model to be trained is trained by minimizing the loss function and back propagation to obtain a trained target speed estimation neural network.

5. A speed estimation device based on convolutional neural network, characterized in that: include: A preprocessing module is used to preprocess the data collected by the inertial measurement unit (IMU) installed on the mobile carrier to obtain a training data set; A training module, configured to input the training data set into a speed estimation neural network to be trained, so as to obtain a trained target speed estimation neural network; An estimation module is used to input the data to be estimated detected by the IMU into the target speed estimation neural network to obtain a speed estimation result of the mobile carrier; an optimization module, configured to smooth the speed estimation result based on acceleration to obtain an optimized speed estimation result; The velocity constraint module is used to construct the observation equation based on the optimized velocity estimation results to constrain the error accumulation of the inertial navigation system; The observation equation is: ； Where, represents the velocity observation residual; Indicates the velocity calculated by the inertial system; represents the observation noise; represents the system residual; The orientation rotation matrix representing the installation angle calculation of the IMU coordinate system relative to the vehicle coordinate system; Represents the direction rotation matrix between the navigation coordinate system and the IMU coordinate system; represents the velocity in the carrier coordinate system; represents an antisymmetric matrix; The lever arm vector representing the center of the IMU coordinate system and the center of the vehicle coordinate system; Represents the gyroscope observation in the carrier coordinate system.

Images