Positioning method of dynamic fusion of inertial navigation, vision and prior information

Abstract

The application discloses a positioning method of inertial navigation, vision and prior information dynamic fusion, aiming at improving the precision and robustness of position and pose estimation in dynamic and complex environment, comprising: 1, collecting device IMU data in real time, converting to world coordinate system and removing gravity to obtain pretreated IMU data; 2, applying neural network architecture search technology to obtain optimal neural network; 3, training the optimal neural network by using the pretreated IMU data to form a training data set and inferring pose incremental estimation and covariance; 4, obtaining prior direction and covariance based on prior information; 5, introducing visual information and device IMU data fusion and constructing VIO system, and outputting pose estimation and covariance; 6, the output results of steps 3 to 5 are used as measurement information of a factor graph for multi-source information fusion; 7, the parameters of the optimal neural network are updated by using the factor graph fusion result, the device IMU data is corrected according to step 1, and then is sent into the optimal neural network for inference, and then steps 4 to 7 are repeated.

Description

Technical Field

[0001] This invention belongs to the field of mobile positioning and navigation technology, specifically covering related fields such as visual inertial odometry (VIO), sensor fusion technology, factor graphs, and neural network architecture search (NAS). It combines inertial measurement unit (IMU) data with visual information, and in particular refers to a positioning method, electronic device, and storage medium that dynamically fuses inertial navigation, vision, and prior information, aiming to improve the accuracy and robustness of position and pose estimation in dynamic and complex environments. Background Technology

[0002] Technological developments in the field of mobile positioning and navigation have mainly focused on improving positioning accuracy and system robustness, resulting in a variety of innovative methods.

[0003] Traditional inertial navigation systems (SINS) acquire position information through double integration of IMU measurements, while methods such as pedestrian dead reckoning (PDR) and zero velocity update (ZUPT) are widely used to reduce the accumulation of errors.

[0004] In recent years, Heuristic Drift Elimination (HDE) algorithms and their improved version, iHDE, have significantly improved positioning accuracy in complex environments by utilizing the dominant orientation within buildings for orientation correction. With the rapid development of deep learning, data-driven methods have gradually become mainstream. For example, the RONIN (Robust Neural Inertial Navigation in the Wild) framework utilizes deep learning models to capture the time-series characteristics of IMU data, enhancing robustness in real-world environments; TLIO (Tight Learned Inertial Odometry) combines neural networks with extended Kalman filters to further improve the accuracy of displacement estimation. CTIN (Robust Contextual Transformer Network for Inertial Navigation) introduces an attention mechanism to enhance focus on key features. For mobile device applications, IMUNet (Inertial Measurement Unit Network) employs a lightweight network architecture designed to optimize performance and reduce computational complexity.

[0005] In summary, this field is developing towards greater intelligence, precision, and adaptability, continuously driving the advancement of inertial navigation systems through deep learning and sensor fusion technologies.

[0006] Existing inertial navigation technologies suffer from several significant drawbacks: First, traditional inertial navigation systems are significantly affected by sensor drift and accumulated errors, especially in complex indoor environments, where positioning accuracy is often severely compromised. Although techniques such as the iHDE algorithm can mitigate accumulated errors caused by sensor drift to some extent, their correction effect on non-ideal paths (such as curved paths) remains limited, potentially leading to additional orientation errors and further impacting positioning accuracy. Furthermore, while data-driven neural network methods can alleviate trajectory error accumulation to some extent, they fail to effectively address the problem of accumulated heading deviation. This limitation restricts their application in long-duration navigation tasks, challenging the adaptability and stability of existing technologies in complex and dynamic environments.

[0007] Therefore, how to further improve navigation accuracy and robustness, especially its effectiveness in complex environments, remains a key challenge in current technological development. Summary of the Invention

[0008] The purpose of this invention is to provide a positioning method that dynamically fuses inertial navigation, vision, and prior information, thereby solving the problems existing in the prior art and aiming to improve the accuracy and robustness of position and pose estimation in dynamic and complex environments.

[0009] To achieve the above objectives, the solution of the present invention is:

[0010] A localization method that dynamically fuses inertial navigation, vision, and prior information, comprising:

[0011] Step 1: Collect device IMU data in real time from the user's mobile device. Then, through a preprocessing step, transform the device IMU data from the device coordinate system to the world coordinate system and remove gravity to obtain preprocessed IMU data. Wherein, IMU stands for Inertial Measurement Unit;

[0012] Step 2: Apply neural network architecture search techniques to search and optimize the layer types, connection methods, and hyperparameters of the neural network in the search space to obtain the optimal neural network. ;

[0013] Step 3: Construct a training dataset from a preset amount of preprocessed IMU data, and then optimize the neural network. The optimal neural network after training is used to perform inference to obtain the pose increment estimate and its covariance.

[0014] Step 4: Obtain the prior direction and its covariance based on prior information;

[0015] Step 5: When visual information is available, the visual information is fused with the device IMU data from Step 1 to construct a VIO system. The VIO system outputs the pose estimate and its covariance in real time; where VIO stands for Visual Inertial Odometry.

[0016] Step 6: Using the output results of steps 3 to 5 as measurement information, perform multi-source information fusion using factor graphs;

[0017] Step 7: Update the parameters of the optimal neural network using the fusion results output by the factor graph in Step 6, and correct the device IMU data according to Step 1. Then, send the corrected results into the optimal neural network for inference, and repeat Steps 4 to 7 until the end.

[0018] The preprocessing step in step 1 specifically includes:

[0019] 1.1 Device IMU data includes time. The collected accelerometer and gyroscope data are the raw acceleration. Original angular velocity ;

[0020] 1.2 If the device IMU data is used to propagate the three-dimensional pose between consecutive time steps, the device IMU data will be updated in three-dimensional space as follows:

[0021] Location update ;

[0022] Speed ​​Update ;

[0023] Posture update ;

[0024] Among them, for time and the previous moment : and Indicates location, and Indicates speed, and Represents the rotation matrix. and These represent the initial acceleration and initial angular velocity, respectively. and These represent the error biases of the accelerometer and gyroscope, respectively. Represents the gravitational acceleration measured at rest. Indicates time and the previous moment The time interval; In Lie algebra Exponential mapping on;

[0025] 1.3 Perform a world coordinate system transformation. The net acceleration in the world coordinate system is expressed as:

[0026] ;

[0027] Angular velocity in the world coordinate system is expressed as:

[0028] ;

[0029] The resulting preprocessed IMU data is represented as follows:

[0030] .

[0031] Step 2 specifically involves:

[0032] 2.1 Defining the Search Space for:

[0033] ;

[0034] in, Indicates a convolutional layer; Indicates a linear layer; Indicates the activation function; Indicates the pooling layer; Indicates other layers;

[0035] 2.2 Defining Candidate Network Architectures for:

[0036] ;

[0037] in, Indicates the first in the network Layer configuration; the configuration of each layer is represented as ,in Indicates the first in this layer One parameter;

[0038] 2.3 Define the loss function to minimize The loss of the model is expressed by the following formula:

[0039] ;

[0040] in, Indicates the total number of samples; express The calculation method; Indicates the first The true label of each sample Indicates the first The model prediction output for each sample;

[0041] 2.4 At least one of reinforcement learning, evolutionary algorithm, Bayesian optimization, and brute-force search is used to search for network architectures, and the performance of candidate network architectures is evaluated by minimizing the loss function; after several iterations, the optimal neural network is selected. and its parameters .

[0042] Step 3 specifically involves:

[0043] 3.1 Optimal Neural Network Input data The dimension is represented as:

[0044] ;

[0045] in, Represent the space of real numbers; Indicates batch size; Indicates the number of channels; Indicates the sequence length;

[0046] 3.2 During the training process, a training dataset is constructed from a predetermined amount of preprocessed IMU data, which is used to optimize the neural network. Training; Optimal Neural Network The output is the pose estimate and its covariance, expressed as:

[0047] ;

[0048] in, and These represent the pose estimate and covariance of the network output during training, respectively. This represents the input data during the training process; These represent the parameters of the optimal neural network.

[0049] Each round optimizes the neural network During training, the parameters are updated, and the function that minimizes the loss function is ultimately saved. The smallest set of parameters is denoted as the final parameter. ;

[0050] 3.3 During the inference process, the optimal trained neural network is used. Perform pose estimation; define input data For the IMU data to be inferred, the inference process is represented as follows:

[0051] ;

[0052] ;

[0053] in, and These represent the optimal neural network. The pose increment estimate and its covariance obtained through reasoning; and These represent the optimal neural network. The position increment covariance and attitude increment covariance are obtained through reasoning.

[0054] Step 4 specifically involves:

[0055] 4.1 Based on device IMU data and its time intervals The IMU yaw angle is calculated and expressed as:

[0056] ;

[0057] in, and Representing time respectively and the previous moment IMU yaw angle; Indicates time The angular velocity component of the gyroscope in the z-axis direction;

[0058] 4.2 Obtain the prior yaw angle provided by prior information based on the known prior orientation angle or nonlinear regression model. The covariance of the prior direction is calculated using a dynamic covariance scheme. The difference between the IMU yaw angle and the prior yaw angle is:

[0059] ;

[0060] right Perform periodic processing to limit it to the range of [-180°, 180°];

[0061] By comparison The covariance of prior information compared to the preset difference threshold is expressed as:

[0062] ;

[0063] in, The difference threshold is represented by the value of the maximum covariance when the prior is unreliable. When the prior is reliable, the smaller covariance value is taken as... .

[0064] Preferably, in step 4.2, the prior yaw angle obtained based on the prior information is obtained using a nonlinear regression model. Specifically:

[0065] 4.2.1 Collect the location and orientation data of the mobile device over a period of time, and record it as the raw trajectory data:

[0066] ;

[0067] in, , and They represent the first The horizontal coordinate, vertical coordinate, and yaw angle of the mobile device at each moment; This indicates the total duration of the period.

[0068] 4.2.2 Smooth the original trajectory data;

[0069] 4.2.3 The smoothed trajectory data after establishing the nonlinear regression model is represented as follows:

[0070] ;

[0071] in, Indicates the predicted direction angle; Represents a nonlinear function; and They represent , All are related to time Related functions; Indicates model parameters;

[0072] By minimizing the predicted direction angle Compared with actual direction angle The error is used to train a nonlinear regression model, which is expressed as:

[0073] ;

[0074] in, This represents the optimal parameter that minimizes the mean square error between the predicted and actual orientation angles. Indicates calculation from arrive The sum of the mean square errors is taken as the minimum value to obtain the optimal parameters;

[0075] 4.2.4 Using the trained nonlinear regression model to process new location data The prior yaw angle is calculated. , is represented as:

[0076] .

[0077] Step 5 specifically involves:

[0078] 5.1 Construct a VIO system and output the conversion matrix of the mobile device through the VIO system. , is represented as:

[0079] ;

[0080] in, This represents the rotation matrix output by the VIO system; This represents the translation matrix output by the VIO system;

[0081] 5.2 Calculate the error covariance of the VIO system output, specifically:

[0082] 5.2.1 Covariance of VIO System Output Represented as:

[0083] ;

[0084] in, This represents the position covariance of the VIO system output; This represents the attitude covariance output by the VIO system;

[0085] 5.2.2 Location Covariance Represented as:

[0086] ;

[0087] in, Represents the positional covariance within the image plane, based on the number of image feature points. and distribution uniformity To estimate, This means constructing the input data into a diagonal matrix. This represents a manually set adjustment coefficient for the number of feature points. This represents a manually set uniformity adjustment coefficient; The position covariance in the depth direction is defined as follows:

[0088] ;

[0089] in, and These represent manually set adjustment coefficients, used to control the influence of VIO trajectory smoothness and IMU acceleration stability on the depth-direction position covariance, respectively. Indicates the positional smoothness of the VIO trajectory; This indicates the acceleration stability of the IMU in the out-of-plane direction;

[0090] Positional smoothness The standard deviation of the displacement increment between adjacent time steps is used for calculation, and it is expressed as:

[0091] ;

[0092] in, This indicates the time step calculated by the VIO system. arrive The relative displacement increment; The modulus representing the displacement increment; This represents the average value of the displacement increment within the sliding window; when the position change is smooth, Take the smaller value, and vice versa. Take the larger value;

[0093] Acceleration stability The acceleration is calculated using the standard deviation of the rate of change of acceleration between adjacent time steps, and is expressed as:

[0094] ;

[0095] in, The magnitude of the rate of change of acceleration between adjacent time steps. and These represent the time steps. and IMU acceleration measurements; The average value of the rate of change of acceleration within the sliding window is calculated using the following formula:

[0096] ;

[0097] Uniformity of distribution The calculation method is as follows:

[0098] By dividing the image captured by the mobile device's camera into equal parts A grid, and make Indicates the first line, number The number of feature points in the column grid is obtained. , This indicates the calculation of variance. The value of the feature point is inversely proportional to the uniformity of the feature point distribution;

[0099] 5.2.3 Attitude Covariance The smoothness of pose changes is estimated by the smoothness of the pose change. , is represented as:

[0100] ;

[0101] in, Indicates the incremental rotation angle; The average value representing the change in rotation angle within the sliding window is calculated using the following formula:

[0102] ;

[0103] The attitude covariance matrix is:

[0104] ;

[0105] in, An adjustment coefficient representing attitude smoothness;

[0106] 5.2.4 Combining the above components, the final VIO covariance matrix is ​​obtained. .

[0107] Step 6 specifically involves:

[0108] 6.1 Defining the state of the factor graph for:

[0109] ;

[0110] in, , and These represent the position, velocity, and attitude output by the factor map, respectively.

[0111] 6.2 Construct IMU factors for a factor graph using device IMU data to establish motion constraints on the state between adjacent time points;

[0112] 6.3 Utilizing Optimal Neural Networks The neural network measurement factor of the output pose increment estimate and its covariance construct factor graph;

[0113] 6.4 Utilizing Prior Yaw Angle The prior measurement factors of its covariance construct factor plot; prior yaw angle The error term is:

[0114] ;

[0115] in, The error term representing the prior yaw angle; This indicates the extraction of the yaw angle from the rotation matrix;

[0116] 6.5 Constructing VIO measurement factors from a factor map using pose estimation and covariance output by the VIO system;

[0117] 6.6 Incremental optimization of the factor graph is performed using a sliding window optimization method;

[0118] 6.7 In real-time data processing, the device IMU data is corrected using the latest optimization results of the factor graph.

[0119] Preferably, step 6.6 specifically involves:

[0120] 6.6.1 Assume the current sliding window contains At that moment, at that moment The factor graph contains several state nodes and measurement factors;

[0121] time The state node is defined as follows:

[0122] ;

[0123] in, Indicates time Location; Indicates time speed; Indicates time rotation matrix;

[0124] 6.6.2 At time The objective function for sliding window optimization includes the weighted sum of errors of all measurement factors, and the optimization objective is expressed as:

[0125] ;

[0126] in, Representing state IMU factor residuals; Representing state The residual of the neural network measurement factor; Representing state The prior measurement factor residuals; Representing state The VIO measurement factor residuals; each factor has a corresponding covariance matrix. Used for weighting;

[0127] 6.6.3 Sliding window optimization uses incremental updates at each time step. The specific execution process is as follows:

[0128] 6.6.3.1 At every new moment Add a new state node And new measurement factors, new state nodes Including location ,speed ,attitude Acceleration bias and angular velocity Bias, new measurement factors include new IMU factors, new neural network measurement factors, new prior measurement factors, and new VIO measurement factors;

[0129] 6.6.3.2 To control computational complexity, the sliding window size is kept constant. It is necessary to remove state nodes and measurement factors outside the window, that is, remove the oldest state nodes. And its associated IMU factor, neural network measurement factor, prior measurement factor and VIO measurement factor;

[0130] 6.6.3.3 The iSAM2 algorithm is used to incrementally optimize the factor graph within the sliding window. iSAM2 uses the optimization result from the previous time step as the initial value to achieve fast updates. The goal is to minimize the sum of residuals to obtain the time step. Optimization status The calculation formula is as follows:

[0131] .

[0132] Preferably, step 7 specifically involves:

[0133] 7.1 Assume that at time... The length of the preprocessed IMU data input to the optimal neural network is , for time to The data are sequentially optimized using a factor graph sliding window to obtain state variables, which are then substituted into step 1.2 to calculate the pose propagation between time steps. Finally, the raw acceleration measured by the IMU is used in step 1.3. and original angular velocity The processed IMU data after correction at each time point is represented as follows:

[0134] ;

[0135] in, and These represent the corrected acceleration and angular velocity, respectively.

[0136] 7.2 The corrected IMU input data sequence is generated as follows:

[0137] ;

[0138] in, and Representing time respectively and Corrected IMU data;

[0139] 7.3 The corrected IMU input data sequence Input to the optimal neural network Perform real-time pose inference and output the time step. The pose estimation is ,in and Representing time respectively By optimal neural network Output the position and orientation; then repeat steps 4 to 7 until finished.

[0140] After adopting the above technical solution, the present invention has the following technical effects:

[0141] (1) By introducing Neural Architecture Search (NAS), the neural network design is optimized, enabling the system to infer displacement information more accurately and reducing the positioning inaccuracy caused by IMU drift and accumulated errors;

[0142] (2) By combining the displacement inferred by the correction neural network with the heading provided by the prior information and the IMU integral heading, the heading weight can be dynamically adjusted, which effectively solves the problem of poor orientation correction effect of traditional methods in complex environments, thereby enhancing the robustness of orientation estimation.

[0143] (3) The visual inertial odometry (VIO) system, which combines visual information, enables the system to maintain high positioning accuracy in most scenarios with good visual features, and improves the system's adaptability to complex indoor environments.

[0144] (4) By fusing VIO, heading correction information and pose covariance inferred by neural network through factor graph, the system can achieve more accurate state estimation, which improves the stability and reliability of the overall system.

[0145] (5) The present invention can form a navigation strategy suitable for nonlinear paths by integrating behavioral patterns and prior knowledge, overcoming the limitation of traditional methods that can only handle linear paths. Attached Figure Description

[0146] Figure 1 This is an overall system block diagram of a specific embodiment of the present invention. Detailed Implementation

[0147] To further explain the technical solution of the present invention, the present invention will be described in detail below through specific embodiments.

[0148] See Figure 1 This invention discloses a positioning method that dynamically fuses inertial navigation, vision, and prior information, comprising:

[0149] Step 1: Data Acquisition and Preprocessing

[0150] Data from the inertial measurement unit (IMU) is collected in real time from the user's mobile device (usually a mobile phone, tablet, etc.), and is denoted as device IMU data. This includes data from the accelerometer and gyroscope, i.e., the current acceleration and angular velocity values. The device IMU data captures the motion information of the user carrying the mobile device in different ways (such as handheld, pocket-carried, or worn on the chest), ensuring that the invention is applicable to various scenarios and has greater adaptability. Since the reference frame of the original data is the device coordinate system, in order to ensure that the data can be uniformly processed in the world coordinate system, step 1 also needs to perform a preprocessing step to transform the device IMU data in the device coordinate system to the world coordinate system and remove the influence of gravity, obtaining preprocessed IMU data for subsequent analysis and processing steps. ;

[0151] Step 2, Neural Network Search

[0152] The Neural Architecture Search (NAS) technique is applied to explore and optimize the layer types, connections, and hyperparameters of neural networks within the search space, obtaining parameters suitable for processing pre-processed IMU data in subsequent steps. Optimal Neural Network This ensures that the resulting neural network model has excellent ability to predict position and orientation information;

[0153] Step 3: Neural Network Inference

[0154] The training dataset consists of a predetermined number of preprocessed IMU data points, which are used to optimize the neural network. The optimal neural network after training is used to perform inference to obtain the pose increment estimate and its covariance.

[0155] Step 4, Direction Correction

[0156] Obtain the prior direction and its covariance based on prior information;

[0157] Step 5, VIO system

[0158] When visual information is available, it is introduced and deeply fused with the device IMU data from step 1 (i.e., the IMU data before preprocessing) to construct a Visual-Inertial Odometry (VIO) system, aiming to significantly improve the accuracy and reliability of pose estimation. The VIO system outputs high-precision pose estimation and its covariance in real time. The VIO system can maintain stable tracking performance in complex and changing environments, thus providing a solid foundation for subsequent pose estimation.

[0159] Step 6: Multi-source information fusion

[0160] Using the outputs of steps 3 to 5 as measurement information, multi-source information fusion is performed using factor graphs;

[0161] Step 7: Update the parameters of the optimal neural network using the fusion results output by the factor graph in Step 6, and correct the device IMU data according to Step 1. Then, send the corrected results into the optimal neural network for inference, and repeat Steps 4 to 7 until the end.

[0162] The following illustrates specific embodiments of the present invention.

[0163] The preprocessing step in step 1 above is specifically as follows:

[0164] 1.1 Raw Data Acquisition

[0165] Define the inertial measurement unit at time 10:00. The collected accelerometer and gyroscope data are the raw acceleration. Original angular velocity Both are values ​​in the device coordinate system, and after subsequent processing steps, unnecessary components and noise can be removed to more accurately describe the user's real motion state.

[0166] 1.2 Status Update

[0167] Using device IMU data to propagate 3D pose across consecutive time steps, the device IMU data is updated in 3D space as follows:

[0168] Location update ;

[0169] Speed ​​Update ;

[0170] Pose update (using Lie group-Lie algebra representation) ;

[0171] Among them, for time and the previous moment : and Indicates location, and Indicates speed, and Represents the rotation matrix (representing the rotation attitude from the device coordinate system to the world coordinate system). and These represent the initial acceleration and initial angular velocity, respectively. and These represent the error biases of the accelerometer and gyroscope, respectively. They can be obtained from the equipment manufacturer or calculated by yourself. Represents the gravitational acceleration measured at rest. Indicates time and the previous moment The time interval; In Lie algebra Exponential mapping on;

[0172] 1.3 World Coordinate System Transformation

[0173] Using rotation matrix After converting the acceleration and angular velocity data to the world coordinate system, the net acceleration in the world coordinate system is expressed as: Angular velocity in the world coordinate system is expressed as This allows the motion representation of all devices to be unified to the world coordinate system, facilitating collaborative calculations by multiple sensors; ultimately, preprocessed IMU data is obtained. .

[0174] In step 1, by acquiring and preprocessing IMU data, the present invention can effectively extract net acceleration and angular velocity information from the raw sensor data and convert it to the standard world coordinate system to obtain preprocessed IMU data for subsequent analysis and processing steps.

[0175] Step 2 above specifically refers to:

[0176] 2.1 Defining the Search Space

[0177] Define the search space for:

[0178] ;

[0179] in, This indicates a convolutional layer, such as standard convolution, depthwise separable convolution, etc. Indicates a linear layer; Represents the activation function, such as , , wait; This indicates the pooling layer, such as max pooling, average pooling, etc. This represents other layers, such as Fourier transform, single-head attention, multi-head attention, etc.; usually, in order to reduce the amount of search computation, existing experience should be used as much as possible to filter necessary elements to reduce the search space.

[0180] 2.2 Representation of Network Architecture

[0181] Candidate network architecture is defined using a structured vector. for:

[0182] ;

[0183] in, Indicates the first in the network The configuration of each layer includes its type, parameters, etc.; the configuration of each layer is represented as a set. ,in Indicates the first in this layer Each parameter is a specific numerical value.

[0184] 2.3 Performance Evaluation

[0185] The performance of each candidate network architecture is evaluated to facilitate subsequent optimization, specifically:

[0186] Evaluate each candidate network architecture When considering performance, minimizing the loss function is typically used. The loss representing the model can be expressed as cross-entropy loss or mean squared error loss.

[0187] ;

[0188] in, Indicates the total number of samples; express The calculation method, such as calculating cross-entropy or mean squared error, yields a single value; Indicates the first The true label of each sample Indicates the first The model prediction output for each sample;

[0189] 2.4 Optimization Algorithm

[0190] At least one of reinforcement learning, evolutionary algorithm, Bayesian optimization, and brute-force search is used to search for network architectures, and the performance of candidate network architectures is evaluated by minimizing the loss function. After several iterations, the optimal neural network is selected. and its parameters Specifically:

[0191] Define reward signals through reinforcement learning. Feedback on network performance to optimize search strategies is represented as The smaller the value, the better;

[0192] Evolutionary algorithms are used to cross-reference and mutate candidate network architectures in the search space to generate new candidate network architectures, resulting in a variety of possible candidate network architectures, represented as follows:

[0193] ;

[0194] in, and These represent two distinct candidate network architectures used for crossover and mutation, initially set manually. When a candidate network architecture contains only one vector, it becomes a specific element in the search space. This indicates the operation of crossing and mutating two dissimilar candidate network architectures;

[0195] By employing Bayesian optimization, a surrogate model is constructed to estimate the performance of each candidate network architecture and then the optimal neural network is selected. , is represented as:

[0196] ;

[0197] in, This represents the optimal neural network, i.e., the candidate network architecture with the largest reward signal value in the search space; This represents the operation of selecting the candidate network architecture with the largest reward signal value in the search space.

[0198] In step 2, by employing neural network architecture search technology and combining at least one of multiple optimization methods, the optimal neural network model can be effectively explored and constructed. The results of this stage will lay a solid foundation for subsequent training and inference, thereby ensuring efficient and accurate analysis and prediction of the IMU data preprocessed in step 1, and ensuring that the selected network architecture achieves the best balance in terms of accuracy, computational complexity, and training efficiency.

[0199] The following example illustrates the specific implementation of step 2:

[0200] ① First, manually define the search space. It includes basic modules such as , and And other necessary connection layers (such as linear layers) and convolutional layers Based on this, the candidate network architecture is defined as having three layers, with each layer corresponding to one module, as follows:

[0201] ;

[0202] At the same time, the parameter set of the layer is defined as:

[0203] .

[0204] ② Based on the limitations of the search space and the number of layers, candidate network architectures are generated through combination. Since the search space is relatively small at this point, reinforcement learning, evolutionary algorithms, and Bayesian optimization are no longer used; instead, a brute-force search is employed to directly find all possible network structures. The candidate network architecture consists of three layers, and the modules in the search space include… , and Then, all possible candidate networks can be constructed directly, including the following combinations:

[0205] ;

[0206] ;

[0207] ;

[0208] ;

[0209] ;

[0210] ;

[0211] These combinations constitute the constructed candidate network set. .

[0212] ③ To evaluate the performance of each candidate network architecture in the inertial navigation localization task, an appropriate loss function is selected to calculate the network error. Commonly used loss functions include cross-entropy loss. or mean square error loss The specific choice depends on the requirements of the inertial navigation localization task. For each candidate network architecture, if the task is a classification problem, then cross-entropy loss is selected. And calculate If the task is a regression problem, then choose the mean squared error loss. And calculate Obviously Sourced from candidate network set .

[0213] ④ Sort the candidate network architectures according to their loss values. The candidate network architecture with the smallest loss value is the optimal neural network, denoted as . ;

[0214] For example, if This network architecture performs best for inertial navigation localization tasks among all candidate network architectures;

[0215] The final output of the optimal neural network and the corresponding parameters It will be used in the subsequent training and inference stages of the model.

[0216] Step 3 above specifically refers to:

[0217] 3.1 Input Data Format

[0218] use The format will optimize the neural network. Input data The dimension is represented as:

[0219] ;

[0220] in, Represent the space of real numbers; This indicates the batch size, representing the number of samples input into the network at one time. This indicates the number of channels; in this scenario, it represents 6-axis preprocessed IMU data. This represents the sequence length, usually corresponding to the sampling rate, and indicates the number of data points for each sample in the time dimension; here, it can be understood as each batch being stacked. indivual Data, and then input it to the network all at once. Data from each batch.

[0221] 3.2 Training Process

[0222] During training, a training dataset consisting of a predetermined amount of preprocessed IMU data is used to optimize the neural network. Training; Optimal Neural Network The output is the pose estimate and its covariance, expressed as:

[0223] ;

[0224] in, and These represent the pose estimate and covariance of the network output during training, respectively. This represents the input data during the training process;

[0225] Note that network optimization is achieved through... This is achieved through changes in the neural network, therefore each round optimizes the neural network. During training, the parameters are updated, and the function that minimizes the loss function is ultimately saved. The smallest set of parameters is denoted as the final parameter. .

[0226] 3.3 Reasoning Process

[0227] During the inference process, the optimal trained neural network is used. Perform pose estimation; define input data For the IMU data to be inferred, the inference process is represented as follows:

[0228] ;

[0229] ;

[0230] in, and These represent the optimal neural network. The pose increment estimate and its covariance obtained through reasoning; It refers to the pose of a mobile device, which typically includes its position. and posture It consists of two parts: the former is the relative displacement of the mobile device in the world coordinate system, and the latter is the rotation matrix of the mobile device in the world coordinate system, representing the relative attitude transformation of the mobile device in the world coordinate system. and These represent the optimal neural network. The position increment covariance and attitude increment covariance are obtained through reasoning.

[0231] In step 3, the optimal neural network obtained in step 2 is used. The preprocessed IMU data obtained in step 1 Training and inference operations can output accurate pose estimates and their covariance, which can be used for subsequent orientation correction and VIO system fusion.

[0232] Step 4 above specifically refers to:

[0233] 4.1 Calculate the IMU yaw angle

[0234] Based on the inertial measurement unit at time... The collected device IMU data is combined with the time interval The numerical integration method is used to obtain the angular change of the mobile device in the world coordinate system (i.e., time). IMU yaw angle ), represented as:

[0235] ;

[0236] in, and Representing time respectively and the previous moment The yaw angle calculated from data collected by the inertial measurement unit; Indicates time The angular velocity component of the gyroscope in the z-axis direction.

[0237] 4.2 Calculate the prior yaw angle

[0238] Prior yaw angle This estimation of the user's possible direction of travel is based on prior information. This prior information can take many forms; any information known in advance falls into this category. For example, the orientation angle of a building (i.e., the prior orientation angle) provides a reliable geographical reference point, and this naturally occurring structural feature helps improve positioning accuracy. Simultaneously, the historical heading of the mobile device is also an important source of prior information, reflecting the movement trends and habits of the mobile device in specific environments, and is of significant value for predicting future movement directions. Furthermore, by establishing a regression model through smoothed historical trajectory data, regularities and patterns can be extracted from past data. In this embodiment, a dynamic covariance scheme is used to calculate the covariance of the prior direction. The difference between the IMU yaw angle and the prior yaw angle is:

[0239] ;

[0240] right Perform periodic processing to limit it to the range of [-180°, 180°];

[0241] By comparison The covariance of prior information compared to the preset difference threshold is expressed as:

[0242] ;

[0243] in, The difference threshold is represented by the value of the maximum covariance when the prior is unreliable. When the prior is reliable, the smaller covariance value is taken as... ;

[0244] Regarding the choice of covariance values:

[0245] When the prior information is reliable, the covariance should be small to assign higher weight to the prior information; this should be set based on practical experience. The value of the covariance should be small, such as between 0.1 and 1; when the prior is unreliable, the covariance should be larger to reduce the weight of the prior information, typically by [missing value]. The value is set to It can be 10 to 100 times greater than the actual value, and the specific value can be adjusted according to the actual test.

[0246] Furthermore, for the calculation of the prior trajectory direction, this invention adopts the assumption of pose continuity and utilizes a nonlinear regression model, specifically:

[0247] 4.2.1 Data Preparation

[0248] Collect the location and orientation data of the mobile device over a period of time, and record it as the raw trajectory data:

[0249] ;

[0250] in, , and They represent the first The horizontal coordinate, vertical coordinate, and yaw angle of the mobile device at each moment; This indicates the total duration of the period.

[0251] 4.2.2 Data Smoothing

[0252] Smoothing the raw trajectory data can reduce the impact of measurement errors and outliers. Common methods include moving average and Kalman filtering.

[0253] 4.2.3 Establishing a nonlinear regression model

[0254] Assuming the smoothed trajectory data satisfies a certain nonlinear relationship, a nonlinear regression model can be established as follows:

[0255] ;

[0256] in, Indicates the predicted direction angle; Represents a nonlinear function; and They represent , All are related to time Related functions; Indicates model parameters;

[0257] By minimizing the predicted direction angle Compared with actual direction angle The error is used to train a nonlinear regression model, which is expressed as:

[0258] ;

[0259] in, This represents the optimal parameter that minimizes the mean square error between the predicted and actual orientation angles. Indicates calculation from arrive The sum of the mean square errors is taken as the minimum value to obtain the optimal parameters;

[0260] 4.2.4 Calculation of Prior Trajectory Direction

[0261] Using a trained nonlinear regression model, new location data The prior yaw angle is calculated. , is represented as:

[0262] .

[0263] The above is merely an explanation of how to calculate the prior yaw angle based on a regression model. In this approach, when the building orientation is stored on a map, the building orientation angle obtained from the map information can be directly used during implementation. As a priori yaw angle, i.e. .

[0264] Step 5 above specifically refers to:

[0265] 5.1 VIO Pose Estimation

[0266] The pose estimation, i.e., the transformation matrix of the mobile device, is output through the VIO system. , is represented as:

[0267] ;

[0268] in, This represents the rotation matrix, which contains the rotation information output by the VIO system. This represents the translation matrix, which contains the displacement information output by the VIO system.

[0269] 5.2 VIO Error Covariance

[0270] The following uses ARCore's implementation as an example to illustrate the calculation of relevant parameters of the VIO system. Since ARCore can acquire image feature points, the number of feature points, their distribution uniformity, IMU stability, and pose change smoothness are used to estimate the covariance. These characteristics can also be used in other VIO solutions, making it a general solution for VIO. The covariance of the VIO system is divided into two parts: position and pose. The entire covariance estimation process is described in matrix form:

[0271] 5.2.1 VIO covariance matrix form

[0272] Covariance of VIO system output It should include the covariance of position and attitude, and the overall form is a A matrix representing the measurement uncertainties of three-dimensional position and three-dimensional attitude:

[0273] ;

[0274] in, This represents the position covariance of the VIO system output; This represents the attitude covariance output by the VIO system;

[0275] 5.2.2 Location Covariance Construction

[0276] Location covariance It can be decomposed into the covariance in the image plane and the covariance in the depth direction, expressed as:

[0277] ;

[0278] in, The positional covariance within the image plane (x, y axes) is based on the number of image feature points. and distribution uniformity To estimate, This means constructing the input data into a diagonal matrix. This represents the manually set adjustment coefficient for the number of feature points (recommended range: 50~200). This represents the uniformity adjustment coefficient set manually (recommended range: 0.1~1.0). The position covariance in the depth direction (z-axis direction) is defined as follows:

[0279] ;

[0280] in, and These represent manually set adjustment coefficients used to control the influence of VIO trajectory smoothness and IMU acceleration stability on the depth-direction position covariance (both values ​​range from 0.1 to 1.0 and can be adjusted according to specific circumstances). Indicates the positional smoothness of the VIO trajectory; This indicates the acceleration stability of the IMU in the out-of-plane direction;

[0281] Positional smoothness The standard deviation of the displacement increment between adjacent time steps is used for calculation, and it is expressed as:

[0282] ;

[0283] in, This indicates the time step calculated by the VIO system. arrive The relative displacement increment; The modulus representing the displacement increment; This represents the average value of the displacement increment within the sliding window; when the position change is smooth, Take the smaller value, and vice versa. Take the larger value;

[0284] Acceleration stability The acceleration is calculated using the standard deviation of the rate of change of acceleration between adjacent time steps, and is expressed as:

[0285] ;

[0286] in, The magnitude of the rate of change of acceleration between adjacent time steps. and These represent the time steps. and IMU acceleration measurements; The average value of the rate of change of acceleration within the sliding window is calculated using the following formula:

[0287] ;

[0288] Uniformity of distribution The calculation method is as follows:

[0289] By dividing the image captured by the mobile device's camera into equal parts A grid, and make Indicates the first line, number The number of feature points in the column grid is obtained. , This indicates the calculation of variance. The value of the feature point is inversely proportional to the uniformity of the feature point distribution;

[0290] 5.2.3 Attitude Covariance Construction

[0291] Posture covariance The smoothness of pose changes is estimated by the smoothness of the pose change. , is represented as:

[0292] ;

[0293] in, Indicates the incremental rotation angle; The average value representing the change in rotation angle within the sliding window is calculated using the following formula:

[0294] ;

[0295] The attitude covariance matrix is:

[0296] ;

[0297] in, The adjustment coefficient representing attitude smoothness is recommended to be in the range of 0.1 to 10.

[0298] 5.2.4 Complete VIO Covariance Matrix Representation

[0299] Combining the above components, we obtain the final VIO covariance matrix. .

[0300] In step 5 above, a VIO system was constructed by introducing visual information and combining it with the original IMU data. Through accurate state estimation and observation updates, the system can achieve high-precision position and pose estimation in dynamic and complex environments. This fusion strategy not only improves the robustness of localization but also lays a solid foundation for subsequent application scenarios.

[0301] In step 6 above, each information source is represented as a node using a factor graph and corresponding factors are introduced. During the optimization process, the weights of different information sources are adaptively adjusted. Specifically:

[0302] 6.1 State Definition

[0303] Define the state of the factor graph for:

[0304] ;

[0305] in, , and These represent the position, velocity, and attitude output by the factor map, respectively.

[0306] 6.2 IMU Factor

[0307] IMU factors are constructed using IMU data to establish motion constraints on the states between adjacent time points. When implementing GTSAM, ImuFactor will be used to implement IMU factors and process IMU data.

[0308] 6.3 Neural Network Measurement Factors

[0309] Using optimal neural networks The neural network measurement factor of the output pose increment estimate and its covariance construct factor graph;

[0310] 6.4 Prior Measurement Factors

[0311] Using prior yaw angle The prior measurement factors of its covariance construct factor plot, for the prior yaw angle The constraints are also calculated using the yaw angle error, but the three-dimensional representation of the attitude is fully preserved.

[0312] 6.5 VIO Measurement Factor

[0313] VIO measurement factors are constructed using the pose estimate and covariance output by the VIO system to create a factor map.

[0314] 6.6 Factor Plot Sliding Window Optimization

[0315] Sliding window optimization is an incremental optimization method for factor graphs, commonly used in real-time localization and attitude estimation. In factor graph fusion schemes, sliding window optimization primarily updates the current state estimate by continuously introducing new factors and state nodes while removing old nodes and factors. The following is a description of sliding window optimization in factor graph fusion schemes:

[0316] 6.6.1 State Node Definition

[0317] Assuming the current sliding window contains At that moment, at that moment The factor graph contains several state nodes and measurement factors;

[0318] time The state node is defined as follows:

[0319] ;

[0320] in, Indicates time Location; Indicates time speed; Indicates time The rotation matrix represents the three-dimensional attitude (including yaw, pitch and roll).

[0321] 6.6.2 Objective Function for Sliding Window Optimization

[0322] At any moment The objective function for sliding window optimization includes the weighted sum of errors of all measurement factors, and the optimization objective is expressed as:

[0323] ;

[0324] in, Representing state The IMU factor residual (i.e., the error between the state and the measurement factor); Representing state The residual of the neural network measurement factor; Representing state The prior measurement factor residuals; Representing state The VIO measurement factor residuals; each of the above factors has a corresponding covariance matrix. Used for weighting to control the influence of different measurements in optimization;

[0325] 6.6.3 Incremental Update Process

[0326] Sliding window optimization uses incremental updates at each time step. The specific execution process is as follows:

[0327] 6.6.3.1 Add new state nodes and measurement factors

[0328] In every new moment Add a new state node And new measurement factors, new state nodes Including location ,speed ,attitude Acceleration bias and angular velocity Bias, new measurement factors include new IMU factors, new neural network measurement factors, new prior measurement factors, and new VIO measurement factors;

[0329] 6.6.3.2 Remove old state nodes and measurement factors

[0330] To control computational complexity, the sliding window size is kept constant. It is necessary to remove state nodes and measurement factors outside the window, that is, remove the oldest state nodes. And its associated IMU factor, neural network measurement factor, prior measurement factor and VIO measurement factor;

[0331] 6.6.3.3 Incremental Optimization

[0332] This embodiment uses GTSAM, therefore the incremental optimization uses the iSAM2 algorithm to incrementally optimize the factor graph within the sliding window. iSAM2 uses the optimization result of the previous time step as the initial value to achieve fast updates. The goal is to obtain the time step by minimizing the sum of residuals. Optimization status The calculation formula is as follows:

[0333] ;

[0334] The factor graph optimization results can be calculated using the Gauss-Newton or Levenberg-Marquardt methods after each incremental update.

[0335] Step 7 above is actually IMU data correction, specifically:

[0336] 7.1 Sensor Data Correction

[0337] Assume at time... The length of the preprocessed IMU data input to the optimal neural network is , for time to The data are sequentially optimized using a factor graph sliding window to obtain state variables, which are then substituted into step 1.2 to calculate the pose propagation between time steps. Finally, the raw acceleration measured by the IMU is used in step 1.3. and original angular velocity The processed IMU data after correction at each time point is represented as follows:

[0338] ;

[0339] in, and These represent the corrected acceleration and angular velocity, respectively.

[0340] 7.2 Corrected IMU Input Data

[0341] The corrected IMU input data sequence is generated as follows:

[0342] ;

[0343] in, and Representing time respectively and Corrected IMU data;

[0344] 7.3 Neural Network Inference

[0345] Corrected IMU input data sequence Input to the optimal neural network Perform real-time pose inference and output the time step. The pose estimation is ,in and Representing time respectively By optimal neural network Output the position and orientation; then repeat steps 4 to 7 until finished.

[0346] The entire process of sliding window optimization and IMU data correction can be summarized as the following closed-loop process:

[0347] (I) Factor graph sliding window optimization, updating the latest pose and sensor noise parameters;

[0348] (II) Use the noise parameters obtained by factor map optimization to denoise the IMU data, and use the latest pose to correct the IMU data;

[0349] (III) Input the corrected IMU data into the neural network to obtain pose estimation;

[0350] (IV) The factor graph uses the output of the neural network as a new measurement to continue optimization.

[0351] That is, the entire process can be represented as:

[0352] → Reasoning → →iSAM2 Optimization→ →IMU data denoising and correction.

[0353] Through the above solution, the present invention has the following technical effects:

[0354] (1) Significantly improve positioning accuracy: By introducing Neural Architecture Search (NAS) and optimizing the neural network design, the system can more accurately infer displacement information and reduce positioning inaccuracy caused by IMU drift and accumulated errors;

[0355] (2) Enhance the robustness of orientation estimation: By combining the displacement inferred by the correction neural network with the heading provided by the prior information and the IMU integral heading, the heading weight can be dynamically adjusted, which effectively solves the problem of poor orientation correction effect of traditional methods in complex environments, thereby enhancing the robustness of orientation estimation.

[0356] (3) Improve adaptability in complex environments: The visual inertial odometry (VIO) system, which combines visual information, enables the system to maintain high positioning accuracy in most scenarios with good visual features, thereby improving the system's adaptability to complex indoor environments.

[0357] (4) Accurate state estimation: By fusing VIO, heading correction information and pose covariance inferred by neural network through expansion factor graph, the system can achieve more accurate state estimation, which improves the stability and reliability of the overall system.

[0358] (5) Effective handling of nonlinear scenarios: This invention can form a navigation strategy suitable for nonlinear paths by integrating behavioral patterns and prior knowledge, overcoming the limitation of traditional methods that can only handle linear paths.

[0359] In summary, by comprehensively utilizing neural networks, visual information, and factor graph technology, this invention significantly improves the performance of inertial navigation systems in terms of positioning accuracy, orientation estimation, and adaptability to complex environments, ensuring reliability and effectiveness in long-term navigation and positioning tasks.

[0360] Several key points of this invention are:

[0361] ① To address the problem that traditional methods often rely on direct integration of IMU data, which is susceptible to sensor drift and noise leading to accumulated errors, this invention introduces Neural Architecture Search (NAS). NAS technology is used to construct an optimal neural network architecture, enabling the network to be optimized for specific tasks and thus infer displacement information. Compared to traditional methods, the NAS-designed neural network can more effectively capture the time-series characteristics of IMU data, improving the accuracy of displacement estimation and reducing error accumulation.

[0362] ② To address the problem that existing technologies have limited heading correction effects on non-ideal paths, easily introduce direction errors, and affect positioning accuracy, this invention combines the displacement inferred by the neural network with prior information and the IMU integral heading for joint correction. By fusing multi-source information and dynamically adjusting the heading weights, the system's heading estimation accuracy in complex environments is enhanced.

[0363] ③ To address the problem that traditional IMU systems are prone to losing positioning accuracy due to accumulated errors, this invention combines visual information with IMU data to construct a visual inertial odometry (VIO) system. The VIO system enhances the accuracy of pose estimation through visual information and provides more reliable positioning results in most common scenarios.

[0364] ④ To address the problem that traditional filtering methods often fail to effectively integrate multi-source information, leading to information loss and reduced accuracy, this invention fuses VIO, heading correction information, and pose covariance inferred from a neural network through a factor graph, providing a dynamic state estimation method that can adjust position and attitude estimation based on real-time sensor input, significantly improving the robustness and accuracy of the overall system.

[0365] Through the above improvements, the present invention effectively enhances the positioning accuracy and robustness of inertial navigation systems in complex environments, overcomes many limitations of traditional technologies, and meets the requirements for high precision and high reliability in practical applications.

[0366] The present invention also discloses an electronic device, including a processor, a memory, and an application program; the application program is stored in the memory and configured to be executed by the processor using the methods described above.

[0367] The present invention also discloses a computer-readable storage medium storing a computer program; when the computer program is executed in a computer, it causes the computer to perform the above-described method. The storage medium may include various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0368] The above embodiments and figures are not intended to limit the product form and style of the present invention. Any appropriate changes or modifications made by those skilled in the art should be considered as not departing from the patent scope of the present invention.

Claims

1. A positioning method that dynamically fuses inertial navigation, vision, and prior information, characterized in that... include: Step 1: Collect device IMU data in real time from the user's mobile device. Then, through a preprocessing step, transform the device IMU data from the device coordinate system to the world coordinate system and remove gravity to obtain preprocessed IMU data. Wherein, IMU stands for Inertial Measurement Unit; Step 2: Apply neural network architecture search techniques to search and optimize the layer types, connection methods, and hyperparameters of the neural network in the search space to obtain the optimal neural network. ; Step 3: Construct a training dataset from a preset amount of preprocessed IMU data, and then use it to optimize the neural network. The optimal neural network after training is used to perform inference to obtain the pose increment estimate and its covariance. Step 4: Obtain the prior direction and its covariance based on prior information; Step 5: When visual information is available, the visual information is fused with the device IMU data from Step 1 to construct a VIO system. The VIO system outputs the pose estimate and its covariance in real time; where VIO stands for Visual Inertial Odometry. Step 6: Using the output results of steps 3 to 5 as measurement information, perform multi-source information fusion using factor graphs; Step 7: Update the parameters of the optimal neural network using the fusion results output by the factor graph in Step 6, and correct the device IMU data according to Step 1. Then, send the corrected results into the optimal neural network for inference, and repeat Steps 4 to 7 until the end.

2. The positioning method for dynamic fusion of inertial navigation, vision, and prior information as described in claim 1, characterized in that... The preprocessing step in step 1 specifically includes: 1.1 Device IMU data includes time. The collected accelerometer and gyroscope data are the raw acceleration. Original angular velocity ; 1.2 If the device IMU data is used to propagate the three-dimensional pose between consecutive time steps, the device IMU data will be updated in three-dimensional space as follows: Location update ; Speed ​​Update ; Posture update ; Among them, for time and the previous moment : and Indicates location, and Indicates speed, and Represents the rotation matrix. and These represent the initial acceleration and initial angular velocity, respectively. and These represent the error biases of the accelerometer and gyroscope, respectively. Represents the gravitational acceleration measured at rest. Indicates time and the previous moment The time interval; In Lie algebra Exponential mapping on; 1.3 Perform a world coordinate system transformation. The net acceleration in the world coordinate system is expressed as: ； Angular velocity in the world coordinate system is expressed as: ； The resulting preprocessed IMU data is represented as follows: 。 3. The positioning method for dynamic fusion of inertial navigation, vision, and prior information as described in claim 1, characterized in that... Step 2 specifically involves: 2.1 Defining the Search Space for: ； in, Indicates a convolutional layer; Indicates a linear layer; Indicates the activation function; Indicates the pooling layer; Indicates other layers; 2.2 Defining Candidate Network Architectures for: ； in, Indicates the first in the network Layer configuration; the configuration of each layer is represented as ,in Indicates the first in this layer One parameter; 2.3 Define the loss function to minimize The loss of the model is expressed by the following formula: ； in, Indicates the total number of samples; express The calculation method; Indicates the first The true label of each sample Indicates the first The model prediction output for each sample; 2.4 At least one of reinforcement learning, evolutionary algorithm, Bayesian optimization, and brute-force search is used to search for network architectures, and the performance of candidate network architectures is evaluated by minimizing the loss function; after several iterations, the optimal neural network is selected. and its parameters .

4. The positioning method for dynamic fusion of inertial navigation, vision, and prior information as described in claim 1, characterized in that... Step 3 specifically involves: 3.1 Optimal Neural Network Input data The dimension is represented as: ； in, Represents the space of real numbers; Indicates batch size; Indicates the number of channels; Indicates the sequence length; 3.2 During the training process, a training dataset is constructed from a predetermined amount of preprocessed IMU data, which is used to optimize the neural network. Training; Optimal Neural Network The output is the pose estimate and its covariance, expressed as: ； in, and These represent the pose estimate and covariance of the network output during training, respectively. This represents the input data during the training process; These represent the parameters of the optimal neural network. Each round of testing the optimal neural network During training, the parameters are updated, and the function that minimizes the loss function is ultimately saved. The smallest set of parameters is denoted as the final parameter. ; 3.3 During the inference process, the optimal trained neural network is used. Perform pose estimation; define input data For the IMU data to be inferred, the inference process is represented as follows: ； ； in, and These represent the optimal neural network. The pose increment estimate and its covariance obtained through reasoning; and These represent the optimal neural network. The position increment covariance and attitude increment covariance are obtained through reasoning.

5. The positioning method for dynamic fusion of inertial navigation, vision, and prior information as described in claim 1, characterized in that... Step 4 specifically involves: 4.1 Based on device IMU data and its time intervals The IMU yaw angle is calculated and expressed as: ； in, and Representing time respectively and the previous moment IMU yaw angle; Indicates time The angular velocity component of the gyroscope in the z-axis direction; 4.2 Obtain the prior yaw angle provided by prior information based on the known prior orientation angle or nonlinear regression model. The covariance of the prior direction is calculated using a dynamic covariance scheme. The difference between the IMU yaw angle and the prior yaw angle is: ； right Perform periodic processing to limit it to the range of [-180°, 180°]; By comparison The covariance of prior information compared to the preset difference threshold is expressed as: ； in, The difference threshold is represented by the value of the maximum covariance when the prior is unreliable. When the prior is reliable, the smaller covariance value is taken as... .

6. The positioning method for dynamic fusion of inertial navigation, vision, and prior information as described in claim 5, characterized in that... In step 4.2, the prior yaw angle obtained based on the prior information from the nonlinear regression model is described. Specifically: 4.2.1 Collect the location and orientation data of the mobile device over a period of time, and record it as the raw trajectory data: ； in, , and They represent the first The horizontal coordinate, vertical coordinate, and yaw angle of the mobile device at each moment; This indicates the total duration of the period. 4.2.2 Smooth the original trajectory data; 4.2.3 The smoothed trajectory data after establishing the nonlinear regression model is represented as follows: ； in, Indicates the predicted direction angle; Represents a nonlinear function; and They represent , All are related to time Related functions; Indicates model parameters; By minimizing the predicted direction angle Compared with actual direction angle The error is used to train a nonlinear regression model, which is expressed as: ； in, This represents the optimal parameter that minimizes the mean square error between the predicted and actual orientation angles. Indicates calculation from arrive The sum of the mean square errors is taken as the minimum value to obtain the optimal parameters; 4.2.4 Using the trained nonlinear regression model to process new location data The prior yaw angle is calculated. , is represented as: 。 7. The positioning method for dynamic fusion of inertial navigation, vision, and prior information as described in claim 1, characterized in that... Step 5 specifically involves: 5.1 Construct a VIO system and output the conversion matrix of the mobile device through the VIO system. , is represented as: ； in, This represents the rotation matrix output by the VIO system; This represents the translation matrix output by the VIO system; 5.2 Calculate the error covariance of the VIO system output, specifically: 5.2.1 Covariance of VIO System Output Represented as: ； in, This represents the position covariance of the VIO system output; This represents the attitude covariance output by the VIO system; 5.2.2 Location Covariance Represented as: ； in, Represents the positional covariance within the image plane, based on the number of image feature points. and distribution uniformity To estimate, This means constructing the input data into a diagonal matrix. This represents a manually set adjustment coefficient for the number of feature points. This represents a manually set uniformity adjustment coefficient; The position covariance in the depth direction is defined as follows: ； in, and These represent manually set adjustment coefficients, used to control the influence of VIO trajectory smoothness and IMU acceleration stability on the depth-direction position covariance, respectively. Indicates the positional smoothness of the VIO trajectory; This indicates the acceleration stability of the IMU in the out-of-plane direction; Positional smoothness The standard deviation of the displacement increment between adjacent time steps is used for calculation, and it is expressed as: ； in, This indicates the time step calculated by the VIO system. arrive The relative displacement increment; The modulus representing the displacement increment; This represents the average value of the displacement increment within the sliding window; when the position change is smooth, Take the smaller value, otherwise Take the larger value; Acceleration stability The acceleration is calculated using the standard deviation of the rate of change of acceleration between adjacent time steps, and is expressed as: ； in, The magnitude of the rate of change of acceleration between adjacent time steps. and These represent the time steps. and IMU acceleration measurements; The average value of the rate of change of acceleration within the sliding window is calculated using the following formula: ； Uniformity of distribution The calculation method is as follows: By dividing the image captured by the mobile device's camera into equal parts A grid, and make Indicates the first line, number The number of feature points in the column grid is obtained. , This indicates the calculation of variance. The value of the feature point is inversely proportional to the uniformity of the feature point distribution; 5.2.3 Attitude Covariance The smoothness of pose changes is estimated by the smoothness of the pose change. , is represented as: ； in, Indicates the incremental rotation angle; The average value representing the change in rotation angle within the sliding window is calculated using the following formula: ； The attitude covariance matrix is: ； in, An adjustment coefficient representing attitude smoothness; 5.2.4 Combining the above components, the final VIO covariance matrix is ​​obtained. .

8. The positioning method for dynamic fusion of inertial navigation, vision, and prior information as described in claim 1, characterized in that... Step 6 specifically involves: 6.1 Defining the state of the factor graph for: ； in, , and These represent the position, velocity, and attitude output by the factor map, respectively. 6.2 Construct IMU factors for a factor graph using device IMU data to establish motion constraints on the state between adjacent time points; 6.3 Utilizing Optimal Neural Networks The neural network measurement factor of the output pose increment estimate and its covariance construct factor graph; 6.4 Utilizing Prior Yaw Angle The prior measurement factors of its covariance construct factor plot; 6.5 Constructing VIO measurement factors from a factor map using pose estimation and covariance output by the VIO system; 6.6 Incremental optimization of the factor graph is performed using sliding window optimization.

9. The positioning method for dynamic fusion of inertial navigation, vision, and prior information as described in claim 8, characterized in that... Step 6.6 specifically involves: 6.6.1 Assume the current sliding window contains At that moment, at that moment The factor graph contains several state nodes and measurement factors; time The state node is defined as follows: ； in, Indicates time Location; Indicates time speed; Indicates time rotation matrix; 6.6.2 At time The objective function for sliding window optimization includes the weighted sum of errors of all measurement factors, and the optimization objective is expressed as: ； in, Representing state IMU factor residuals; Representing state The residual of the neural network measurement factor; Representing state The prior measurement factor residuals; Representing state VIO measurement factor residuals; Each factor has a corresponding covariance matrix. Used for weighting; 6.6.3 Sliding window optimization uses incremental updates at each time step. The specific execution process is as follows: 6.6.3.1 At every new moment Add a new state node And new measurement factors, new state nodes Including location ,speed ,attitude Acceleration bias and angular velocity Bias, new measurement factors include new IMU factors, new neural network measurement factors, new prior measurement factors, and new VIO measurement factors; 6.6.3.2 To control computational complexity, the sliding window size is kept constant. It is necessary to remove state nodes and measurement factors outside the window, that is, remove the oldest state nodes. And its associated IMU factor, neural network measurement factor, prior measurement factor and VIO measurement factor; 6.6.3.3 The iSAM2 algorithm is used to incrementally optimize the factor graph within the sliding window. iSAM2 uses the optimization result from the previous time step as the initial value to achieve fast updates. The goal is to minimize the sum of residuals to obtain the time step. Optimization status The calculation formula is as follows: 。 10. The positioning method for dynamic fusion of inertial navigation, vision, and prior information as described in claim 2, characterized in that... Step 7 specifically involves: 7.1 Assume that at time... The length of the preprocessed IMU data input to the optimal neural network is , for time to The data are sequentially optimized using a factor graph sliding window to obtain state variables, which are then substituted into step 1.2 to calculate the pose propagation between time steps. Finally, the raw acceleration measured by the IMU is used in step 1.

3. and original angular velocity The corrected IMU data obtained after processing are represented as follows: ； in, and These represent the corrected acceleration and angular velocity, respectively. 7.2 The corrected IMU input data sequence is generated as follows: ； in, and Representing time respectively and Corrected IMU data; 7.3 Convert the corrected IMU input data sequence Input to the optimal neural network Perform real-time pose inference and output the time step. The pose estimation is ,in and Representing time respectively By optimal neural network Output the position and orientation; then repeat steps 4 to 7 until finished.

Images