Positioning method for dynamically fusing inertial navigation, vision and prior information

Abstract

The invention discloses an inertial navigation, vision and prior information dynamic fusion positioning method, and aims to improve the precision and robustness of position and pose estimation in dynamic and complex environments, and the method comprises the following steps: 1, collecting equipment IMU data in real time, converting the equipment IMU data into a world coordinate system, and removing gravity to obtain preprocessed IMU data; 2, applying a neural network architecture search technology to obtain an optimal neural network; 3, forming a training data set by the preprocessed IMU data to train an optimal neural network, and reasoning pose increment estimation and covariance; 4, acquiring a prior direction and a covariance thereof based on the prior information; 5, visual information is introduced to be fused with IMU data of equipment, a VIO system is constructed, and pose estimation and covariance are output through the VIO system; 6, performing multi-source information fusion by taking output results of the steps 3-5 as measurement information of the factor graph; and 7, updating parameters of the optimal neural network by using a factor graph fusion result, correcting the IMU data of the equipment according to the step 1, sending the data into the optimal neural network for reasoning, and repeating the steps 4 to 7.

Description

Technical Field

[0001] The present invention belongs to the field of mobile positioning and navigation technology, specifically covering related fields such as visual inertial odometry (VIO), sensor fusion technology, factor graphs, and neural network architecture search (NAS), combining inertial measurement unit (IMU) data with visual information, and particularly refers to a positioning method, electronic device, and storage medium that dynamically fuses inertial navigation, vision, and prior information, aiming to improve the accuracy and robustness of position and pose estimation in dynamic and complex environments. Background Art

[0002] Technological developments in the field of mobile positioning and navigation have mainly focused on improving positioning accuracy and system robustness, and a variety of innovative methods have emerged.

[0003] The traditional inertial navigation system (SINS) obtains position information through double integration of IMU measurements, while methods such as pedestrian dead reckoning (PDR) and zero velocity update (ZUPT) are widely used to reduce the accumulation of errors.

[0004] In recent years, the heuristic drift elimination (HDE) algorithm and its improved version iHDE have significantly improved the positioning accuracy in complex environments by using the dominant direction in the building for direction correction. With the rapid development of deep learning, data-driven methods have gradually become mainstream. For example, the RONIN (Robust Neural Inertial Navigation in the Wild) framework uses deep learning models to capture the time series characteristics of IMU data and enhances robustness in real environments; TLIO (Tight Learned Inertial Odometry) combines neural networks with extended Kalman filters to further improve the accuracy of displacement estimation. CTIN (Robust Contextual Transformer Network for Inertial Navigation) introduces an attention mechanism to enhance focus on key features. For applications on mobile devices, IMUNet (Inertial Measurement Unit Network) uses a lightweight network architecture designed to optimize performance and reduce computational complexity.

[0005] In summary, the field is moving towards a more intelligent, precise, and adaptable direction, using deep learning and sensor fusion technology to continuously promote the advancement of inertial navigation systems.

[0006] Existing inertial navigation technology has several significant defects: First, the traditional inertial navigation system is significantly affected by sensor drift and cumulative error, especially in complex indoor environments, where positioning accuracy is often seriously affected; although technologies such as the iHDE algorithm can mitigate the cumulative error caused by sensor drift to a certain extent, the correction effect on non-ideal paths (such as curved paths) is still limited, which may lead to additional directional errors and further affect positioning accuracy. In addition, although data-driven neural network methods can alleviate the problem of trajectory error accumulation to some extent, they fail to effectively solve the problem of cumulative heading offset. This defect limits its application effect in long-term navigation tasks, making the adaptability and stability of existing technologies in complex and dynamic environments challenged.

[0007] Therefore, how to further improve navigation accuracy and robustness, especially its effectiveness in complex environments, remains a key challenge in current technological development. Summary of the invention

[0008] The purpose of the present invention is to provide a positioning method that dynamically integrates inertial navigation, vision and prior information to solve the problems existing in the prior art and to improve the accuracy and robustness of position and posture estimation in dynamic and complex environments.

[0009] In order to achieve the above object, the solution of the present invention is: A positioning method for dynamically integrating inertial navigation, vision and prior information, comprising: Step 1: Collect device IMU data in real time from the mobile device carried by the user, and then convert the device IMU data in the device coordinate system to the world coordinate system and remove gravity through the preprocessing step to obtain preprocessed IMU data. Among them, IMU stands for inertial measurement unit; Step 2: Apply neural network architecture search technology to search and optimize the layer type, connection method and hyperparameters of the neural network in the search space to obtain the optimal neural network ; Step 3: A training data set is formed by a preset number of pre-processed IMU data, and the optimal neural network is trained. Perform training and use the trained optimal neural network for inference to obtain the pose increment estimate and its covariance; Step 4: Obtain the prior direction and its covariance based on the prior information; Step 5: When visual information is available, the visual information is introduced and fused with the IMU data of the device in step 1 to build a VIO system, which outputs the pose estimate and its covariance in real time; where VIO stands for visual inertial odometry; Step 6: Using the output results of steps 3 to 5 as measurement information, a factor graph is used to perform multi-source information fusion; Step 7: Use the fusion results of the factor graph output in step 6 to update the parameters of the optimal neural network, and correct the device IMU data according to step 1, and then send the corrected results to the optimal neural network for reasoning, and then repeat steps 4 to 7 until the end.

[0010] The pre-processing steps in step 1 are specifically: 1.1 Device IMU data includes time The collected accelerometer and gyroscope data are the raw acceleration , original angular velocity ; 1.2 Use the device IMU data to propagate the 3D posture between consecutive time steps. The device IMU data is updated in 3D space as follows: Location Updates ; Speed ​​Update ; Posture Update ; Among them, for the time and the previous moment : and Indicates location, and Indicates speed, and represents the rotation matrix, and Respectively represent the original acceleration and original angular velocity; and Represent the error bias of accelerometer and gyroscope respectively; Represents the gravitational acceleration measured at rest Indicates time and the previous moment time interval; Representation in Lie algebra Exponential mapping on ; 1.3 Perform world coordinate system conversion. The net acceleration in the world coordinate system is expressed as: ; The angular velocity in the world coordinate system is expressed as: ; The pre-processed IMU data is expressed as follows: .

[0011] The step 2 specifically includes: 2.1 Defining the Search Space for: ; in, represents the convolutional layer; represents a linear layer; represents the activation function; represents the pooling layer; Indicates other layers; 2.2 Defining candidate network architectures for: ; in, Indicates the network The configuration of each layer is represented by ,in Indicates the first parameters; 2.3 Defining the Minimization Loss Function Represents the loss of the model, and its formula is: ; in, Indicates the total number of samples; express Calculation method of Indicates The true labels of samples, Indicates The model prediction output corresponding to the samples; 2.4 Use at least one of reinforcement learning, evolutionary algorithms, Bayesian optimization, and brute force search to search for network architectures, and use the minimization loss function to evaluate the performance of candidate network architectures; after several iterations, select the optimal neural network and its parameters .

[0012] The step 3 is specifically: 3.1 The optimal neural network Input data The dimensions are expressed as: ; in, represents the real number space; Indicates the batch size; Indicates the number of channels; Indicates the length of the sequence; 3.2 During the training process, a training data set is formed by a preset number of pre-processed IMU data to optimize the neural network. Training; optimal neural network The output of is the pose estimate and its covariance, expressed as: ; in, and They represent the pose estimation and covariance of the network output during the training process respectively; Represents the input data during training; represents the parameters of the optimal neural network; Each round of the optimal neural network During training, its parameters are updated and finally the function that minimizes the loss is saved. The smallest set of parameters is recorded as the final parameters ; 3.3 Use the best trained neural network during inference Perform pose estimation; define input data is the IMU data to be inferred, and the inference process is expressed as: ; ; in, and Represent the optimal neural network The inferred pose increment estimate and its covariance; and Represent the optimal neural network The position increment covariance and attitude increment covariance are obtained by inference.

[0013] The step 4 is specifically: 4.1 Based on device IMU data and its time interval Calculate the IMU yaw angle, expressed as: ; in, and Respectively indicate time and the previous moment IMU yaw angle; Indicates time The angular velocity component of the gyroscope in the z-axis direction; 4.2 Obtaining the prior yaw angle provided by the prior information based on the known prior heading angle or nonlinear regression model ; The covariance of the prior direction is calculated using the dynamic covariance scheme. The difference between the IMU yaw angle and the prior yaw angle is: ; right Perform periodic processing and limit it to the range of [-180°, 180°]; By comparison The covariance of the prior information is determined by the preset difference threshold, which is expressed as: ; in, represents the difference threshold, then the larger covariance value when the prior is unreliable is , when the prior is credible, the smaller covariance is .

[0014] Preferably, in step 4.2, the prior yaw angle provided by the prior information is obtained based on the nonlinear regression model. , specifically: 4.2.1 Collect the location and direction data of the mobile device over a period of time and record it as the original trajectory data: ; in, , and Respectively represent The horizontal coordinate, vertical coordinate and yaw angle of the mobile device at each moment; Indicates the total duration of the period; 4.2.2 Smoothing the original trajectory data; 4.2.3 Establish the trajectory data after nonlinear regression model smoothing, expressed as: ; in, Represents the predicted direction angle; represents a nonlinear function; and Respectively , It's all about time Related functions; represents the model parameters; By minimizing the predicted direction angle The actual direction angle The error is used to train the nonlinear regression model, which is expressed as: ; in, It represents the optimal parameter that minimizes the mean square error between the predicted direction angle and the actual direction angle; Indicates the calculation from arrive The sum of the mean square errors and the minimum value are used to obtain the optimal parameters; 4.2.4 Using the trained nonlinear regression model to predict new position data Calculate the prior yaw angle , expressed as: .

[0015] The step 5 is specifically: 5.1 Build a VIO system and output the transformation matrix of the mobile device through the VIO system , expressed as: ; in, Represents the rotation matrix output by the VIO system; Represents the translation matrix output by the VIO system; 5.2 Calculate the error covariance of the VIO system output, specifically: 5.2.1 Covariance of VIO System Output It is expressed as: ; in, represents the position covariance of the VIO system output; represents the attitude covariance output by the VIO system; 5.2.2 Position Covariance It is expressed as: ; in, Represents the position covariance in the image plane, based on the number of image feature points and distribution uniformity To estimate, Indicates that the input data is constructed into a diagonal matrix. Indicates the artificially set feature point quantity adjustment coefficient, Indicates the artificially set uniformity adjustment coefficient; represents the position covariance in the depth direction, which is defined as follows: ; in, and It represents the artificially set adjustment coefficient, which is used to control the influence of VIO trajectory smoothness and IMU acceleration stability on the depth direction position covariance; Indicates the position smoothness of the VIO trajectory; Represents the acceleration stability of the IMU in the out-of-plane direction; Position smoothness It is calculated by the standard deviation of the displacement increments between adjacent time steps and is expressed as: ; in, represents the time step calculated by the VIO system arrive The relative displacement increment; represents the modulus of displacement increment; Represents the average value of the displacement increment within the sliding window; when the position changes smoothly, Take the smaller value, otherwise Take the larger value; Acceleration stability It is calculated by the standard deviation of the rate of change of acceleration between adjacent time steps and is expressed as: ; in, represents the modulus of the rate of change of acceleration in adjacent time steps, and Respectively represent the time step and IMU acceleration measurements; It represents the average value of the acceleration change rate within the sliding window, and its calculation formula is: ; Distribution uniformity The calculation method is: By dividing the image captured by the mobile device's camera into grids, and let Indicates Row, No. The number of feature points in the column grid is obtained , represents the calculation of variance, The value of is inversely proportional to the uniformity of the feature point distribution; 5.2.3 Attitude Covariance It is estimated by the smoothness of posture change. The smoothness measure of posture change is , expressed as: ; in, Indicates the incremental rotation angle; Represents the average value of the rotation angle change within the sliding window, and its calculation formula is: ; The attitude covariance matrix is: ; in, The adjustment coefficient representing the smoothness of the posture; 5.2.4 Combining the above components, we get the final VIO covariance matrix .

[0016] The step 6 specifically includes: 6.1 Defining the state of the factor graph for: ; in, , and Respectively represent the position, velocity and posture output by the factor graph; 6.2 Use the IMU data of the device to construct the IMU factors of the factor graph to establish motion constraints on the states between adjacent moments; 6.3 Using the Optimal Neural Network The output pose increment estimate and its covariance construct the neural network measurement factor of the factor graph; 6.4 Using the Prior Yaw Angle and its covariance to construct the prior measurement factor of the factor map; prior yaw angle The error term is: ; in, represents the error term of the prior yaw angle; Indicates the extraction of the yaw angle in the rotation matrix; 6.5 Use the pose estimate and covariance output by the VIO system to construct the VIO measurement factor of the factor graph; 6.6 Incremental optimization of factor graphs using sliding window optimization; 6.7 In real-time data processing, the device IMU data is corrected using the latest optimization results of the factor graph.

[0017] Preferably, the step 6.6 specifically includes: 6.6.1 Assume that the current sliding window contains a moment, at a moment ,The factor graph contains several state nodes and measurement factors; time The state node is defined as: ; in, Indicates time location; Indicates time speed; Indicates time The rotation matrix of 6.6.2 At time , the objective function of sliding window optimization contains the weighted error sum of all measurement factors, and the optimization objective is expressed as: ; in, Indicates status IMU factor residual; Indicates status The neural network measurement factor residuals; Indicates status The prior measurement factor residuals of ; Indicates status VIO measurement factor residuals; each factor has a corresponding covariance matrix For weighting; 6.6.3 Sliding Window Optimization by Incremental Update at Each Moment Execution, the specific process is: 6.6.3.1 At each new moment , add a new state node and new measurement factors, new state nodes Include Location ,speed ,attitude , acceleration bias and angular velocity Bias, the new measurement factors include new IMU factors, new neural network measurement factors, new prior measurement factors and new VIO measurement factors; 6.6.3.2 To control the computational complexity, keep the sliding window size fixed to , it is necessary to remove the state nodes and measurement factors outside the window, that is, remove the earliest state node and its associated IMU factors, neural network measurement factors, prior measurement factors, and VIO measurement factors; 6.6.3.3 Use the iSAM2 algorithm to incrementally optimize the factor graph in the sliding window. iSAM2 uses the optimization result of the previous moment as the initial value to achieve fast update. The goal is to minimize the residual sum to achieve the moment Optimization status , and its calculation formula is: .

[0018] Preferably, the step 7 specifically includes: 7.1 Assume that at time The length of the pre-processed IMU data input into the optimal neural network is , for the moment to The data is optimized by factor graph sliding window to obtain state variables and substituted into step 1.2 to calculate the pose propagation between each time step, and then the original acceleration measured by IMU is calculated by step 1.3. and the original angular velocity The IMU data corrected at each moment is expressed as: ; in, and Respectively represent the corrected acceleration and angular velocity; 7.2 Generate the corrected IMU input data sequence as: ; in, and Respectively indicate time and Corrected IMU data; 7.3 Input the calibrated IMU data sequence Input to the optimal neural network Perform real-time pose reasoning in the output time The pose estimate is ,in and Respectively indicate time By optimal neural network Output the position and posture; then repeat steps 4 to 7 until the end.

[0019] After adopting the above technical solution, the present invention has the following technical effects: (1) By introducing Neural Network Architecture Search (NAS), the neural network design is optimized, enabling the system to infer displacement information more accurately and reduce positioning inaccuracy caused by IMU drift and accumulated errors; (2) The displacement inferred by the neural network is jointly corrected with the heading provided by the prior information and the IMU integrated heading, which can dynamically adjust the heading weight, effectively solving the problem that the traditional method has poor direction correction effect in complex environments, thereby enhancing the robustness of direction estimation; (3) The visual inertial odometry (VIO) system combined with visual information enables the system to maintain high positioning accuracy in most scenes with good visual features, improving the system's adaptability to complex indoor environments; (4) By fusing VIO, heading correction information, and the pose covariance inferred by the neural network through the factor graph, the system can achieve more accurate state estimation, thereby improving the stability and reliability of the overall system. (5) The present invention can form a navigation strategy suitable for nonlinear paths by integrating behavioral patterns and prior knowledge, overcoming the limitation that traditional methods can only process linear paths. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is an overall system block diagram of a specific embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0022] See also Figure 1 The present invention discloses a positioning method for dynamically integrating inertial navigation, vision and prior information, comprising: Step 1: Data collection and preprocessing Inertial measurement unit (IMU) data is collected in real time from a mobile device (generally a mobile phone, tablet, etc.) carried by the user, recorded as device IMU data, including data from an accelerometer and a gyroscope, that is, current acceleration values ​​and angular velocity values; the device IMU data captures the motion information of the user when carrying the mobile device in different ways (such as hand-held, pocket-carried, or worn on the chest, etc.), ensuring that the present invention can be applied to various scenarios and has stronger adaptability; since the reference system of the original data is the device coordinate system, in order to ensure that the data can be uniformly processed in the world coordinate system, step 1 also needs to convert the device IMU data of the device coordinate system to the world coordinate system through a preprocessing step and remove the influence of gravity, so as to obtain preprocessed IMU data for subsequent analysis and processing steps ; Step 2: Neural Network Search Apply Neural Architecture Search (NAS) technology to explore and optimize the layer types, connection methods and hyperparameters of the neural network in the search space to obtain the pre-processed IMU data suitable for processing in the subsequent steps. The optimal neural network , ensuring that the obtained neural network model has excellent ability to predict position and posture information; Step 3: Neural Network Inference The training data set is composed of a preset number of pre-processed IMU data, and the optimal neural network Perform training and use the trained optimal neural network for inference to obtain the pose increment estimate and its covariance; Step 4: Direction Correction Obtaining prior directions and their covariances based on prior information; Step 5: VIO system When visual information is available, the visual information is introduced and deeply integrated with the device IMU data in step 1 (i.e., the IMU data before preprocessing) to build a visual-inertial odometry (VIO) system, which aims to significantly improve the accuracy and reliability of pose estimation. The VIO system outputs high-precision pose estimation and its covariance in real time. The VIO system can maintain stable tracking performance in complex and changing environments, thus providing a solid foundation for subsequent pose estimation. Step 6: Multi-source information fusion Using the output results of steps 3 to 5 as measurement information, multi-source information fusion is performed using factor graphs; Step 7: Use the fusion results of the factor graph output in step 6 to update the parameters of the optimal neural network, and correct the device IMU data according to step 1, and then send the corrected results to the optimal neural network for reasoning, and then repeat steps 4 to 7 until the end.

[0023] Specific embodiments of the present invention are shown below.

[0024] The preprocessing steps in step 1 above are specifically: 1.1 Raw data collection Define the inertial measurement unit at time The collected accelerometer and gyroscope data are the raw acceleration , original angular velocity , both are values ​​in the device coordinate system, and after subsequent processing steps, unnecessary components and noise can be removed to more accurately describe the user's actual motion state.

[0025] 1.2 Status Update Using the device IMU data to propagate the 3D posture between consecutive time steps, the device IMU data is updated in 3D space as follows: Location Updates ; Speed ​​Update ; Posture update (expressed using Lie group-Lie algebra) ; Among them, for the time and the previous moment : and Indicates location, and Indicates speed, and Represents the rotation matrix (representing the rotation posture from the device coordinate system to the world coordinate system), and Respectively represent the original acceleration and original angular velocity; and They represent the error bias of the accelerometer and gyroscope respectively, which can be obtained from the equipment manufacturer or calculated by yourself; Represents the gravitational acceleration measured at rest Indicates time and the previous moment time interval; Representation in Lie algebra Exponential mapping on ; 1.3 World Coordinate System Conversion Using the rotation matrix The acceleration and angular velocity data are converted to the world coordinate system. The net acceleration in the world coordinate system is expressed as , the angular velocity in the world coordinate system is expressed as ; Thus, the motion description of all devices can be unified into the world coordinate system to facilitate the collaborative calculation of multiple sensors; finally, the pre-processed IMU data is obtained .

[0026] As described above, in step 1, by collecting and preprocessing the IMU data, the present invention can effectively extract the net acceleration and angular velocity information from the raw sensor data, and convert it into a standard world coordinate system to obtain preprocessed IMU data for subsequent analysis and processing steps.

[0027] The above step 2 is specifically: 2.1 Defining the Search Space Defining the search space for: ; in, Represents convolutional layers, such as standard convolution, depth-wise separable convolution, etc. represents a linear layer; represents the activation function, such as , , wait; Represents pooling layers, such as maximum pooling, average pooling, etc.; Represents other layers, such as Fourier transform, single-head attention, multi-head attention, etc.; usually, in order to reduce the amount of search calculation, we should use existing experience as much as possible and screen necessary elements to reduce the search space.

[0028] 2.2 Representation of Network Architecture Define candidate network architectures via a structured vector for: ; in, Indicates the network Layer configuration, including type, parameters, etc.; the configuration of each layer is represented as a set ,in Indicates the first A parameter, usually a specific value.

[0029] 2.3 Performance Evaluation The performance of each candidate network architecture is evaluated for subsequent optimization, specifically: Evaluate each candidate network architecture When minimizing the loss function Represents the loss of the model, such as cross entropy loss or mean squared error loss: ; in, Indicates the total number of samples; express The calculation method of , such as calculating cross entropy or mean square error, results in a value; Indicates The true labels of samples, Indicates The model prediction output corresponding to the samples; 2.4 Optimization Algorithm Use at least one of reinforcement learning, evolutionary algorithms, Bayesian optimization, and brute force search to search for network architectures, and use the minimization loss function to evaluate the performance of candidate network architectures; after several iterations, select the optimal neural network and its parameters ; Specifically: Defining reward signals through reinforcement learning Feedback network performance to optimize the search strategy, expressed as , the smaller the value, the better; Through the evolutionary algorithm, the candidate network architectures in the search space are cross-linked and mutated to generate new candidate network architectures to obtain various possible candidate network architectures, which are expressed as: ; in, and Represents two different candidate network architectures for crossover and mutation, which are initially set manually. When the candidate network architecture contains only one vector, it becomes a specific element in the search space. Indicates the operation of crossing and mutating two different candidate network architectures; Through Bayesian optimization, a proxy model is constructed to estimate the performance of each candidate network architecture and select the optimal neural network. , expressed as: ; in, represents the optimal neural network, that is, the candidate network architecture with the largest reward signal value in the search space; Represents the operation of screening the candidate network architecture with the largest reward signal value in the search space.

[0030] As mentioned above, step 2 can effectively explore and build a neural network model with optimal performance by adopting neural network architecture search technology combined with at least one of multiple optimization methods. The results of this stage will lay a solid foundation for subsequent training and reasoning, thereby ensuring efficient and accurate analysis and prediction of the IMU data preprocessed in step 1, and ensuring that the selected network architecture achieves the best balance in accuracy, computational complexity, and training efficiency.

[0031] The following example illustrates the specific implementation of step 2: ① First, manually define the search space , which includes basic modules such as , and And other necessary connection layers (such as linear layers and convolutional layers ); On this basis, the candidate network architecture is defined as three layers, each layer corresponds to a module, expressed as: ; At the same time, the parameter set of the layer is defined as: .

[0032] ② According to the search space and layer number restrictions, the candidate network architecture is generated by combination. At this time, the search space is small, so the reinforcement learning, evolutionary algorithm and Bayesian optimization mentioned above are no longer used. Instead, brute force search is used to directly search out all possible network structures; the candidate network architecture consists of three layers, and the modules in the search space include , and , then all possible candidate networks are directly constructed, including the following combinations: ; ; ; ; ; ; These combinations are the candidate network sets constructed .

[0033] ③ In order to evaluate the performance of each candidate network architecture in the inertial navigation positioning task, select a suitable loss function to calculate the network error. Commonly used loss functions include cross entropy loss or mean square error loss The specific choice depends on the requirements of the inertial navigation positioning task. For each candidate network architecture, if the task is a classification problem, the cross entropy loss is selected. And calculate If the task is a regression problem, choose the mean square error loss And calculate , obviously From the candidate network set .

[0034] ④ Sort the candidate network architectures according to the loss value. The candidate network architecture with the smallest loss value is the optimal neural network, denoted as ; For example, if , this network architecture performs best for inertial navigation positioning tasks among all candidate network architectures; The optimal neural network output at the end And the corresponding parameters It will be used in the subsequent training and reasoning stages of the model.

[0035] The above step 3 is specifically: 3.1 Input Data Format use The optimal neural network is in the format of Input data The dimensions are expressed as: ; in, represents the real number space; Represents the batch size, which represents the number of samples input to the network at one time; Indicates the number of channels, which in this scenario indicates 6-axis pre-processed IMU data; Indicates the sequence length, usually corresponding to the sampling rate, representing the number of data points for each sample in the time dimension; here it can be understood that each batch is stacked indivual Data, and then input it into the network at one time batches of data.

[0036] 3.2 Training Process During the training process, a training data set is formed by a preset number of pre-processed IMU data to optimize the neural network. Training; optimal neural network The output of is the pose estimate and its covariance, expressed as: ; in, and They represent the pose estimation and covariance of the network output during the training process respectively; Represents the input data during training; Note that in this process, the network is optimized by Therefore, each round of the optimal neural network During training, its parameters are updated and finally the function that minimizes the loss is saved. The smallest set of parameters is recorded as the final parameters .

[0037] 3.3 Reasoning Process During inference, the optimal neural network after training is used Perform pose estimation; define input data is the IMU data to be inferred, and the inference process is expressed as: ; ; in, and Represent the optimal neural network The inferred pose increment estimate and its covariance; is the pose of the mobile device, usually including the position and posture There are two parts. The former is the relative displacement of the mobile device in the world coordinate system, and the latter is the rotation matrix of the mobile device in the world coordinate system, which represents the relative posture transformation of the mobile device in the world coordinate system. and Represent the optimal neural network The position increment covariance and attitude increment covariance are obtained by inference.

[0038] Above, step 3 adopts the optimal neural network obtained in step 2 Preprocess the IMU data obtained in step 1 Training and inference operations can output accurate pose estimation and its covariance for subsequent direction correction and VIO system fusion.

[0039] The above step 4 is specifically: 4.1 Calculating IMU yaw angle

[0040] Based on the inertial measurement unit at time The collected device IMU data is combined with the time interval , the numerical integration method is used to obtain the angle change of the mobile device in the world coordinate system (i.e., the time IMU yaw angle ), expressed as: ; in, and Respectively indicate time and the previous moment The yaw angle calculated from the data collected by the inertial measurement unit; Indicates time The angular velocity component of the gyroscope in the z-axis direction.

[0041] 4.2 Calculating the Prior Yaw Angle

[0042] Prior yaw angle It is the possible direction of travel of the user estimated based on prior information, where there are multiple modes of prior information, and any pre-known information belongs to this category, for example: the orientation angle of the building (i.e., the prior orientation angle) can provide a reliable geographic reference point. This naturally existing structural feature helps to improve the accuracy of positioning; at the same time, the historical heading of the mobile device is also one of the important sources of prior information, which reflects the movement trend and habits of the mobile device in a specific environment, and is of great value for predicting future movement directions; in addition, by smoothing the historical trajectory data to establish a regression model, regularities and patterns can be extracted from past data. In this embodiment, the dynamic covariance scheme is used to calculate the covariance of the prior direction, and the difference between the IMU yaw angle and the prior yaw angle is: ; right Perform periodic processing and limit it to the range of [-180°, 180°]; By comparison The covariance of the prior information is determined by the preset difference threshold, which is expressed as: ; in, represents the difference threshold, then the larger covariance value when the prior is unreliable is , when the prior is credible, the smaller covariance is ; Regarding the choice of covariance value: When the prior is credible, the covariance should be small to give higher weight to the prior information, and it is set according to actual experience. The value of , such as a small value between 0.1 and 1; when the prior is unreliable, the covariance should be larger to reduce the weight of the prior information, usually The value is set to The specific value can be adjusted according to actual tests.

[0043] In addition, for the calculation of the prior trajectory direction, the present invention adopts the assumption of posture continuity and uses a nonlinear regression model, specifically: 4.2.1 Data Preparation Collect the position and direction data of the mobile device over a period of time and record it as the original trajectory data: ; in, , and Respectively represent The horizontal coordinate, vertical coordinate and yaw angle of the mobile device at each moment; Indicates the total duration of the period; 4.2.2 Data smoothing Smoothing the original trajectory data to reduce the impact of measurement errors and outliers. Commonly used methods include moving average and Kalman filtering. 4.2.3 Establishing a nonlinear regression model Assuming that the smoothed trajectory data satisfies a certain nonlinear relationship, the nonlinear regression model is established as follows: ; in, Represents the predicted direction angle; represents a nonlinear function; and Respectively , It's all about time Related functions; represents the model parameters; By minimizing the predicted direction angle The actual direction angle The error is used to train the nonlinear regression model, which is expressed as: ; in, It represents the optimal parameter that minimizes the mean square error between the predicted direction angle and the actual direction angle; Indicates the calculation from arrive The sum of the mean square errors and the minimum value are used to obtain the optimal parameters; 4.2.4 Calculation of prior trajectory direction Using the trained nonlinear regression model, the new location data Calculate the prior yaw angle , expressed as: .

[0044] The above is only to illustrate the calculation of the prior yaw angle based on the established regression model. In the case where the building orientation is stored in the map, the building orientation angle obtained from the map information can be directly used in implementation. As the prior yaw angle, .

[0045] The above step 5 is specifically: 5.1 VIO Pose Estimation Output pose estimation through the VIO system, that is, the transformation matrix of the mobile device , expressed as: ; in, Represents the rotation matrix, which represents the rotation information output by the VIO system; Represents the translation matrix, which represents the displacement information output by the VIO system.

[0046] 5.2 VIO Error Covariance The following takes the ARCore implementation plan as an example to illustrate the calculation of relevant parameters of the VIO system. Since ARCore can obtain image feature points, the number of feature points, distribution uniformity, IMU stability and smoothness of posture changes are used to estimate the covariance. Since these characteristics can also be used in other VIO solutions, it is also a general solution for VIO; the covariance of the VIO system is divided into two parts: position and posture, and the entire covariance estimation process is described in matrix form: 5.2.1 VIO covariance matrix form Covariance of VIO system output It should include the covariance of position and attitude, and the overall form is a Matrix, representing the measurement uncertainty of 3D position and 3D attitude: ; in, represents the position covariance of the VIO system output; represents the attitude covariance output by the VIO system; 5.2.2 Position Covariance Structure Position covariance It can be decomposed into the covariance in the image plane and the covariance in the depth direction, expressed as: ; in, Represents the position covariance in the image plane (x, y axis direction), based on the number of image feature points and distribution uniformity To estimate, Indicates that the input data is constructed into a diagonal matrix. Indicates the manually set feature point quantity adjustment coefficient (the recommended range is 50~200). Indicates the manually set uniformity adjustment coefficient (the recommended range is 0.1~1.0); Represents the position covariance in the depth direction (z-axis direction), which is defined as follows: ; in, and Represents the artificially set adjustment coefficient, which is used to control the influence of VIO trajectory smoothness and IMU acceleration stability on the depth direction position covariance (both values ​​are between 0.1 and 1.0, and can be adjusted according to specific circumstances); Indicates the position smoothness of the VIO trajectory; Represents the acceleration stability of the IMU in the out-of-plane direction; Position smoothness It is calculated by the standard deviation of the displacement increments between adjacent time steps and is expressed as: ; in, represents the time step calculated by the VIO system arrive The relative displacement increment; represents the modulus of displacement increment; Represents the average value of the displacement increment within the sliding window; when the position changes smoothly, Take the smaller value, otherwise Take the larger value; Acceleration stability It is calculated by the standard deviation of the rate of change of acceleration between adjacent time steps and is expressed as: ; in, represents the modulus of the rate of change of acceleration in adjacent time steps, and Respectively represent the time step and IMU acceleration measurements; It represents the average value of the acceleration change rate within the sliding window, and its calculation formula is: ; Distribution uniformity The calculation method is: By dividing the image captured by the mobile device's camera into grids, and let Indicates Row, No. The number of feature points in the column grid is obtained , represents the calculation of variance, The value of is inversely proportional to the uniformity of the feature point distribution; 5.2.3 Attitude Covariance Structure Attitude Covariance It is estimated by the smoothness of posture change. The smoothness measure of posture change is , expressed as: ; in, Indicates the incremental rotation angle; Represents the average value of the rotation angle change within the sliding window, and its calculation formula is: ; The attitude covariance matrix is: ; in, Indicates the adjustment coefficient of posture smoothness, the recommended range is 0.1~10; 5.2.4 Complete VIO Covariance Matrix Representation Combining the above components, we get the final VIO covariance matrix .

[0047] In step 5, the VIO system is constructed by introducing visual information and combining it with the original IMU data. Through accurate state estimation and observation update, the system can achieve high-precision position and pose estimation in dynamic and complex environments. This fusion strategy not only improves the robustness of positioning, but also lays a solid foundation for subsequent application scenarios.

[0048] In step 6 above, each information source is represented as a node using a factor graph and the corresponding factors are introduced. The weights of different information are adaptively adjusted during the optimization process. Specifically: 6.1 State Definition Define the state of the factor graph for: ; in, , and Represent the position, velocity and posture of the factor graph output respectively.

[0049] 6.2 IMU Factor The IMU factor of the factor graph is constructed using the device IMU data to establish motion constraints on the states between adjacent moments. When using GTSAM, ImuFactor will be used to implement the IMU factor and process the IMU data.

[0050] 6.3 Neural Network Measurement Factors Using the best neural network The output pose increment estimate and its covariance construct the neural network measurement factor of the factor graph; 6.4 A priori measurement factors Using the prior yaw angle The prior measurement factor of the factor map constructed by its covariance is The constraints are also calculated using the yaw angle error, but the three-dimensional representation of the attitude is fully preserved.

[0051] 6.5 VIO measurement factors The VIO measurement factor of the factor graph is constructed using the pose estimate and its covariance output by the VIO system; 6.6 Factor Graph Sliding Window Optimization Sliding window optimization is an incremental optimization method for factor graphs, which is often used for real-time positioning and attitude estimation. The sliding window optimization of the factor graph fusion scheme mainly gradually updates the current state estimate by continuously introducing new factors and state nodes and removing old nodes and factors. The following is a description of the sliding window optimization in the factor graph fusion scheme: 6.6.1 Status Node Definition Assume that the current sliding window contains a moment, at a moment ,The factor graph contains several state nodes and measurement factors; time The state node is defined as: ; in, Indicates time location; Indicates time speed; Indicates time The rotation matrix represents the three-dimensional posture (including yaw, pitch and roll angles); 6.6.2 Objective Function of Sliding Window Optimization At the moment , the objective function of sliding window optimization contains the weighted error sum of all measurement factors, and the optimization objective is expressed as: ; in, Indicates status The IMU factor residual (i.e. the error between the state and the measurement factor); Indicates status The neural network measurement factor residuals; Indicates status The prior measurement factor residuals of ; Indicates status VIO measurement factor residuals; each of the above factors has a corresponding covariance matrix Used for weighting to control the influence of different measures in the optimization; 6.6.3 Incremental Update Process Sliding window optimization incrementally updates the Execution, the specific process is: 6.6.3.1 Adding new status nodes and measurement factors In every new moment , add a new state node and new measurement factors, new state nodes Include Location ,speed ,attitude , acceleration bias and angular velocity Bias, the new measurement factors include new IMU factors, new neural network measurement factors, new prior measurement factors and new VIO measurement factors; 6.6.3.2 Removing old state nodes and measurement factors To control the computational complexity, the sliding window size is kept fixed at , it is necessary to remove the state nodes and measurement factors outside the window, that is, remove the earliest state node and its associated IMU factors, neural network measurement factors, prior measurement factors, and VIO measurement factors; 6.6.3.3 Incremental Optimization This embodiment is implemented using GTSAM, so the incremental optimization uses the iSAM2 algorithm to incrementally optimize the factor graph in the sliding window. iSAM2 uses the optimization result of the previous moment as the initial value to achieve fast update. The goal is to minimize the residual sum to achieve the moment Optimization status , and its calculation formula is: ; The calculation of factor graph optimization results after each incremental update can be achieved using the Gauss-Newton or Levenberg-Marquardt method.

[0052] The above step 7 is actually the IMU data correction, specifically: 7.1 Sensor Data Correction Assume that at time The length of the pre-processed IMU data input into the optimal neural network is , for the moment to The data is optimized by factor graph sliding window to obtain state variables and substituted into step 1.2 to calculate the pose propagation between each time step, and then the original acceleration measured by IMU is calculated by step 1.3. and the original angular velocity The IMU data corrected at each moment is expressed as: ; in, and Respectively represent the corrected acceleration and angular velocity; 7.2 Corrected IMU input data The corrected IMU input data sequence is generated as: ; in, and Respectively indicate time and Corrected IMU data; 7.3 Neural Network Inference Input the calibrated IMU data sequence Input to the optimal neural network Perform real-time pose reasoning in the output time The pose estimate is ,in and Respectively indicate time By optimal neural network Output the position and posture; then repeat steps 4 to 7 until the end.

[0053] Above, the entire sliding window optimization and IMU data correction process can be summarized as the following closed-loop process: (I) Factor graph sliding window optimization to update the latest pose and sensor noise parameters; (II) De-noise the IMU data using the noise parameters obtained by factor graph optimization, and correct the IMU data using the latest pose; (III) Input the corrected IMU data into the neural network to obtain pose estimation; (IV) The factor graph continues to optimize the output of the neural network as a new measurement value.

[0054] That is, the whole process can be expressed as: → Reasoning → →iSAM2 Optimization→ →IMU data denoising and correction.

[0055] Through the above scheme, the present invention has the following technical effects: (1) Significantly improve positioning accuracy: By introducing Neural Network Architecture Search (NAS) and optimizing the neural network design, the system can more accurately infer displacement information and reduce positioning inaccuracy caused by IMU drift and accumulated errors. (2) Enhanced robustness of direction estimation: The displacement inferred by the neural network is jointly corrected with the heading provided by the prior information and the IMU integrated heading, which can dynamically adjust the heading weight. This effectively solves the problem that the traditional method has poor direction correction effect in complex environments, thereby enhancing the robustness of direction estimation. (3) Improve adaptability in complex environments: The visual inertial odometry (VIO) system combined with visual information enables the system to maintain high positioning accuracy in most scenes with good visual feature conditions, thereby improving the system's adaptability to complex indoor environments; (4) Accurate state estimation: By fusing VIO, heading correction information, and the pose covariance inferred by the neural network through the expanded factor graph, the system can achieve more accurate state estimation, thereby improving the stability and reliability of the overall system. (5) Effectively handle nonlinear scenarios: The present invention can form a navigation strategy suitable for nonlinear paths by integrating behavioral patterns and prior knowledge, thus overcoming the limitation that traditional methods can only handle linear paths.

[0056] In summary, the present invention significantly improves the performance of the inertial navigation system in terms of positioning accuracy, direction estimation and adaptability to complex environments by comprehensively utilizing neural networks, visual information and factor graph technology, ensuring its reliability and effectiveness in long-term navigation and positioning tasks.

[0057] Several key points of the present invention are: ① In view of the problem that traditional methods often rely on direct integration of IMU and are affected by sensor drift and noise, resulting in cumulative errors, this invention introduces neural network architecture search (NAS) and constructs the optimal neural network architecture through NAS technology, so that the network can be optimized for specific tasks and then infer displacement information. Compared with traditional methods, the neural network designed by NAS can more effectively capture the time series characteristics of IMU data, improve the accuracy of displacement estimation, and reduce the accumulation of errors.

[0058] ② In view of the problem that the existing technology has limited effect on heading correction on non-ideal paths, easily introduces direction errors and affects positioning accuracy, the present invention jointly corrects the displacement inferred by the neural network with the prior information and the IMU integrated heading, and dynamically adjusts the heading weight by fusing multi-source information, thereby enhancing the system's direction estimation accuracy in complex environments.

[0059] ③ In response to the problem that traditional IMU systems easily lose positioning accuracy due to accumulated errors, the present invention combines visual information with IMU data to construct a visual inertial odometry (VIO) system. The VIO system enhances the accuracy of pose estimation through visual information and provides more reliable positioning results in most common scenarios.

[0060] ④ In view of the problem that traditional filtering methods often cannot effectively integrate multi-source information, resulting in information loss and reduced accuracy, the present invention fuses VIO, heading correction information and the pose covariance inferred by the neural network through a factor graph, providing a dynamic state estimation method that can adjust the position and pose estimation according to real-time sensor input, significantly improving the robustness and accuracy of the overall system.

[0061] Through the above improvements, the present invention effectively improves the positioning accuracy and robustness of the inertial navigation system in complex environments, overcomes many limitations of traditional technologies, and meets the requirements for high accuracy and high reliability in practical applications.

[0062] The present invention also discloses an electronic device, comprising a processor, a memory and an application program; the application program is stored in the memory and is configured so that the processor executes the above method.

[0063] The present invention also discloses a computer-readable storage medium, which stores a computer program; when the computer program is executed in a computer, the computer executes the above method. The storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0064] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A positioning method that dynamically integrates inertial navigation, vision and prior information, characterized in that include: Step 1: Collect device IMU data in real time from the mobile device carried by the user, and then convert the device IMU data in the device coordinate system to the world coordinate system and remove gravity through the preprocessing step to obtain preprocessed IMU data. Among them, IMU stands for inertial measurement unit; Step 2: Apply neural network architecture search technology to search and optimize the layer type, connection method and hyperparameters of the neural network in the search space to obtain the optimal neural network ; Step 3: A training data set is formed by a preset number of pre-processed IMU data, and the optimal neural network is trained. Perform training and use the trained optimal neural network for inference to obtain the pose increment estimate and its covariance; Step 4: Obtain the prior direction and its covariance based on the prior information; Step 5: When visual information is available, the visual information is introduced and fused with the IMU data of the device in step 1 to build a VIO system, which outputs the pose estimate and its covariance in real time; where VIO stands for visual inertial odometry; Step 6: Using the output results of steps 3 to 5 as measurement information, a factor graph is used to perform multi-source information fusion; Step 7: Use the fusion results of the factor graph output in step 6 to update the parameters of the optimal neural network, and correct the device IMU data according to step 1, and then send the corrected results to the optimal neural network for reasoning, and then repeat steps 4 to 7 until the end.

2. The positioning method of dynamic fusion of inertial navigation, vision and prior information as claimed in claim 1, characterized in that The pre-processing steps in step 1 are specifically: 1.1 Device IMU data includes time The collected accelerometer and gyroscope data are the raw acceleration , original angular velocity ; 1.2 Use the device IMU data to propagate the 3D posture between consecutive time steps. The device IMU data is updated in 3D space as follows: Location Updates ; Speed ​​Update ; Posture Update ; Among them, for the time and the previous moment : and Indicates location, and Indicates speed, and represents the rotation matrix, and Respectively represent the original acceleration and original angular velocity; and Represent the error bias of accelerometer and gyroscope respectively; Represents the gravitational acceleration measured at rest Indicates time and the previous moment time interval; Representation in Lie algebra Exponential mapping on ; 1.3 Perform world coordinate system conversion. The net acceleration in the world coordinate system is expressed as: ； The angular velocity in the world coordinate system is expressed as: ； The pre-processed IMU data is expressed as follows: 。 3. The positioning method of dynamic fusion of inertial navigation, vision and prior information as claimed in claim 1, characterized in that The step 2 specifically includes: 2.1 Defining the Search Space for: ； in, represents the convolutional layer; represents a linear layer; represents the activation function; represents the pooling layer; Indicates other layers; 2.2 Defining candidate network architectures for: ； in, Indicates the network The configuration of each layer is represented by ,in Indicates the first parameters; 2.3 Defining the Minimization Loss Function Represents the loss of the model, and its formula is: ； in, Indicates the total number of samples; express Calculation method of Indicates The true labels of samples, Indicates The model prediction output corresponding to the samples; 2.4 Use at least one of reinforcement learning, evolutionary algorithms, Bayesian optimization, and brute force search to search for network architectures, and use the minimization loss function to evaluate the performance of candidate network architectures; after several iterations, select the optimal neural network and its parameters .

4. The positioning method of dynamic fusion of inertial navigation, vision and prior information as claimed in claim 1, characterized in that The step 3 is specifically: 3.1 The optimal neural network Input data The dimensions are expressed as: ； in, represents the real number space; Indicates the batch size; Indicates the number of channels; Indicates the length of the sequence; 3.2 During the training process, a training data set is formed by a preset number of pre-processed IMU data to optimize the neural network. Training; optimal neural network The output of is the pose estimate and its covariance, expressed as: ； in, and They represent the pose estimation and covariance of the network output during the training process respectively; Represents the input data during training; represents the parameters of the optimal neural network; Each round of the optimal neural network During training, its parameters are updated and finally the function that minimizes the loss is saved. The smallest set of parameters is recorded as the final parameters ; 3.3 Use the best trained neural network during inference Perform pose estimation; define input data is the IMU data to be inferred, and the inference process is expressed as: ； ； in, and Represent the optimal neural network The inferred pose increment estimate and its covariance; and Represent the optimal neural network The position increment covariance and attitude increment covariance are obtained by inference.

5. The positioning method of dynamic fusion of inertial navigation, vision and prior information as claimed in claim 1, characterized in that The step 4 is specifically: 4.1 Based on device IMU data and its time interval Calculate the IMU yaw angle, expressed as: ； in, and Respectively indicate time and the previous moment IMU yaw angle; Indicates time The angular velocity component of the gyroscope in the z-axis direction; 4.2 Obtaining the prior yaw angle provided by the prior information based on the known prior heading angle or nonlinear regression model ; The covariance of the prior direction is calculated using the dynamic covariance scheme. The difference between the IMU yaw angle and the prior yaw angle is: ； right Perform periodic processing and limit it to the range of [-180°, 180°]; By comparison The covariance of the prior information is determined by the preset difference threshold, which is expressed as: ； in, represents the difference threshold, then the larger covariance value when the prior is unreliable is , when the prior is credible, the smaller covariance is .

6. The positioning method of dynamic fusion of inertial navigation, vision and prior information as claimed in claim 5, characterized in that In step 4.2, the prior yaw angle provided by the prior information is obtained based on the nonlinear regression model. , specifically: 4.2.1 Collect the location and direction data of the mobile device over a period of time and record it as the original trajectory data: ； in, , and Respectively represent The horizontal coordinate, vertical coordinate and yaw angle of the mobile device at each moment; Indicates the total duration of the period; 4.2.2 Smoothing the original trajectory data; 4.2.3 Establish the trajectory data after nonlinear regression model smoothing, expressed as: ； in, Represents the predicted direction angle; represents a nonlinear function; and Respectively , It's all about time Related functions; represents the model parameters; By minimizing the predicted direction angle The actual direction angle The error is used to train the nonlinear regression model, which is expressed as: ； in, It represents the optimal parameter that minimizes the mean square error between the predicted direction angle and the actual direction angle; Indicates the calculation from arrive The sum of the mean square errors and the minimum value are used to obtain the optimal parameters; 4.2.4 Using the trained nonlinear regression model to predict new position data Calculate the prior yaw angle , expressed as: 。 7. The positioning method of dynamic fusion of inertial navigation, vision and prior information as claimed in claim 1, characterized in that The step 5 is specifically: 5.1 Build a VIO system and output the transformation matrix of the mobile device through the VIO system , expressed as: ； in, Represents the rotation matrix output by the VIO system; Represents the translation matrix output by the VIO system; 5.2 Calculate the error covariance of the VIO system output, specifically: 5.2.1 Covariance of VIO System Output It is expressed as: ； in, represents the position covariance of the VIO system output; represents the attitude covariance output by the VIO system; 5.2.2 Position Covariance It is expressed as: ； in, Represents the position covariance in the image plane, based on the number of image feature points and distribution uniformity To estimate, Indicates that the input data is constructed into a diagonal matrix. Indicates the artificially set feature point quantity adjustment coefficient, Indicates the artificially set uniformity adjustment coefficient; represents the position covariance in the depth direction, which is defined as follows: ； in, and It represents the artificially set adjustment coefficient, which is used to control the influence of VIO trajectory smoothness and IMU acceleration stability on the depth direction position covariance; Indicates the position smoothness of the VIO trajectory; Represents the acceleration stability of the IMU in the out-of-plane direction; Position smoothness It is calculated by the standard deviation of the displacement increments between adjacent time steps and is expressed as: ； in, represents the time step calculated by the VIO system arrive The relative displacement increment; represents the modulus of displacement increment; Represents the average value of the displacement increment within the sliding window; when the position changes smoothly, Take the smaller value, otherwise Take the larger value; Acceleration stability It is calculated by the standard deviation of the rate of change of acceleration between adjacent time steps and is expressed as: ； in, represents the modulus of the rate of change of acceleration in adjacent time steps, and Respectively represent the time step and IMU acceleration measurements; It represents the average value of the acceleration change rate within the sliding window, and its calculation formula is: ； Distribution uniformity The calculation method is: By dividing the image captured by the mobile device's camera into grids, and let Indicates Row, No. The number of feature points in the column grid is obtained , represents the calculation of variance, The value of is inversely proportional to the uniformity of the feature point distribution; 5.2.3 Attitude Covariance It is estimated by the smoothness of posture change. The smoothness measure of posture change is , expressed as: ； in, Indicates the incremental rotation angle; Represents the average value of the rotation angle change within the sliding window, and its calculation formula is: ； The attitude covariance matrix is: ； in, The adjustment coefficient representing the smoothness of the posture; 5.2.4 Combining the above components, we get the final VIO covariance matrix .

8. The positioning method of dynamic fusion of inertial navigation, vision and prior information as claimed in claim 1, characterized in that The step 6 specifically includes: 6.1 Defining the state of the factor graph for: ； in, , and Represent the position, velocity and posture of the factor graph output respectively; 6.2 Use the IMU data of the device to construct the IMU factors of the factor graph to establish motion constraints on the states between adjacent moments; 6.3 Using the Optimal Neural Network The output pose increment estimate and its covariance construct the neural network measurement factor of the factor graph; 6.4 Using the Prior Yaw Angle and their covariances to construct the factor graph for the prior measurement factors; 6.5 Use the pose estimate and covariance output by the VIO system to construct the VIO measurement factor of the factor graph; 6.6 Incremental Optimization of Factor Graphs Using Sliding Window Optimization.

9. The positioning method of dynamic fusion of inertial navigation, vision and prior information as claimed in claim 8, characterized in that The step 6.6 is specifically: 6.6.1 Assume that the current sliding window contains a moment, at a moment ,The factor graph contains several state nodes and measurement factors; time The state node is defined as: ； in, Indicates time location; Indicates time speed; Indicates time The rotation matrix of 6.6.2 At time , the objective function of sliding window optimization contains the weighted error sum of all measurement factors, and the optimization objective is expressed as: ； in, Indicates status IMU factor residual; Indicates status The neural network measurement factor residuals; Indicates status The prior measurement factor residuals of ; Indicates status VIO measurement factor residuals; Each factor has a corresponding covariance matrix For weighting; 6.6.3 Sliding Window Optimization by Incremental Update at Each Moment Execution, the specific process is: 6.6.3.1 At each new moment , add a new state node and new measurement factors, new state nodes Include Location ,speed ,attitude , acceleration bias and angular velocity Bias, the new measurement factors include new IMU factors, new neural network measurement factors, new prior measurement factors and new VIO measurement factors; 6.6.3.2 To control the computational complexity, keep the sliding window size fixed to , it is necessary to remove the state nodes and measurement factors outside the window, that is, remove the earliest state node and its associated IMU factors, neural network measurement factors, prior measurement factors, and VIO measurement factors; 6.6.3.3 Use the iSAM2 algorithm to incrementally optimize the factor graph in the sliding window. iSAM2 uses the optimization result of the previous moment as the initial value to achieve fast update. The goal is to minimize the residual sum to achieve the moment Optimization status , and its calculation formula is: 。 10. The positioning method of dynamic fusion of inertial navigation, vision and prior information as claimed in claim 2, characterized in that The step 7 is specifically: 7.1 Assume that at time The length of the pre-processed IMU data input into the optimal neural network is , for the moment to The data is optimized by factor graph sliding window to obtain state variables and substituted into step 1.2 to calculate the pose propagation between each time step, and then the original acceleration measured by IMU is calculated by step 1.

3. and the original angular velocity The IMU data corrected at each moment is processed and expressed as: ； in, and Respectively represent the corrected acceleration and angular velocity; 7.2 Generate the corrected IMU input data sequence as: ； in, and Respectively indicate time and Corrected IMU data; 7.3 Input the calibrated IMU data sequence Input to the optimal neural network Perform real-time pose reasoning in the output time The pose estimate is ,in and Respectively indicate time By optimal neural network Output the position and posture; then repeat steps 4 to 7 until the end.

Images