Positioning method and electronic device

Abstract

A positioning method and an electronic device, relating to the technical field of positioning. The method comprises: acquiring reference information of an object to be positioned at at least one moment, wherein the reference information comprises inertial measurement unit (IMU) information, visual information, and global navigation satellite system (GNSS) information (S102); on the basis of the IMU information, performing IMU state propagation to obtain IMU positioning data (S104); on the basis of the GNSS information, performing GNSS state propagation to obtain GNSS positioning data (S106); and by means of a square root inverse sliding window filter (SRI-SWF), positioning said object on the basis of the IMU positioning data, the GNSS positioning data, and the visual information (S108).

Description

Positioning method and electronic device Technical Field

[0001] The present disclosure relates to the field of positioning technology, and in particular to a positioning method and electronic equipment. Background Art

[0002] Visual Inertial Odometry (VIO) has been widely used in fields such as autonomous driving, mobile robotics, and augmented reality. For example, in autonomous driving scenarios, the VIO algorithm implemented using a camera and an inertial measurement unit (IMU) can be used to achieve six-degree-of-freedom positioning. However, since the VIO algorithm only calculates inter-frame pose estimates, when there is an error in the motion estimation between two frames and no additional global pose constraints are introduced, cumulative errors will occur on the trajectory. The most intuitive manifestation is that when returning to the origin after trajectory movement, the position estimated by the VIO algorithm does not return to the origin due to the accumulated error.

[0003] By adding the global position constraints of the Global Navigation Satellite System (GNSS) to the VIO system, the cumulative error of the visual inertial odometry can be reduced. Summary of the Invention

[0004] The present disclosure provides a positioning method and electronic device that, at least to some extent, address the numerical instability issue associated with adding GNSS global position constraints to a VIO system in the related art when the Hessian matrix condition number exceeds 10^9. Other features and advantages of the present disclosure will become apparent from the detailed description below, or may be learned in part through practice of the present disclosure.

[0005] According to a first aspect of the present disclosure, a positioning method is provided, comprising: obtaining reference information of an object to be positioned at at least one moment, the reference information comprising inertial measurement unit (IMU) information, visual information, and global navigation satellite system (GNSS) information; performing IMU state propagation based on the IMU information to obtain IMU positioning data; performing GNSS state propagation based on the GNSS information to obtain GNSS positioning data; and positioning the object to be positioned based on the IMU positioning data, the GNSS positioning data, and the visual information using a square root inverse sliding window filter (SRI-SWF).

[0006] According to a second aspect of the present disclosure, a positioning system is provided, including: an information acquisition module for acquiring reference information of an object to be positioned at at least one moment, the reference information including inertial measurement unit (IMU) information, visual information, and global navigation satellite system (GNSS) information; an IMU measurement module for performing IMU state propagation based on the IMU information to obtain IMU positioning data; a GNSS measurement module for performing GNSS state propagation based on the GNSS information to obtain GNSS positioning data; and a positioning module for positioning the object to be positioned based on the IMU positioning data, the GNSS positioning data, and the visual information using a square root inverse sliding window filter (SRI-SWF).

[0007] According to a third aspect of the present disclosure, an electronic device is provided, including: a memory for storing instructions; and a processor for calling the instructions stored in the memory to implement the above positioning method.

[0008] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed by a processor, the above-mentioned positioning method is implemented.

[0009] According to a fifth aspect of the present disclosure, a computer program product is provided. The computer program product stores instructions, which, when executed by a computer, enable the computer to implement the above positioning method.

[0010] According to a sixth aspect of the present disclosure, a chip is provided, comprising at least one processor and an interface; the interface is configured to provide program instructions or data to the at least one processor; and the at least one processor is configured to execute the program instructions to implement the above-mentioned positioning method.

[0011] The positioning method and electronic device provided in the embodiments of the present disclosure use a square root inverse sliding window filter (SRI-SWF) to fuse IMU information, visual information, and GNSS information to locate the object to be located. The present disclosure integrates GNSS information to reduce the cumulative error of the visual inertial odometer while having superior numerical stability.

[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0014] Obviously, the drawings described below are only some embodiments of the present disclosure. A person skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0015] FIG1 shows a flow chart of an exemplary positioning method in an embodiment of the present disclosure;

[0016] FIG2 shows a schematic diagram of an exemplary SRI-SWF architecture in an embodiment of the present disclosure;

[0017] FIG3 shows a flowchart of an exemplary GNSS measurement update in an embodiment of the present disclosure;

[0018] FIG4 shows an exemplary positioning error diagram of an embodiment of the present disclosure when using different GNSS measurement values;

[0019] FIG5 shows an exemplary x-axis extrinsic calibration error diagram in an embodiment of the present disclosure;

[0020] FIG6 shows an exemplary velocity error calibration diagram without external parameters in an embodiment of the present disclosure;

[0021] FIG7 shows an exemplary velocity error calibration diagram with external parameters in an embodiment of the present disclosure;

[0022] FIG8 shows a trajectory diagram of exemplary algorithms in an embodiment of the present disclosure;

[0023] FIG9 shows a position error diagram of each exemplary algorithm in an embodiment of the present disclosure;

[0024] FIG10 shows a schematic diagram of an exemplary positioning system in an embodiment of the present disclosure;

[0025] FIG11 shows a structural block diagram of an exemplary electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0027] The solution of adding the global position constraint of GNSS to the VIO system can reduce the cumulative error of the visual inertial odometer. Specifically, in the related art, the global position constraint of GNSS can generally be added to the VIO system based on a loose coupling or tight coupling solution. The loose coupling solution refers to the position and velocity calculated by the GNSS receiver as observations integrated into the VIO, while the tight coupling solution refers to the use of the original measurement information of GNSS (such as pseudorange, Doppler shift, etc.) as observations integrated into the VIO.

[0028] The above loosely coupled scheme cannot provide positioning information when the number of satellite observations is less than 4.

[0029] The above-mentioned tightly coupled schemes include the optimization-based GVINS (GNSS Visual-Inertial Navigation System) and the filtering-based InGVIO (invariant GNSS Visual-Inertial Odometry). Among them, the GVINS scheme has a large amount of computation, which is not conducive to lightweight deployment on portable devices. The InGVIO scheme does not support GNSS-IMU online external parameter calibration, and does not accurately align the filter velocity state when the Doppler shift measurement is updated, thereby introducing unmodeled errors. At the same time, both GVINS and InGVIO have numerical instability problems when the Hessian matrix condition number is >10^9, which may lead to a decrease in accuracy or even estimation divergence. In addition, neither GVINS nor InGVIO can utilize the single-difference pseudorange and double-difference carrier phase measurements of GNSS.

[0030] The defects of the above solutions and the proposed solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed in the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0031] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0032] This exemplary implementation is described in detail below with reference to the accompanying drawings and examples.

[0033] First, an embodiment of the present disclosure provides a positioning method, which can be executed by any electronic device with computing and processing capabilities.

[0034] FIG1 shows a flow chart of a positioning method in an embodiment of the present disclosure. As shown in FIG1 , the positioning method provided in the embodiment of the present disclosure includes steps S102 - S108 .

[0035] In S102 , reference information of an object to be positioned at at least one moment is acquired, where the reference information includes inertial measurement unit (IMU) information, visual information, and global navigation satellite system (GNSS) information.

[0036] In some embodiments, the IMU state information of the object at a moment includes at least one of the position, velocity, rotation angle, acceleration bias, and gyroscope bias of the object in the world coordinate system at that moment.

[0037] In some embodiments, the visual information of an object at a moment in time includes information about each visual feature point in an image captured at that moment, where the image was captured by a camera associated with the object, such as a camera mounted on the object. Exemplarily, the information about a visual feature point includes the actual position of the visual feature point in three-dimensional space, or the position of the visual feature point in the camera coordinate system.

[0038] In some embodiments, the GNSS information of the object at a moment includes pseudorange, Doppler shift, single-difference pseudorange, double-difference carrier phase, etc.

[0039] It should be noted that GNSS includes but is not limited to at least one of the following systems: GPS (Global Positioning System), GLONASS (a global satellite navigation system), BDS (BeiDou Navigation Satellite System) and Galileo (a global satellite navigation system).

[0040] In S104, IMU state propagation is performed based on the IMU information to obtain IMU positioning data.

[0041] IMU state propagation can be a recursive calculation process of the system state using the IMU information (e.g., acceleration and angular velocity data) continuously collected by the IMU through the laws of physics (primarily Newton's laws of motion and the law of conservation of angular momentum). In other words, in S104, the angular velocity and acceleration of the carrier are continuously monitored and the IMU positioning data is obtained through integration. The IMU positioning data can be continuous position, velocity, and attitude changes over a short period of time.

[0042] In S106 , GNSS status propagation is performed based on the GNSS information to obtain GNSS positioning data.

[0043] GNSS state propagation is the process of performing positioning solutions based on satellite signals received by the receiver, including updating and maintaining the receiver's positioning state based on the received GNSS information.

[0044] In S108 , the object to be positioned is positioned based on the IMU positioning data, the GNSS positioning data and the visual information through a square root inverse sliding window filter (SRI-SWF).

[0045] The present disclosure uses the SRI-SWF filter to perform state estimation based on the information obtained in the above steps. The SRI-SWF can effectively handle the state estimation problem of large nonlinear systems and improve the stability and estimation accuracy of the filter to a certain extent.

[0046] In some embodiments, the SRI-SWF filter in S108 can determine a state vector based on IMU positioning data, GNSS positioning data, and visual information to perform state estimation. The state estimation includes SLAM (Simultaneous Localization and Mapping) features, historical IMU clones, parameter state vectors, and IMU additional states. The historical IMU clones include the cloned IMU states at each moment. The parameter state vectors include external parameters from the IMU to the camera, external parameters from the GNSS receiver to the IMU, GNSS receiver clock error, and GNSS receiver clock drift. The IMU additional states include the velocity, acceleration bias, and gyroscope bias of the object in the world coordinate system.

[0047] In some embodiments, GNSS initialization can be performed based on GNSS information to obtain the transformation relationship between the visual inertial odometry (VIO) coordinate system and the Earth-centered Earth-fixed (ECEF) coordinate system, the GNSS receiver clock error, and the GNSS receiver clock drift; SLAM features of the object to be located are extracted from the visual information; based on the IMU information and the transformation relationship between the VIO coordinate system and the ECEF coordinate system, the historical IMU clone and parameter state vector are determined; based on the IMU information, the GNSS receiver clock error, and the GNSS receiver clock drift, the IMU additional state is determined.

[0048] In some embodiments, the architecture of SRI-SWF may be as shown in Figure 2, wherein the sensor input includes an inertial measurement unit (IMU), a camera, and a global satellite navigation system (GNSS).

[0049] The inertial measurement unit is used for IMU state propagation in square root inverse sliding window filtering (SRI-SWF); the camera is used for visual feature point extraction and optical flow tracking; and the global satellite navigation system is used for GNSS initialization, propagation, and update.

[0050] Initialization includes VIO initialization (VI-Initialization) and GNSS initialization (GNSS-Initialization). VI-Initialization means using images and IMU to complete conventional VIO initialization; GNSS-Initialization means using GNSS to solve the VIO-ECEF coordinate system transformation and GNSS clock error and drift initialization after VIO initialization.

[0051] The measurement types include conventional Msckf feature point updates in VIO, delayed initialization, SLAM feature point updates, and raw measurement updates from GNSS.

[0052] The backend SRI-SWF filtering includes IMU state propagation, GNSS state propagation, measurement update, and sliding window marginalization.

[0053] The following is a detailed introduction to the process based on SRI-SWF filtering:

[0054] At time k, the state vector x of the filter k Including the current SLAM feature x S 、Historical IMU clone x C , parameter state vector x P And the IMU additional state x E .

[0055] Where, for j = 1, ..., n, p fj Represents point feature f j The state of is represented by inverse depth parameterization of the camera pose first observed within the current sliding window.

[0056] x Ci Represents the IMU state cloned at time i, where is a quaternion describing the rotation from the IMU coordinate system {Ii} to the local VIO coordinate system {W}, and is the position of {Ii} in {W}.

[0057] It is the external parameter from IMU to camera. It is the external parameter from the GNSS receiver to the IMU.

[0058] For j=1,…,N-1, represents the receiver clock bias relative to GNSS galaxy j. is the receiver clock drift, which is the same for every constellation.

[0059] is the velocity of {I} in {W}, and b ak and b gk are the accelerometer and gyroscope biases at time k.

[0060] The following describes the IMU state propagation process.

[0061] Given an inertial measurement u k , Using inertial kinematics it is propagated as:

[0062] in, w k，k+1 is the Gaussian noise that affects IMU measurement, with covariance Q k .

[0063] Linearizing the above formula (7) around the state estimate, we get the error state propagation equation:

[0064] Among them, Φ k+1,k and G k+1,k is the corresponding Jacobian matrix.

[0065] In the IMU propagation step, a new attitude state is added to the current state vector

[0066] The prior (upper triangular) square root information matrix and residual vector are defined as follows:

[0067] in, and They are the Jacobian matrices Φ k+1，k Block columns relative to clone state and extra IMU state.

[0068] in, is not an upper triangular matrix before the next marginalization step.

[0069] The marginalization steps are described below.

[0070] At time k, historical SLAM features The oldest clone and additional IMU states Be marginalized. The error state vector containing the above states to be marginalized is defined as follows:

[0071] Will As the remaining state, the state vector is transformed as follows through a permutation matrix:

[0072] After performing the full QR decomposition, we can obtain the a priori (upper triangular) square root information matrix and the residual vector, where we only keep

[0073] The measurement update process is described below.

[0074] In general, a measurement can be modeled as follows:

[0075] where h is the nonlinear function describing the measurement model, z is the obtained measurement value (e.g., SLAM feature re-observation value and MSCKF feature measurement value), n and Σ are the white Gaussian noise of the measurement value and its covariance, respectively.

[0076] By linearizing formula (13), we can obtain the linearized measurement constraint in, denote the measurement Jacobian matrix and the residual, respectively.

[0077] Finally, the state can be updated by performing a QR decomposition:

[0078] Within the SRI-SWF framework, as shown in FIG2 , in addition to the visual inertial measurements in VIO, the disclosed embodiment also deeply integrates the pseudorange, Doppler shift, single-difference pseudorange, and double-difference carrier phase measurements of GNSS.

[0079] In some embodiments, in the above step S106, GNSS state propagation is performed based on the GNSS information to obtain IMU positioning data, including: calculating the GNSS receiver clock error and GNSS receiver clock drift based on the pseudorange and Doppler frequency shift in the GNSS information; and performing GNSS state propagation based on the GNSS receiver clock error and GNSS receiver clock drift to obtain IMU positioning data.

[0080] In some embodiments, a GNSS receiver clock error and a GNSS receiver clock drift are calculated based on the pseudorange and Doppler frequency shift in the GNSS information, including calculating the residual of the pseudorange based on the pseudorange in the GNSS information; calculating the residual of the Doppler frequency shift based on the Doppler frequency shift in the GNSS information; and determining the GNSS receiver clock error and the GNSS receiver clock drift based on the residual of the pseudorange and the residual of the Doppler frequency shift.

[0081] In some embodiments, the GNSS receiver clock error and the GNSS receiver clock drift are determined based on the residual of the pseudorange and the residual of the Doppler frequency shift, including applying a random sampling consistency RANSAC algorithm to perform data cleaning on the residual of the pseudorange and the residual of the Doppler frequency shift; and determining the GNSS receiver clock error and the GNSS receiver clock drift based on the residual of the pseudorange and the residual of the Doppler frequency shift after data cleaning.

[0082] In some embodiments, the above method may further include determining the transformation relationship between the GNSS coordinate system and the VIO coordinate system based on a filtering method; determining the transformation relationship between the visual inertial odometry VIO coordinate system and the Earth-centered Earth-fixed ECEF coordinate system based on the transformation relationship between the GNSS coordinate system and the VIO coordinate system, the transformation relationship between the VIO coordinate system and the North-East Earth (NED) coordinate system, and the transformation relationship between the NED coordinate system and the ECEF coordinate system.

[0083] 3 , the measurement update of GNSS data may include steps S301 - S306 .

[0084] In S301, it is determined whether the transformations of the VIO system and the ECEF system have been solved;

[0085] If no solution is found, go to S302 to solve the transformation between the VIO system and the ECEF system, and then go to S303;

[0086] If the solution has been found, the process directly proceeds to S303 to determine whether the GNSS has been initialized.

[0087] If there is no initialization, go to S304 to initialize GNSS, and then go to S305;

[0088] If initialized, directly proceed to S305 to perform GNSS prediction; then, execute S306 to perform GNSS update.

[0089] The following describes in detail the measurement updates of raw GNSS data.

[0090] First, we introduce the reference frame initialization.

[0091] GNSS updates require conversion from the local VIO world frame to the ECEF frame { E R W E p W}, receiver clock deviation and receiver clock drift See the above formula (5), in order to initialize the transformation { E R W E p W}, the transformation variables can be inserted into the filter to adaptively converge the 4-degree-of-freedom (DOF) transformation. The position and velocity of the receiver in the VIO world frame can be expressed as: W p r = W p I + W R I I p r (16)

[0092] Using the first GNSS measurement as a reference point, the GNSS single point positioning (SPP) output is used to obtain the receiver's position and velocity in the North-East frame (NED, {N}). At time k, the following geometric constraints can be applied:

[0093] The position and velocity of the receiver at time k can be predicted from the GNSS measurements at time i by IMU integration. N R W is the yaw angle of 1DOF, the transformation state can be expressed as [θ E p W ].

[0094] By augmenting the transformed states with infinite covariance priors in the state vector, a standard measurement update is performed, as in Equation (18) above. When the covariance matrix P θθ Yaw angle is below the decision threshold When , the transformation converges. After initialization, the transformation state is marginalized. Given { N R W N p W}and{ E R N E p N}, get the transformation { E R W E p W}as follows: E R W = E R N N R W , E p W = E p N + E R N N p W (19)

[0095] The following describes the calculation process of GNSS receiver clock error and clock drift.

[0096] The disclosed embodiment uses pseudorange and Doppler shift measurements to initialize GNSS receiver clock errors. and receiver clock drift First, the pseudorange measurement can be calculated at time k The residuals are as follows:

[0097] Here, c represents the speed of light. and Receiver and satellite s respectively j clock difference. and are tropospheric delay, ionospheric delay and Sagnac term, respectively. pr represents the pseudorange measurement noise.

[0098] Similarly, Doppler shift measurement The residuals are as follows:

[0099] Where λ is the GNSS carrier wavelength, is the unit direction vector from the receiver to the satellite in {E}. represents the speed of satellite sx in {E}. and are the clock drifts of the receiver and satellite sx respectively. d To measure noise.

[0100] By using all GNSS measurement residuals, see the above formula (20) and formula (21), solve the following linear constraint AX = b, where X is and of stacking.

[0101] First, the RANSAC algorithm is applied to remove erroneous constraints. Then, the aforementioned least-squares problem is solved to obtain initial guesses for the GNSS receiver clock error and drift. These clock errors and drift are then incorporated into the filter state via delayed initialization.

[0102] After initialization, the GNSS clock error and clock drift status are propagated. and

[0103] The prior (upper triangular) square root information matrix is ​​updated based on formula (10) as follows:

[0104] in, Denotes the Jacobian matrix Φ k+1,k Block column relative to parameter status.

[0105] In some embodiments, the present disclosure performs GNSS measurement updates based on pseudorange, Doppler shift, single-difference pseudorange, and double-difference carrier phase in GNSS information.

[0106] In some embodiments, GNSS measurement updates are performed based on pseudorange, Doppler shift, single-difference pseudorange, and double-difference carrier phase in GNSS information, including using IMU integration to model delayed and asynchronous GNSS measurements, where the GNSS measurements include pseudorange measurements, Doppler shift measurements, single-difference pseudorange measurements, and double-difference carrier phase measurements.

[0107] The GNSS measurement update process is described in detail below.

[0108] Due to the latency and asynchronous arrival of GNSS messages, the GNSS measurement time is usually earlier than the filter state time. Since the velocity of each cloned pose is not preserved in the sliding window, the velocity state at the GNSS measurement time cannot be obtained by interpolation. In the embodiment of the present disclosure, assuming the filter state at time i and the GNSS measurement value at time k, the following constraints are obtained through IMU integration:

[0109] in, and is the pre-integration term. For the time interval [t k ,t i ], these terms are calculated as follows:

[0110] Pseudorange measurement (Psr): Substituting equations (16) and (24) into equation (20) defines the full residual of the pseudorange measurement.

[0111] Doppler shift measurement (Dopp): Substituting equations (17) and (24) into equation (21) defines the full residual of the Doppler shift measurement.

[0112] Single-difference pseudorange measurement (Dpsr): For a base station, the pseudorange measurement of the base station at time k can be defined as follows:

[0113] When the distance between the base station and the receiver is close, the time delays in the ionosphere and troposphere can be approximately offset by subtracting equation (20) from equation (26):

[0114] Double-difference carrier phase measurement (DDCP): Carrier phase measurement is usually more accurate than pseudorange measurement. The residual of carrier phase measurement is given by:

[0115] Where N is the integer ambiguity, is the carrier phase measurement value, n cp By using the base station and reference satellite s i The residual error of the double-difference carrier phase measurement can be defined as:

[0116] in, and N dd is the double-difference integer ambiguity obtained by the RTK algorithm.

[0117] The Jacobian matrix of the above GNSS measurements can be calculated as:

[0118] The overall measurement covariance matrix Σ is as follows:

[0119] Among them, ∑ m is the GNSS raw measurement covariance matrix, ∑ g is the IMU integrated measurement noise.

[0120] In some embodiments, the present disclosure may also use Doppler shift update to converge to obtain GNSS-IMU extrinsic parameters when the IMU excitation is greater than a preset threshold.

[0121] IMU excitation can be external motion signals or dynamic environmental changes sensed by the IMU. An IMU contains sensors such as accelerometers and gyroscopes, which can measure an object's acceleration and angular velocity, as well as their changes over time. This data reflects the object's state of motion.

[0122] The external GNSS-IMU is represented in the state vector as Equation (5) above. The disclosed embodiment can use Doppler shift measurement to estimate extrinsic parameters and update the extrinsic parameter estimate when the IMU excitation is sufficient. The Jacobian matrix of the Doppler shift residual with respect to the extrinsic parameter is as follows:

[0123] The inventors collected real-world data sets on their own to verify the solutions of the embodiments of the present disclosure. Among them, the true value data is provided by RTK-INS (real-time dynamic carrier phase differential global navigation satellite system and inertial navigation system combination). The sensor platform includes two downward-looking cameras with a resolution of 1920×1080 and an operating frequency of 20Hz, an IMU (inertial measurement unit) with an operating frequency of 250Hz, and a Unicore UM982 GNSS receiver with an operating frequency of 5Hz. The self-built base station is located 3 kilometers away from the test site and broadcasts the RTCM (Radio Technical Commission for Maritime, International Maritime Wireless Technology Committee Transmission Protocol) stream in real time, which can then be calculated using single-difference pseudorange and double-difference carrier phase observations.

[0124] In order to explore the impact of GNSS measurement updates, the inventors evaluated the performance of the embodiment of the present disclosure (SRI-GVINS system) using different types of GNSS measurements. The inventors selected 10 sets of outdoor drone data sets, covering different flight altitudes from 20 meters to 60 meters, trajectory lengths from 400 meters to 3 kilometers, and flight speeds from 4 meters per second to 12 meters per second. Figure 4 shows the position error of the system in a data set with a flight altitude of 50 meters, a trajectory length of 3 kilometers, and a flight speed of 8 meters per second, where curve 401 represents the error of the VIO solution, curve 402 represents the error of the VIO+Dopp solution, curve 403 represents the error of the VIO+Dopp+Psr solution, and curve 404 represents the error of the VIO+Dopp+Ddcp solution.

[0125] PSR and Dopp measurements: Since PSR and Dopp measurements do not require base station support, the following configurations can be used for testing in the embodiments of this disclosure: a VIO system without GNSS measurement fusion (VIO), a system with loosely coupled VIO and GPS velocity fusion (VIO+GPS Velocity), a system with tightly coupled VIO and Dopp fusion (VIO+Dopp), and a system with tightly coupled VIO and both Dopp and PSR fusion (VIO+Dopp+Psr). The root mean square error (RMSE) of the translation and body velocity estimates for each dataset is shown in Table 1.

[0126] Table 1

[0127] As shown in Table 1, the VIO+Dopp+Psr method performs best in terms of average position and velocity accuracy, demonstrating its ability to effectively eliminate VIO drift. It is also observed that the VIO+Dopp method achieves higher accuracy than the VIO+GPS vel method. It is worth noting that, on long trajectory datasets, adding Psr to the VIO+Dopp fusion further improves position accuracy, but may result in a slight decrease in position accuracy on short trajectory datasets.

[0128] Dpsr and Ddcp measurements: With base station support, Dpsr and Ddcp measurement updates can be performed. The inventors conducted experiments in the following configurations: VIO tightly coupled with Dopp and Dpsr (VIO+Dopp+Dpsr), and VIO tightly coupled with Dopp and Ddcp (VIO+Dopp+Ddcp).

[0129] Table 2

[0130] The accuracy results are shown in Table 2. When the RTK module successfully resolves integer ambiguities, the VIO+Dopp+Ddcp method achieves the highest position accuracy. However, when integer ambiguities cannot be resolved, the VIO+Dopp+Dpsr method still demonstrates higher position accuracy than the VIO+Dopp+Psr method. Regarding velocity accuracy, adding Psr, Dpsr, and Ddcp measurements to the VIO+Dopp fusion achieves comparable accuracy.

[0131] The inventors also conducted ablation experiments, excluding IMU integration from GNSS measurement updates in the VIO+Dopp+Psr method. The results in Table 2 show that excluding IMU integration significantly degrades both position and velocity accuracy.

[0132] To study the impact of online GNSS-IMU extrinsic parameter estimation, the inventors conducted experiments using accurate and poor initial GNSS-IMU extrinsic parameter parameters, and enabled and disabled the GNSS-IMU extrinsic parameter calibration function, respectively.

[0133] Table 3

[0134] Table 3 shows the results of online GNSS-IMU extrinsic parameter estimation under different configurations. Given poor initial extrinsic parameters (x-axis error of 0.3 meters), enabling GNSS-IMU extrinsic parameter estimation improves all velocity accuracies and almost all position accuracies. Given accurate initial extrinsic parameters, enabling GNSS-IMU extrinsic parameter estimation has a relatively small effect, with only a slight improvement in average position and velocity accuracy.

[0135] Figures 5, 6, and 7 demonstrate how the estimated error of GNSS-IMU extrinsic calibration gradually converges to almost zero given poor initial GNSS-IMU extrinsic parameters. Furthermore, the velocity error is significantly reduced with GNSS-IMU extrinsic calibration compared to without extrinsic calibration. These results confirm that GNSS-IMU extrinsic estimation can effectively mitigate the effects of extrinsic parameter degradation, thereby improving positioning accuracy.

[0136] Table 4

[0137] Table 4 shows the CPU usage and runtime of the VIO+Dopp+Dpsr method. The inventors replayed the dataset on a laptop equipped with an Intel Core i7-10750U 2.60GHz×12 core CPU and 16GB of memory, and conducted actual flight tests on a drone platform equipped with a Qualcomm RB5 processor. As shown in Table 4, compared to the VIO method, the fusion of Dopp and Dpsr measurements increased the runtime of the estimator algorithm by 13.77%, accounting for 2.76% of the total processing time. The CPU usage only increased slightly from 22.77% to 23.01%, indicating that the computational cost brought by the GNSS measurement update is almost negligible. In addition, the actual flight tests show that the disclosed embodiments can achieve real-time performance even on resource-constrained devices.

[0138] In addition, the inventors also compared the embodiment of the present disclosure with two other state-of-the-art methods, GVINS and InGVIO, on public data sets. The experiment used three public data sets, which contained IMU (inertial measurement unit) data, binocular image data, Doppler (Dopp) and pseudorange (Psr) measurement data. The comparison of the root mean square error of translation (meters) and the average time (milliseconds) is shown in Table 5, which shows the accuracy and efficiency of each algorithm. Figures 8 and 9 show the trajectory and position errors of the three algorithms. It should be noted that all trajectories affected by unreliable RTK (real-time dynamic carrier phase difference) data have been eliminated in the analysis.

[0139] Table 5

[0140] The results in Table 5 show that the disclosed embodiment exhibits a competitive advantage in accuracy due to its excellent numerical stability and ability to update GNSS measurements using IMU integration. Furthermore, thanks to the efficiency gained from the effective use of the square root form, the disclosed embodiment achieves the lowest computation time.

[0141] This disclosed embodiment deeply integrates visual, inertial, and raw GNSS data (including pseudorange, Doppler shift, single-difference pseudorange, and double-difference carrier phase) within an efficient SRI-SWF framework. It uses a filter-based initialization method to adaptively converge reference frame transformations, leverages IMU integration techniques to update state using delayed and asynchronous GNSS measurements, and simultaneously calibrates extrinsic parameters between the GNSS and IMU online. Extensive experimental results demonstrate that this disclosed embodiment outperforms other approaches in terms of accuracy and efficiency.

[0142] In the embodiments of the present disclosure, the terms “first,” “second,” and “third” are used for descriptive purposes only and should not be understood as indicating or implying relative importance.

[0143] In this disclosure, the term "and / or" simply describes an association relationship between related objects, indicating that three possible relationships exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0144] Furthermore, although the steps of the methods of the present disclosure are depicted in a particular order in the drawings, this does not require or imply that the steps must be performed in this particular order, or that all illustrated steps must be performed to achieve desired results.

[0145] In some embodiments, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0146] Based on the same inventive concept, the present disclosure also provides a positioning system, as described in the following embodiments. Since the principles of the positioning system embodiments are similar to those of the above-mentioned method embodiments, the implementation of the positioning system embodiments can refer to the implementation of the above-mentioned method embodiments, and the repeated parts will not be repeated.

[0147] FIG10 shows a schematic diagram of a positioning system according to an embodiment of the present disclosure. As shown in FIG10 , the positioning system includes:

[0148] The information acquisition module 1002 is used to obtain reference information of the object to be positioned at at least one moment, the reference information including inertial measurement unit (IMU) information, visual information and global navigation satellite system (GNSS) information; the IMU measurement module 1004 is used to perform IMU state propagation based on the IMU information to obtain IMU positioning data; the GNSS measurement module 1006 is used to perform GNSS state propagation based on the GNSS information to obtain GNSS positioning data; the positioning module 1008 is used to locate the object to be positioned based on the IMU positioning data, GNSS positioning data and visual information through a square root inverse sliding window filter (SRI-SWF).

[0149] In some embodiments, the positioning system may further include a GNSS measurement update module.

[0150] The GNSS measurement update module is used to update the GNSS measurement based on the pseudorange, Doppler frequency shift, single-difference pseudorange and double-difference carrier phase in the GNSS information.

[0151] In some embodiments, the GNSS measurement update module is configured to use IMU integration to model delayed and asynchronous GNSS measurements, wherein the GNSS measurements include pseudorange measurements, Doppler shift measurements, single-difference pseudorange measurements, and double-difference carrier phase measurements.

[0152] In some embodiments, the positioning system may further include an external parameter calibration module.

[0153] The extrinsic parameter calibration module is used to use Doppler shift update to converge to obtain GNSS-IMU extrinsic parameters when the IMU excitation is greater than a preset threshold.

[0154] In some embodiments, the GNSS measurement module 1006 is used to calculate the GNSS receiver clock error and GNSS receiver clock drift based on the pseudorange and Doppler frequency shift in the GNSS information; perform GNSS state propagation based on the GNSS clock error and clock drift to obtain IMU positioning data.

[0155] In some embodiments, a GNSS receiver clock error and a GNSS receiver clock drift are calculated based on the pseudorange and Doppler frequency shift in the GNSS information, including: calculating the residual of the pseudorange based on the pseudorange in the GNSS information; calculating the residual of the Doppler frequency shift based on the Doppler frequency shift in the GNSS information; and determining the GNSS receiver clock error and the GNSS receiver clock drift based on the residual of the pseudorange and the residual of the Doppler frequency shift.

[0156] In some embodiments, determining a GNSS receiver clock error and a GNSS receiver clock drift based on the residual of the pseudorange and the residual of the Doppler frequency shift includes: applying a random sampling consistency RANSAC algorithm to perform data cleaning on the residual of the pseudorange and the residual of the Doppler frequency shift; and determining the GNSS receiver clock error and the GNSS receiver clock drift based on the residual of the pseudorange and the residual of the Doppler frequency shift after data cleaning.

[0157] In some embodiments, the positioning system may further include a coordinate conversion module.

[0158] The coordinate conversion module is used to determine the transformation relationship between the GNSS coordinate system and the VIO coordinate system based on a filtering method; based on the transformation relationship between the GNSS coordinate system and the VIO coordinate system, the transformation relationship between the VIO coordinate system and the North-East Earth (NED) coordinate system, and the transformation relationship between the NED coordinate system and the ECEF coordinate system, the transformation relationship between the visual inertial odometry VIO coordinate system and the Earth-centered Earth-fixed ECEF coordinate system is determined.

[0159] In some embodiments, the positioning system may further include a data processing module.

[0160] The data processing module is used to extract the real-time positioning and map creation SLAM features of the object to be located from the visual information; based on the IMU information and the transformation relationship between the VIO coordinate system and the ECEF coordinate system, the historical IMU clone and parameter state vector are determined. The historical IMU clone includes the cloned IMU state at each moment. The parameter state vector includes the external parameters from the IMU to the camera, the external parameters from the GNSS receiver to the IMU, the GNSS receiver clock error, and the GNSS receiver clock drift; based on the IMU information, the GNSS receiver clock error, and the GNSS receiver clock drift, the additional IMU state is determined;

[0161] The positioning module 1008 is used to apply SRI-SWF to fuse SLAM features, historical IMU clones, parameter state vectors and IMU additional states to locate the object that needs to be positioned.

[0162] The concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0163] Regarding the positioning system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the positioning method, and will not be elaborated here.

[0164] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, such division is not mandatory.

[0165] In fact, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0166] Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0167] The electronic device provided by an embodiment of the present disclosure is described below with reference to Figure 11. The electronic device 1100 shown in Figure 11 is merely an example and should not limit the functions and scope of use of the embodiment of the present disclosure.

[0168] FIG11 shows a schematic diagram of the architecture of an electronic device 1100 provided by an embodiment of the present disclosure. As shown in FIG11 , the electronic device 1100 includes but is not limited to: at least one processor 1110 and at least one memory 1120 .

[0169] The memory 1120 is used to store instructions.

[0170] In some embodiments, the memory 1120 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 11201 and / or a cache memory unit 11202 , and may further include a read-only memory unit (ROM) 11203 .

[0171] In some embodiments, the memory 1120 may also include a program / utility 11204 having a set (at least one) of program modules 11205, such program modules 11205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0172] In some embodiments, the memory 1120 may store an operating system, which may be a real-time operating system (RTX), LINUX, UNIX, WINDOWS, or OS X.

[0173] In some embodiments, data may also be stored in the memory 1120 .

[0174] As an example, the processor 1110 may read data stored in the memory 1120 . The data may be stored at the same storage address as the instruction, or the data may be stored at a different storage address from the instruction.

[0175] The processor 1110 is configured to call instructions stored in the memory 1120 to implement the steps of various exemplary embodiments of the present disclosure described in the "Exemplary Method" section above. For example, the processor 1110 may execute the steps of the aforementioned method embodiment.

[0176] It should be noted that the processor 1110 may be a general-purpose processor or a dedicated processor. The processor 1110 may include one or more processing cores, and the processor 1110 executes various functional applications and data processing by running instructions.

[0177] In some embodiments, the processor 1110 may include a central processing unit (CPU) and / or a baseband processor.

[0178] In some embodiments, the processor 1110 may determine an instruction based on the priority identifier and / or function category information carried in each control instruction.

[0179] In the present disclosure, the processor 1110 and the memory 1120 may be provided separately or integrated together.

[0180] As an example, the processor 1110 and the memory 1120 may be integrated on a single board or a system on chip (SOC).

[0181] 11 , the electronic device 1100 is implemented as a general-purpose computing device and may further include a bus 1130 .

[0182] The bus 1130 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0183] The electronic device 1100 may also communicate with one or more external devices 1140 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 1100, and / or any device that enables the electronic device 1100 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication may be performed through an input / output (I / O) interface 1150.

[0184] Furthermore, the electronic device 1100 can also communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through the network adapter 1160 .

[0185] As shown in FIG. 11 , the network adapter 1160 communicates with other modules of the electronic device 1100 through the bus 1130 .

[0186] It should be understood that although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0187] It should be understood that the structures illustrated in the embodiments of the present disclosure do not constitute a specific limitation on the electronic device 1100. In other embodiments of the present disclosure, the electronic device 1100 may include more or fewer components than those shown in FIG11, or may combine or separate certain components, or may have different component arrangements. The components shown in FIG11 may be implemented in hardware, software, or a combination of software and hardware.

[0188] The present disclosure also provides a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed by a processor, the positioning method described in the above method embodiment is implemented.

[0189] The computer-readable storage medium in the embodiments of the present disclosure is a computer instruction that can be sent, propagated or transmitted for use by or in conjunction with an instruction execution system, apparatus or device.

[0190] As an example, computer readable storage media are non-volatile storage media.

[0191] In some embodiments, more specific examples of computer-readable storage media in the present disclosure may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, a USB flash drive, a mobile hard disk, or any suitable combination of the foregoing.

[0192] In the embodiments of the present disclosure, the computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer instructions (readable program codes).

[0193] Such a propagated data signal may take any of a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof.

[0194] In some examples, computing instructions contained on a computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0195] The embodiments of the present disclosure further provide a computer program product, which stores instructions. When the instructions are executed by a computer, the computer implements the positioning method described in the above method embodiment.

[0196] The above instructions may be program codes. In specific implementation, the program codes may be written in any combination of one or more programming languages.

[0197] Programming languages ​​include object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages.

[0198] The program code may execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0199] Where a remote computing device is involved, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0200] The embodiment of the present disclosure further provides a chip, comprising at least one processor and an interface;

[0201] An interface for providing program instructions or data to at least one processor;

[0202] At least one processor is configured to execute program instructions to implement the positioning method described in the above method embodiment.

[0203] In some embodiments, the chip may further include a memory for storing program instructions and data, and the memory may be located inside or outside the processor.

[0204] Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments may be implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software, which may be collectively referred to herein as a "circuit," "module," or "system."

[0205] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein.

[0206] This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered as exemplary only, with the true scope of the present disclosure being indicated by the appended claims.

Claims

1. A positioning method, characterized in that: include: Acquiring reference information of an object to be positioned at at least one moment, the reference information including inertial measurement unit (IMU) information, visual information, and global navigation satellite system (GNSS) information; Based on the IMU information, IMU state propagation is performed to obtain IMU positioning data; Based on the GNSS information, GNSS status propagation is performed to obtain GNSS positioning data; The object to be positioned is positioned based on the IMU positioning data, the GNSS positioning data and the visual information through a square root inverse sliding window filter (SRI-SWF).

2. The method according to claim 1, characterized in that The method further comprises: GNSS measurement update is performed based on the pseudorange, Doppler frequency shift, single-difference pseudorange and double-difference carrier phase in the GNSS information.

3. The method according to claim 2, characterized in that The updating of GNSS measurements based on the pseudorange, Doppler shift, single-difference pseudorange, and double-difference carrier phase in the GNSS information includes: IMU integration is used to model delayed and asynchronous GNSS measurements, including pseudorange measurements, Doppler shift measurements, single-difference pseudorange measurements, and double-difference carrier phase measurements.

4. The method according to claim 1, wherein The method further comprises: When the IMU excitation is greater than the preset threshold, the Doppler shift is used to update and converge to obtain the GNSS-IMU external parameters.

5. The method according to claim 1, wherein Based on the GNSS information, GNSS status propagation is performed to obtain IMU positioning data, including: Calculating the GNSS receiver clock error and the GNSS receiver clock drift based on the pseudorange and Doppler shift in the GNSS information; Based on the GNSS receiver clock difference and the GNSS receiver clock drift, GNSS state propagation is performed to obtain IMU positioning data.

6. The method according to claim 5, characterized in that The calculating, based on the pseudorange and Doppler shift in the GNSS information, the GNSS receiver clock error and the GNSS receiver clock drift, includes: Calculating the residual of the pseudorange based on the pseudorange in the GNSS information; Calculating a Doppler shift residual based on the Doppler shift in the GNSS information; The GNSS receiver clock error and the GNSS receiver clock drift are determined based on the residual of the pseudorange and the residual of the Doppler frequency shift.

7. The method according to claim 6, characterized in that The determining the GNSS receiver clock error and the GNSS receiver clock drift based on the residual of the pseudorange and the residual of the Doppler frequency shift includes: Applying a random sampling consistency RANSAC algorithm to perform data cleaning on the residual of the pseudorange and the residual of the Doppler frequency shift; The GNSS receiver clock error and the GNSS receiver clock drift are determined based on the residual of the pseudorange and the residual of the Doppler frequency shift after data cleaning.

8. The method according to claim 5, characterized in that The method further comprises: Based on the filtering method, the transformation relationship between the GNSS coordinate system and the VIO coordinate system is determined; Based on the transformation relationship between the GNSS coordinate system and the VIO coordinate system, the transformation relationship between the VIO coordinate system and the North-East Earth (NED) coordinate system, and the transformation relationship between the NED coordinate system and the ECEF coordinate system, the transformation relationship between the visual inertial odometry VIO coordinate system and the Earth-centered Earth-fixed ECEF coordinate system is determined.

9. The method according to claim 8, characterized in that The method further comprises: Extracting the real-time positioning and map creation SLAM features of the object to be located from the visual information; Based on the IMU information and the transformation relationship between the VIO coordinate system and the ECEF coordinate system, determine a historical IMU clone and a parameter state vector, wherein the historical IMU clone includes the cloned IMU state at each moment, and the parameter state vector includes the external parameters from the IMU to the camera, the external parameters from the GNSS receiver to the IMU, the GNSS receiver clock error, and the GNSS receiver clock drift; Determining an additional IMU state based on the IMU information, the GNSS receiver clock error, and the GNSS receiver clock drift; The method of positioning the object to be positioned based on the IMU positioning data, the GNSS positioning data, and the visual information by using a square root inverse sliding window filter (SRI-SWF) includes: The SRI-SWF is applied to fuse the SLAM features, the historical IMU clones, the parameter state vectors, and the IMU additional states to locate the object that needs to be located.

10. An electronic device, characterized in that: include: a memory for storing instructions; A processor, configured to call instructions stored in the memory to implement the positioning method according to any one of claims 1 to 9.

Images