Integrated navigation method for generating GNSS random model with assistance of fisheye image

Abstract

The invention provides an integrated navigation method for generating a GNSS (Global Navigation Satellite System) random model with the assistance of a fisheye image, which comprises the following steps: acquiring GNSS observation value information of a base station and a moving station, constructing a single-difference observation equation, and estimating a receiver clock error by utilizing a predicted carrier pose and robustness to obtain a single-difference residual error; converting the satellite feature vector projection to a pixel plane matrix by using an omnidirectional model, and associating a fisheye image with GNSS features; constructing a deep neural network based on an attention mechanism, inputting a fisheye image and GNSS features, and outputting a GNSS random model reflecting NLOS and multipath effect influence; and fusing the EKF with the GNSS and IMU observation values, and inputting the GNSS random model to obtain a pose estimation value at a corresponding moment. The GNSS random model suitable for the urban complex scene can be generated, the interference of the NLOS and the multipath effect in the urban complex scene is remarkably reduced, and the GNSS positioning performance is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of navigation technology, and in particular to a combined navigation method for generating GNSS stochastic models with the assistance of fisheye images. Background Technology

[0002] Global Navigation Satellite System (GNSS) provides global, high-precision navigation and positioning services and has been widely used in various fields such as vehicle positioning and autonomous navigation of unmanned systems. However, in complex urban environments, frequent obstruction by objects such as tall buildings and trees can cause frequent occurrences of non-line-of-sight signals and multipath effects, resulting in a significant deterioration in the positioning performance of GNSS systems.

[0003] For non-line-of-sight (NLOS) and multipath signals, machine learning methods such as decision trees and support vector products can be used to construct signal classification models for identification and detection based on indicators such as satellite elevation angle, signal-to-noise ratio, and pseudorange residuals. Currently, machine learning-based methods have achieved some success in NLOS identification, but their performance is limited by the limited feature dimensions of GNSS signals. Based on the physical mechanism of NLOS generation, GNSS systems can expand the input feature dimensions by introducing external information reflecting the observation environment to achieve accurate identification of anomalous observations. 3D maps are commonly used external auxiliary information. By analyzing the geometric relationship between station-satellite vectors and 3D maps, and using methods such as shadow matching and ray tracing, NLOS observations can be effectively processed, improving positioning accuracy. However, maintaining high-precision, high-timeliness 3D maps requires significant costs. In addition to 3D maps, the sky view provided by fisheye cameras can also reflect the GNSS observation environment, assisting in the identification of NLOS signals, thereby eliminating or downweighting anomalous observations and improving positioning accuracy. Fisheye cameras can provide observation environment information at a relatively low hardware cost. However, because image semantics can only convey very limited information, it cannot reflect the degree of interference of occlusion on the signal, nor can it clearly indicate the accuracy of semantic recognition. This leads to the loss of most of the information in the image during the semantic acquisition process, failing to fully exploit the high-order features in the image that reflect the GNSS observation environment, severely limiting the performance of related methods.

[0004] In summary, in complex urban scenarios, NLOS and multipath signals severely impact the positioning performance of GNSS systems. Machine learning-based NLOS identification methods can achieve some success, but their performance is limited by the finite feature dimensions of GNSS. 3D map-based solutions can effectively detect NLOS observations; however, maintaining timely 3D maps is costly. Fisheye cameras can provide environmental information at a lower cost, but current methods only utilize occlusion semantic information extracted from images, failing to fully exploit high-dimensional features within the images. Summary of the Invention

[0005] This invention provides a combined navigation method for generating GNSS random models with the assistance of fisheye images, in order to overcome the deficiencies in the existing technology.

[0006] In a first aspect, the present invention provides a method for integrated navigation that uses fisheye images to assist in generating GNSS stochastic models, comprising: Obtain GNSS observation information from the base station and rover, construct a single-difference observation equation, use the predicted carrier pose to robustly estimate the receiver clock error, and obtain the single-difference residual; An omnidirectional model is used to project and transform satellite feature vectors onto a pixel plane matrix, thus associating fisheye images with GNSS features; A deep neural network is constructed based on the attention mechanism. The input is a fisheye image and GNSS features, and the output is a GNSS stochastic model that reflects the effects of NLOS and multipath effects. By fusing GNSS and IMU observations with EKF and inputting them into the GNSS stochastic model, the pose estimate for the corresponding time moment is obtained.

[0007] According to the present invention, a combined navigation method for generating a GNSS stochastic model assisted by fisheye images acquires GNSS observation information from a base station and a rover, constructs a single-difference observation equation, uses the predicted carrier pose to robustly estimate the receiver clock error, and obtains the single-difference residual, including: The single-difference observation equation can be expressed as:

[0008] in, Indicates a simple difference operator; Indicates the geometric distance between the station and the satellite; Indicates receiver clock bias; This indicates the observation error introduced by NLOS; This indicates the observation error introduced by multipath propagation; Indicates observation noise; Represents the observation values ​​at the base station; Compared with the mobile observations, neglecting the error term in the base station observations, the single-difference observation equation is approximated as follows:

[0009] Among them, superscript Indicates a mobile station; The carrier pose is predicted using an IMU, electronic compass, and visual laser odometry. Substituting the predicted carrier pose into the single-difference observation equation, the receiver clock error is estimated using median estimation and random sample consensus robust estimation. Based on the receiver clock error, the single-difference residual is obtained:

[0010] in, This represents the single-difference residual.

[0011] According to the present invention, a combined navigation method for generating a GNSS stochastic model assisted by fisheye images employs an omnidirectional model to project and transform satellite feature vectors onto a pixel plane matrix, and associates fisheye images with GNSS features, including: Based on predicted carrier location ,attitude The transformation equation for converting satellite coordinates from the ECEF coordinate system to the carrier coordinate system is as follows:

[0012] in, For the carrier coordinate system, Using the ECEF coordinate system, express The satellite positions under the system, express The satellite positions under the system; Based on the rotation and translation relationships between the carrier coordinate system and the camera coordinate system, the position satellite in the carrier coordinate system is transformed to the camera coordinate system. The corresponding transformation equation is:

[0013] in, For the camera coordinate system, express The satellite positions under the system, , These represent the rotational and translational relationships between the carrier coordinate system and the camera coordinate system, respectively. An omnidirectional projection model is used to project the satellite position in the camera coordinate system onto the normalized spherical coordinate system. Department:

[0014] in, These are the satellite's coordinates in a normalized spherical coordinate system. These are the satellite's coordinates in the camera's coordinate system; Normalized spherical coordinate system along the optical axis Translation of the origin A new coordinate system is obtained. The satellite is The coordinates of the system are:

[0015] Will Satellite positions projected onto a normalized plane coordinate system On the tether, the satellite is The coordinates of the system are:

[0016] Using a camera pinhole model, the pixel coordinates corresponding to the satellite's position can be obtained by projection. The pixel coordinate system is represented as follows:

[0017] in, , This represents the pixel coordinates after the satellite is projected onto the pixel plane. , , , This represents the focal length and optical center offset of the camera pinhole model.

[0018] According to the present invention, a combined navigation method for generating a GNSS stochastic model assisted by fisheye images is provided. This method constructs a deep neural network based on an attention mechanism, takes a fisheye image and GNSS features as input, and outputs a GNSS stochastic model reflecting the effects of NLOS and multipath effects. The method includes: Image modules are constructed using multiple cascaded self-attention layers to extract coarse features that provide high resolution and fine-grained features that provide low resolution from fisheye images. A fusion module is constructed based on a multi-head cross-attention layer, which tightly integrates fisheye image features and GNSS features extracted by the image processing module; The GNSS stochastic model prediction values ​​are output using the perceptron layer.

[0019] According to the present invention, a combined navigation method for generating a GNSS stochastic model assisted by fisheye images is provided, which constructs a fusion module based on a multi-head cross-attention layer, and tightly fuses fisheye image features and GNSS features extracted by the image processing module, including: Image features Using GNSS features as query and value Using the key, a cross-attention mechanism is applied to calculate the information fusion result. The formula for calculating information fusion is as follows:

[0020] in, , and This represents the query, key, and value matrix in the attention layer. , and This represents the linear transformation matrix corresponding to the query, key, and value matrices. , and This represents the bias matrix corresponding to the query, key, and value matrices.

[0021] According to the integrated navigation method for generating a GNSS stochastic model assisted by fisheye images provided by the present invention, after constructing a deep neural network based on an attention mechanism, inputting fisheye images and GNSS features, and outputting a GNSS stochastic model reflecting the effects of NLOS and multipath effects, the method further includes: By leveraging the correlation between the image processing module and the semantic segmentation task in the network, a transfer learning strategy is adopted to pre-train the image processing module in the network using a semantic segmentation dataset; The pose reference results obtained by post-processing high-precision integrated navigation equipment are applied, and the single-difference pseudorange residuals of GNSS observations are calculated in reverse. The single-difference pseudorange residuals are used as the label values ​​of the GNSS stochastic model in network training, and MSE is used as the loss function for network training.

[0022] Secondly, the present invention also provides a combined navigation system for generating GNSS stochastic models with the assistance of fisheye images, comprising: The prediction module is used to acquire GNSS observation information from the base station and the rover, construct a single-difference observation equation, and use the predicted carrier pose to robustly estimate the receiver clock error to obtain the single-difference residual. The association module is used to project and transform satellite feature vectors onto a pixel plane matrix using an omnidirectional model, and associate fisheye images with GNSS features; The generation module is used to build a deep neural network based on the attention mechanism. It takes fisheye images and GNSS features as input and outputs a GNSS stochastic model that reflects the effects of NLOS and multipath effects. The estimation module is used to fuse EKF with GNSS and IMU observations, input them into the GNSS stochastic model, and obtain the pose estimate for the corresponding time.

[0023] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the combined navigation method for generating GNSS stochastic models with fisheye images as described above.

[0024] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the combined navigation method for generating GNSS stochastic models with fisheye images as described above.

[0025] This invention provides a combined navigation method for generating GNSS stochastic models using fisheye images. Addressing the issue of NLOS and multipath signal interference with GNSS system positioning performance, it proposes a method for generating stochastic models by fusing satellite features and fisheye images. This method can tightly integrate information from multiple sensors, predict the observation quality of GNSS signals, and accurately adjust the GNSS stochastic model. Furthermore, it constructs a stochastic model prediction network that correlates GNSS and fisheye images, building an image processing module and an information fusion module based on self-attention and cross-attention mechanisms, respectively. The image processing module, which cascades multiple self-attention layers, can extract high-resolution coarse features and low-resolution fine-grained features from fisheye images. In the information fusion module, image features are used as queries and values, and GNSS features are used as keys, achieving a tight correlation between the two types of information and accurately predicting the GNSS stochastic model. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is one of the flowcharts of the integrated navigation method for generating GNSS random models with the assistance of fisheye images provided by the present invention; Figure 2 This is the second flowchart of the integrated navigation method for generating GNSS random models with the assistance of fisheye images provided by the present invention; Figure 3 This is the network structure diagram provided by the present invention; Figure 4 This is an example diagram of transfer learning provided by the present invention; Figure 5 This is a schematic diagram of the integrated navigation system for generating GNSS random models with the assistance of fisheye images, provided by the present invention. Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] To address the problem of NLOS and multipath effects interfering with the positioning performance of GNSS systems in existing technologies, this invention constructs a stochastic model prediction network for observations associated with GNSS fisheye images. Based on self-attention and cross-attention mechanisms, an image processing module and an information fusion module are constructed respectively, which can closely integrate information from multiple sensors, predict the observation quality of GNSS signals, and achieve accurate adjustment of the stochastic model of the GNSS system.

[0030] Figure 1 This is one of the flowcharts illustrating the integrated navigation method for generating GNSS stochastic models with fisheye image assistance provided in this embodiment of the invention, such as... Figure 1 As shown, it includes: Step 100: Obtain GNSS observation information from the base station and rover, construct a single-difference observation equation, use the predicted carrier pose to robustly estimate the receiver clock error, and obtain the single-difference residual; Step 200: Use an omnidirectional model to project and transform the satellite feature vectors onto the pixel plane matrix, and associate the fisheye image with GNSS features; Step 300: Construct a deep neural network based on the attention mechanism, input fisheye image and GNSS features, and output a GNSS stochastic model that reflects the effects of NLOS and multipath effects; Step 400: Fuse the EKF with GNSS and IMU observations, input them into the GNSS stochastic model, and obtain the pose estimate for the corresponding time.

[0031] Specifically, such as Figure 2 As shown, it includes: Step 1, GNSS Feature Generation Module: This module addresses various error factors in the generation, propagation, and reception of GNSS signals. It constructs single-difference observations through inter-station differential analysis to mitigate the impact of satellite-end errors and atmospheric errors. Using the predicted carrier pose, it further robustly estimates the receiver clock error to obtain the single-difference residual. The single-difference residual, composed of NLOS, multipath effects, and observation noise, is strongly correlated with the anomaly degree of GNSS observations and is used as a GNSS feature in this invention. Furthermore, the signal-to-noise ratio (SNR) of GNSS observations reflects the intensity of the observations and is also used as a GNSS feature in this invention.

[0032] Step 2: Satellite Projection Module: Addressing the issue of correlating fisheye image information with GNSS features, the satellite positions in the Earth-Centered, Earth-Fixed (ECEF) coordinate system are projected onto a unit sphere centered on the carrier, based on the carrier's pose. Furthermore, an omnidirectional model is used to project the satellite positions on the sphere onto a pixel plane. This projection process implicitly incorporates the satellite's elevation and azimuth angles, thus achieving the correlation between the fisheye image and GNSS features.

[0033] Step 3: Predicting a Random GNSS Model Based on a Neural Network: To address the issue of GNSS observations being severely affected by NLOS and multipath signals in complex urban scenarios, a deep neural network is constructed to generate a random GNSS model. This model utilizes fisheye images and GNSS feature data to generate a random GNSS model that reflects the impact of NLOS and multipath signals, making it more suitable for complex urban scenarios compared to traditional signal-to-noise ratio and elevation angle random models.

[0034] In one embodiment, step 1 is implemented as follows: Step 1.1, Constructing the Single-Difference Observation Equation: After acquiring observations from the rover and the base station, the system performs inter-station observation difference analysis to reduce the impact of satellite-side errors, atmospheric errors, and ionospheric errors. The single-difference observation equation can be expressed as:

[0035] in, Indicates a simple difference operator; Indicates the geometric distance between the station and the satellite; Indicates receiver clock bias; This indicates the observation error introduced by NLOS; This indicates the observation error introduced by multipath propagation; Indicates observation noise; This represents the observation value from the base station.

[0036] Considering that base stations generally use high-precision receivers and choke antennas, the error term in base station observations is negligible compared to mobile observations. Therefore, the single-difference observation equation can be approximated as:

[0037] Among them, superscript Let represent the rover station. In this case, the single-difference observation equation depends only on the geometric distance, receiver clock error, and the error term experienced by the rover station.

[0038] Step 1.2, predict carrier pose: based on IMU mechanical arrangement, electronic compass, visual laser odometry and other methods, the carrier pose is recursively calculated in a short time.

[0039] Step 1.3, Robust estimation of receiver clock bias and single-difference residuals: After substituting the predicted carrier position information into the single-difference observation equation, the receiver clock bias can be robustly estimated using methods such as median estimation and random sample consistency. The single-difference residuals can be obtained based on the receiver clock bias:

[0040] in, This represents the single-difference residual, which is strongly correlated with errors introduced by NLOS, multipath effects, and observation noise, and is suitable as a GNSS characteristic reflecting the quality of observations.

[0041] In one embodiment, step 2 is implemented as follows: Step 2.1, Project the satellite position onto the carrier system: For the carrier coordinate system, Using the ECEF coordinate system. Based on the predicted carrier position. ,attitude The transformation equation for converting satellite coordinates from the ECEF coordinate system to the carrier coordinate system is as follows:

[0042] in, express The satellite positions under the system; express The location of the satellites under the system.

[0043] Step 2.2, Projecting the satellite position onto the camera coordinate system: based on the carrier coordinate system and the camera coordinate system (denoted as...). The rotation and translation relationships between the two coordinate systems are used to transform the position of the satellite in the carrier system to the camera coordinate system. The transformation equation is:

[0044] in, For the camera coordinate system, express The satellite positions under the system, , These represent the rotational and translational relationships between the carrier coordinate system and the camera coordinate system, respectively.

[0045] Step 2.3, Omnidirectional Model Projection: First, project the camera coordinate system... Satellite positions projected onto normalized spherical coordinate system Department:

[0046] in, These are the satellite's coordinates in a normalized spherical coordinate system. These are the satellite's coordinates in the camera's coordinate system.

[0047] Normalized spherical coordinate system along the optical axis Translation of the origin A new coordinate system is obtained. At this time, the satellite is The coordinates of the system are:

[0048] Will Satellite positions projected onto a normalized plane coordinate system The satellite is now in contact with the tether. The coordinates of the system are:

[0049] At this point, the satellite's position has been projected onto the normalized plane in front of the camera's optical center. Using the camera pinhole model, the pixel coordinates corresponding to the satellite's position can be obtained through projection. The pixel coordinate system is represented as follows:

[0050] in, , This represents the pixel coordinates after the satellite is projected onto the pixel plane; , , , This represents the focal length and optical center offset of the camera pinhole model.

[0051] After the above steps, The satellite positions under the system are projected onto the camera's pixel plane, thereby enabling the association between GNSS features and fisheye image features.

[0052] In one embodiment, step 3 is implemented as follows: Step 3.1, Construct a neural network based on the attention mechanism: The network structure is as follows Figure 3As shown, multiple cascaded self-attention layers are used to extract high-resolution coarse features and low-resolution fine-grained features from fisheye images. A multi-head cross-attention layer is then constructed to tightly fuse fisheye image features and GNSS features. Finally, a perceptron layer is used to output the prediction results.

[0053] The construction of the multi-head cross-attention layer here tightly integrates fisheye image features and GNSS features, specifically including: Image features Using GNSS features as query and value Using the key, a cross-attention mechanism is applied to calculate the information fusion result. The formula for calculating information fusion is as follows:

[0054] in, , and This represents the query, key, and value matrix in the attention layer. , and This represents the linear transformation matrix corresponding to the query, key, and value matrices. , and This represents the bias matrix corresponding to the query, key, and value matrices.

[0055] Step 3.2, Pre-training the image processing module using a transfer learning strategy: Considering that the image processing module in the proposed network has a similar task to the semantic segmentation network, this invention adopts a transfer learning strategy, as implemented below. Figure 4 As shown, using a semantic segmentation dataset to pre-train the parameters of the image processing module in the network significantly reduces the network's requirement for training data.

[0056] Step 3.3, Training the network: In order to obtain a neural network capable of accurately predicting stochastic models of GNSS observations in complex urban scenarios, a large amount of data in the urban environment is used to train the network.

[0057] The network training here specifically includes: By leveraging the correlation between the image processing module and the semantic segmentation task in the network, a transfer learning strategy is adopted to pre-train the image processing module in the network using a semantic segmentation dataset; The pose reference results obtained by post-processing high-precision integrated navigation equipment are applied, and the single-difference pseudorange residuals of GNSS observations are calculated in reverse. The single-difference pseudorange residuals are used as the label values ​​of the GNSS stochastic model in network training, and MSE is used as the loss function for network training.

[0058] In summary, the present invention has the following beneficial effects: (1) This invention provides a method for generating a GNSS stochastic model based on neural networks and fisheye images. In response to the problem of NLOS and multipath signal interference with the positioning performance of GNSS systems, a method for generating a stochastic model that integrates satellite features and fisheye images is proposed. This method can closely integrate information from multiple sensors, predict the observation quality of GNSS signals, and achieve accurate adjustment of the GNSS stochastic model. (2) This invention constructs a stochastic model prediction network that associates GNSS and fisheye images. Based on self-attention and cross-attention mechanisms, an image processing module and an information fusion module are constructed respectively. Among them, the image processing module is connected in series with multiple self-attention layers, which can extract high-resolution coarse features and low-resolution fine-grained features from fisheye images. In the information fusion module, image features are used as Query and Value, and GNSS features are used as Key to achieve close association between the two types of information and accurately predict GNSS stochastic models.

[0059] The following describes the integrated navigation system for generating GNSS random models with the assistance of fisheye images provided by the present invention. The integrated navigation system for generating GNSS random models with the assistance of fisheye images described below can be referred to in correspondence with the integrated navigation method for generating GNSS random models with the assistance of fisheye images described above.

[0060] Figure 5 This is a schematic diagram of the structure of the integrated navigation system for generating GNSS random models with the assistance of fisheye images, as provided in an embodiment of the present invention. Figure 5 As shown, it includes: a prediction module 51, an association module 52, a generation module 53, and an estimation module 54, wherein: The prediction module 51 is used to acquire GNSS observation information from the base station and the rover, construct a single-difference observation equation, and use the predicted carrier pose to robustly estimate the receiver clock error to obtain the single-difference residual. The association module 52 is used to project and transform the satellite feature vector onto the pixel plane matrix using an omnidirectional model, and associate the fisheye image with the GNSS features. The generation module 53 is used to construct a deep neural network based on an attention mechanism, input the fisheye image and GNSS features, and output a GNSS stochastic model that reflects the effects of NLOS and multipath effects. The estimation module 54 is used to fuse GNSS and IMU observations with EKF, input the GNSS stochastic model, and obtain the pose estimate at the corresponding time.

[0061] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640. The processor 610, communication interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a combined navigation method that uses fisheye images to assist in generating a GNSS stochastic model. This method includes: acquiring GNSS observation information from a base station and a rover; constructing a single-difference observation equation; using the predicted carrier pose to robustly estimate the receiver clock error and obtain the single-difference residual; using an omnidirectional model to project and transform satellite feature vectors onto a pixel plane matrix, and associating the fisheye image with GNSS features; constructing a deep neural network based on an attention mechanism, inputting the fisheye image and GNSS features, and outputting a GNSS stochastic model reflecting the effects of NLOS and multipath effects; fusing GNSS and IMU observations using an EKF (Electronic Keyframe Function) and inputting it into the GNSS stochastic model to obtain the pose estimate for the corresponding time.

[0062] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0063] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the integrated navigation method for generating a GNSS stochastic model with fisheye image assistance provided by the above methods. The method includes: acquiring GNSS observation information of a base station and a rover; constructing a single-difference observation equation; using the predicted carrier pose to robustly estimate the receiver clock error and obtain the single-difference residual; using an omnidirectional model to project and transform the satellite feature vector onto a pixel plane matrix and associate the fisheye image with the GNSS features; constructing a deep neural network based on an attention mechanism, inputting the fisheye image and GNSS features, and outputting a GNSS stochastic model reflecting the effects of NLOS and multipath effects; fusing GNSS and IMU observations with EKF and inputting them into the GNSS stochastic model to obtain the pose estimate at the corresponding time.

[0064] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a combined navigation method for generating a GNSS stochastic model with fisheye image assistance, as provided by the methods described above. The method includes: acquiring GNSS observation information from a base station and a rover; constructing a single-difference observation equation; using the predicted carrier pose to robustly estimate the receiver clock error and obtain the single-difference residual; using an omnidirectional model to project and transform the satellite feature vector onto a pixel plane matrix, and associating the fisheye image with GNSS features; constructing a deep neural network based on an attention mechanism, inputting the fisheye image and GNSS features, and outputting a GNSS stochastic model reflecting the effects of NLOS and multipath effects; fusing GNSS and IMU observations with an EKF and inputting the EKF into the GNSS stochastic model to obtain the pose estimate at the corresponding time.

[0065] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0066] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for integrated navigation that uses fisheye images to assist in generating GNSS stochastic models, characterized in that, include: Obtain GNSS observation information from the base station and rover, construct a single-difference observation equation, use the predicted carrier pose to robustly estimate the receiver clock error, and obtain the single-difference residual; An omnidirectional model is used to project and transform satellite feature vectors onto a pixel plane matrix, thus associating fisheye images with GNSS features; A deep neural network is constructed based on the attention mechanism. The input is a fisheye image and GNSS features, and the output is a GNSS stochastic model that reflects the effects of NLOS and multipath effects. By fusing GNSS and IMU observations with EKF and inputting them into the GNSS stochastic model, the pose estimate at the corresponding time moment is obtained.

2. The integrated navigation method for generating GNSS stochastic models with fisheye image assistance according to claim 1, characterized in that, Obtain GNSS observation information from the base station and rover, construct a single-difference observation equation, use the predicted carrier pose to robustly estimate the receiver clock error, and obtain the single-difference residuals, including: The single-difference observation equation can be expressed as: in, Indicates a simple difference operator; Indicates the geometric distance between the station and the satellite; Indicates receiver clock bias; This indicates the observation error introduced by NLOS; This indicates the observation error introduced by multipath propagation; Indicates observation noise; Represents the observation values ​​at the base station; Compared with the mobile observations, neglecting the error term in the base station observations, the single-difference observation equation is approximated as follows: Among them, superscript Indicates a mobile station; The carrier pose is predicted using an IMU, electronic compass, and visual laser odometry. Substituting the predicted carrier pose into the single-difference observation equation, the receiver clock error is estimated using median estimation and random sample consensus robust estimation. Based on the receiver clock error, the single-difference residual is obtained: in, This represents the single-difference residual.

3. The integrated navigation method for generating GNSS stochastic models with fisheye image assistance according to claim 1, characterized in that, An omnidirectional model is used to project and transform satellite feature vectors onto a pixel plane matrix, and fisheye images are associated with GNSS features, including: Based on predicted carrier location ,attitude The transformation equation for converting satellite coordinates from the ECEF coordinate system to the carrier coordinate system is as follows: in, For the carrier coordinate system, Using the ECEF coordinate system, express The satellite positions under the system, express The satellite positions under the system; Based on the rotation and translation relationships between the carrier coordinate system and the camera coordinate system, the position satellite in the carrier coordinate system is transformed to the camera coordinate system. The corresponding transformation equation is: in, For the camera coordinate system, express The satellite positions under the system, , These represent the rotational and translational relationships between the carrier coordinate system and the camera coordinate system, respectively. An omnidirectional projection model is used to project the satellite position in the camera coordinate system onto the normalized spherical coordinate system. Department: in, These are the satellite's coordinates in a normalized spherical coordinate system. These are the satellite's coordinates in the camera's coordinate system; Normalized spherical coordinate system along the optical axis Translation of the origin A new coordinate system is obtained. The satellite is The coordinates of the system are: Will Satellite positions projected onto a normalized plane coordinate system On the tether, the satellite is The coordinates of the system are: Using a camera pinhole model, the pixel coordinates corresponding to the satellite's position can be obtained by projection. The pixel coordinate system is represented as follows: in, , This represents the pixel coordinates after the satellite is projected onto the pixel plane. , , , This represents the focal length and optical center offset of the camera pinhole model.

4. The integrated navigation method for generating GNSS stochastic models with fisheye image assistance according to claim 1, characterized in that, A deep neural network based on an attention mechanism is constructed. Inputting fisheye images and GNSS features, it outputs a GNSS stochastic model reflecting the effects of NLOS and multipath propagation, including: Image modules are constructed using multiple cascaded self-attention layers to extract coarse features that provide high resolution and fine-grained features that provide low resolution from fisheye images. A fusion module is constructed based on a multi-head cross-attention layer, which tightly integrates fisheye image features and GNSS features extracted by the image processing module; The GNSS stochastic model prediction values ​​are output using the perceptron layer.

5. The integrated navigation method for generating GNSS stochastic models with fisheye image assistance according to claim 4, characterized in that, A fusion module is constructed based on a multi-head cross-attention layer, which tightly integrates fisheye image features and GNSS features extracted by the image processing module, including: Image features Using GNSS features as query and value Using the cross-attention mechanism as the key, the information fusion result is calculated. The formula for information fusion is as follows: in, , and This represents the query, key, and value matrix in the attention layer. , and This represents the linear transformation matrix corresponding to the query, key, and value matrices. , and This represents the bias matrix corresponding to the query, key, and value matrices.

6. The integrated navigation method for generating GNSS stochastic models with fisheye image assistance according to claim 4, characterized in that, After constructing a deep neural network based on the attention mechanism, taking fisheye images and GNSS features as input, and outputting a GNSS stochastic model reflecting the effects of NLOS and multipath effects, the following are also included: By leveraging the correlation between the image processing module and the semantic segmentation task in the network, a transfer learning strategy is adopted to pre-train the image processing module in the network using a semantic segmentation dataset; The pose reference results obtained by post-processing high-precision integrated navigation equipment are applied, and the single-difference pseudorange residuals of GNSS observations are calculated in reverse. The single-difference pseudorange residuals are used as the label values ​​of the GNSS stochastic model in network training, and MSE is used as the loss function for network training.

7. A combined navigation system for generating GNSS stochastic models with the assistance of fisheye images, characterized in that, include: The prediction module is used to acquire GNSS observation information from the base station and the rover, construct a single-difference observation equation, and use the predicted carrier pose to robustly estimate the receiver clock error to obtain the single-difference residual. The association module is used to project and transform satellite feature vectors onto a pixel plane matrix using an omnidirectional model, and associate fisheye images with GNSS features; The generation module is used to build a deep neural network based on the attention mechanism. It takes fisheye images and GNSS features as input and outputs a GNSS stochastic model that reflects the effects of NLOS and multipath effects. The estimation module is used to fuse EKF with GNSS and IMU observations, input them into the GNSS stochastic model, and obtain the pose estimate for the corresponding time.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the combined navigation method for generating GNSS random models with the assistance of fisheye images as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the combined navigation method for generating GNSS stochastic models with the assistance of fisheye images as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the combined navigation method for generating GNSS stochastic models with the assistance of fisheye images as described in any one of claims 1 to 6.

Images