A fish-eye image assisted generation GNSS random model combination navigation method

Abstract

This invention provides a combined navigation method for generating GNSS stochastic models using fisheye images, comprising: acquiring GNSS observation information from a base station and a rover; constructing a single-difference observation equation; robustly estimating receiver clock errors using predicted carrier pose to obtain single-difference residuals; projecting satellite feature vectors onto a pixel plane matrix using an omnidirectional model to correlate fisheye images with GNSS features; constructing a deep neural network based on an attention mechanism, inputting fisheye images and GNSS features, and outputting a GNSS stochastic model reflecting the effects of NLOS and multipath effects; fusing GNSS and IMU observations using an EKF (Extended Kinematics Function) and inputting the model to obtain pose estimates for the corresponding time. This invention can generate GNSS stochastic models suitable for complex urban scenarios, significantly reducing interference from NLOS and multipath effects in complex urban environments, and effectively improving GNSS positioning performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of navigation technology, and in particular to a combined navigation method for generating a GNSS random model assisted by a fisheye image. BACKGROUND

[0002] The global navigation satellite system (GNSS) can provide global and high-precision navigation and positioning services, and has been widely used in vehicle positioning, autonomous navigation of unmanned systems and other fields. However, in complex urban scenarios, frequent occlusion of high-rise buildings, trees and other objects can cause non-line-of-sight (NLOS) signals and multipath effects to frequently occur, resulting in a serious decline in the positioning performance of the GNSS system.

[0003] For non-line-of-sight (NLOS) and multipath signals, machine learning methods such as decision trees and support vector products can be used to construct a signal classification model based on indicators such as satellite elevation angle, signal-to-noise ratio and pseudo-range residual to identify and detect. At present, the method based on machine learning has achieved certain results in identifying NLOS, but its performance is restricted by the limited feature dimension of GNSS signals. According to the physical mechanism of NLOS generation, GNSS systems can expand the input feature dimension by introducing external information reflecting the observation environment to accurately identify abnormal observation values. 3D maps are commonly used external auxiliary information. By analyzing the geometric relationship between the station-star vector and the 3D map, using shadow matching and beam tracking methods, NLOS observation values can be effectively processed, and positioning accuracy can be improved. However, maintaining a high-precision and high-timeliness 3D map requires a high cost. In addition to 3D maps, the sky view provided by fisheye cameras can also reflect the GNSS observation environment, assist in identifying NLOS signals, and then eliminate or reduce the weight of abnormal observation values to improve positioning accuracy. Fisheye cameras can provide observation environment information at a relatively low hardware cost. However, since the information that can be transmitted by the image semantics is very limited, it cannot reflect the degree of interference of occlusion in the image on the signal, nor can it explicitly indicate the accuracy of semantic recognition. This type of scheme loses most of the information in the image in the process of obtaining semantics, and does not fully exploit the high-order features in the image that can reflect the GNSS observation environment, which severely restricts the performance of related methods.

[0004] In summary, in the urban complex scene, NLOS and multipath signals will seriously affect the positioning performance of the GNSS system. The NLOS identification method based on machine learning can achieve certain effect, but its performance is restricted by the limited feature dimension of GNSS. The scheme based on 3D map can effectively detect NLOS observation values, however, maintaining a high timeliness of 3D map requires a high cost. The fisheye camera can provide observation environment information at a lower cost, but the current scheme only uses the occlusion semantic information extracted from the image, and does not fully exploit the high-dimensional features in the image. SUMMARY

[0005] The application provides a combined navigation method for generating a GNSS random model assisted by a fisheye image, to solve the defects in the prior art.

[0006] In a first aspect, the application provides a combined navigation method for generating a GNSS random model assisted by a fisheye image, comprising:

[0007] Obtaining GNSS observation value information of a reference station and a rover station, constructing a single-difference observation equation, using a predicted carrier pose to robustly estimate a receiver clock error, and obtaining a single-difference residual;

[0008] Using an omnidirectional model to project and convert satellite feature vectors onto a pixel plane matrix, and associating the fisheye image with GNSS features;

[0009] Constructing a deep neural network based on an attention mechanism, inputting the fisheye image and GNSS features, and outputting a GNSS random model reflecting the influence of NLOS and multipath effects;

[0010] Fusing GNSS and IMU observation values using an EKF, inputting the GNSS random model, and obtaining a pose estimation value at the corresponding time.

[0011] According to the combined navigation method for generating a GNSS random model assisted by a fisheye image provided by the application, GNSS observation value information of a reference station and a rover station is obtained, a single-difference observation equation is constructed, a predicted carrier pose is used to robustly estimate a receiver clock error, and a single-difference residual is obtained, comprising:

[0012] The single-difference observation equation can be expressed as:

[0013]

[0014] wherein, represents a single-difference operator; represents the geometric distance between the station and the satellite; represents the receiver clock error; represents the observation value error introduced by NLOS; represents the observation value error introduced by multipath; represents an observation noise; represents a reference station observation value;

[0015] Compared with the flow observation value, the error term in the reference station observation value is ignored, and the single difference observation equation is approximately obtained:

[0016]

[0017] wherein the superscript represents a flow station;

[0018] The predicted carrier pose is obtained by using an IMU, an electronic compass and a visual laser odometry;

[0019] The predicted carrier pose is substituted into the single difference observation equation, the receiver clock error is estimated by using the median estimation and the random sample consensus robust estimation, and the single difference residual is obtained based on the receiver clock error:

[0020]

[0021] wherein, represents a single difference residual.

[0022] According to the fish-eye image assisted generation GNSS random model combination navigation method provided by the application, the satellite feature vector is projected and converted to the pixel plane matrix by using the omnidirectional model, the fish-eye image is associated with the GNSS feature, and the method comprises the following steps:

[0023] Based on the predicted carrier position , attitude , the satellite coordinates in the ECEF coordinate system are converted to the carrier coordinate system, and the corresponding conversion equation is:

[0024]

[0025] wherein, is the carrier coordinate system, is the ECEF coordinate system, represents the satellite position in the carrier coordinate system, represents the satellite position in the carrier coordinate system;

[0026] Based on the rotation relationship and the translation relationship between the carrier coordinate system and the camera coordinate system, the satellite position in the carrier coordinate system is converted to the camera coordinate system, and the corresponding conversion equation is:

[0027]

[0028] wherein, is the camera coordinate system, represents the satellite position in the carrier coordinate system, 、 R and t represent the rotation relationship and translation relationship between the carrier coordinate system and the camera coordinate system, respectively;

[0029] The satellite position in the camera coordinate system is projected onto the normalized spherical coordinate system using an omnidirectional projection model :

[0030]

[0031] wherein, is the coordinate of the satellite in the normalized spherical coordinate system, is the coordinate of the satellite in the camera coordinate system;

[0032] The origin of the normalized spherical coordinate system is translated along the optical axis to obtain a new coordinate system , and the coordinate of the satellite in the coordinate system is:

[0033]

[0034] The satellite position in the camera coordinate system is projected onto the normalized planar coordinate system , and the coordinate of the satellite in the coordinate system is:

[0035]

[0036] The pixel coordinate corresponding to the satellite position can be obtained by using a camera pinhole model, and the pixel coordinate system is represented as:

[0037]

[0038] wherein, , represents the pixel coordinate of the satellite after being projected onto the pixel plane, 、 、 、 represents the focal length and optical center offset of the camera pinhole model.

[0039] According to the fish-eye image assisted generation of a GNSS random model combined navigation method provided by the application, a deep neural network is constructed based on an attention mechanism, fish-eye images and GNSS features are input, and a GNSS random model reflecting the influence of NLOS and multipath effects is output, including:

[0040] An image module is constructed using multiple serial self-attention layers to extract coarse features with high resolution and fine-grained features with low resolution from the fish-eye images; ​​​​

[0041] The fusion module is constructed based on a multi-head cross attention layer, and fisheye image features extracted by the image processing module are closely fused with GNSS features;

[0042] The perception layer is used to output a GNSS random model prediction value.

[0043] According to the combination navigation method for generating a GNSS random model with the aid of a fisheye image provided by the application, a fusion module is constructed based on a multi-head cross attention layer, and fisheye image features extracted by an image processing module are closely fused with GNSS features, including:

[0044] The image features are taken as query and value, the GNSS features are taken as key, and a cross attention mechanism is applied to calculate an information fusion result, and a calculation formula of information fusion is as follows: The image features are taken as query and value, the GNSS features are taken as key, and a cross attention mechanism is applied to calculate an information fusion result, and a calculation formula of information fusion is as follows:

[0045]

[0046] wherein, 、 and denote query, key and value matrices in the attention layer, 、 and denote linear transformation matrices corresponding to the query, key and value matrices, 、 and denote bias matrices corresponding to the query, key and value matrices.

[0047] According to the combination navigation method for generating a GNSS random model with the aid of a fisheye image provided by the application, a deep neural network is constructed based on an attention mechanism, fisheye images and GNSS features are input, and a GNSS random model reflecting influences of NLOS and multipath effects is output, and the method further includes:

[0048] The correlation between the image processing module and the semantic segmentation task in the network is utilized, a transfer learning strategy is adopted, and the image processing module in the network is pre-trained using a semantic segmentation dataset;

[0049] The pose reference result obtained after post-processing of the high-precision integrated navigation equipment is applied, and the single-difference pseudo-range residual error of the GNSS observation value is calculated reversely;

[0050] The single-difference pseudo-range residual error is taken as a label value of the GNSS random model in network training, MSE is used as a loss function, and network training is performed.

[0051] ​In a second aspect, the present application also provides a fish-eye image assisted GNSS random model generation combined navigation system, comprising:

[0052] a prediction module configured to acquire GNSS observation value information of the reference station and the rover station, construct a single-difference observation equation, and obtain a single-difference residual by using a predicted carrier pose and robustly estimating a receiver clock error;

[0053] a correlation module configured to project and convert a satellite feature vector to a pixel plane matrix by using an omnidirectional model, and correlate the fish-eye image with the GNSS feature;

[0054] a generation module configured to construct a deep neural network based on an attention mechanism, input the fish-eye image and the GNSS feature, and output a GNSS random model reflecting the influence of NLOS and multipath effects;

[0055] an estimation module configured to fuse GNSS and IMU observation values by using an EKF, input the GNSS random model, and obtain a pose estimation value at a corresponding time.

[0056] In a third aspect, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the fish-eye image assisted GNSS random model generation combined navigation method according to any one of the above aspects when executing the program.

[0057] In a fourth aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the fish-eye image assisted GNSS random model generation combined navigation method according to any one of the above aspects.

[0058] The fish-eye image assisted GNSS random model generation combined navigation method provided by the present application proposes a random model generation method fusing satellite features and fish-eye images to solve the problem of NLOS and multipath signal interference on the positioning performance of a GNSS system, can tightly fuse a plurality of sensor information, predict the observation quality of a GNSS signal, and accurately adjust the GNSS random model; and further constructs a random model prediction network correlating GNSS and fish-eye images, and constructs an image processing module and an information fusion module based on self-attention and cross-attention mechanisms. The image processing module is connected in series with a plurality of self-attention layers, can extract coarse features with high resolution and fine-grained features with low resolution from the fish-eye image; in the information fusion module, the image features are used as Query and Value, and the GNSS features are used as Key, to realize the close correlation of the two kinds of information and accurately predict the GNSS random model. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to make the technical solutions in the present application or the prior art clearer, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0060] Figure 1 is one of the flowcharts of the fish-eye image assisted GNSS random model generated combined navigation method provided by the present application;

[0061] Figure 2 is the second flowchart of the fish-eye image assisted GNSS random model generated combined navigation method provided by the present application;

[0062] Figure 3 is the network structure provided by the present application;

[0063] Figure 4 is the transfer learning example diagram provided by the present application;

[0064] Figure 5 is the structure diagram of the fish-eye image assisted GNSS random model generated combined navigation system provided by the present application;

[0065] Figure 6 is the structure diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0066] In order to make the technical solutions in the present application or the prior art clearer, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0067] In view of the problem that NLOS and multipath effect interfere with the positioning performance of the GNSS system in the prior art, the present application constructs an observation value random model prediction network associated with the GNSS fish-eye image, constructs an image processing module and an information fusion module based on self-attention and cross-attention mechanisms, can tightly fuse multiple sensor information, predict the observation quality of the GNSS signal, and realize accurate adjustment of the GNSS system random model.

[0068] Figure 1 is one of the flowcharts of the fish-eye image assisted GNSS random model generated combined navigation method provided by the present application, as shown in Figure 1 , including:

[0069] Step 100: Obtain GNSS observation information of reference stations and rover stations, construct single-difference observation equations, use predicted carrier pose to robustly estimate receiver clock error, and obtain single-difference residuals;

[0070] Step 200: Project satellite feature vectors onto the pixel plane matrix using the omnidirectional model to convert and associate fish-eye images with GNSS features;

[0071] Step 300: Construct a deep neural network based on the attention mechanism, input fish-eye images and GNSS features, and output a GNSS stochastic model reflecting the influence of NLOS and multipath effects;

[0072] Step 400: Fuse GNSS and IMU observations using EKF, input the GNSS stochastic model, and obtain pose estimation values at corresponding time.

[0073] Specifically, as shown in Figure 2 includes:

[0074] Step 1: GNSS feature generation module: For various error factors in the generation, propagation, and reception of GNSS signals, single-difference observations are constructed to weaken the influence of satellite errors and atmospheric errors. The predicted carrier pose is used to further robustly estimate the receiver clock error, and single-difference residuals are obtained. Single-difference residuals are composed of NLOS, multipath effects, and observation noise, and are strongly related to the degree of abnormality of GNSS observations. In this invention, they are used as GNSS features. In addition, the signal-to-noise ratio of GNSS observations can reflect the strength of observations, which is used as a GNSS feature in this invention.

[0075] Step 2: Satellite projection module: For the problem of associating fish-eye image information with GNSS features, the satellite position in the Earth-Centered, Earth-Fixed (ECEF) coordinate system is projected onto the unit sphere centered on the carrier based on the carrier pose, and then the omnidirectional model is used to project the satellite position on the sphere onto the pixel plane. This projection process implicitly includes the satellite's elevation angle and azimuth angle information, achieving the association of fish-eye images with GNSS features.

[0076] Step 3: Predict GNSS stochastic model based on neural network: For the problem of GNSS observations being severely affected by NLOS and multipath signals in urban complex scenarios, a deep neural network is constructed to generate a GNSS stochastic model. Fish-eye images and GNSS feature data are used to drive the generation of a GNSS stochastic model that can reflect the influence of NLOS and multipath signals. Compared with traditional signal-to-noise ratio and elevation angle stochastic models, it is more suitable for urban complex scenarios.

[0077] In one embodiment, the implementation of step 1 is as follows:

[0078] Step 1.1, Constructing single-difference observation equation: After the system obtains the observation values from the flow station and the reference station, the inter-station observation value difference is differentiated to weaken the influence of satellite end error, atmospheric error and ionospheric error. The single-difference observation equation can be expressed as:

[0079]

[0080] wherein, represents a single-difference operator; represents the geometric distance between the station and the satellite; represents the receiver clock error; represents the observation error introduced by NLOS; represents the observation error introduced by multipath; represents the observation noise; represents the reference station observation value.

[0081] Considering that the reference station generally uses a high-precision receiver and a choke ring antenna, compared with the flow observation value, the error term in the reference station observation value can be ignored, and the single-difference observation equation can be approximated as:

[0082]

[0083] wherein, the superscript represents the flow station. At this time, the single-difference observation equation is only related to the geometric distance, the receiver clock error and the error term suffered by the flow station.

[0084] Step 1.2, Predicting carrier pose: Based on the IMU mechanical arrangement, electronic compass, visual laser odometry and other methods, the carrier pose is recursively predicted in a short time.

[0085] Step 1.3, Robustly estimating receiver clock error and single-difference residual: After the predicted carrier position information is substituted into the single-difference observation equation, the receiver clock error can be robustly estimated by using the median estimation, random sample consensus and other methods. Based on the receiver clock error, the single-difference residual can be obtained:

[0086]

[0087] wherein, represents the single-difference residual, which is strongly related to the errors introduced by NLOS, multipath effect and observation noise, and is suitable for reflecting the GNSS features of observation value quality.

[0088] In one embodiment, the implementation of step 2 is as follows:

[0089] Step 2.1, Projecting satellite position to carrier system: is the carrier coordinate system, ECEF coordinate system. Based on the predicted carrier position , attitude , the conversion equation from satellite coordinates in ECEF coordinate system to carrier coordinate system is:

[0090]

[0091] where, represents the satellite position in the carrier coordinate system; represents the satellite position in the camera coordinate system.

[0092] Step 2.2, satellite position projection to camera system: based on the rotation and translation relationship between the carrier coordinate system and the camera coordinate system (denoted as ), the satellite position in the carrier coordinate system is converted to the camera coordinate system. The conversion equation is:

[0093]

[0094] where, is the camera coordinate system, represents the satellite position in the camera coordinate system, , respectively represents the rotation relationship and translation relationship between the carrier coordinate system and the camera coordinate system.

[0095] Step 2.3, omnidirectional model projection: first, project the satellite position in the camera coordinate system to the normalized spherical coordinate system :

[0096]

[0097] where, is the satellite coordinate in the normalized spherical coordinate system, is the satellite coordinate in the camera coordinate system.

[0098] Translate the origin of the normalized spherical coordinate system by to get a new coordinate system , at this time the satellite coordinate in is:

[0099]

[0100] Project the satellite position in to the normalized planar coordinate system , at this time the satellite coordinate in is:

[0101]

[0102] At this time, the satellite position has been projected onto the normalized plane in front of the camera pinhole, and the pixel coordinates corresponding to the satellite position can be obtained by using the camera pinhole model. The pixel coordinate system is represented as:

[0103]

[0104] wherein, , represents the pixel coordinates of the satellite projected onto the pixel plane; , , , represents the focal length and optical center offset of the camera pinhole model.

[0105] After the above steps are processed, the satellite position under the system is projected onto the pixel plane of the camera, and the association of the GNSS features and the fisheye image features is realized based on this.

[0106] In one embodiment, the specific implementation of step 3 is as follows:

[0107] Step 3.1, constructing a neural network based on an attention mechanism: the network structure is as shown in Figure 3 , using multiple self-attention layers in series to extract coarse features providing high resolution and fine-grained features providing low resolution from the fisheye image, further constructing a multi-head cross-attention layer to tightly integrate the fisheye image features and the GNSS features, and finally using a perception machine layer to output the prediction result.

[0108] Here, the multi-head cross-attention layer is constructed to tightly integrate the fisheye image features and the GNSS features, which specifically includes:

[0109] The image features are taken as the query and the value, the GNSS features are taken as the key, the cross-attention mechanism is applied to calculate the information fusion result, and the calculation formula of the information fusion is:

[0110]

[0111] wherein, , and represent the query, key and value matrices in the attention layer, , and represent the linear transformation matrices corresponding to the query, key and value matrices, , With The bias matrix corresponding to the query, key and value matrix is represented.

[0112] Step 3.2, pre-training the image processing module using the transfer learning strategy: considering that the image processing module in the proposed network is similar to the task of the semantic segmentation network, the application adopts the transfer learning strategy, and the implementation scheme is as shown in Figure 4 The parameters of the image processing module in the network are pre-trained using the semantic segmentation data set, which significantly reduces the demand of the network on the training data.

[0113] Step 3.3, training the network: in order to obtain a neural network capable of accurately predicting the GNSS observation random model in a complex urban scene, a large amount of data in an urban environment is used for network training.

[0114] The network training specifically includes:

[0115] By utilizing the correlation between the image processing module in the network and the semantic segmentation task, the transfer learning strategy is adopted to pre-train the image processing module in the network using the semantic segmentation data set;

[0116] The pose reference result obtained by applying the high-precision integrated navigation equipment post-processing is used to calculate the single-difference pseudo-range residual error of the GNSS observation value;

[0117] The single-difference pseudo-range residual error is used as the label value of the GNSS random model in the network training, and the MSE is used as the loss function for network training.

[0118] In summary, the application has the following beneficial effects:

[0119] (1) The application provides a GNSS random model generation method based on a neural network and a fisheye image, and proposes a random model generation method that fuses satellite features and fisheye images to solve the problem of NLOS and multipath signal interference GNSS system positioning performance, which can tightly integrate various sensor information, predict the observation quality of the GNSS signal, and accurately adjust the GNSS random model.

[0120] (2) The application constructs a random model prediction network associated with GNSS and fisheye images, and constructs an image processing module and an information fusion module based on self-attention and cross-attention mechanisms. Among them, the image processing module is connected with multiple self-attention layers, which can extract coarse features with high resolution and fine-grained features with low resolution from the fisheye image; in the information fusion module, the image features are used as Query and Value, and the GNSS features are used as Key, to realize the close association of the two kinds of information, and accurately predict the GNSS random model.

[0121] The fish-eye image assisted GNSS random model combined navigation system provided by the present application is described below, and the fish-eye image assisted GNSS random model combined navigation system described below can be correspondingly referred to the fish-eye image assisted GNSS random model combined navigation method described above.

[0122] Figure 5 is a structural schematic diagram of the fish-eye image assisted GNSS random model combined navigation system provided by the embodiment of the present application, as Figure 5 shown, comprising: a prediction module 51, an association module 52, a generation module 53 and an estimation module 54, wherein:

[0123] The prediction module 51 is used to obtain GNSS observation value information of the reference station and the rover station, construct a single-difference observation equation, robustly estimate the receiver clock error by using the predicted carrier pose, and obtain a single-difference residual; the association module 52 is used to project and convert the satellite feature vector to the pixel plane matrix by using an omnidirectional model, and associate the fish-eye image with the GNSS feature; the generation module 53 is used to construct a deep neural network based on an attention mechanism, input the fish-eye image and the GNSS feature, and output a GNSS random model reflecting the influence of NLOS and multipath effect; and the estimation module 54 is used to fuse GNSS and IMU observation values by using EKF, input the GNSS random model, and obtain the pose estimation value at the corresponding time.

[0124] Figure 6 An example of an entity structure schematic diagram of an electronic device is shown in Figure 6 , which can include a processor 610, a communications interface 620, a memory 630 and a communications bus 640, wherein the processor 610, the communications interface 620 and the memory 630 complete mutual communication through the communications bus 640. The processor 610 can invoke the logic instructions in the memory 630 to execute the fish-eye image assisted GNSS random model combined navigation method, which includes: obtaining GNSS observation value information of the reference station and the rover station, constructing a single-difference observation equation, robustly estimating the receiver clock error by using the predicted carrier pose, and obtaining a single-difference residual; projecting and converting the satellite feature vector to the pixel plane matrix by using an omnidirectional model, and associating the fish-eye image with the GNSS feature; constructing a deep neural network based on an attention mechanism, inputting the fish-eye image and the GNSS feature, and outputting a GNSS random model reflecting the influence of NLOS and multipath effect; and fusing GNSS and IMU observation values by using EKF, inputting the GNSS random model, and obtaining the pose estimation value at the corresponding time.

[0125] In addition, the logic instructions in the memory 630 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0126] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the fish-eye image assisted generation of GNSS random model combination navigation method provided by the above-mentioned methods. The method comprises: obtaining GNSS observation value information of a reference station and a rover station, constructing a single difference observation equation, using a predicted carrier pose to robustly estimate a receiver clock error to obtain a single difference residual; projecting and converting satellite feature vectors to a pixel plane matrix using an omnidirectional model, correlating fish-eye images and GNSS features; constructing a deep neural network based on an attention mechanism, inputting fish-eye images and GNSS features, and outputting a GNSS random model reflecting the influence of NLOS and multipath effects; fusing GNSS and IMU observation values using EKF, inputting the GNSS random model, and obtaining a pose estimation value at a corresponding time.

[0127] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the fish-eye image assisted generation of GNSS random model combination navigation method provided by the above-mentioned methods. The method comprises: obtaining GNSS observation value information of a reference station and a rover station, constructing a single difference observation equation, using a predicted carrier pose to robustly estimate a receiver clock error to obtain a single difference residual; projecting and converting satellite feature vectors to a pixel plane matrix using an omnidirectional model, correlating fish-eye images and GNSS features; constructing a deep neural network based on an attention mechanism, inputting fish-eye images and GNSS features, and outputting a GNSS random model reflecting the influence of NLOS and multipath effects; fusing GNSS and IMU observation values using EKF, inputting the GNSS random model, and obtaining a pose estimation value at a corresponding time.

[0128] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0129] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0130] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A fish-eye image aided GNSS random model generation method for integrated navigation, characterized in that, The method comprises the following steps: Obtaining GNSS observation value information of the reference station and the mobile station, constructing a single-difference observation equation, using a predicted carrier pose to robustly estimate a receiver clock error, and obtaining a single-difference residual error. An omnidirectional model is used to project and convert satellite feature vectors onto a pixel plane matrix, and the fisheye image and the GNSS feature are associated. A deep neural network is constructed based on an attention mechanism, the fisheye image and the GNSS feature are input, and a GNSS stochastic model reflecting the influence of NLOS and multipath effects is output. An EKF is used to fuse GNSS and IMU observation values, the GNSS stochastic model is input, and a pose estimation value at a corresponding time is obtained. Obtaining GNSS observation value information of the reference station and the mobile station, constructing a single-difference observation equation, using a predicted carrier pose to robustly estimate a receiver clock error, and obtaining a single-difference residual error, comprising: The single-difference observation equation is represented as: wherein, denotes the single-difference operator; denotes the geometric distance between the station and the satellite; denotes the receiver clock bias; denotes the observation error introduced by NLOS; denotes the observation error introduced by multipath; denotes the observation noise; denotes the reference station observation; Compared with the mobile observation value, the error term in the reference station observation value is ignored, and the single-difference observation equation is approximately: wherein the superscript represents a flow station; A predicted carrier pose is obtained by using an IMU, an electronic compass and a visual laser odometry. The predicted carrier pose is substituted into the single-difference observation equation, a median estimation and a random sample consensus are used to robustly estimate a receiver clock error, and a single-difference residual error is obtained based on the receiver clock error: wherein denotes the single-difference residuals.

2. The fish-eye image aided GNSS random model assisted integrated navigation method according to claim 1, characterized in that, An omnidirectional model is used to project and convert satellite feature vectors onto a pixel plane matrix, and the fisheye image and the GNSS feature are associated, comprising: Predicted carrier position , attitude , the satellite coordinates in the ECEF coordinate system are converted to the carrier coordinate system, and the corresponding conversion equation is: wherein, is a carrier coordinate system, is an ECEF coordinate system, denotes the satellite position in the ECI system, denotes the satellite position in the ECEF system; Based on the rotation relationship and the translation relationship between the carrier coordinate system and the camera coordinate system, the position satellite in the carrier coordinate system is converted to the camera coordinate system, and the corresponding conversion equation is: wherein, is a camera coordinate system, denotes a satellite position in the camera coordinate system, , denote a rotation relationship and a translation relationship between the carrier coordinate system and the camera coordinate system, respectively. The satellite position in the camera coordinate system is projected to the normalized spherical coordinate system using the omnidirectional projection model On: wherein, is the coordinate of the satellite in the normalized spherical coordinate system, is the coordinate of the satellite in the camera coordinate system; The normalized spherical coordinate system is translated along the optical axis to the origin of the coordinate system , obtaining a new coordinate system , the coordinates of the satellite in the coordinate system are ​ The satellite position in the ECEF system is projected to the normalized plane coordinate system The satellite position in the ECEF system is projected to the normalized plane coordinate system The satellite position in the ECEF system is projected to the normalized plane coordinate system The satellite position in the ECEF system is projected to the normalized plane coordinate system A pixel coordinate corresponding to the satellite position is obtained by using a camera pinhole model projection, and the pixel coordinate system is represented as: wherein, , denotes the pixel coordinates of the satellite projection onto the pixel plane, , , , denotes the focal length and the optical center shift of the camera pinhole model.

3. The fish-eye image aided GNSS random model assisted integrated navigation method according to claim 1, characterized in that, A deep neural network is constructed based on an attention mechanism, the fisheye image and the GNSS feature are input, and a GNSS stochastic model reflecting the influence of NLOS and multipath effects is output, comprising: A plurality of serial self-attention layers are used to construct an image module, coarse features with high resolution and fine-grained features with low resolution are extracted from the fisheye image; A multi-head cross-attention layer is used to construct a fusion module, and the fisheye image features extracted by the image processing module and the GNSS features are tightly fused; A perception layer is used to output a GNSS stochastic model prediction value.

4. The fish-eye image aided GNSS random model assisted integrated navigation method according to claim 3, characterized in that, A multi-head cross-attention layer is used to construct a fusion module, and the fisheye image features extracted by the image processing module and the GNSS features are tightly fused, comprising: The image features are extracted As the query and value, GNSS features are used As the key, the cross-attention mechanism is applied to calculate the information fusion result, and the calculation formula of information fusion is: wherein, , and denote query, key and value matrices in the attention layer, , and denote linear transformation matrices corresponding to the query, key and value matrices, , and denote bias matrices corresponding to the query, key and value matrices.

5. The fish-eye image aided GNSS random model assisted integrated navigation method according to claim 3, characterized in that, After constructing a deep neural network based on an attention mechanism, inputting the fisheye image and the GNSS feature, and outputting a GNSS stochastic model reflecting the influence of NLOS and multipath effects, the following steps are further included: The relevance between the image processing module and the semantic segmentation task in the network is used, a transfer learning strategy is adopted, and the image processing module in the network is pre-trained using a semantic segmentation dataset; The pose reference result obtained by applying a high-precision integrated navigation device is post-processed, and the single-difference pseudo-range residual error of the GNSS observation value is calculated reversely; The single-difference pseudo-range residual error is used as the label value of the GNSS stochastic model in network training, the MSE is used as the loss function, and the network is trained.

6. A fish-eye image aided GNSS random model generation integrated navigation system based on the fish-eye image aided GNSS random model generation integrated navigation method according to any one of claims 1 to 5, characterized in that, The method comprises the following steps: The prediction module is configured to acquire GNSS observation value information of the reference station and the rover station, construct a single-difference observation equation, robustly estimate a receiver clock error by using a predicted carrier pose, and obtain a single-difference residual; The association module is configured to project and convert a satellite feature vector to a pixel plane matrix by using an omnidirectional model, and associate the fisheye image with the GNSS feature; The generation module is configured to construct a deep neural network based on an attention mechanism, input the fisheye image and the GNSS feature, and output a GNSS stochastic model reflecting the influence of NLOS and multipath effects. The estimation module is configured to fuse GNSS and IMU observation values by using an EKF, input the GNSS stochastic model, and obtain a pose estimation value at a corresponding moment.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the fish-eye image assisted generation of the GNSS stochastic model combined navigation method according to any one of claims 1 to 5 when executing the program.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the fish-eye image assisted generation of the GNSS stochastic model combined navigation method according to any one of claims 1 to 5 when executed by the processor.

9. A computer program product comprising a computer program, characterized in that, The computer program implements the fish-eye image assisted generation of the GNSS stochastic model combined navigation method according to any one of claims 1 to 5 when executed by the processor.

Images