Satellite-based positioning
By combining image processing and machine learning technology, multipath and non-line-of-sight signals in GNSS signals are detected and classified, the problem of GNSS positioning decreases in accuracy in complex environments is solved, and high-precision positioning is achieved under harsh conditions.
Patent Information
- Application Number
- CN202410637914.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-24
- Filing Date
- 2024-05-22
- Publication Date
- 2024-11-26
AI Technical Summary
In open sky environments, GNSS signals are susceptible to multipath and non-line-of-sight signals, resulting in reduced positioning accuracy, especially in severe weather and complex terrain environments.
Using image-based methods and signal fusion technology, images are captured through a panoramic 360° camera and combined with machine learning, GNSS signals are detected and classified, and local errors are estimated, thereby improving positioning accuracy.
The accuracy and reliability of GNSS positioning are significantly improved in complex environments, especially in harsh conditions such as night or fog and rain, which can more accurately calculate geospatial locations.
Smart Images

Figure CN119024385A_ABST
Abstract
Description
Technical Field
[0001] The present invention generally relates to determining the current location of a device using signals from a global navigation satellite system (GNSS). More specifically, the present invention relates to classifying the quality of GNSS signals to allow for a more accurate determination of location. Background Art
[0002] GNSS positioning in an open sky environment is an established, accurate and reliable technology. However, it is well known that GNSS signals can be reflected (or deflected) and diffracted by certain surfaces. This can result in the reception of multipath or diffracted GNSS signals, which adversely affects positioning performance if these signals are not properly processed. Therefore, the method and system according to the present invention include the detection of multipath or diffracted GNSS signals. These signals can be classified according to their quality, and / or the "local error" of these signals can be estimated, which can advantageously improve positioning accuracy.
[0003] Multipath signals and diffracted signals can include line-of-sight (LOS) and non-line-of-sight (NLOS) signals. NLOS signal propagation occurs outside the typical line of sight between the transmitter and the receiver, such as in ground reflections. Obstacles that typically cause NLOS propagation include buildings, trees, hills, mountains, and in some cases high-voltage power lines. Some of these obstacles reflect certain signal frequencies, while other obstacles absorb or destroy the signal.
[0004] In the case of GNSS signals used to determine geospatial position, reflected signals can cause problems. Reflected (multipath) signals have a longer travel distance than direct signals and can therefore lead to GNSS receiving devices making erroneous assumptions about their position. GNSS products are often used near signal obstructions like trees and buildings. Therefore, improved positioning performance in these challenging environments is a desired feature.
[0005] Detecting multipath and NLOS signals remains a challenging problem for conventional GNSS technology. In some solutions, cameras (including fisheye cameras and infrared cameras) combined with computer vision (CV) techniques have been used to detect NLOS signals, which are then excluded or down-weighted in the positioning solution.
[0006] For example, US2022 / 0018973A1 discloses a method for determining NLOS GNSS signals, in which a camera oriented toward a GNSS satellite is used to determine whether the satellite has a line of sight, which then allows the LOS signal to be distinguished from the NLOS signal. The solution described in US2022 / 0018973A1 includes segmenting images captured by the camera based on radio frequency (RF) characteristics.
[0007] Unfortunately, the performance of visual methods may be limited by prevailing weather and lighting conditions and are therefore not suitable for all applications. Summary of the invention
[0008] It is therefore an object of the present invention to provide an improved method and an improved system for deriving a geospatial position based on GNSS signals.
[0009] A particular object is to provide a method and system which allow better positioning in difficult areas such as urban canyons or forests.
[0010] A particular object is to provide a method and system which allow better positioning in poor lighting conditions, for example at night or during fog or rain.
[0011] This application proposes a "hybrid" method to filter detected NLOS GNSS signals or estimate the local errors of the detected signals. Local errors are errors due to local effects such as diffraction and multipath (MP) and can be expressed as double or single difference residuals. The proposed "hybrid" method is a fusion of image-based and signal-based methods. This includes combining GNSS signals with additional visual information such as panoramic 360° camera images. Artificial intelligence is applied to estimate signal quality, which can be used to exclude certain signals or apply a weighting process to the signals.
[0012] A first aspect of the invention relates to a computer-implemented method for processing satellite signals to derive a geospatial position, in particular fully automatically and in real time. The method comprises:
[0013] - receiving a plurality of GNSS signals from a plurality of GNSS satellites via a GNSS antenna;
[0014] - capturing a digital image using an imaging device oriented at least partially toward a plurality of GNSS satellites, the digital image comprising a plurality of pixels, a first subset of pixels imaging the sky and a second subset of pixels imaging obstacles that are at least partially inadmissible to GNSS signals and / or have reflective surfaces;
[0015] - determining the orientation of the image; and
[0016] - Calculate geospatial location.
[0017] According to this aspect of the invention, the method further comprises: for each GNSS satellite of at least a subset of the plurality of GNSS satellites:
[0018] -Extract signal features from corresponding GNSS signals;
[0019] - processing the image to extract image features, wherein processing the image comprises: (a) using image segmentation, and / or (b) applying machine learning to compute an image feature vector or an image embedding vector for at least a third subset of pixels of the image, the third subset of pixels being defined based on positions of corresponding GNSS satellites;
[0020] - combining the extracted signal features and the extracted image features to receive a feature combination; and
[0021] - Deriving signal classification and / or estimated local errors of corresponding GNSS signals based on feature combinations of multiple GNSS satellites and by applying machine learning.
[0022] Then, a geospatial position is calculated based on at least a subset of the GNSS signals and based on the signal classification and / or the estimated local error of each GNSS signal of the subset.
[0023] The pixel subsets should be understood as not being exclusive. In particular, each pixel of the third subset may also be a pixel of the first subset or the second subset. An embedding vector can be considered a special case of a feature vector.
[0024] According to some embodiments of the method, extracting the signal features includes applying machine learning to calculate a signal feature vector or a signal embedding vector of the corresponding GNSS signal.
[0025] According to some embodiments of the method, processing the image comprises applying machine learning to compute an image feature vector or an image embedding vector for at least a third subset of pixels of the image.
[0026] According to some embodiments of the method, the extracted signal features include signal feature vectors or signal embedding vectors, the extracted image features include image feature vectors or image embedding vectors, combining the extracted signal features and the extracted image features includes combining the signal feature vectors or signal embedding vectors with the image feature vectors or image embedding vectors, respectively, and the feature combination is a combined feature vector or a combined embedding vector.
[0027] According to some embodiments, the method further comprises:
[0028] - based on the orientation of the image and the known satellite positions, projecting at least a subset of the plurality of GNSS satellites onto the image such that each projected GNSS satellite corresponds to a pixel or a coherent set of pixels of the digital image;
[0029] - determining potential GNSS signal quality values for at least a plurality of pixels of said image; and
[0030] - Assigning to each projected GNSS satellite a potential GNSS signal quality value for the corresponding pixel or corresponding set of coherent pixels.
[0031] According to some embodiments of the method, processing the image includes: applying machine learning to calculate an image feature vector for at least a third subset of pixels of the image, and the third subset of pixels is defined based on the position of the corresponding projected GNSS satellites in the image. For example, the second subset of pixels images an obstacle with a reflective surface, and the third subset of pixels is also defined by the position of the reflective surface in the image relative to the position of the corresponding projected GNSS satellites in the image.
[0032] According to some embodiments, the method further comprises:
[0033] - for at least a subset of the plurality of GNSS satellites, centering the image on a GNSS satellite in the subset of GNSS satellites based on the orientation of the image and the known satellite position;
[0034] - determining, for at least a plurality of pixels of the image, a potential GNSS signal quality value; and
[0035] - assigning to each GNSS satellite in the subset a potential GNSS signal quality value for the pixel or set of coherent pixels at the center of the image.
[0036] In one embodiment, processing the image includes applying machine learning to compute an image embedding vector for the image, such as where the second subset of pixels images an obstacle having a reflective surface.
[0037] According to other embodiments of the method, the derived signal classification and / or the estimated local error is also based on a potential GNSS signal quality value.
[0038] In one embodiment, processing the image includes identifying at least a second subset of pixels in the image (e.g., where processing the image includes using image segmentation), and determining a potential GNSS signal quality value for a GNSS satellite is based on a relative position of a pixel or set of coherent pixels corresponding to the respective GNSS satellite relative to the second subset of pixels in the image.
[0039] According to other embodiments of the method, determining a potential GNSS signal quality value includes extracting an image feature vector for each GNSS satellite of at least a subset of the projected satellites, the image feature vector including the potential GNSS signal quality value, generating a signal feature vector for the satellite signal of each GNSS satellite of at least the subset of the projected satellites, combining the extracted signal features and the extracted image features includes: combining the image feature vector and the signal feature vector of each GNSS satellite of at least the subset of the projected satellites into a combined feature vector of the corresponding GNSS satellite, and deriving the signal classification and / or the estimated local error is performed by a classifier module specifically implemented as a neural network or a support vector machine and based on the combined feature vector of at least the subset of the projected satellites as input.
[0040] In one embodiment, a combined feature vector of multiple time points is generated, wherein the classifier module is a recurrent neural network, and in order to take into account the behavior of the GNSS signal over time, the classifier module calculates the signal classification and / or the estimated local error at multiple time points based on the combined feature vector.
[0041] According to some embodiments of the method, a signal classification is derived for a GNSS signal of each GNSS satellite in a subset of GNSS satellites, and a geospatial location is calculated based on at least the subset of GNSS signals for which the signal classification is derived and their respective signal classifications. For example, calculating the geospatial location includes weighting the GNSS signals based on their respective signal classifications.
[0042] In one embodiment, calculating the signal classification includes: detecting multipath signals, non-line-of-sight signals and / or diffraction signals. Optionally, calculating the geospatial position includes down-weighting the detected multipath signals, non-line-of-sight signals and / or diffraction signals, respectively. Alternatively, the subset of GNSS signals from which the geospatial position is calculated does not include any of the detected multipath signals, non-line-of-sight signals and / or diffraction signals, respectively.
[0043] According to some embodiments of the method, an estimated local error is derived for a GNSS signal of each GNSS satellite in a subset of GNSS satellites, and a geospatial position is calculated based on the subset of GNSS signals for which the estimated local error is derived and their corresponding estimated local errors, wherein the method further comprises:
[0044] - based on the orientation of the image and the known satellite positions, projecting at least a subset of the plurality of GNSS satellites onto the image such that each projected GNSS satellite corresponds to a pixel or a coherent set of pixels of the digital image;
[0045] - determining, for at least a plurality of pixels of the image, a potential GNSS signal quality value; and
[0046] - Assigning to each projected GNSS satellite a potential GNSS signal quality value for the corresponding pixel or corresponding set of coherent pixels.
[0047] According to some embodiments of the method, extracting signal features from the GNSS signal includes: considering further information about the GNSS signal, wherein the further information includes at least a pseudorange. For example, the further information also includes Doppler shift, pseudorange standard deviation, phase standard deviation, lock time count, parity check, carrier-to-noise density ratio, satellite ground height, satellite age, satellite generation and / or satellite signal history. Alternatively or additionally, each GNSS signal includes pseudorandom noise, and each pseudorandom noise is correlated to obtain a pseudorange.
[0048] A second aspect of the invention relates to a system for processing satellite signals to derive a geospatial position, in particular a method according to the first aspect. The system comprises:
[0049] - a GNSS antenna configured to receive GNSS signals from a plurality of GNSS satellites;
[0050] - a measurement engine configured to correlate pseudo-random noise of the GNSS signal to obtain pseudo-ranges;
[0051] - a SLAM unit configured to determine the orientation of the imaging device when capturing the image, e.g. using visual SLAM based on image data captured by the imaging device;
[0052] - a signal processing module configured to extract signal features from the GNSS signal;
[0053] - an image processing module configured to process the image to extract image features by using image segmentation and / or by applying machine learning to compute an image feature vector or an image embedding vector for at least a third subset of pixels of the image, the third subset of pixels being defined based on positions of corresponding GNSS satellites;
[0054] a classifier module, which is specifically implemented as a neural network or a support vector machine and is configured to derive a signal classification and / or an estimated local error of a corresponding GNSS signal based on feature combinations of a plurality of GNSS satellites and by applying machine learning, each feature combination being a combination of extracted signal features and extracted image features of a corresponding GNSS signal; and
[0055] - a positioning engine configured to calculate a geospatial position based on a subset of GNSS signals and / or pseudoranges, wherein the positioning engine is configured to calculate the geospatial position further based on signal classification and / or based on an estimated local error for each GNSS signal in the subset.
[0056] A third aspect relates to a computer program product comprising a program code stored on a machine-readable medium or embodied by an electromagnetic wave comprising program code segments, and having computer executable instructions for executing the method according to the first aspect of the invention, in particular when executed in a system according to the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In the following, the present invention is described in detail by referring to exemplary embodiments accompanied by the accompanying drawings, in which:
[0058] Figures 1a to 1e The problem of NLOS signals and multipath GNSS signals is shown;
[0059] Figure 2 A flow chart illustrating an exemplary embodiment of the method according to the invention is shown;
[0060] Figure 3a is a panoramic image captured by a camera according to an exemplary embodiment of the system of the present invention;
[0061] Figure 3b shows the partitioning of the panoramic image according to the classification of potential signal quality;
[0062] Figure 4 shows components and data flows between components according to an exemplary embodiment of a system of the present invention;
[0063] Figure 5 shows a setup for training artificial intelligence according to an exemplary embodiment of the system of the present invention; and
[0064] Figures 6 to 11 Various modes of operation of an exemplary embodiment of a system according to the present invention are shown. DETAILED DESCRIPTION
[0065] Figures 1a to 1e The general problem of multipath and non-line-of-sight (NLOS) signals in satellite-based positioning is shown. GNSS signals are basically electromagnetic wave signals, which are blocked by structures such as buildings and reflected by glass or metal surfaces. Therefore, especially in urban environments, GNSS signals suffer from blocking, reflection and attenuation, resulting in a variety of received signals, such as multipath signals from both LOS satellites and NLOS satellites. Therefore, pseudorange errors may be introduced due to multipath effects, diffraction and NLOS reception of signals, which may lead to severe degradation of positioning applications or navigation applications.
[0066] exist Figure 1aIn the figure, a single satellite of a global navigation satellite system (GNSS satellite 2) is shown, which provides a signal that allows the derivation of a geospatial position in combination with signals from other GNSS satellites (not shown here). Antennas configured to receive GNSS signals (GNSS antennas 1, 1') are positioned at two locations in an urban environment. The task of the antenna is to determine the current geospatial position of itself or of a device to which it is connected. Such a device may be, for example, a measurement system (e.g. a GNSS pole), a handheld user device (e.g. a smartphone) or a vehicle.
[0067] The first GNSS antenna 1 has a LOS reception position. This means that the antenna 1 is positioned with direct line of sight (LOS) relative to a GNSS satellite 2 so that it receives a direct signal 20 from the satellite 2. In addition, the antenna 1 receives an indirect multipath signal 21 from the same satellite 2. This multipath signal 21 is deflected from a reflective surface of one of the buildings (e.g., from a window) and may negatively affect the determination of the antenna position.
[0068] The second GNSS antenna 1' has a NLOS reception position, i.e. it is positioned with obstructed line of sight relative to the GNSS satellite 2. Therefore, for this antenna 1', the satellite 2 is a non-line-of-sight (NLOS) satellite and no direct signal can be received from it. Nevertheless, the antenna 1' receives a reflected signal from the NLOS satellite 2 which, if not detected, could negatively affect the determination of the antenna position. Again, an indirect multipath signal 21 is received which is deflected from one of the buildings. In addition, the antenna 1' receives a diffracted signal 22 from the top of another building.
[0069] Figure 1b to Figure 1e A GNSS antenna 1, a GNSS satellite 2 and a building are each shown in a top view. The surface of the building can deflect the signal from the satellite 2. In addition to the direct signal 20, the deflection may or may not result in multipath signals 21, depending on the relative position and orientation of the building to the GNSS antenna 1 and the satellite 2.
[0070] exist Figure 1b and Figure 1c In FIG. 1 , the GNSS antenna 1 receives only direct signals 20 from the GNSS satellites 2. Figure 1d and Figure 1e , the different orientations of the respective buildings result in a signal deflection towards the GNSS antenna 1 , such that the antenna 1 receives multipath signals 21 in addition to the direct signal 20 .
[0071] Figure 2 A flow chart illustrating the steps of an exemplary embodiment of a method 100 for processing satellite signals to derive a geospatial position according to the present invention is shown.
[0072] For each satellite of the plurality of GNSS satellites, the GNSS antenna receives 110 one or more signals. A slightly different frequency (frequency band) may be received from each satellite, and furthermore, different satellite constellations (GPS, GLONASS, GALILEO...) operate at different frequencies.
[0073] A camera (e.g. a panoramic camera mounted to a GNSS antenna or a device including a GNSS antenna) captures 120 a panoramic image oriented (at least partially) toward a plurality of GNSS satellites. Some portions of the image will image the sky, while other portions will image obstacles that do not allow GNSS signals. This allows "seeing" the environment.
[0074] The orientation of the camera (or image) is derived 130, e.g. determined or estimated. If the position and orientation of the entire GNSS system is known from the measurements, the orientation of the camera in the GNSS system and therefore the orientation of the image is also known. For example, deriving 130 the orientation of the camera / image may involve the use of a SLAM algorithm. With the image orientation known, the positions of the satellites in the sky can be mapped onto the image, and it can be determined by relatively simple processing 140 whether the satellite is in clear sky, covered by an object, or partially blocked, e.g. behind a tree canopy. This is described, for example, in US2022 / 0018973A1.
[0075] This application proposes an algorithm that does not simply segment the image, but is trained in an end-to-end manner to implicitly learn and understand the relevant physical and geometric properties of the surrounding environment and their impact on signal quality. These properties can be based on depth images or depth estimates, such as using an additional monocular depth estimation algorithm.
[0076] Furthermore, in the hybrid approach, physical information from the signal itself is processed simultaneously. Typically, multipath and NLOS signals can be detected from GNSS observations such as pseudorange and carrier-to-noise ratio (CNR or carrier-to-noise density ratio) or by integrating a discriminator directly into the signal tracking loop to detect multipath and NLOS signals from the results of the signal correlation output. The proposed solution goes a step further: instead of performing a simple classification, machine learning is applied to determine which features of the surrounding 150 affect the quality of the signal and to what extent.
[0077] Thus, not only can NLOS signals be detected, but all received signals (i.e., including LOS signals) can also be classified 160 with respect to their error or probability of error. The classification can be binary (e.g., low multipath error vs. high multipath error) or more finely graded. In some embodiments, a machine learning algorithm can be used to estimate 165 the local error for each satellite. Typically, the local error is primarily based on multipath error, but may also include errors due to diffraction. For example, the local error can be modeled as a regression problem.
[0078] Machine learning can be trained to identify relevant features from several inputs, namely, GNSS signals (including pseudoranges, carrier-to-noise ratios, Doppler shifts, etc.), panoramic images, orientation estimates, and the known positions of satellites.
[0079] Using the received GNSS signals and a classification of the signals or their local errors, a geospatial position may be calculated 170 with high accuracy.
[0080] Optionally, the AI model can be applied based on the current surroundings and / or measurement history. Such a system needs to know its current surroundings. For example, the system's camera provides images that allow the ML classifier to recognize the surrounding scene in real time, for example, "surrounded by skyscrapers in an inner city" or "inside a forest". Such images can be Figure 2 A panoramic image 120 or additional images are captured during the described method. Cameras and / or devices for distance assessment regarding the system's surroundings allow real-time ML classification of scenes, such as "within less than one meter of a reflective wall" or "location under an awning." Alternatively, the system can be configured to assess the current situation and its similarity to past situations. This can include situations of the same device as well as other devices and situations at the same location or other locations. For example, this can include a history of trajectories or statistics about available satellites and their attributes (such as height above the ground). This situation awareness can then be used to run a situation awareness machine learning model on the system. This can allow processing that depends on the category of the current scene in a given hybrid approach to place more (or less) weight on the image than on a time series-based ML model. This can also allow processing with different ML models for inner city, roadside, and forest environments. This can allow processing that only considers satellites above a certain height above the ground.
[0081] Figure 3a and Figure 3b The steps of image processing are further shown. Figure 3aA panoramic image 30 is shown showing the sky 31 in the background and buildings in the foreground causing signal blocking obstacles 33. Depending on the specific camera position on the GNSS device, some obstacles 35 in the image may be caused by the GNSS device itself. In the example shown, the positions 32 of the GNSS satellites are also projected into the image.
[0082] The panoramic image is processed with an image-based AI module to obtain an embedding, which can then be used in a second AI module. The image-based AI module is particularly embodied as a neural network. The obtained embedding encodes for each pixel of the image and, as a result, for each location in the sky 31, the potential signal quality of the satellites located in that area.
[0083] Such a model is trained on collected data where satellites and their local errors (e.g., calculated as "double difference residuals" or "single difference residuals") are mapped into images. The model is then trained to predict this local error. Since there is very sparse information per pixel and image, a large amount of training data is required. Such data can be acquired by using two or more reference rovers, i.e., one rover in the field including a panoramic camera tracked by a laser tracker to acquire ground truth information.
[0084] Figure 3b Shows from Figure 3a 30. The embedding includes four signal classifications 41-44 ranging from "blocked" 41 and "poor" 42 to "fair" 43 and "good" 44. Thus, each satellite may be assigned one of the four classifications 41-44.
[0085] Figure 4 The data flow between components of an exemplary embodiment of a system 10 according to the present invention is shown. The system includes a GNSS antenna 1 that receives "raw" GNSS signals from a plurality of GNSS satellites (not shown here), the "raw" signals including pseudo-random noise. The GNSS antenna 1 provides the GNSS signals to a measurement engine 5, which processes the signals. A correlator within the measurement engine 5 correlates the pseudo-random noise to obtain a pseudo-range. The processing of the GNSS signals thus produces an output including a pseudo-range, and in some embodiments may also include a carrier-to-noise ratio (CNR), a Doppler shift, a pseudo-range standard deviation, a phase standard deviation, a lock time count, or a parity check.
[0086] The output of the measurement engine 5 is provided to the positioning engine 8 which is configured to calculate and output a geospatial position based thereon.
[0087] In order to improve the accuracy of the geospatial position calculated in difficult surroundings including multipath and NLOS signals (e.g. urban environments), the system 10 according to the invention comprises additional components. These components include a digital panoramic camera 3, which captures panoramic images, e.g. as a video stream, while being at least partially oriented towards the satellite. The SLAM unit 9 determines the orientation of the camera when capturing these images. The image data and orientation are provided to the image-based AI module 4.
[0088] As mentioned above about Figure 3b As described, the image-based AI module 4 can be specifically implemented as a neural network. The image-based AI module 4 processes the panoramic image to obtain an embedding, which encodes the potential signal quality of each pixel. Alternatively, the embedding can encode the potential signal quality of each satellite. Therefore, the signal quality of a satellite whose position corresponds to a certain pixel or pixel group of the image can be classified. Optionally, embeddings do not need to be obtained for all image pixels. For example, those parts of the image that are obscured by part of the system itself (for example, a GNSS antenna mounted above the camera) can be identified and ignored during image processing. Moreover, the horizon can be determined and embeddings obtained only for those pixels that are above the horizon in the image. For example, as Figure 3b As shown, the embedding may include four signal classifications "blocked", "poor", "fair", and "good" that may be assigned to satellites.
[0089] The system 10 also includes a signal-based AI module 6, which may be or include a support vector machine or a second neural network. The signal-based AI module 6 receives the output of the measurement engine 5, such as the pseudorange, CNR and Doppler shift for each satellite and channel, and the embedding obtained in the image-based AI module 4. The output of the measurement engine 5 and the embedding of the image-based AI module 4 are processed together in the signal-based AI module 6, and based on this, further information about the signal quality is derived. This information may include a classification of the signal or a regression of its actual error.
[0090] Combined with the satellite's position in the sky, these two sets of features can be correlated and weighted in a single neural network to predict signal quality. This is done for each satellite individually. Multiple satellites can be processed at once or sequentially per epoch, and one frequency band or multiple bands can be used.
[0091] The derived regression or signal quality of each satellite is then provided to the positioning engine 8. The positioning engine 8 can use the signal quality to exclude certain satellite signals from consideration, or weight the satellite signals based on the signal quality of the satellite signals, that is, give a greater weight to satellite signals with high quality than to satellite signals with low quality. As a result, multipath or NLOS signals will be identified as low-quality signals and not used at all or given a smaller weight (down-weighted), so that positioning is improved. Down-weighting can be done by a fixed factor, which can also be case-specific. In any scenario, fewer satellite signals can be received in total, and more care must be taken when discarding these satellite signals. When there are still a large number of good LOS satellites, any multipath and NLOS signals can be discarded. However, in difficult situations, even poor signals may still be beneficial and should / can be considered in the final positioning solution, but weighted down. Even if multipath or NLOS signals have been used, regression of actual errors can also be used to correct or improve positioning. For example, if the error can be accurately regressed, each signal can be weighted by 1 / "estimated error", that is, the higher the error, the lower the weight.
[0092] Figure 5 Shows the training Figure 4 An image-based AI module of an exemplary system of . In a high multipath error environment 51, such as between tall buildings and / or under trees, a GNSS RTK rover (e.g., Leica GS18) comprising a GNSS antenna, a measurement engine, and a panoramic camera receives GNSS signals including pseudo-random noise from satellites Sa, Sb. These GNSS signals are converted into pseudo-ranges by the measurement engine. The camera captures a 360° video of the environment, while the orientation and position of the rover are statically fixed or continuously monitored, such as tracked by a laser tracker (not shown here). Pseudo-ranges, video, and orientation and position are provided as inputs to a machine learning model.
[0093] Two rovers of the same type are provided as references in a low multipath error environment 52, for example on top of one of the tall buildings surrounding the first rover. The pseudoranges from the first rover and from the two reference rovers are provided to GNSS data analysis software that allows quality control and performance analysis of a network of GNSS reference stations, such as the Leica "SpiderQC" GNSS data analysis software. The software calculates the local errors (residuals) due to multipath and diffraction and provides these to the machine learning model.
[0094] Using the known positions of the three rovers and hence the known baselines A, B between them, the signal quality Q of the satellite can be derived in a generally known manner (e.g. by differencing).
[0095] Figures 6 to 11 Several modes of operation of an exemplary embodiment of a system according to the invention are shown. Figure 6 A first operating mode is shown. For clarity, the mode is described for only one of three satellites. However, in practice a larger number of satellites may be used.
[0096] Using the image 30 with the satellite's positions 32 projected therein, features can be extracted from the image of each satellite. In the illustrated embodiment, this is done using hand-crafted features. Features can be extracted as feature vectors 62, i.e., as an abstract description of the surroundings of a feature in the image 30, including, for example, the distribution of color and brightness values in an area (e.g., about 30×30 pixels) around the feature.
[0097] Thus, the extracted feature vector 62 can be fused with the handcrafted feature vector 64 (describing the characteristics of the signal) for each satellite signal 60 into a combined feature vector 66. The combined feature vector 66 includes the values of both the image feature vector 62 and the signal feature vector 64. This is repeated for each satellite, resulting in a set of combined feature vectors 66 from multiple satellites. The set of combined feature vectors 66 can then be fed into another module 15 (the "classifier module") to regress a signal classification 68, i.e., a "signal quality for each satellite" value.
[0098] Alternatively or additionally, the local error for each satellite may be estimated. Ideally, the signal quality is the estimated local error, i.e., the error due to multipath and diffraction. In another embodiment, the signal quality is simply a classification, such as "good" / "poor", "NLOS" / "LOS", "good" / "medium" / "poor", or "NLOS" / "high MP" / "low MP". In another embodiment, there may be multiple signals per satellite and eigenvectors from the signals. That is, a separate eigenvector is calculated for each satellite and frequency band. Figure 6 In the embodiment shown, only one (joint) signal eigenvector 66 is calculated for each satellite.
[0099] As shown here, module 15 can be or include a neural network (NN). Alternatively, it can include a traditional machine-learning (ML) algorithm, such as a random forest or a support vector machine (SVM). The signal classification 68 (or the estimated local error) can then be provided to the positioning engine to calculate the geospatial position based on at least a subset of the GNSS signals and the corresponding signal classification (or the corresponding estimated local error). For example, the signal classification 68 can be used to weight each signal, and the local error can be directly used to correct the calculated position.
[0100] Figure 7 A second mode of operation is shown, which is a modification of the first mode. Figure 6 ) illustrates this modification. In this embodiment, the hand-crafted image and / or signal features 62, 64 are replaced by neural networks 4 and 6, respectively. The first neural network 4 (image-based neural network) is trained on the image 30, and the second neural network 6 (signal-based neural network) is trained on the signal 60. The image-based neural network 4 generates an embedding vector 72 from the image 30, while the signal-based neural network 6 generates an embedding vector 74 from the signal 60. The two embedding vectors 72, 74 can then be fused into a combined embedding vector 76 for the corresponding projection satellite. For example, the fusion can be performed by either of the two neural networks 4 and 6. In addition, a transformer architecture, such as software over the air (SOTA), can be used for the integration of signal and image features.
[0101] Similar to the first embodiment, the set of combined embedding vectors 76 from multiple satellites can then be fed into a neural network to regress a "signal quality per satellite" value. Alternatively or additionally, the local error for each satellite can be estimated. The embedding vectors 72, 74 can be viewed as a special case of feature vectors. In particular, the image embedding vector 72 is generated for the entire image. Taking into account the satellite positions and obstacles in the image, the image embedding vector 72 can also describe where reflections of the GNSS signal may occur.
[0102] Figure 8 A third mode of operation is shown. According to this embodiment, the image 30, which is a 360° equirectangular projection, is rotated so that the center 36 of the image is aligned with one of the azimuths of the satellite. Therefore, it is not necessary to decode the satellite positions into images, but instead a separate image (from the same equirectangular projection) is generated for each satellite. Then, an embedding vector 72 for the satellite is generated from the image 30 using an image-based neural network 4. For the signal 60, hand-crafted features 64 can be used, as shown in the first embodiment of the system. Alternatively, a signal-based neural network 6 can be used, as shown in the second embodiment of the system.
[0103] Centering the image on the satellite may include taking into account or ignoring the altitude of the satellite. If the altitude is ignored, it cannot be derived directly from the input image 30, but needs to be derived from the signal 60. This involves measuring the Doppler shift of the signal: the higher the Doppler shift in the signal, the faster the GNSS satellite is moving relative to the GNSS antenna receiving its signal 60. Satellites near the horizon move faster than satellites near the zenith. Therefore, the faster the GNSS satellite, the lower it is in the image. Therefore, the higher the measured Doppler shift in the signal, the lower the satellite is in the image. This implicit approach allows to assign pixels of the image and therefore potential GNSS signal quality values with sufficient accuracy to obtain a signal classification 68 or estimate a local error.
[0104] Fig. 9 A fourth embodiment of the system is shown. Here, the behavior of the signal over time is taken into account by using a recurrent neural network (RNN) or other recurrent architectures such as long short-term memory (LSTM). In addition, a transformer architecture (e.g., as software over the air (SOTA)) can be used to process data sequences. The combined feature vector 66, 66', 66" (which can be as described with respect to Figure 6 , Figure 7 or Figure 8 The combined feature vectors at t0 (66), t1 (66') and t2 (66") are fed into a recurrent neural network to obtain the signal quality of each satellite at t0 (68), t1 (68') and t2 (68").
[0105] In conclusion, Figures 6 to 9 Four exemplary embodiments or modifications are shown with combined feature vectors (e.g., handcrafted), combined embedding vectors (e.g., generated by a neural network), panoramic centering by satellite azimuth, and operating modes captured by the temporal behavior of RNNs. Multiple possible signal quality metrics can be predicted, such as local error (regression) or classification (e.g., binary or ternary classification).
[0106] Additionally, the image features themselves can be enhanced with depth estimates and / or semantic segmentation masks. This provides geometric properties of the surrounding surface. Dedicated neural networks can be trained accordingly.
[0107] Fig.10A fifth exemplary mode of operation is shown. In addition to the neural network generated or handcrafted features and embeddings described above, depth estimation can also be performed to compute geometric features based on the image 30, thereby producing a depth or depth estimated image 38. For example, a dedicated neural network can be trained to perform monocular depth estimation. Alternatively, the image 30 can be captured as a depth image. The resulting depth or depth estimated image 38 is used as an additional input layer to the image 30, which can be used by the neural network 4 to generate a better image embedding 72.
[0108] Fig.11 A sixth mode of operation is shown. In this embodiment, instead of using a dedicated neural network to perform depth estimation (as in Fig.10 ), using only one image correlation neural network 4 (as in Figure 7 However, during the training 80 of the neural network 4, a different loss may be used to explicitly force the neural network 4 to learn the semantic segmentation features 86 and / or the depth estimation features 84 so that these features are obtained as part of the embedding vector 72. In the previous embodiment, the signal classification loss 82 may be used for direct training. The signal classification loss 82 is (as with respect to Figure 6 In the first embodiment described in the previous section, a local error loss (e.g., MSE of the local error for each satellite / signal) or a classification loss (e.g., cross entropy loss) for binary or ternary classification problems can be selected. In the embodiment shown here, for some epochs during training, a loss based on semantic segmentation labels and / or depth estimates is selected.
[0109] Optionally, outlier-based training can be performed in advance and / or on the fly. Training of ML models can be performed based on the sensitivity of each GNSS signal's contribution to the overall positioning accuracy of the system, rather than explicitly requiring ground truth. In the presence of many satellite signals, the impact of a single signal (or a group of signals from a few satellites with similar characteristics, such as nearby satellites or satellites at a considerable altitude above the ground) on positioning accuracy is evaluated by "turning on" or "turning off" these signals. For example, a small impact produces the label "reliable signal" while a large impact produces the label "outlier". These labels (along with time series and image data) are used in the training and evaluation pipeline of the end-to-end ML model, or before its deployment, and / or on the deployed ML model for retraining or fine-tuning purposes.
[0110] Although the present invention has been described above with reference to some preferred embodiments, it must be understood that many modifications and combinations of different features of the embodiments are possible. All these modifications are within the scope of the appended claims.
Claims
1. A computer-implemented method (100) for processing satellite signals to derive a geospatial position, in particular fully automatically and in real time, the method (100) include: Receiving (110) a plurality of GNSS signals (20-22, 60) from a plurality of GNSS satellites (2) via a GNSS antenna (1); capturing (120) a digital image (30) using an imaging device (3) oriented at least partially toward the plurality of GNSS satellites (2), the digital image (30) comprising a plurality of pixels, a first subset of pixels imaging the sky (31) and a second subset of pixels imaging obstacles (33) that are at least partially non-GNSS signal-tolerant and / or have reflective surfaces; determining (130) an orientation of the image (30); as well as Calculate (170) the geospatial location, Features The method (100) further comprises: for each GNSS satellite of at least a subset of the plurality of GNSS satellites (2), Extracting signal features (64, 74) from corresponding GNSS signals (20-22, 60); processing (140) the image (30) to extract image features (62, 72), wherein processing the image comprises Use image segmentation, and / or Applying machine learning to compute an image feature vector (62) or an image embedding vector (72) of at least a third subset of pixels of the image (30), the third subset of pixels being defined based on positions of corresponding GNSS satellites (2); combining the extracted signal features (64, 74) and the extracted image features (62, 72) to receive a feature combination (66, 76); and Based on a combination of features (66, 76) of a plurality of GNSS satellites (2) and by applying machine learning to derive (160, 165) a signal classification (68) and / or an estimated local error of a corresponding GNSS signal (20-22, 60), wherein calculating (170) the geospatial position is based on at least a subset of the GNSS signals (20-22, 60) and on the signal classification (68) and / or the estimated local error of each GNSS signal (20-22, 60) in the subset.
2. The method (100) according to claim 1, in, Extracting the signal features (64, 74) includes applying machine learning to calculate a signal feature vector (64) or a signal embedding vector (74) of the corresponding GNSS signal (20-22, 60).
3. The method (100) according to claim 1 or claim 2, in, Processing (140) the image includes applying machine learning to compute an image feature vector (62) or an image embedding vector (72) for at least the third subset of pixels of the image (30).
4. The method (100) according to any one of the preceding claims, in, The extracted signal features (64, 74) include a signal feature vector (64) or a signal embedding vector (74); The extracted image features (62, 72) include an image feature vector (62) or an image embedding vector (72); Combining the extracted signal features (64, 74) and the extracted image features (62, 72) comprises: combining the signal feature vector (64) or the signal embedding vector (74) with the image feature vector (62) or the image embedding vector (72), respectively; and The feature combination is a combined feature vector (66) or a combined embedding vector (76).
5. The method (100) according to any one of claims 1 to 4, wherein the method include: projecting at least a subset of the plurality of GNSS satellites (2) onto the image (30) based on the orientation of the image and known satellite positions, such that each projected GNSS satellite (2) corresponds to a pixel or a coherent set of pixels of the digital image (30); determining (150) potential GNSS signal quality values (41-44) for at least a plurality of pixels of the image (30); as well as Each projected GNSS satellite (2) is assigned a potential GNSS signal quality value (41-44) for a corresponding pixel or a corresponding set of coherent pixels.
6. The method (100) according to claim 5, in, Processing (140) the image includes: applying machine learning to calculate an image feature vector (62) for at least the third subset of pixels of the image (30); and The third subset of pixels is defined based on the position (32) of the corresponding projected GNSS satellite (2) in the image (30), In particular, the second subset of pixels images an obstacle (33) having a reflective surface, and the third subset of pixels is also defined by the position of the reflective surface in the image (30) relative to the position (32) of the corresponding projected GNSS satellite (2) in the image (30).
7. The method (100) according to any one of claims 1 to 4, wherein the method include: for at least a subset of the plurality of GNSS satellites (2), centering the image (30) on a satellite in the subset of the GNSS satellites (2) based on the orientation of the image and a known satellite position; determining (150) potential GNSS signal quality values (41-44) for at least a plurality of pixels of the image (30); as well as Each of the GNSS satellites (2) in the subset is assigned a potential GNSS signal quality value (41-44) for the pixel or set of coherent pixels at the center of the image (30).
8. The method (100) according to claim 7, in, Processing (140) the image comprises applying machine learning to compute an image embedding vector (72) of the image (30), in particular wherein the second subset of pixels images an obstacle (33) having a reflective surface.
9. The method (100) according to any one of claims 5 to 8, in, Deriving (160, 165) the signal classification (68) and / or the estimated local error is also based on the potential GNSS signal quality value (41-44).
10. The method (100) according to claim 9, in, Processing (140) the image (30) comprises: identifying at least the second subset of pixels in the image (30), in particular wherein processing (140) the image comprises using image segmentation; and Determining (150) the potential GNSS signal quality values (41-44) of the GNSS satellites (2) is based on the relative positions of the pixels or the sets of coherent pixels corresponding to the respective GNSS satellites with respect to the second subset of pixels in the image (30).
11. The method (100) according to claim 5, in, Determining (150) the potential GNSS signal quality values (41-44) comprises: extracting an image feature vector (62) for each GNSS satellite (2) of at least a subset of the projected satellites, the image feature vector (62) comprising the potential GNSS signal quality values (41-44); generating a signal feature vector (64) for the satellite signal (60) of each GNSS satellite (2) of at least a subset of the projected satellites; Combining the extracted signal features and the extracted image features comprises: combining the image feature vector (62) and the signal feature vector (64) of each GNSS satellite (2) in at least the subset of projected satellites into a combined feature vector (66) of the corresponding GNSS satellite (2); and Deriving (160, 165) the signal classification (68) and / or the estimated local error is performed by a classifier module (15) embodied as a neural network or a support vector machine and based on a combined feature vector (66) of at least a subset of the projected satellites as input.
12. The method (100) according to claim 11, in, Generate multiple time points (t 0 ,t 1 ,t 2 )’s combined feature vector (66, 66’, 66”); The classifier module (15) is a recurrent neural network; and In order to take into account the behavior of the GNSS signal over time, the classifier module (15) calculates (160, 165) at multiple time points (t 0 ,t 1 ,t 2 ) of the signal classification (68, 68', 68") and / or the estimated local error.
13. The method (100) according to any one of the preceding claims, in, deriving (160) a signal classification (68) for the GNSS signal (20-22, 60) of each GNSS satellite (2) in the subset of GNSS satellites; as well as calculating (170) the geospatial position based on at least a subset of the GNSS signals (20-22, 60) for which the signal classification (68) was derived (160) and based on their respective signal classifications (68), In particular, calculating (170) the geospatial position includes weighting the GNSS signals (20-22, 60) based on respective signal classifications (68) of the GNSS signals.
14. The method (100) according to claim 13, in, Calculating (160) the signal classification (68) comprises detecting multipath signals, non-line-of-sight signals and / or diffraction signals, in particular wherein Calculating (170) the geospatial location comprises: respectively weighting down the detected multipath signals, non-line-of-sight signals and / or diffracted signals; or The subset of the GNSS signals (20-22) from which the geospatial position is calculated (170) does not include any detected multipath signals, non-line-of-sight signals, and / or diffracted signals, respectively.
15. The method (100) according to any one of the preceding claims, in, deriving (165) an estimated local error for the GNSS signal (20-22, 60) of each GNSS satellite (2) in the subset of GNSS satellites; as well as calculating (170) the geospatial position based on a subset of the GNSS signals (20-22, 60) for which the estimated local errors were derived (165) and their corresponding estimated local errors, Wherein, the method comprises projecting at least a subset of the plurality of GNSS satellites (2) onto the image (30) based on the orientation of the image and known satellite positions, such that each projected GNSS satellite (2) corresponds to a pixel or a coherent set of pixels of the digital image (30); determining (150) potential GNSS signal quality values (41-44) for at least a plurality of pixels of the image (30); and Each projected GNSS satellite (2) is assigned the potential GNSS signal quality value (41-44) of a corresponding pixel or a corresponding set of coherent pixels.
16. The method (100) according to any one of the preceding claims, in, Extracting signal features (64, 74) from the GNSS signals (20-22, 60) comprises taking into account further information about the GNSS signals (20-22, 60), wherein the further information comprises at least a pseudorange, in particular wherein The further information may also include Doppler shift, pseudorange standard deviation, phase standard deviation, lock time count, parity check, carrier-to-noise density ratio, satellite ground height, satellite age, satellite generation and / or satellite signal history; and / or Each GNSS signal (20-22, 60) includes pseudo-random noise, and each pseudo-random noise is correlated to obtain the pseudo-range.
17. A system (10) for processing satellite signals to derive a geospatial position, in particular a method (100) according to any one of the preceding claims, said system include: A GNSS antenna (1) configured to receive (110) GNSS signals (20-22, 60) from a plurality of GNSS satellites (2); a measurement engine (5) configured to correlate pseudo-random noise of the GNSS signals (20-22, 60) to obtain pseudo-ranges; as well as a positioning engine (8) configured to calculate (170) a geospatial position based on a subset of the GNSS signals (20-22, 60) and / or pseudoranges, Features a SLAM unit (9) configured to determine (130) the orientation of the imaging device (3) while capturing the image (30), in particular using visual SLAM based on image data captured by the imaging device (3); a signal processing module (6), the signal processing module being configured to extract signal features (64, 74) from the GNSS signal (20-22, 60); an image processing module (4) configured to process (140) the image (30) to extract image features (62, 72) by using image segmentation and / or by applying machine learning to calculate an image feature vector (62) or an image embedding vector (72) for at least a third subset of pixels of the image (30), the third subset of pixels being defined based on positions of corresponding GNSS satellites (2); and a classifier module (15) embodied as a neural network or a support vector machine and configured to derive (160, 165) a signal classification (68) and / or an estimated local error of a corresponding GNSS signal (20-22, 60) based on feature combinations (66, 76) of a plurality of GNSS satellites (2) and by applying machine learning, each feature combination (66, 76) being a combination of extracted signal features (64, 74) and extracted image features (62, 72) of a corresponding GNSS signal (20-22, 60); The positioning engine (8) is configured to calculate (170) the geospatial position further based on the signal classification (68) and / or the estimated local error for each GNSS signal (20-22, 60) of the subset.
18. A computer program product comprising a program code, the program code being stored on a machine-readable medium or being embodied by an electromagnetic wave comprising program code segments, and having computer executable instructions for executing a method (100) according to any one of claims 1 to 16, in particular when executed in a system (10) according to claim 17.
Citation Information
Patent Citations
GNSS satellite line of sight detection
US20220018973A1
Cited By
Integrated navigation method for generating GNSS random model with assistance of fisheye image
CN120871208A
