A Navigation Satellite Positioning Method Based on Deep Learning and Fisheye Camera Image Segmentation

Abstract

This invention discloses a navigation satellite positioning method based on deep learning and fisheye camera image segmentation, belonging to the field of navigation satellite technology. This invention uses a fisheye camera with sky orientation to assist in determining NLOS signals; after image segmentation, the satellite positions are projected onto a pixel plane, and the satellite visibility is determined based on the projection results. NLOS signals are then eliminated, and positioning is finally achieved. This invention uses deep learning-based sky segmentation on fisheye camera images to identify LOS and NLOS navigation satellite signals, and eliminates NLOS navigation satellite signals, ultimately improving the average positioning accuracy by approximately 31.9%.

Description

Technical Field

[0001] This invention relates to the field of navigation satellite technology, and in particular to a navigation satellite positioning method based on deep learning and fisheye camera image segmentation. Background Technology

[0002] Multipath propagation is a significant source of error in satellite navigation signal measurement, caused by the reflection and refraction of satellite signals by certain objects. Depending on whether a direct satellite signal is received, multipath signals can be categorized into line-of-sight (LOS) signals and non-line-of-sight (NLOS) signals. The magnitude of multipath errors is highly dependent on the environment, making it difficult to describe using mathematical models. Compared to carrier phase observations, multipath errors have a greater impact on pseudorange observations. Generally, they introduce centimeter-level errors into carrier phase observations, while introducing meter-level errors into pseudorange observations, and under extreme conditions, the impact can reach tens or even hundreds of meters.

[0003] In complex urban environments, the effects of multipath errors are amplified due to obstruction, reflection, and scattering from buildings and other obstacles, as well as interference from the complex electromagnetic environment. Accurately detecting non-line-of-sight signals and further eliminating or compensating for these detected signals can mitigate the multipath effect and effectively improve positioning accuracy in complex urban environments.

[0004] Therefore, those skilled in the art are dedicated to developing a navigation satellite positioning method based on deep learning and fisheye camera image segmentation. This method can accurately detect non-line-of-sight signals, further eliminate detected signals or perform compensation, mitigate the effects of multipath effects, and effectively improve positioning accuracy in complex urban environments. Summary of the Invention

[0005] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is to mitigate the effects of multipath effect and effectively improve the satellite positioning accuracy in complex urban environments.

[0006] To achieve the above objectives, this invention provides a navigation satellite positioning method based on deep learning and fisheye camera image segmentation, comprising the following steps:

[0007] Step 1: The RGB image of the up-and-down fisheye camera that has been acquired or received is used as the object to be processed. For each of the three channels of data of each pixel in the RGB image, a weighted average method is used to process the data to calculate the gray value of the corresponding pixel and obtain the gray map corresponding to the converted up-and-down fisheye camera RGB image.

[0008] Step 2: Use the Sobel operator to extract the gradient map. Based on the grayscale image of the RGB image from the fisheye camera, perform derivative operations on the grayscale values ​​in both the horizontal and vertical directions to obtain the gradient map of the fisheye camera grayscale image.

[0009] Step 3: Use a deep learning-based method to divide the sky region into sky and non-sky regions;

[0010] Step 4: Project the navigation satellite constellation in the zenith region onto the fisheye camera image and compare it with the binarized fisheye camera image segmented based on the deep learning method; when the navigation satellite constellation falls into the sky region, it is determined that the navigation satellite constellation is a LOS satellite signal, otherwise it is an NLOS satellite signal; remove the NLOS satellite signal and keep only the LOS satellite signal, and then continue to use the traditional navigation satellite positioning method to locate the receiver.

[0011] Furthermore, in step 1, the weighted average coefficients of the three channels of the RGB image data from the upward-facing fisheye camera are ωR, ωG, and ω... B , has ωR+ωG+ω B =1 and 0≤ω R ≤1、0≤ω G ≤1、0≤ω B ≤1.

[0012] Further, in step 1, ω is selected. R =0.2126, ω G =0.7152, ω B =0.0722.

[0013] Further, in step 1, the RGB data (R) of each pixel in the RGB image of the upward-facing fisheye camera... i G i B i The grayscale data is grey. i =ω R *R i +ω G *G i +ω B *B i .

[0014] Furthermore, step 3 includes the following steps:

[0015] Step 3.1: Use gamma transform to enhance the contrast between the sky area and the non-sky area;

[0016] Step 3.2: Use the collected and calibrated fisheye camera images of the sky and non-sky areas to construct the training set for the deep learning network, and finally generate binarized images from the fisheye camera images;

[0017] Step 3.3: Train the parameters of the deep learning network using deep learning methods. The input of the network is a normal fisheye camera image, and the output of the network is a binarized fisheye camera image.

[0018] Step 3.4: Use the octet algorithm to determine the connectivity between the sky region and the non-sky region, and finally determine the entire sky region.

[0019] Furthermore, in step 3.2, the sky area is designated as region 1, and the non-sky area is designated as region 0.

[0020] Furthermore, in step 3.2, a gradient threshold is set. When the gradient value of an image pixel is less than the gradient threshold, it corresponds to a sky region; conversely, when the gradient value of an image pixel is greater than the gradient threshold, it corresponds to a non-sky region.

[0021] Furthermore, the gradient threshold is 0.6.

[0022] Furthermore, in step 3.3, a fully convolutional neural network is selected for training and learning.

[0023] Furthermore, in step 4, an orthogonal projection model is used to project the satellite constellation.

[0024] In a preferred embodiment of the present invention, in complex urban environments, the impact of multipath errors is amplified due to obstruction, reflection, and scattering caused by buildings and other obstacles, as well as interference from the complex electromagnetic environment in cities. If non-line-of-sight (NLOS) signals can be accurately detected, and the detected results can be further eliminated or compensated for, mitigating the effects of multipath, thus effectively improving positioning accuracy in complex urban environments. Addressing the aforementioned shortcomings of existing technologies, the present invention primarily focuses on using a sky-facing fisheye camera to assist in determining NLOS signals. After image segmentation, satellite positions are projected onto a pixel plane, and satellite visibility is determined based on the projection results. NLOS signals are then eliminated, ultimately achieving positioning.

[0025] 1. The RGB image acquired or received from the up-and-down fisheye camera is used as the object to be processed. For each of the three channels of data of each pixel in the RGB image, a weighted average method is used to calculate the gray value of the corresponding pixel and obtain the grayscale image corresponding to the converted up-and-down fisheye camera RGB image.

[0026] 2. Use the Sobel operator to extract the gradient map. Based on the grayscale image of the fisheye camera, perform derivative operations on the grayscale values ​​in both the horizontal and vertical directions to obtain the gradient map of the fisheye camera grayscale image.

[0027] 3. A deep learning-based method is used to divide the sky region into sky and non-sky regions. The specific steps include:

[0028] (1) First step, use gamma transformation to enhance the contrast between the sky area and the non-sky area;

[0029] (2) The second step is to use the fisheye camera images of the sky region and the non-sky region that have been collected and labeled to construct the training set of the deep learning network. The sky region is labeled as region 1 and the non-sky region is labeled as region 0. Finally, the fisheye camera images are generated into binarized images.

[0030] (3) The third step is to train the parameters of the deep learning network using deep learning methods. The input of the network is a normal fisheye camera image, and the output of the network is a binarized fisheye camera image.

[0031] (4) The fourth step is to use the 8-connectivity algorithm to determine the connectivity between the sky region and the non-sky region, and finally determine the entire sky region.

[0032] 4. Project the navigation satellite constellation in the zenith region onto the fisheye camera image and compare it with the binarized fisheye camera image segmented using a deep learning method. When a navigation satellite constellation falls within the sky region, it is determined to be a LOS (Left In Sight) satellite signal; otherwise, it is a NLOS (Non-Low In Sight) satellite signal. The NLOS satellite signals are discarded, retaining only the LOS satellite signals, and then the traditional navigation satellite positioning method is used to continue positioning the receiver.

[0033] Compared with the prior art, the present invention has the following obvious substantive features and significant advantages:

[0034] This invention identifies LOS and NLOS navigation satellite signals by performing deep learning-based sky segmentation on fisheye camera images, and removes NLOS navigation satellite signals, ultimately improving the positioning accuracy by an average of approximately 31.9%.

[0035] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of a fully convolutional network model structure according to a preferred embodiment of the present invention;

[0037] Figure 2 This is an example diagram of sky image segmentation results based on deep learning and fisheye camera images, according to a preferred embodiment of the present invention. Detailed Implementation

[0038] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0039] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components has been appropriately exaggerated in the drawings.

[0040] This embodiment relates to a navigation satellite positioning method based on deep learning and fisheye camera image segmentation, specifically including the following steps:

[0041] Step 1: Assume that the RGB data of each pixel in the acquired fisheye camera RGB image can be represented as (R... i G i B i ), i = 1, 2, ..., I, where I is the total number of pixels. The range of RGB data is R. i ∈[0,255]、G i ∈[0,255]、B i ∈[0,255]. Let the weighted average coefficients of the three RGB channels be ω. R ω G ω B , has ω R +ω G +ω B =1 and 0≤ω R ≤1、0≤ω G ≤1、0≤ω B ≤1. In this embodiment, ω is selected. R =0.2126, ω G =0.7152, ω B =0.0722. The corresponding grayscale data is calculated by weighted averaging the RGB data of each pixel in the RGB image from the fisheye camera. i =ω R *R i +ω G *G i +ω B *B i This yields the grayscale image corresponding to the converted fisheye camera RGB image.

[0042] Step 2: Based on the RGB image grayscale obtained in Step 1, the Sobel operator is used to extract the gradient map of the image. In this embodiment, the following two Sobel operator kernels in the horizontal and vertical directions are used to calculate the gradient, and the gradient in the horizontal direction is g. x The gradient in the vertical direction is g. y .

[0043] Table 1. Kernel of Sobel Operator in the Horizontal Direction

[0044] -1 0 +1

[0045] Table 2. Sobel operator kernel in the vertical direction

[0046] -1 0 +1

[0047] Step 3: Calculate the magnitude g and direction angle θ of the gradient using the following formulas:

[0048]

[0049]

[0050] And obtain the gradient map of the grayscale image from the fisheye camera.

[0051] Step 4: Use gamma transform to enhance the contrast between sky and non-sky areas. Gamma transform is performed on a single pixel basis, specifically as follows:

[0052] s = c·g γ

[0053] Where g is the gradient value of the pixel in the grayscale image of the fisheye camera obtained in step 3, parameters c and γ are constants, and s is the gradient value of the pixel after gamma transformation and normalization. In this embodiment, the value of parameter c is set to 200, and the value of parameter γ is set to 0.8.

[0054] Step 5: Set the gradient threshold s th When the gradient value of an image pixel is less than the gradient threshold s th When the gradient value of the image pixel is greater than the gradient threshold s, the region containing the pixel is designated as region 1, corresponding to the sky region; conversely, when the gradient value of the image pixel is greater than the gradient threshold s, the region containing the pixel is designated as region 1, corresponding to the sky region. th At that time, the region where the image pixel is located is marked as region 0, corresponding to the non-sky region, and a binarized image is generated from the fisheye camera image. In this embodiment, a gradient threshold s is set. th It is 0.6.

[0055] Step 6: Based on the binarized fisheye camera images acquired and processed in Steps 1-5, train the parameters of a deep learning network using deep learning methods. The network input is a normal RGB fisheye camera image, and the network output is the binarized sky segmentation image obtained after processing in Steps 1-5. (Example:...) Figure 1 As shown, in this embodiment, a fully convolutional neural network is selected for training and learning, and a total of 4,000 fisheye camera images are used for training.

[0056] Step 7: Use the octal fill algorithm to determine the connectivity between sky and non-sky regions. Scan each pixel starting from the top left corner of the image. For each pixel marked 0, check if its eight surrounding pixels are also marked 0. If any of the surrounding pixels are marked 0, mark them as the same region and assign them the same label number. If none of the surrounding pixels are marked 0, mark the pixel as a new region and assign it a new label number. For each pixel marked 1, check if its eight surrounding pixels are also marked 1. If any of the surrounding pixels are marked 1, mark them as the same region and assign them the same label number. If none of the surrounding pixels are marked 1, mark the pixel as a new region and assign it a new label number.

[0057] Step 8: Based on the projection model of the corresponding fisheye camera, project the GNSS navigation satellite constellation received by the receiver onto the binarized fisheye camera image obtained above. In this embodiment, an orthogonal projection model is used for satellite constellation projection. When a GNSS navigation satellite constellation falls into the area marked 1 (sky area), the GNSS navigation satellite signal is determined to be a LOS satellite signal; when a GNSS navigation satellite constellation falls into the area marked 0 (non-sky area), the GNSS navigation satellite signal is determined to be an NLOS satellite signal.

[0058] Step 9: Eliminate the GNSS navigation satellite received signals identified as NLOS satellite signals from all received signals, retaining only the GNSS navigation satellite received signals identified as LOS satellite signals. Use the retained LOS satellite signals to continue locating the receiver using traditional navigation satellite positioning methods.

[0059] By following the steps above, the navigation satellite positioning process based on deep learning and fisheye camera image segmentation is completed.

[0060] Compared with existing technologies, the navigation satellite positioning results obtained by this invention have smaller positioning errors and better positioning performance. Furthermore, as can be seen from the test results of the embodiments, such as... Figure 2 As shown, regardless of the different environments and scenarios such as open sky, shaded trees, city, and urban canyon, this method can effectively improve positioning accuracy compared to traditional navigation satellite positioning methods, and also verifies the stability and robustness of this processing method in different scenarios.

[0061] Table 3. Changes in the number of visible satellites (unit: satellites)

[0062]

[0063]

[0064] Table 4. Comparison of Positioning Error and Improvement Degree (Unit: meters)

[0065]

[0066] Table 5. Comparison of Positioning Error Performance in Different Scenarios (Unit: meters)

[0067]

[0068] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A navigation satellite positioning method based on deep learning and fisheye camera image segmentation, characterized in that, Includes the following steps: Step 1: The RGB image of the up-and-down fisheye camera that has been acquired or received is used as the object to be processed. For each of the three channels of data of each pixel in the RGB image, a weighted average method is used to process the data to calculate the gray value of the corresponding pixel and obtain the gray map corresponding to the converted up-and-down fisheye camera RGB image. Step 2: Use the Sobel operator to extract the gradient map. Based on the grayscale image of the RGB image from the fisheye camera, perform derivative operations on the grayscale values ​​in both the horizontal and vertical directions to obtain the gradient map of the fisheye camera grayscale image. Step 3: Use a deep learning-based method to divide the sky region into sky and non-sky regions; Step 4: Project the navigation satellite constellation in the zenith region onto the fisheye camera image and compare it with the binarized fisheye camera image segmented based on the deep learning method; when the navigation satellite constellation falls into the sky region, it is determined that the navigation satellite constellation is a LOS satellite signal, otherwise it is an NLOS satellite signal; remove the NLOS satellite signal and keep only the LOS satellite signal, and then continue to use the traditional navigation satellite positioning method to locate the receiver.

2. The navigation satellite positioning method based on deep learning and fisheye camera image segmentation as described in claim 1, characterized in that, In step 1, the weighted average coefficients of the three channels of the RGB image data from the fisheye camera are ω. R ω G ω B , has ω R +ω G +ω B =1 and 0≤ω R ≤1、0≤ω G ≤1、0≤ω B ≤1.

3. The navigation satellite positioning method based on deep learning and fisheye camera image segmentation as described in claim 2, characterized in that, In step 1, ω is selected. R =0.2126, ω G =0.7152, ω B =0.0722.

4. The navigation satellite positioning method based on deep learning and fisheye camera image segmentation as described in claim 2, characterized in that, In step 1, the RGB data (R) of each pixel in the RGB image of the fisheye camera is... i G i B i The grayscale data is grey. i =ω R *R i +ω G *G i +ω B *B i .

5. The navigation satellite positioning method based on deep learning and fisheye camera image segmentation as described in claim 1, characterized in that, Step 3 includes the following steps: Step 3.1: Use gamma transform to enhance the contrast between the sky area and the non-sky area; Step 3.2: Use the collected and calibrated fisheye camera images of the sky and non-sky areas to construct the training set for the deep learning network, and finally generate binarized images from the fisheye camera images; Step 3.3: Train the parameters of the deep learning network using deep learning methods. The input of the network is a normal fisheye camera image, and the output of the network is a binarized fisheye camera image. Step 3.4: Use the octet algorithm to determine the connectivity between the sky region and the non-sky region, and finally determine the entire sky region.

6. The navigation satellite positioning method based on deep learning and fisheye camera image segmentation as described in claim 5, characterized in that, In step 3.2, the sky area is designated as region 1, and the non-sky area is designated as region 0.

7. The navigation satellite positioning method based on deep learning and fisheye camera image segmentation as described in claim 5, characterized in that, In step 3.2, a gradient threshold is set. When the gradient value of an image pixel is less than the gradient threshold, it corresponds to a sky region; conversely, when the gradient value of an image pixel is greater than the gradient threshold, it corresponds to a non-sky region.

8. The navigation satellite positioning method based on deep learning and fisheye camera image segmentation as described in claim 7, characterized in that, The gradient threshold is 0.

6.

9. The navigation satellite positioning method based on deep learning and fisheye camera image segmentation as described in claim 5, characterized in that, In step 3.3, a fully convolutional neural network is selected for training and learning.

10. The navigation satellite positioning method based on deep learning and fisheye camera image segmentation as described in claim 1, characterized in that, Step 4 involves using an orthogonal projection model to project the satellite constellation.

Images