Sky segmentation and nlos satellite identification method based on fisheye polarization camera
By combining a fisheye polarization camera with a deep learning model, the system effectively distinguishes between the real sky and highly reflective surfaces, solving the problem of misjudgment of NLOS signals and improving the accuracy and robustness of GNSS positioning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 郭佳涵
- Filing Date
- 2026-04-24
- Publication Date
- 2026-07-14
AI Technical Summary
Existing technologies struggle to effectively distinguish between the real sky and highly reflective surfaces (such as glass curtain walls) in urban environments, leading to misinterpretation of NLOS signals and impacting GNSS positioning accuracy.
A fisheye polarization camera is used to acquire RGB intensity images in multiple polarization directions to obtain linear polarization degree and polarization feature images. After fusion, an attention-enhanced U-Net deep learning model is constructed to generate a sky segmentation mask and remove reflective areas to achieve NLOS satellite recognition.
It significantly improves the accuracy of NLOS satellite identification and enhances positioning precision. It is suitable for consumer devices such as smartphones and drones and is robust to changes in lighting and weather conditions.
Smart Images
Figure CN122391252A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of interdisciplinary technology of satellite navigation and computer vision, specifically involving a sky segmentation and NLOS satellite identification method based on a fisheye polarization camera. Background Technology
[0002] In complex environments such as urban canyons and densely populated areas with tall buildings, Global Navigation Satellite Systems (GNSS) are susceptible to signal reflection and obstruction, resulting in non-line-of-sight (NLOS) signals. NLOS signals significantly increase pseudorange measurement errors, sometimes reaching tens of meters, severely reducing positioning accuracy.
[0003] Existing NLOS identification methods mainly include the following categories: (1) Hardware-enhanced methods: such as choke-ring antennas and anti-multipath antennas, although they can effectively suppress multipath effects, they are costly and bulky, making them difficult to integrate into consumer devices such as smartphones.
[0004] (2) Signal processing methods: such as MEDLL, Strobe correlator, etc., have high hardware requirements and their performance degrades significantly in strong multipath environments.
[0005] (3) 3D city model-assisted method: It relies on high-precision 3D maps and ray tracing algorithms, which have high data acquisition and update costs, poor real-time performance, and are difficult to meet the low power consumption requirements of consumer devices.
[0006] Visual aids: Sky images are captured using a fisheye camera, and LOS regions are identified through image segmentation. A sky-facing view is obtained using a fisheye camera; after segmenting the sky region, it is determined whether the satellite signal is NLOS. A fisheye camera facing the sky is used to assist in determining NLOS signals; after image segmentation, the satellite position is projected onto a pixel plane to determine satellite visibility.
[0007] However, the aforementioned method based on ordinary fisheye cameras has a key drawback: it struggles to distinguish between the real sky and highly reflective surfaces such as glass curtain walls. In urban environments, glass curtain walls and smooth walls visually exhibit similar brightness and color characteristics to the sky, making it difficult for ordinary RGB images to differentiate them effectively, leading to numerous misclassifications. For example, when a satellite projection point falls on a glass curtain wall area, existing methods incorrectly identify it as a LOS signal, when in fact the signal has been reflected by the glass and is an NLOS signal.
[0008] In recent years, polarization imaging technology has demonstrated unique advantages in the field of specular reflection suppression. Studies have shown that specular and diffuse light have different polarization characteristics: reflected light from smooth surfaces (such as glass and water) typically has a high degree of linear polarization (DoLP), while scattered light from the real sky has a low degree of polarization. This physical property can be used to effectively distinguish between the real sky and reflective areas on glass. However, there is currently no technical solution combining polarization imaging with fisheye cameras for GNSS NLOS identification. Furthermore, how to effectively integrate polarization features with deep learning models to achieve end-to-end sky segmentation and NLOS identification remains a pressing technical problem to be solved. Summary of the Invention
[0009] The purpose of this application is to provide a sky segmentation and NLOS satellite identification method based on a fisheye polarization camera, so as to solve the problem in the prior art that it is difficult to distinguish between the real sky and highly reflective surfaces (such as glass curtain walls).
[0010] This application provides a sky segmentation and NLOS satellite identification method based on a fisheye polarization camera, including: RGB intensity images corresponding to multiple polarization directions are acquired using a fisheye polarization camera, and linear polarization degree images and polarization feature images corresponding to the intensity images corresponding to the multiple polarization directions are obtained. The RGB intensity image is fused with the corresponding linear polarization degree image and polarization feature image to obtain a multi-channel input image; Construct a deep learning model based on attention-enhanced U-Net to obtain AttU-PNet, and train the AttU-PNet to obtain the trained AttU-PNet; The multi-channel input image is used as the input to the trained AttU-PNet to obtain the sky segmentation mask output by AttU-PNet. Based on the linear polarization image, a reflective area mask is generated, and the sky segmentation mask is fused with the reflective area mask to obtain a real sky mask; Obtain the coordinates of the satellite projection point, and based on the satellite projection point coordinates and the real sky mask, obtain the NLOS satellite identification result.
[0011] Furthermore, after obtaining the NLOS satellite identification results, the process also includes: Based on the NLOS satellite identification results, the satellite signals corresponding to NLOS are removed, and the satellite signals corresponding to the remaining LOS are used for GNSS positioning calculation.
[0012] Furthermore, RGB intensity images corresponding to multiple polarization directions are acquired using a fisheye polarization camera, including: By using a fisheye polarization camera to capture RGB intensity images at polarization directions of 0°, 45°, 90°, and 135°, RGB intensity images corresponding to multiple polarization directions are obtained.
[0013] Further, obtaining the linear polarization degree image and polarization feature image corresponding to the intensity image of the multiple polarization directions includes: Based on the intensity images corresponding to the multiple polarization directions, obtain the Stokes vector; Based on the Stokes vector, a linear polarization degree image and a polarization feature image are obtained.
[0014] Furthermore, based on the intensity images corresponding to the multiple polarization directions, the Stokes vector is obtained as follows: ; ; ; in, The first Stokes vector, The second Stokes vector, The third Stokes vector, This is an RGB intensity image in the 0° polarization direction. This is an RGB intensity image along a 45° polarization direction. This is an RGB intensity image along a 90° polarization direction. This is an RGB intensity image along a 135° polarization direction.
[0015] Furthermore, based on the Stokes vector, a linear polarization degree image and a polarization feature image are obtained, including: Based on the Stokes vector, the linear polarization degree image is obtained as follows: ; in, This is a linear polarization degree image. The first Stokes vector, The second Stokes vector, The third Stokes vector; Based on the Stokes vector, the polarization feature image is obtained as follows: ; in, This is a polarization feature image.
[0016] Further, the RGB intensity image is fused with the corresponding linear polarization degree image and polarization feature image to obtain a multi-channel input image, including: The R, G, and B color channels of the RGB intensity image are fused with the DoLP polarization feature channel of the linear polarization degree image and the AoP polarization feature channel of the polarization feature image to obtain a multi-channel input image.
[0017] Furthermore, a deep learning model based on attention-enhanced U-Net is constructed to obtain AttU-PNet, and AttU-PNet is trained to obtain the trained AttU-PNet, including: A U-Net is constructed, and a polarization attention gating mechanism is introduced at the skip connections in the U-Net to obtain AttU-PNet; AttU-PNet is trained using a combination of weighted cross-entropy loss and Dice loss.
[0018] Further, based on the linear polarization image, a reflective area mask is generated, and the sky segmentation mask is fused with the reflective area mask to obtain a realistic sky mask, including: Based on the aforementioned linear polarization image, when hour, Mark as reflective area, otherwise The non-reflective areas are marked to obtain a reflective area mask; among them, For the preset threshold, This represents the degree of linear polarization at pixel coordinates (x, y). This represents the value of the reflective area mask at pixel coordinates (x, y). It is a binary mask, where a value of 1 indicates that the pixel belongs to the reflective area, and a value of 0 indicates that the pixel belongs to the non-reflective area. The sky segmentation mask is fused with the reflective area mask to obtain the true sky mask as follows: ; in, To mask the real sky, Divide the sky into a mask. For reflective areas, ∧ represents the logical AND operation, and ¬ represents the logical NOT operation.
[0019] Further, the coordinates of the satellite projection points are obtained, and based on the satellite projection point coordinates and the real sky mask, the NLOS satellite identification result is obtained, including: Project the satellite line-of-sight vector onto the image pixel plane to obtain the coordinates of the satellite projection point; Determine whether the coordinates of the satellite projection point are located within the real sky mask. If so, determine that the satellite signal to be identified is the satellite signal corresponding to LOS; otherwise, determine that the satellite signal to be identified is the satellite signal corresponding to NLOS. All satellite signals to be identified are evaluated to obtain NLOS satellite identification results.
[0020] The beneficial effects of this application are as follows: This invention provides a sky segmentation and NLOS satellite recognition method based on a fisheye polarization camera, comprising: acquiring RGB intensity images in multiple polarization directions to obtain linear polarization degree images and polarization feature images; fusing the above images to obtain a multi-channel input image; constructing and training an attention-enhanced U-Net model to output a sky segmentation mask; fusing the reflective area mask generated by the linear polarization degree image with the sky segmentation mask to obtain a real sky mask; and realizing NLOS satellite recognition based on the satellite projection point coordinates and the real sky mask. By utilizing polarization characteristics, the method effectively distinguishes between the real sky and glass reflections, significantly improving the accuracy of NLOS recognition, and is suitable for high-precision positioning of mobile devices in complex urban environments. Attached Figure Description
[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0022] Figure 1 A flowchart of a sky segmentation and NLOS satellite identification method based on a fisheye polarization camera provided in this application.
[0023] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0024] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0025] The embodiments of this application are described in detail below with reference to the accompanying drawings.
[0026] like Figure 1 As shown, this application provides a sky segmentation and NLOS satellite identification method based on a fisheye polarization camera, including: S101. Acquire RGB intensity images corresponding to multiple polarization directions using a fisheye polarization camera, and obtain linear polarization degree images and polarization feature images corresponding to the intensity images corresponding to the multiple polarization directions. S102. The RGB intensity image is fused with the corresponding linear polarization degree image and polarization feature image to obtain a multi-channel input image; S103. Construct a deep learning model based on attention-enhanced U-Net to obtain AttU-PNet, and train AttU-PNet to obtain the trained AttU-PNet; S104. Use the multi-channel input image as the input to the trained AttU-PNet to obtain the sky segmentation mask output by AttU-PNet. S105. Based on the linear polarization image, generate a reflective area mask and fuse the sky segmentation mask with the reflective area mask to obtain a real sky mask. S106. Obtain the coordinates of the satellite projection point, and obtain the NLOS satellite identification result based on the coordinates of the satellite projection point and the real sky mask.
[0027] This invention aims to solve the problem of difficulty in distinguishing between the real sky and highly reflective surfaces (such as glass curtain walls) in the prior art, and provides a sky segmentation and NLOS satellite identification method based on a fisheye polarization camera. By introducing polarization imaging technology to extract specular reflection features and combining it with a deep learning model, high-precision sky segmentation is achieved, which significantly improves the accuracy and robustness of NLOS satellite identification.
[0028] In some possible implementations, after obtaining the NLOS satellite identification results, the following is also included: Based on the NLOS satellite identification results, the satellite signals corresponding to NLOS are removed, and the satellite signals corresponding to the remaining LOS are used for GNSS positioning calculation.
[0029] In some possible implementations, RGB intensity images corresponding to multiple polarization directions are acquired using a fisheye polarization camera, including: By using a fisheye polarization camera to capture RGB intensity images at polarization directions of 0°, 45°, 90°, and 135°, RGB intensity images corresponding to multiple polarization directions are obtained.
[0030] For example, a fisheye polarization camera can be used to acquire image data in complex urban environments. This fisheye polarization camera employs a focal plane polarization sensor (DoFP) to simultaneously acquire intensity images in four different polarization directions (0°, 45°, 90°, and 135°) in a single exposure. The fisheye polarization camera uses a focal plane polarization sensor, with four micro-polarizers of different orientations integrated above each 2×2 pixel unit, enabling the simultaneous acquisition of images in four polarization directions in a single exposure.
[0031] In some possible implementations, obtaining the linear polarization degree image and polarization feature image corresponding to the intensity image corresponding to the multiple polarization directions includes: Based on the intensity images corresponding to the multiple polarization directions, obtain the Stokes vector; Based on the Stokes vector, a linear polarization degree image and a polarization feature image are obtained.
[0032] Optionally, preprocessing can be performed on DoLP and AoP images, including median filtering for noise reduction, normalization, and other operations.
[0033] In some possible implementations, the Stokes vector is obtained based on the intensity image corresponding to the multiple polarization directions: ; ; ; in, The first Stokes vector, The second Stokes vector, The third Stokes vector, This is an RGB intensity image in the 0° polarization direction. This is an RGB intensity image along a 45° polarization direction. This is an RGB intensity image along a 90° polarization direction. This is an RGB intensity image along a 135° polarization direction.
[0034] In some possible implementations, based on the Stokes vector, a linear polarization degree image and a polarization feature image are obtained, including: Based on the Stokes vector, the linear polarization degree image is obtained as follows: ; in, This is a linear polarization degree image. The first Stokes vector, The second Stokes vector, The third Stokes vector; Based on the Stokes vector, the polarization feature image is obtained as follows: ; in, This is a polarization feature image.
[0035] In some possible implementations, the RGB intensity image is fused with the corresponding linear polarization image and polarization feature image to obtain a multi-channel input image, including: The R, G, and B color channels of the RGB intensity image are fused with the DoLP polarization feature channel of the linear polarization degree image and the AoP polarization feature channel of the polarization feature image to obtain a multi-channel input image.
[0036] In some possible implementations, a deep learning model based on attention-enhanced U-Net is constructed to obtain AttU-PNet, and AttU-PNet is trained to obtain the trained AttU-PNet, including: A U-Net is constructed, and a polarization attention gating mechanism is introduced at the skip connections in the U-Net to obtain AttU-PNet; AttU-PNet is trained using a combination of weighted cross-entropy loss and Dice loss.
[0037] For example, a deep learning model based on attention-enhanced U-Net is constructed, named AttU-PNet, and a polarization attention module is introduced into the model, specifically including: (1) Encoder: Uses convolutional neural networks to extract multi-scale features; (2) Polarization Attention Module: A polarization attention gating mechanism is introduced at the skip connection. Attention weights are calculated using DoLP and AoP features, and the encoder features are weighted and fused as follows: ; ; in, For encoder features, The polarization characteristics of the linear polarization degree image. The polarization features of the polarization feature image. Here, σ represents the attention weights, and σ is the sigmoid function; where, For learnable convolutional kernel weights, For bias terms, This indicates element-wise multiplication (Hadamard product).
[0038] (3) Decoder: Gradually upsample to restore spatial resolution and output sky segmentation probability map; (4) Loss function: A combination of weighted cross-entropy loss and Dice loss is used. ; in, For the combined loss function, For cross-entropy loss, For Dice's loss, The weights are the weights for the cross-entropy loss. Let be the weighted weights of the Dice loss, and + =1.
[0039] Optionally, the deep learning model AttU-PNet also includes a spatial attention module to enhance its ability to pay attention to sky boundaries.
[0040] In some possible implementations, a reflective area mask is generated based on the linear polarization image, and the sky segmentation mask is fused with the reflective area mask to obtain a realistic sky mask, including: Based on the aforementioned linear polarization image, when hour, Mark as reflective area, otherwise The non-reflective areas are marked to obtain a reflective area mask; among them, This is a preset threshold value, ranging from 0.3 to 0.6. This represents the degree of linear polarization at pixel coordinates (x, y). This represents the value of the reflective area mask at pixel coordinates (x, y). It is a binary mask, where a value of 1 indicates that the pixel belongs to the reflective area, and a value of 0 indicates that the pixel belongs to the non-reflective area.
[0041] Optional, preset threshold An adaptive threshold method can also be used to determine the threshold, which can be dynamically adjusted according to the polarization statistical characteristics of different scenarios.
[0042] The sky segmentation mask is fused with the reflective area mask. That is, if a pixel is marked as reflective in the reflective mask, it is removed in the final sky mask to obtain the true sky mask. ; in, To mask the real sky, Divide the sky into a mask. For reflective areas, ∧ represents the logical AND operation, and ¬ represents the logical NOT operation.
[0043] In some possible implementations, satellite projection point coordinates are obtained, and based on the satellite projection point coordinates and the real sky mask, NLOS satellite identification results are obtained, including: Project the satellite line-of-sight vector onto the image pixel plane to obtain the coordinates of the satellite projection point; Determine whether the coordinates of the satellite projection point are located within the real sky mask. If so, determine that the satellite signal to be identified is the satellite signal corresponding to LOS; otherwise, determine that the satellite signal to be identified is the satellite signal corresponding to NLOS. For example, (1) the intrinsic parameter matrix and distortion coefficient of the fisheye polarization camera are obtained through camera calibration, and the mapping relationship between the pixel coordinate system and the spatial direction is established; (2) Obtain the satellite ephemeris data at the current moment and calculate the elevation angle and azimuth angle of each satellite in the carrier coordinate system; (3) Based on the camera's extrinsic parameter matrix (attitude information provided by the IMU), the satellite line-of-sight vector is projected onto the image pixel plane to obtain the coordinates of the satellite projection point. ; Optionally, the satellite projection process can use a spherical projection model to maintain the full field of view of the fisheye lens and avoid information loss caused by distortion correction.
[0044] (4) Determine whether the satellite projection point falls on the final real sky mask. Inside: like =1, then the satellite signal is determined to be LOS; if =0, then the satellite signal is determined to be NLOS; where, This represents the value of the real sky mask at the pixel coordinates (u,v) of the satellite projection point. It is a binary mask value, where a value of 1 indicates that the pixel belongs to the real sky area, and a value of 0 indicates that the pixel belongs to the non-sky area (including solid obstructions or glass reflection areas).
[0045] After obtaining the NLOS satellite identification results, the satellite signals determined to be LOS can be used for GNSS positioning calculation, eliminating NLOS signals and improving positioning accuracy.
[0046] All satellite signals to be identified are evaluated to obtain NLOS satellite identification results.
[0047] Compared with the prior art, the present invention has the following beneficial effects: (1) High-precision reflective area identification: The polarization characteristics (DoLP / AoP) are used to effectively distinguish between the real sky and highly reflective surfaces such as glass curtain walls. The reflected light from glass curtain walls has a high DoLP value (usually >0.5), while the DoLP value of the scattered light from the real sky is low (usually <0.3). This physical difference provides a reliable basis for accurately identifying reflective areas.
[0048] (2) Significantly improve NLOS recognition accuracy: By eliminating misjudgments in reflective areas, the NLOS / LOS recognition accuracy is improved to over 99%, which is about 5-10 percentage points higher than the traditional fisheye camera method.
[0049] (3) Strong robustness: The polarization feature is not sensitive to changes in light and weather conditions, and can still maintain stable reflective recognition ability in low light scenarios such as cloudy days and evenings.
[0050] (4) Low cost and easy deployment: It adopts a consumer-grade fisheye polarization camera, which does not rely on 3D city models or high-precision hardware, and is suitable for mobile platforms such as smartphones, drones, and autonomous driving.
[0051] (5) End-to-end processing flow: Polarization feature extraction and deep learning segmentation are integrated into one framework to realize automated processing from the original image to NLOS recognition, which is convenient for engineering implementation.
[0052] To make the technical solutions described in the embodiments of this application easier for those skilled in the art to understand, this application provides illustrative examples.
[0053] This embodiment involves data collection in a CBD area of a city, which includes multiple high-rise buildings with glass curtain walls, areas shaded by trees, and open squares.
[0054] Hardware configuration: Fisheye polarization camera: Sony IMX250MZR focal plane polarization sensor, resolution 1440×1080, frame rate 30fps.
[0055] Installation method: The camera is fixedly mounted on the top of the smartphone with the camera facing the sky, ensuring that the field of view covers the entire upper hemisphere.
[0056] IMU module: Rigidly connected to the camera, used to provide camera attitude information.
[0057] Data acquisition process: Image data was acquired at different times (morning, noon, evening) and under different weather conditions (sunny, cloudy, overcast), for a total of 5,000 sets of images. Each set includes: raw polarized mosaic image; synchronized IMU attitude data; and raw GNSS observation data.
[0058] Polarization feature extraction: The original polarization mosaic image is de-mosaiced to reconstruct the intensity images I0, I2, and I3 in the four polarization directions. 45 I 90 I 135 Then, calculate the Stokes parameters and polarization characteristics according to the formula: ; ; ; ; ; Median filtering (kernel size 3×3) is applied to DoLP and AoP images to remove noise, and then they are normalized to the [0,1] interval.
[0059] The RGB image (obtained from S0 conversion) is stacked with the polarization feature image to construct a 5-channel input image: [ ].
[0060] The dataset annotation adopted a semi-automatic annotation strategy: for solid occlusions such as buildings and trees, manual annotation was performed using the Labelme tool; for glass reflection areas, DoLP thresholding was used to assist in annotation: areas with DoLP>0.5 and located on the surface of buildings were marked as "reflective"; finally, three types of labels were generated: sky, solid occlusion, and glass reflection. A total of 2000 images were labeled and divided into training, validation and test sets in an 8:1:1 ratio.
[0061] The network structure can be as follows: Encoder: 4 convolutional blocks, each containing two 3×3 convolutions + BN + ReLU, with max pooling with a stride of 2; Polarization attention module: introduced at 4 skip connections, with the input being encoder features F_enc, DoLP feature map, and AoP feature map, and attention weights generated through 1×1 convolutions; Decoder: 4 upsampling blocks, each containing upsampling + skip connections + two 3×3 convolutions; Output layer: 1×1 convolution + softmax, outputting a sky probability map.
[0062] Training parameters: The optimizer is Adam, and the initial learning rate is 1×10. -3 The learning rate decays to 0.5 when the validation loss does not decrease for 5 consecutive rounds; the batch size is 4; the epochs are 50; the weights of the loss function are α=0.5 and β=0.5; and the input image size is 512×512.
[0063] Transfer learning can be employed, using ImageNet pre-trained weights to initialize the encoder, and the polarization attention module to be randomly initialized. During training, the IoU on the validation set is monitored, and the best model is saved.
[0064] Reflective mask generation: An adaptive thresholding method is used on the DoLP map of the test image. in, and Here are the mean and standard deviation of the DoLP plot. When DoLP(x,y> At that time, it was marked as reflective.
[0065] Taking a certain test image as an example, the DoLP value of the glass curtain wall area is about 0.65, the DoLP value of the sky area is about 0.25, the adaptive threshold is 0.45, and the glass curtain wall area is successfully marked as reflective.
[0066] Final Sky Mask: Perform logical operations between the sky segmentation mask output by the model and the reflection mask. ; That is, retain the pixels that the model identifies as sky and are not marked by the reflective mask.
[0067] Satellite projection: Taking GPS satellite C09 as an example, at a certain moment its elevation angle is 45° and azimuth angle is 120°. The pixel coordinates (u,v) = (320, 240) are calculated using the camera projection model. The final sky mask value at that location is then retrieved. If the location corresponds to the real sky region (M_true_sky=1), then C09 is determined to be LOS.
[0068] If the location corresponds to a reflective area (M_true_sky=0), then C09 is determined to be NLOS.
[0069] In this embodiment, the C09 projection point falls exactly on the glass curtain wall area and is correctly identified as NLOS.
[0070] Positioning Solution: After removing all NLOS satellites, least-squares positioning is performed using the remaining LOS satellites. Experimental results show that, using this method, the positioning error is reduced from an average of 15.3 meters to 2.1 meters, improving accuracy by 86.3%.
[0071] The technical solutions described in the embodiments of this application are compared with existing methods, and the results are shown in Table 1: Table 1 Experimental results show that the technical solution described in this application significantly improves the accuracy of sky segmentation and NLOS recognition by introducing polarization features and removing reflective areas, and ultimately greatly improves the positioning accuracy.
[0072] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0073] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0074] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0075] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.
[0076] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A sky segmentation and NLOS satellite identification method based on a fisheye polarization camera, characterized in that, include: RGB intensity images corresponding to multiple polarization directions are acquired using a fisheye polarization camera, and linear polarization degree images and polarization feature images corresponding to the intensity images corresponding to the multiple polarization directions are obtained. The RGB intensity image is fused with the corresponding linear polarization degree image and polarization feature image to obtain a multi-channel input image; Construct a deep learning model based on attention-enhanced U-Net to obtain AttU-PNet, and train the AttU-PNet to obtain the trained AttU-PNet; The multi-channel input image is used as the input to the trained AttU-PNet to obtain the sky segmentation mask output by AttU-PNet. Based on the linear polarization image, a reflective area mask is generated, and the sky segmentation mask is fused with the reflective area mask to obtain a real sky mask; Obtain the coordinates of the satellite projection point, and based on the satellite projection point coordinates and the real sky mask, obtain the NLOS satellite identification result.
2. The sky segmentation and NLOS satellite identification method based on a fisheye polarization camera according to claim 1, characterized in that, After obtaining the NLOS satellite identification results, the following is also included: Based on the NLOS satellite identification results, the satellite signals corresponding to NLOS are removed, and the satellite signals corresponding to the remaining LOS are used for GNSS positioning calculation.
3. The sky segmentation and NLOS satellite identification method based on a fisheye polarization camera according to claim 1, characterized in that, RGB intensity images corresponding to multiple polarization directions are acquired using a fisheye polarization camera, including: By using a fisheye polarization camera to capture RGB intensity images at polarization directions of 0°, 45°, 90°, and 135°, RGB intensity images corresponding to multiple polarization directions are obtained.
4. The sky segmentation and NLOS satellite identification method based on a fisheye polarization camera according to claim 1, characterized in that, Obtaining the linear polarization degree image and polarization feature image corresponding to the intensity image of the multiple polarization directions includes: Based on the intensity images corresponding to the multiple polarization directions, obtain the Stokes vector; Based on the Stokes vector, a linear polarization degree image and a polarization feature image are obtained.
5. The sky segmentation and NLOS satellite identification method based on a fisheye polarization camera according to claim 4, characterized in that, Based on the intensity images corresponding to the multiple polarization directions, the Stokes vector is obtained as follows: ； ； ； in, The first Stokes vector, The second Stokes vector, The third Stokes vector, This is an RGB intensity image in the 0° polarization direction. This is an RGB intensity image along a 45° polarization direction. This is an RGB intensity image along a 90° polarization direction. This is an RGB intensity image along a 135° polarization direction.
6. The sky segmentation and NLOS satellite identification method based on a fisheye polarization camera according to claim 5, characterized in that, Based on the Stokes vector, obtain the linear polarization degree image and polarization feature image, including: Based on the Stokes vector, the linear polarization degree image is obtained as follows: ； in, This is a linear polarization degree image. The first Stokes vector, The second Stokes vector, The third Stokes vector; Based on the Stokes vector, the polarization feature image is obtained as follows: ； in, This is a polarization feature image.
7. The sky segmentation and NLOS satellite identification method based on a fisheye polarization camera according to claim 1, characterized in that, The RGB intensity image is fused with the corresponding linear polarization degree image and polarization feature image to obtain a multi-channel input image, including: The R, G, and B color channels of the RGB intensity image are fused with the DoLP polarization feature channel of the linear polarization degree image and the AoP polarization feature channel of the polarization feature image to obtain a multi-channel input image.
8. The sky segmentation and NLOS satellite identification method based on a fisheye polarization camera according to claim 1, characterized in that, Construct a deep learning model based on attention-enhanced U-Net to obtain AttU-PNet, and train the AttU-PNet to obtain the trained AttU-PNet, including: A U-Net is constructed, and a polarization attention gating mechanism is introduced at the skip connections in the U-Net to obtain AttU-PNet; AttU-PNet is trained using a combination of weighted cross-entropy loss and Dice loss.
9. The sky segmentation and NLOS satellite identification method based on a fisheye polarization camera according to claim 1, characterized in that, Based on the linear polarization image, a reflective area mask is generated, and the sky segmentation mask is fused with the reflective area mask to obtain a realistic sky mask, including: Based on the aforementioned linear polarization image, when hour, Mark as reflective area, otherwise The non-reflective areas are marked to obtain a reflective area mask; among them, For the preset threshold, This represents the degree of linear polarization at pixel coordinates (x, y). This represents the value of the reflective area mask at pixel coordinates (x, y). It is a binary mask, where a value of 1 indicates that the pixel belongs to the reflective area, and a value of 0 indicates that the pixel belongs to the non-reflective area. The sky segmentation mask is fused with the reflective area mask to obtain the true sky mask as follows: ； in, To mask the real sky, Divide the sky into a mask. For reflective areas, ∧ represents the logical AND operation, and ¬ represents the logical NOT operation.
10. The sky segmentation and NLOS satellite identification method based on a fisheye polarization camera according to claim 1, characterized in that, Obtain the coordinates of the satellite projection point, and based on the satellite projection point coordinates and the real sky mask, obtain the NLOS satellite identification result, including: Project the satellite line-of-sight vector onto the image pixel plane to obtain the coordinates of the satellite projection point; Determine whether the coordinates of the satellite projection point are located within the real sky mask. If so, determine that the satellite signal to be identified is the satellite signal corresponding to LOS; otherwise, determine that the satellite signal to be identified is the satellite signal corresponding to NLOS. All satellite signals to be identified are evaluated to obtain NLOS satellite identification results.