A multi-source fusion reliable navigation positioning method for urban night complex scenes

Abstract

This invention provides a multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios, including: GNSS, INS, and visual data acquisition and preprocessing to achieve quality control of input data; visual feature extraction and initial matching using deep learning technology; identification and removal of dynamic object interference by combining optical flow residuals and IMU motion parameters; estimation of the confidence of visual feature matching point pairs based on LSTM; construction of a GNSS / INS / Vision tightly coupled navigation and positioning model and a filter optimizer by fusing the matching confidence; and output of the final navigation result. This invention improves the robustness of visual matching in complex environments with changing urban nighttime lighting and dynamic object interference, significantly enhancing the continuity and reliability of autonomous perception and navigation and positioning of the vehicle.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation technology for mobile carriers, and in particular to a multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios. Background Technology

[0002] With the rapid development of smart cities and intelligent transportation, the requirements for safety and reliability of autonomous driving systems are increasing. Accurate and continuous navigation and positioning capabilities are fundamental to ensuring the safe and stable operation of vehicles. Only by relying on real-time, high-precision pose information can vehicles achieve reliable path planning and control decisions. However, in the complex environment of urban nighttime, positioning systems face numerous challenges: on the one hand, the poor geometry of the observation satellites weakens the GNSS observation signal, leading to a decrease in fixed resolution; on the other hand, rapid changes in light intensity and interference from dynamic objects in the nighttime environment easily result in a lack of visual features and mismatches, thus causing performance degradation or even failure of the visual positioning system. Therefore, for complex urban nighttime environments, it is urgent to propose a robust navigation and positioning method that takes into account the structural characteristics of the scene. This method improves the quality of visual features through visual enhancement, constructs matching confidence based on the geometric error of feature point trajectory tracking, and integrates the advantages of multi-sensor measurements to establish a robust and reliable tightly coupled navigation and positioning model. This method is of great significance for improving the accuracy and reliability of vehicle positioning in complex urban nighttime environments and can also provide strong technical support for the engineering application and deployment of autonomous driving vehicles.

[0003] Currently, numerous studies have been conducted by scholars both domestically and internationally on visually robust perception methods in the field of integrated navigation, generally showing a trend of evolution from classical image domain methods to end-to-end deep learning methods (Wei et al. 2026). Image domain methods enhance feature representation capabilities by constructing multi-scale spaces and designing gradient operators based on image information, thereby improving detectability and matching ability under repetitive textures and illumination perturbations (Harris et al. 1988; Mistry et al. 2017). However, non-Gaussian illumination variations, dynamic occlusion, and degradation superposition in dynamic urban nighttime scenes can break the assumption boundaries, leading to unstable feature responses and making it difficult to guarantee accurate and verifiable matching relationships in real time. To maintain high matching accuracy and coverage in degraded scenarios, researchers have designed integrated feature extraction and matching networks that combine detection, description, and matching using deep learning techniques. These networks construct multi-scale feature pyramids to cover scale variations and utilize contextual aggregation and cross-feature interaction to expand the effective receptive field and enhance the expressive power of local features (Bhowmik et al. 2020; Liao et al. 2024). While these methods significantly improve detection capabilities in degraded scenarios, their bottleneck lies in insufficient generalization, strong model dependence on training distribution, and difficulty in demonstrating matching accuracy and stability. In multi-sensor integrated navigation, vision-centric integrated navigation schemes have become a research focus for low-cost, high-precision navigation (Xu Shangzhi et al. 2025). Vision / INS combinations can achieve autonomous localization, but due to the lack of absolute position constraints, long-term operation is prone to drift due to error accumulation (Duo Jingyun et al. 2024; Qin et al. 2018). To enhance positioning stability in GNSS-denied environments, researchers have proposed various algorithms, including loose combination (Qin et al. 2025; Niu et al. 2022), semi-compact combination (Xu et al. 2023; Zhang et al. 2025), and compact combination (Liu et al. 2026; Wu et al. 2024), considering the complementary nature of measurements from GNSS, INS, and vision. These algorithms achieve navigation and positioning accuracy from meters to decimeters in large-scale scenarios. In summary, considering the non-Gaussian and time-varying characteristics of observation noise in complex urban nighttime environments, this paper proposes a multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios, leveraging the matching confidence of visual feature points and the measurement advantages of multi-source information.First, GNSS, INS, and visual stereo image data are collected and preprocessed. Second, the images are augmented using an improved Retinexformer, and feature point extraction, matching, and dynamic object removal are achieved using deep learning methods. Then, feature vectors are constructed using various geometric errors of statistical feature points, and the matching confidence is evaluated based on LSTM. Finally, a GNSS / INS / Vision tightly integrated navigation and positioning model is constructed to output continuous and reliable positioning results for the carrier. Summary of the Invention

[0004] In view of this, the present invention provides a multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios, in order to solve the defects of existing navigation and positioning models in complex urban nighttime environments.

[0005] This invention provides a multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios, including:

[0006] Acquisition and preprocessing of GNSS, INS and visual data to achieve quality control of input data;

[0007] Deep learning techniques are used for visual feature extraction and initial matching.

[0008] By combining optical flow residuals and IMU motion parameters, dynamic object interference can be identified and eliminated.

[0009] LSTM is used to estimate the confidence of visual feature matching point pairs;

[0010] Constructing a tightly integrated GNSS / INS / Vision navigation and positioning model by fusing matching confidence scores;

[0011] The final navigation result is obtained by applying a compact combination navigation and positioning model and an extended Kalman filter optimizer.

[0012] According to the present invention, a multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios includes the acquisition and preprocessing of GNSS, INS, and visual data to achieve quality control of input data, comprising:

[0013] A spatiotemporally synchronized multi-sensor integrated acquisition platform is used to acquire multi-source observation data in complex environments such as drastic changes in urban nighttime light intensity and frequent switching between open and closed scenes. The GNSS data includes observation data from the base station and rover, satellite ephemeris data, INS data includes angular velocity and acceleration data measured in real time by the IMU, and visual data includes stereo images.

[0014] Time alignment and quality control are performed on the raw GNSS observations and ephemeris data in the base station and rover data. Double-difference pseudorange observations and double-difference carrier phase observations are generated for filtering measurement updates. The corresponding measurement covariance is output as an absolute constraint in the global coordinate system.

[0015] Perform unit conversion and data integrity checks on the angular velocity and acceleration data of the IMU, and initialize the angular velocity noise and acceleration noise used for filter measurement updates according to the IMU calibration parameters;

[0016] Image enhancement is performed on stereoscopic images of cities with significant nighttime lighting variations to improve image representation and thus enhance the extraction and matching rate of visual features.

[0017] According to the present invention, a multi-source fusion reliable navigation and positioning method for complex urban nighttime scenes is provided, which enhances the image representation of stereo images with large changes in urban nighttime illumination, including:

[0018] Based on Retinex theory, images captured visually under low-light conditions High-quality visual reflectance maps available under ideal lighting conditions and lighting components The product representation enables the understanding of... Accurate estimation can effectively improve image enhancement. To avoid color distortion caused by the high coupling between the three channels in the RGB space, the image channels are separated into HSV spaces, and the illumination component is estimated through the V luminance space. Furthermore, to achieve accurate estimation of the illumination component, degradation factors such as noise and artifacts are comprehensively considered, and a noise perturbation term for the reflection component is introduced. and lighting component disturbance terms The model has been revised, and the revised model is as follows:

[0019]

[0020] To ensure uniform brightness, absence of artifacts, and noise suppression in the enhanced V channel, the overall loss function is designed as a fidelity loss. Loss of smoothness under illumination Exposure control loss and sparsity loss The combination of is defined as:

[0021]

[0022] in, , , and This represents the loss balance parameter.

[0023] Normalize the V channel image to The interval is input into the Retinex Former network. Through the illumination-guided multi-head self-attention mechanism, the global dependency relationship of different brightness regions in the V channel is modeled. Then, through collaborative inference of the depth convolution and channel attention mechanism modules, the illumination component is accurately estimated. At the same time, repair and The noise and artifacts introduced are used to obtain the restored reflection components. ; and thus obtain the enhanced channel image. .

[0024] Enhanced The original H and S channel images are fused to obtain an enhanced HSV image, which is then converted back to the RGB color space to obtain an enhanced binocular RGB image.

[0025] According to the present invention, a multi-source fusion reliable navigation and positioning method for complex urban nighttime scenes is provided, which employs deep learning technology for visual feature extraction and initial matching, including:

[0026] The Superpoint method is used to extract feature points sequentially from the initial binocular images. The binocular image at time 10:00 is and After extracting feature points, Superglue is used to perform feature matching to obtain pairs of feature points with the same name. .

[0027] The present invention provides a multi-source fusion reliable navigation and positioning method for complex urban nighttime scenes, which combines optical flow residuals and IMU motion parameters to identify and eliminate interference from dynamic objects, including:

[0028] Based on the angular velocity and acceleration information of the IMU Perform state propagation and predict the state vector of the carrier. ;

[0029] Detect vehicles and pedestrians in a scene using a large YOLO model, and then analyze the detection bounding boxes. The data is input into the SAM large model to obtain the segmentation range in real time. Feature points within the bounding boxes are extracted and tracked, and inter-frame optical flow velocity is calculated. :

[0030]

[0031] in, and These represent the displacements of the feature point in the horizontal and vertical directions, respectively. For optical flow vector, This represents the number of optical flow traces for the target region.

[0032] Calculate the camera translational velocity and rotation angle using statistical IMU motion data, and derive the target's expected optical flow as follows:

[0033]

[0034] in, Here is the camera pose transformation matrix. The three-dimensional coordinates of the feature points within the target region. The pixel coordinates of the observed feature points. This is a camera projection model.

[0035] By constructing optical flow residuals To determine the dynamic or static state of an object.

[0036] in, This is the actual optical flow vector. To predict the optical flow vector.

[0037] Remove interfering feature points attached to dynamic targets and retain only feature point pairs in static scenes. .

[0038] According to the present invention, a multi-source fusion reliable navigation and positioning method for complex urban nighttime scenes is provided, which estimates the confidence of visual feature matching point pairs based on LSTM, including:

[0039] Based on the feature point pairs obtained in claim 4, forward tracking using optical flow method is performed to obtain... The feature observation trajectory of this feature point is constructed as follows: If a feature point cannot be tracked, the feature trajectory is marked as ended.

[0040] Based on the feature matching points of the left and right images and the inter-frame matching points, the descriptor distances are calculated sequentially. Polar geometric error Forward and backward tracking error Reprojection error Tracking length Local brightness average Local gradient intensity For each feature trajectory Extracting temporal feature vectors ;

[0041] LSTM is used for time-series updates, and the confidence scores corresponding to the observed trajectories of feature points are output. Then, design visual measurement weights. ;

[0042] in, It is the sigmoid activation function. It is the hidden state of the last time step. It is the initial observation noise covariance matrix of feature point j. As a regularization factor;

[0043] The weighted covariance matrices of all feature points are stacked into a single visual observation covariance matrix. , This represents a block diagonal matrix.

[0044] According to the present invention, a multi-source fusion reliable navigation and positioning method for complex urban nighttime scenes provides a temporal feature vector for each feature trajectory, including:

[0045] Similarity distance is calculated based on descriptors of left and right feature points. ;

[0046] Using the calibration parameters of a binocular camera, epipolar line correction was performed on the left and right images, and the epipolar geometric errors of corresponding points on the left and right sides were statistically analyzed. ;

[0047] Based on the corresponding point pairs of inter-frame features, construct the forward and backward tracking error. ;

[0048] Through multi-view Figure 3 Feature points obtained by keratinization The corresponding 3D coordinates are then used to reproject the point onto the image using the pose and intrinsic parameters of the l-th camera, thus obtaining the predicted image point coordinates. The actual observed image point coordinates are Then construct the reprojection error ;

[0049] Tracking length of statistical feature points ;

[0050] To consider the robustness of feature points to brightness variations, a local window is taken centered on the pixel coordinates of the feature point. Statistical analysis of local brightness mean ;

[0051] To consider the robustness of feature points to variations in texture intensity, a local window is taken centered on the pixel coordinates of the feature point. Calculate the average gradient magnitude within a local window. .

[0052] According to the present invention, a multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios is provided, which constructs a GNSS / INS / Vision compact combination navigation and positioning model by fusing and matching confidence scores, including:

[0053] Based on the reliable feature matching point pairs and confidence levels obtained above, the historical camera poses of the same visual feature point under multiple time points or multiple camera observations are used as geometric constraints. For Same-name feature points observed by binoculars at all times Visual observation model of normalized image plane of left and right cameras Represented as:

[0054]

[0055] in, For normalized image plane coordinates, This represents the coordinates in the camera coordinate system. This indicates measurement noise during observation.

[0056] By combining the image points of feature points and the collinear projection relationship between their corresponding 3D world coordinate points, the transformation relationship between the left and right camera coordinate systems and the feature points can be constructed as follows:

[0057]

[0058] in, and They represent The pose and position of the left camera in the global coordinate system at any given time. , for The relative pose of the left camera to the right camera at any given time. , Let represent the coordinates of the visual feature points in the left and right camera coordinate systems, respectively. Therefore, the visual reprojection error state vector of the stereo camera can be constructed as follows:

[0059]

[0060] in, The attitude and position error of the camera in the world coordinate system at a certain moment. Let be the number of cameras. The reprojection error equation for visual measurements can be expressed as:

[0061]

[0062] in, and These are the coordinates of the observed image point and the coordinates of the reprojected image point, respectively. It is the Jacobian matrix corresponding to the camera state vector. This represents the observation noise determined based on the confidence level of the matching points.

[0063] Based on the principles of pseudorange and carrier phase observation, the observation equations for the real-time differential model are determined as follows:

[0064]

[0065] in, This represents the double difference operator, where b and r are the base station and rover station, respectively, and i and j represent the satellite numbers observed simultaneously by the base station and rover station. , These are the ionospheric and tropospheric delay errors, respectively. and These are the carrier wavelength and carrier phase integer ambiguity, respectively. and These are pseudorange observation noise and multipath effect noise, respectively.

[0066] A state model for GNSS is constructed using double-difference carrier phase ambiguity, and the state vector is defined as follows: ;

[0067] The acceleration output based on the INS system at time k. and angular velocity Its measurement model can be expressed as:

[0068]

[0069] in, Represents the rotation matrix. and It is observation noise with a zero-mean Gaussian distribution. The measurement included the Earth's rotation. The impact, This represents the gravitational acceleration in the local coordinate system.

[0070] The error state model of the linearized INS can be expressed as:

[0071]

[0072] in, , and The derivatives of the position, velocity, and attitude errors in the navigation coordinate system n are, in order. ' represents the cross product operator; , These represent the acceleration and angular velocity errors, respectively. and To provide zero-biased noise that conforms to a zero-mean Gaussian distribution; and This includes observation noise for acceleration and angular velocity. In summary, the error state vector of the INS can be expressed as:

[0073] .

[0074] Based on the measurement models of the aforementioned sensors, a tightly integrated GNSS / Vision / INS navigation and positioning model is constructed using MSCKF, and the system's state vector is determined as follows. The continuous state equation is derived as follows:

[0075]

[0076] in, Here is the continuous-time state transition matrix of INS. and These are the INS and GNSS error state process noises, respectively.

[0077] use The fourth-order Runge-Kutta numerical integral propagation of the state variable to be estimated has the following propagation state covariance:

[0078]

[0079] in, and These are the error-state covariance matrices before and after state augmentation, respectively. Represents the discrete state transition matrix. It is time The continuous-time state transition matrix at the given time. Let be the discrete noise covariance matrix.

[0080] When the system receives new images or new GNSS observations, the matrix information of the IMU attitude and bias terms associated with the observation time is augmented into the error state equation. Simultaneously, the error covariance matrix at the corresponding time is also augmented. The augmented covariance matrix is ​​expressed as follows:

[0081]

[0082] in, , These represent the number of images and GNSS observations, respectively. As GNSS observations are recorded, the state variable matrix and error covariance matrix need to be dynamically adjusted. The Jacobian matrix of the camera pose in the latest state relative to the original state can be expressed as:

[0083]

[0084] Therefore, based on the above measurement models, the observation model for the GNSS / INS / Vision tightly integrated navigation system can be determined as follows:

[0085]

[0086]

[0087] in, , These are the double-difference pseudorange and double-difference carrier phase predicted by INS mechanical orchestration, and their Jacobian matrices are respectively... , . Error state vector for RTK positioning and These represent the DD pseudorange observation error noise and DD carrier phase observation error in RTK positioning, respectively.

[0088] According to the present invention, a multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios is provided, which combines a compact combination navigation and positioning model and an extended Kalman filter optimizer to obtain the final navigation result, including:

[0089] Based on the new moment and The state vector contains the raw observations of GNSS pseudorange and carrier phase, visual camera pose, and INS motion parameters. Using the tightly coupled navigation model derived above, combined with extended Kalman filter increments and least squares, the state vector is measured and updated. During operation, the information weights of each sensor are determined by its measurement covariance and Jacobian sensitivity, with the Kalman gain automatically allocating the weights. The recursive process can be summarized as follows:

[0090]

[0091] in, For Kalman filter gain, The observation vector is adjusted by updating the covariance matrix of the measurement noise in the observation equation. The state vector is recursively solved in the compact combination positioning model to obtain an accurate and reliable pose.

[0092] This invention provides a multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios. It enhances the representational capability of visual features through image enhancement and dynamic interference detection and removal. Then, it utilizes Superpoint and Superglue to extract and match feature points, constructing feature trajectories for each feature point. For each trajectory, temporal quality features such as descriptor distance, epipolar error, forward and backward tracking error, reprojection error, tracking length, local brightness, and gradient are extracted at each observation time. This temporal quality sequence is input into an LSTM network to predict the matching confidence of the feature trajectory. Finally, based on the matching confidence, the weights of the corresponding visual residuals or measurement noise covariance are adaptively adjusted. The weighted visual constraints, IMU predictions, and GNSS observations are then input into an extended Kalman filter for fusion and updating, outputting more stable and accurate navigation and positioning results. This method solves the problem of visual feature degradation caused by poor matching reliability and failure to consider dynamic environmental features, brightness intensity changes, and other factors in traditional integrated navigation methods, which leads to positioning system drift. It effectively improves the robustness and accuracy of integrated navigation and positioning in complex urban nighttime environments.

[0093] Beneficial effects of the present invention

[0094] 1. Since the feature matching confidence assessment in the current visual navigation system is not performed, and the reprojection error is used as a constraint, it is easily affected by non-Gaussian factors such as brightness and gradient, which can lead to mismatch and positioning drift. Based on spatial constraints and feature consistency, feature trajectories are constructed, and LSTM network is used to estimate the matching confidence to effectively improve the reliability of matching points and provide scientific visual measurement weights for subsequent filtering optimization.

[0095] 2. A tightly integrated GNSS / INS / Vision navigation and positioning model was constructed. By leveraging the measurement advantages of the three types of sensors, the model effectively overcomes the degradation of visual features caused by changes in scene brightness intensity and interference from dynamic objects. Based on the extended Kalman filter optimizer and the measurement covariance of GNSS, INS and vision, the state vector is measured and updated to output accurate and reliable navigation results, providing a favorable guarantee for reliable navigation in complex urban nighttime environments. Attached Figure Description

[0096] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0097] Figure 1This is a flowchart illustrating the multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios provided by the present invention.

[0098] Figure 2 This is the overall framework diagram provided by the present invention;

[0099] Figure 3 This is the result of image enhancement feature extraction and matching proposed in this invention;

[0100] Figure 4 This is the result of dynamic object detection and removal provided by the present invention;

[0101] Figure 5 This is the feature trajectory error constraint diagram provided by the present invention;

[0102] Figure 6 This is the positioning comparison result provided by the present invention with mainstream navigation methods. Detailed Implementation

[0103] To better understand this application, various aspects of this application will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are merely illustrative of exemplary embodiments of this application and are not intended to limit the scope of this application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0104] In the accompanying drawings, the size, dimensions, and shapes of the elements have been slightly adjusted for ease of illustration. The drawings are for illustrative purposes only and are not strictly to scale. As used herein, the terms “approximately,” “about,” and similar terms are used to indicate approximation, not degree, and are intended to illustrate inherent deviations in measured or calculated values ​​that will be recognized by one of ordinary skill in the art. Furthermore, the order in which the steps are described in this application does not necessarily indicate the order in which these steps occur in actual operation, unless otherwise expressly defined or deduced from the context.

[0105] It should also be understood that expressions such as "comprising," "including," "having," "containing," and / or "comprising" are open-ended rather than closed-ended expressions in this specification, indicating the presence of the stated features, elements, and / or components, but not excluding the presence of one or more other features, elements, components, and / or combinations thereof. Furthermore, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features, not just individual elements in the list. Additionally, when describing embodiments of this application, the word "may" is used to mean "one or more embodiments of this application." And the term "exemplary" is intended to refer to examples or illustrations.

[0106] Unless otherwise specified, all terms used herein (including engineering and technical terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that, unless expressly stated herein, terms defined in common dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or overly formalized meaning.

[0107] It should be noted that, where there is no conflict, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0108] refer to Figures 1-6 This embodiment provides a multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios. Addressing the problem that traditional vision-based integrated navigation methods are prone to positioning drift due to visual degradation caused by changes in brightness intensity and interference from dynamic objects in complex urban nighttime environments, a visual feature matching confidence-driven GNSS / INS / Vision tightly integrated navigation and positioning method is invented. The specific processing steps are as follows:

[0109] Figure 1 This is a flowchart illustrating a multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios provided by an embodiment of the present invention, including:

[0110] Step 100: Acquisition and preprocessing of GNSS, INS and visual data to achieve quality control of input data;

[0111] Step 200: Visual feature extraction and initial matching are performed using deep learning techniques;

[0112] Step 300: Identify and eliminate dynamic object interference by combining optical flow residuals and IMU motion parameters;

[0113] Step 400: Estimate the confidence of visual feature matching point pairs based on LSTM;

[0114] Step 500: Construct a GNSS / INS / Vision tightly coupled navigation and positioning model by fusing matching confidence scores;

[0115] Step 600: Apply the compact combination navigation and positioning model and the extended Kalman filter optimizer to obtain the final navigation result.

[0116] Specifically, such as Figure 2 The overall framework diagram shown illustrates the general concept of this invention as follows:

[0117] First, a spatiotemporally synchronized multi-sensor integrated acquisition platform is used to acquire stereo images, IMU data, and GNSS data under dynamic urban nighttime conditions, and data preprocessing is performed. Time alignment and quality control are applied to the raw GNSS observations and ephemeris data to generate double-difference pseudorange and double-difference carrier phase observations for filter measurement updates, outputting the corresponding measurement covariance as an absolute constraint in the global coordinate system. Unit conversion and integrity checks are performed on the IMU angular velocity and acceleration data, and angular velocity and acceleration noise for filter measurement updates are initialized according to IMU calibration parameters. Image enhancement is performed on stereo images of urban nighttime illumination with significant variations to improve image representation and enhance visual feature extraction and matching rates.

[0118] The image enhancement for stereoscopic images with significant changes in urban nighttime lighting involves using the open-source Retinexformer algorithm to enhance the V-channel images. Fidelity loss, lighting smoothness loss, exposure control loss, and sparsity loss are designed to construct an overall loss to improve image fidelity and visual representation.

[0119] Secondly, Superpoint is used for feature point extraction, and Superglue is used for feature matching. The combination of the two can effectively improve the feature robustness in weak texture and low-light scenes. At the same time, YOLO and SAM large models are used to detect dynamic objects in the scene, and dynamic objects are removed by combining inter-frame optical flow residuals and IMU motion parameters, so as to obtain visual information that can be effectively used for navigation.

[0120] Then, based on the stereo geometric relationship of feature points and the invariance of local features, feature trajectories were defined, and temporal feature vectors were statistically analyzed. LSTM was used to estimate the matching confidence of each feature trajectory, which was then used as the measurement weight of the vision system to improve the reliability of localization.

[0121] Finally, by combining GNSS, INS, and visual measurement models, a compact integrated navigation and positioning model is constructed based on extended Kalman filtering, which outputs accurate and reliable navigation results in real time.

[0122] In one embodiment, image enhancement is performed on stereoscopic images of cities with significant nighttime illumination variations to improve image representation.

[0123] Specifically, this invention implements images captured visually under low-light conditions based on Retinex theory. High-quality visual reflectance maps available under ideal lighting conditions and lighting components The product representation enables the understanding of... Accurate estimation can effectively improve image enhancement. To avoid color distortion caused by the high coupling between the three channels in the RGB space, the image channels are separated into HSV spaces, and the illumination component is estimated through the V luminance space. Furthermore, to achieve accurate estimation of the illumination component, degradation factors such as noise and artifacts are comprehensively considered, and a noise perturbation term for the reflection component is introduced. and lighting component disturbance terms The model has been revised, and the revised model is as follows:

[0124] (1)

[0125] To ensure uniform brightness, absence of artifacts, and noise suppression in the enhanced V channel, the overall loss function is designed as a fidelity loss. Loss of smoothness under illumination Exposure control loss and sparsity loss The combination of is defined as:

[0126] (2)

[0127] in, , , and This represents the loss balance parameter.

[0128] Normalize the V channel image to The interval is input into the Retinex Former network. Through the illumination-guided multi-head self-attention mechanism, the global dependency relationship of different brightness regions in the V channel is modeled. Then, through collaborative inference of the depth convolution and channel attention mechanism modules, the illumination component is accurately estimated. At the same time, repair and The noise and artifacts introduced are used to obtain the restored reflection components. ; and thus obtain the enhanced channel image. .

[0129] Enhanced The original H and S channel images are fused to obtain an enhanced HSV image, which is then converted back to the RGB color space to obtain an enhanced binocular RGB image.

[0130] The Superpoint method is used to extract feature points sequentially from the initial binocular images. The binocular image at time 10:00 is and After extracting feature points, Superglue is used to perform feature matching to obtain pairs of feature points with the same name. ,like Figure 3The diagram shown illustrates image enhancement, feature extraction, and matching.

[0131] In one embodiment, dynamic object interference is identified and eliminated by combining optical flow residuals and IMU motion parameters. The specific implementation process includes:

[0132] Based on the angular velocity and acceleration information of the IMU Perform state propagation and predict the state vector of the carrier. ;

[0133] Detect vehicles and pedestrians in a scene using a large YOLO model, and then analyze the detection bounding boxes. The data is input into the SAM large model to obtain the segmentation range in real time. Feature points within the bounding boxes are extracted and tracked, and inter-frame optical flow velocity is calculated. :

[0134] (3)

[0135] in, and These represent the displacements of the feature point in the horizontal and vertical directions, respectively. For optical flow vector, This represents the number of optical flow traces for the target region.

[0136] Calculate the camera translational velocity and rotation angle using statistical IMU motion data, and derive the target's expected optical flow as follows:

[0137] (4)

[0138] in, Here is the camera pose transformation matrix. The three-dimensional coordinates of the feature points within the target region. The pixel coordinates of the observed feature points. This is a camera projection model.

[0139] By constructing optical flow residuals To determine the dynamic or static state of an object.

[0140] in, This is the actual optical flow vector. To predict the optical flow vector.

[0141] Remove interfering feature points attached to dynamic targets and retain only feature point pairs in static scenes. ,like Figure 4 The diagram shown illustrates the detection of moving vehicles and pedestrians.

[0142] In one embodiment, the confidence level of visual feature matching point pairs is estimated based on LSTM, and the specific implementation process includes:

[0143] Based on the feature point pairs obtained above, forward tracking using the optical flow method is obtained. The feature observation trajectory of this feature point is constructed as follows: If a feature point cannot be tracked, the feature trajectory is marked as ended.

[0144] Based on the feature matching points of the left and right images and the inter-frame matching points, the descriptor distances are calculated sequentially. Polar geometric error Forward and backward tracking error Reprojection error Tracking length Local brightness average Local gradient intensity For each feature trajectory Extracting temporal feature vectors ;

[0145] LSTM is used for time-series updates, and the matching confidence scores corresponding to the observed trajectories of feature points are output. Then, design visual measurement weights. ;

[0146] in, It is the sigmoid activation function. It is the hidden state of the last time step. It is the initial observation noise covariance matrix of feature point j. As a regularization factor;

[0147] The weighted covariance matrices of all feature points are stacked into a single visual observation covariance matrix. , This represents a block diagonal matrix.

[0148] Each feature trajectory is provided with a temporal feature vector, such as... Figure 5 A schematic diagram of error constraints is provided. It includes:

[0149] Similarity distance is calculated based on descriptors of left and right feature points. ;

[0150] Using the calibration parameters of a binocular camera, epipolar line correction was performed on the left and right images, and the epipolar geometric errors of corresponding points on the left and right sides were statistically analyzed. ;

[0151] Based on the corresponding point pairs of inter-frame features, construct the forward and backward tracking error. ;

[0152] Through multi-view Figure 3 Feature points obtained by keratinization The corresponding 3D coordinates are then used to reproject the point onto the image using the pose and intrinsic parameters of the l-th camera, thus obtaining the predicted image point coordinates. The actual observed image point coordinates are Then construct the reprojection error ;

[0153] Tracking length of statistical feature points ;

[0154] To consider the robustness of feature points to brightness variations, a local window is taken centered on the pixel coordinates of the feature point. Statistical analysis of local brightness mean ;

[0155] To consider the robustness of feature points to variations in texture intensity, a local window is taken centered on the pixel coordinates of the feature point. Calculate the average gradient magnitude within a local window. .

[0156] In one embodiment, a tightly integrated GNSS / INS / Vision navigation and positioning model is constructed by fusing matching confidence scores. The specific implementation process includes:

[0157] First, based on the reliable feature matching point pairs and confidence levels obtained above, the historical camera poses of the same visual feature point at multiple times or under multiple camera observations are used as geometric constraints. For Same-name feature points observed by binoculars at all times Visual observation model of normalized image plane of left and right cameras Represented as:

[0158] (5)

[0159] in, For normalized image plane coordinates, This represents the coordinates in the camera coordinate system. This indicates measurement noise during observation.

[0160] By combining the image points of feature points and the collinear projection relationship between their corresponding 3D world coordinate points, the transformation relationship between the left and right camera coordinate systems and the feature points can be constructed as follows:

[0161] (6)

[0162] in, and They represent The pose and position of the left camera in the global coordinate system at any given time. , for The relative pose of the left camera to the right camera at any given time. , Let represent the coordinates of the visual feature points in the left and right camera coordinate systems, respectively. Therefore, the visual reprojection error state vector of the stereo camera can be constructed as follows:

[0163] (7)

[0164] in, The attitude and position error of the camera in the world coordinate system at a certain moment. Let be the number of cameras. The reprojection error equation for visual measurements can be expressed as:

[0165] (8)

[0166] in, and These are the coordinates of the observed image point and the coordinates of the reprojected image point, respectively. It is the Jacobian matrix corresponding to the camera state vector. This represents the observation noise determined based on the confidence level of the matching points.

[0167] Secondly, based on the principles of pseudorange and carrier phase observation, the observation equations for the real-time differential model are determined as follows:

[0168] (9)

[0169] in, This represents the double difference operator, where b and r are the base station and rover station, respectively, and i and j represent the satellite numbers observed simultaneously by the base station and rover station. , These are the ionospheric and tropospheric delay errors, respectively. and These are the carrier wavelength and carrier phase integer ambiguity, respectively. and These are pseudorange observation noise and multipath effect noise, respectively.

[0170] A state model for GNSS is constructed using double-difference carrier phase ambiguity, and the state vector is defined as follows: ;

[0171] Then, based on the acceleration observed by the INS system at time k... and angular velocity Its measurement model can be expressed as:

[0172] (10)

[0173] in, Represents the rotation matrix. and It is observation noise with a zero-mean Gaussian distribution. The measurement included the Earth's rotation. The impact, This represents the gravitational acceleration in the local coordinate system.

[0174] The error state model of the linearized INS can be expressed as:

[0175] (11)

[0176] in, , and The derivatives of the position, velocity, and attitude errors in the navigation coordinate system n are, in order. ' represents the cross product operator; , These represent the acceleration and angular velocity errors, respectively. and To provide zero-biased noise that conforms to a zero-mean Gaussian distribution; and This includes observation noise for acceleration and angular velocity. In summary, the error state vector of the INS can be expressed as... .

[0177] Finally, based on the measurement models of the aforementioned sensors, a GNSS / Vision / INS tightly integrated navigation and positioning model is constructed using MSCKF, and the system's state vector is determined as follows. The continuous state equation is derived as follows:

[0178] (12)

[0179] in, Here is the continuous-time state transition matrix of INS. and These are the INS and GNSS error state process noises, respectively.

[0180] use The fourth-order Runge-Kutta numerical integral propagation of the state variable to be estimated has the following propagation state covariance:

[0181] (13)

[0182] in, and These are the error-state covariance matrices before and after state augmentation, respectively. Represents the discrete state transition matrix. It is time The continuous-time state transition matrix at the given time. Let be the discrete noise covariance matrix.

[0183] When the system receives new images or new GNSS observations, the matrix information of the IMU attitude and bias terms associated with the observation time is augmented into the error state equation. Simultaneously, the error covariance matrix at the corresponding time is also augmented. The augmented covariance matrix is ​​expressed as follows:

[0184] (14)

[0185] in, , These represent the number of images and GNSS observations, respectively. As GNSS observations are recorded, the state variable matrix and error covariance matrix need to be dynamically adjusted. The Jacobian matrix of the camera pose in the latest state relative to the original state can be expressed as:

[0186] (15)

[0187] Therefore, based on the above measurement models, the observation model for the GNSS / INS / Vision tightly integrated navigation system can be determined as follows:

[0188] (16)

[0189] (17)

[0190] in, , These are the double-difference pseudorange and double-difference carrier phase predicted by INS mechanical orchestration, and their Jacobian matrices are respectively... , . Error state vector for RTK positioning and These represent the DD pseudorange observation error noise and DD carrier phase observation error in RTK positioning, respectively.

[0191] In one embodiment, a compact combination navigation and positioning model and an extended Kalman filter optimizer are applied to obtain the final navigation result.

[0192] Based on the new moment and The state vector contains the raw observations of GNSS pseudorange and carrier phase, visual camera pose, and INS motion parameters. Using the tightly coupled navigation model derived above, combined with extended Kalman filter increments and least squares, the state vector is measured and updated. During operation, the information weights of each sensor are determined by its measurement covariance and Jacobian sensitivity, with the Kalman gain automatically allocating the weights. The recursive process can be summarized as follows:

[0193] (18)

[0194] in, For Kalman filter gain, The observation vector is adjusted by updating the covariance matrix of the measurement noise in the observation equation. The state vector is recursively solved in the compact combination positioning model to obtain an accurate and reliable pose.

[0195] Figure 6 The diagram shows the positioning trajectory error statistics of the method of this invention and the current classic combined navigation method in complex urban nighttime environments. The statistics show that the positioning result of the method of this invention is optimal, with better accuracy and reliability, and can provide a reliable technical reference for navigation in complex urban nighttime scenarios.

[0196] According to the embodiments of this application, a multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios is proposed. Through image enhancement, dynamic object removal, matching confidence assessment, and construction of a GNSS / INS / Vision tightly combined navigation and positioning model, continuous and reliable positioning is achieved in complex environments such as changes in urban nighttime light intensity and GNSS signal obstruction. Compared with traditional methods, the positioning accuracy and reliability are significantly improved, and it is suitable for autonomous navigation tasks in complex nighttime environments.

[0197] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios, characterized in that, include: Acquisition and preprocessing of GNSS, INS and visual data to achieve quality control of input data; Deep learning techniques are used for visual feature extraction and initial matching. Combining optical flow residuals and IMU motion parameters, dynamic object interference is identified and eliminated; the confidence of visual feature matching point pairs is estimated based on LSTM; and the matching confidence is fused to construct a GNSS / INS / Vision tightly integrated navigation and positioning model. The final navigation results are obtained by applying a compact combination navigation and positioning model and an extended Kalman filter optimizer. The GNSS, INS, and visual data acquisition and preprocessing include: using a spatiotemporally synchronized multi-sensor integrated platform to collect multi-source data under complex scenarios with significant changes in urban nighttime light intensity and frequent switching between open and enclosed areas; GNSS data including base station and rover observation data and satellite ephemeris data; INS data including real-time angular velocity and acceleration data measured by the IMU; and visual data including stereo imagery. The raw GNSS observations and ephemeris data from the base station and rover are time-aligned and quality-controlled to generate double-difference pseudorange and double-difference carrier phase observations for filter measurement updates, outputting the corresponding measurement covariance as an absolute constraint in the global coordinate system; unit conversion and data integrity checks are performed on the IMU's angular velocity and acceleration data, and angular velocity and acceleration noise for filter measurement updates are initialized according to the IMU calibration parameters; image enhancement is performed on stereo imagery with significant changes in urban nighttime light intensity to improve image representation, thereby improving the extraction and matching rate of visual features.

2. The multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios according to claim 1, characterized in that, Image enhancement is performed on stereoscopic images of cities with significant nighttime illumination variations to improve image representation. This includes: images visually captured under low-light conditions based on Retinex theory. High-quality visual reflectance maps available under ideal lighting conditions and lighting components The product representation enables the understanding of... Accurate estimation can effectively improve image enhancement. To avoid color distortion caused by the high coupling between the three channels in the RGB space, the image channels are separated into HSV space, and the illumination component is estimated through the V luminance space. In addition, to achieve accurate estimation of the illumination component, degradation factors such as noise and artifacts are comprehensively considered, and a noise perturbation term for the reflection component is introduced. and lighting component disturbance terms The model has been revised, and the revised model is as follows: To ensure uniform brightness, absence of artifacts, and noise suppression in the enhanced V channel, the overall loss function is designed as a fidelity loss. Loss of smoothness under illumination Exposure control loss and sparsity loss The combination of is defined as: ,in, , , and Represents the loss balance parameter; normalizes the V-channel image to The interval is input into the Retinex Former network. Through the illumination-guided multi-head self-attention mechanism, the global dependency relationship of different brightness regions in the V channel is modeled. Then, through collaborative inference of the depth convolution and channel attention mechanism modules, the illumination component is accurately estimated. At the same time, repair and The noise and artifacts introduced are used to obtain the restored reflection components. ; and thus obtain the enhanced channel image. ; will be enhanced The original H and S channel images are fused to obtain an enhanced HSV image, which is then converted back to the RGB color space to obtain an enhanced binocular RGB image.

3. The multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios according to claim 1, characterized in that, Visual feature extraction and initial matching are performed using deep learning techniques, including: The Superpoint method is used to extract feature points sequentially from the initial binocular images. The binocular image at time 10:00 is and After extracting feature points, Superglue is used for feature matching to obtain pairs of feature points with the same name. .

4. The multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios according to claim 1, characterized in that, Combining optical flow residuals and IMU motion parameters to identify and eliminate interference from dynamic objects, including: Based on the angular velocity and acceleration information of the IMU Perform state propagation and predict the state vector of the carrier. ; Utilize YOLO large model to detect vehicles and pedestrians in the scene, and then convert the detection bounding boxes The data is input into the SAM large model to obtain the segmentation range in real time; feature points within the bounding box are extracted and tracked, and inter-frame optical flow velocity is calculated. ,in, and These represent the displacements of the feature point in the horizontal and vertical directions, respectively. For optical flow vector, The target region optical flow tracking number is calculated; the camera translation velocity and rotation angle are calculated from the statistical IMU motion data, and the expected optical flow of the target is derived as follows: ,in, Here is the camera pose transformation matrix. The three-dimensional coordinates of the feature points within the target area. The pixel coordinates of the observed feature points. For camera projection model; by constructing optical flow residuals To determine the dynamic or static state of an object; among which, This is the actual optical flow vector. To predict optical flow vectors, interference feature points attached to dynamic targets are removed, and only feature point pairs in static scenes are retained. .

5. The multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios according to claim 1, characterized in that, The confidence level of visual feature matching point pairs is estimated based on LSTM, including: Based on the feature point pairs obtained in claim 4, forward tracking using optical flow method is performed to obtain... The feature observation trajectory of this feature point is constructed as follows: If a feature point cannot be tracked, the feature trajectory is terminated; the descriptor distances are then calculated sequentially. Polar geometric error Forward and backward tracking error Reprojection error Tracking length Local brightness average Local gradient intensity For each trajectory Extracting temporal feature vectors ; LSTM is used for time-series updates, and the confidence scores corresponding to the observed trajectories of feature points are output. Then, design visual measurement weights. ;in, It is the sigmoid activation function. It is the hidden state of the last time step. It is the initial observation noise covariance matrix of feature point j. The regularization factor is used; the weighted covariance matrices of all feature points are stacked into a single visual observation covariance matrix. , This represents a block diagonal matrix.

6. The multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios according to claim 1, characterized in that, A tightly integrated GNSS / INS / Vision navigation and positioning model is constructed by fusing matching confidence scores, including: Based on the reliable feature matching point pairs and confidence levels obtained in claim 5, the historical camera poses of the same visual feature point under multiple time points or multiple camera observations are used as geometric constraints; for Same-name feature points observed by binoculars at all times Visual observation model of normalized image plane of left and right cameras Represented as: , in, For normalized image plane coordinates, This represents the coordinates in the camera coordinate system. This represents measurement noise during observation; combining the image points of feature points and the collinear projection relationship between their corresponding 3D world coordinate points, the transformation relationship between the left and right camera coordinate systems and the feature points can be constructed as follows: ， in, and They represent The pose and position of the left camera in the global coordinate system at any given time. , for The relative pose of the left camera to the right camera at any given time. , Let represent the coordinates of the visual feature points in the left and right camera coordinate systems, respectively; therefore, the visual reprojection error state vector of the stereo camera can be constructed as follows: ,in, The attitude and position error of the camera in the world coordinate system at a certain moment. Let be the number of cameras; the reprojection error equation for visual measurements can be expressed as: ,in, and These are the coordinates of the observed image point and the coordinates of the reprojected image point, respectively. It is the Jacobian matrix corresponding to the camera state vector. The observation noise is determined based on the matching point confidence level; based on the pseudorange and carrier phase observation principles, the observation equation of the real-time differential model is determined as follows: ， in, This represents the double difference operator, where b and r are the base station and rover station, respectively, and i and j represent the satellite numbers observed simultaneously by the base station and rover station. , These are the ionospheric and tropospheric delay errors, respectively. and These are the carrier wavelength and carrier phase integer ambiguity, respectively. and These are pseudorange observation noise and multipath effect noise, respectively; a GNSS state model is constructed using double-difference carrier phase ambiguity, and the state vector is defined as follows: Acceleration output based on INS system observation at time k and angular velocity Its measurement model can be expressed as: , in, Represents the rotation matrix. and It is observation noise with a zero-mean Gaussian distribution. The measurement included the Earth's rotation. The impact, The gravitational acceleration in the local coordinate system; the error state model of the linearized INS can be expressed as: , in, , and The derivatives of the position, velocity, and attitude errors in the navigation coordinate system n are, in order. ' represents the cross product operator; , These represent the acceleration and angular velocity errors, respectively. and To provide zero-biased noise that conforms to a Gaussian distribution with zero mean; and The noise is for the observations of acceleration and angular velocity; therefore, the error state vector of the INS can be expressed as: ； Based on the measurement models of the aforementioned sensors, a tightly integrated GNSS / Vision / INS navigation and positioning model is constructed using MSCKF, and the system's state vector is determined as follows. The continuous state equation is derived as follows: ， in, Here is the continuous-time state transition matrix of INS. and These are the INS and GNSS error state process noises, respectively; using The fourth-order Runge-Kutta numerical integral propagation of the state variable to be estimated has the following propagation state covariance: , in, and These are the error-state covariance matrices before and after state augmentation, respectively. Represents the discrete state transition matrix. It is time The continuous-time state transition matrix at the given time. Let be the discrete noise covariance matrix. When the system receives a new image or a new GNSS observation, the matrix information of the IMU attitude and bias terms associated with the observation time is augmented into the error state equation. At the same time, the error covariance matrix at the corresponding time is also augmented. The augmented covariance matrix is ​​expressed as: ,in, , These represent the number of images and GNSS observations, respectively. When GNSS observations are recorded, the state variable matrix and error covariance matrix need to be dynamically adjusted; where... The Jacobian matrix of the camera pose in the latest state relative to the original state can be expressed as: ， Based on the above measurement models, the observation model for the GNSS / INS / Vision tightly integrated navigation system can be determined as follows: , , in, , These are the double-difference pseudorange and double-difference carrier phase predicted by INS mechanical orchestration, and their Jacobian matrices are respectively... , ; Error state vector for RTK positioning and These represent the DD pseudorange observation error noise and DD carrier phase observation error in RTK positioning, respectively.

7. The multi-source fusion reliable navigation and positioning method for complex urban nighttime scenarios according to claim 1, characterized in that, The final navigation results are obtained by applying a compact combination navigation and positioning model and an extended Kalman filter optimizer, including: Combining the new era and The state vector includes the original observations of GNSS pseudorange and carrier phase, visual camera pose, and INS motion parameters. Based on the compact combination navigation model derived in claim 5, the measurement and update of the state vector are achieved by combining extended Kalman filter increments and least squares. During operation, the information weights of each sensor are determined by its measurement covariance and Jacobian sensitivity, and the Kalman gain automatically completes the weight allocation. The recursive process can be summarized as follows: , in, For Kalman filter gain, The observation vector is adjusted by updating the covariance matrix of the measurement noise in the observation equation. The state vector is recursively solved in the compact combination positioning model to obtain an accurate and reliable pose.

Images