Aviation scene positioning and mapping method based on implicit neural rendering and optical flow assistance

Abstract

The invention relates to the technical field of aviation target measurement, in particular to an aviation scene positioning and mapping method based on implicit neural rendering and optical flow assistance, and the method comprises the steps: extracting a key frame based on a monocular video stream, obtaining a dense depth value through the key frame in combination with TSDF model rendering, and estimating the posture of a camera; performing unsupervised dense scene measurement according to the adjacent key frames and the camera attitude, and performing training by adopting a loss function to obtain a depth map of the current key frame; obtaining a TSDF voxel grid through a TSDF model, fusing the depth map into the TSDF voxel grid, reconstructing a scene three-dimensional model, and forming a positioning and mapping model; an image data set is collected and preprocessed, a training data set is constructed, and a positioning and mapping model is trained; and carrying out aviation scene positioning and mapping based on the trained positioning and mapping model. Through positioning and dense mapping, the precision of positioning and reconstruction is improved, and then the perception capability of the aviation cockpit is improved.

Description

Technical Field

[0001] This invention relates to the field of aviation target measurement technology, specifically to an aviation scene localization and mapping method based on implicit neural rendering and optical flow assistance. Background Technology

[0002] The intelligent aircraft cockpit is a mobile interactive space centered on the pilot. The cockpit human-machine interaction transforms the interaction between humans and complex machinery into natural interaction, and the interaction mode changes from mechanical passive to active. Perception, decision-making, and execution are the core steps of the robotic arm's navigation in the intelligent cockpit. "Perception capability" serves as the core input for subsequent steps and is the foundation for autonomous action.

[0003] Visual SLAM (Simultaneous Localization and Mapping) technology can construct 3D environmental maps and simultaneously determine its own position within the map. Its localization accuracy and mapping quality directly impact the robotic arm's path planning and obstacle avoidance performance. While traditional visual SLAM technology has made significant progress in improving tracking and map construction accuracy, most systems still focus on enhancing real-time performance. This makes visual SLAM technology highly sensitive to noise and outliers during map reconstruction, resulting in lower map reconstruction quality.

[0004] Compared to near-field sensing and LiDAR (Light Detection and Ranging)-based solutions, which are not only expensive but also only applicable to indoor environments, monocular methods have significant advantages. However, since depth values ​​cannot be read by the sensor and there is no real and reliable depth map label data for training, it is difficult for monocular cameras to achieve unsupervised real-time dense reconstruction.

[0005] To address these technical shortcomings, and with the improvement of computing resources and the advancement of deep learning, many scholars have combined visual SLAM with DNN (Deep Neural Network) networks to achieve object detection and semantic segmentation, endowing the system with semantic-level environmental understanding capabilities. They employ monocular depth estimation and end-to-end networks such as variational autoencoders for dense mapping to achieve monocular pose estimation and dense mapping. However, current SLAM systems rarely use unsupervised methods for depth estimation. In real-world scenarios, unsupervised depth estimation based solely on geometric constraints is difficult to achieve ideal results due to interference factors such as specular reflection and lighting. Furthermore, in practical engineering applications, the cost and difficulty of acquiring labeled data are high. Summary of the Invention

[0006] To address the challenge that existing technologies, relying solely on geometrically constrained unsupervised depth estimation due to interference in real-world scenarios, struggle to achieve high-precision real-time localization and dense mapping of cockpit scenes, this invention aims to provide an aviation scene localization and mapping method based on implicit neural rendering and optical flow assistance. The specific technical solution employed is as follows:

[0007] Acquire monocular video stream and TSDF model, extract keyframes based on monocular video stream, render dense depth values ​​by combining keyframes with TSDF model, and estimate camera pose;

[0008] Unsupervised dense scene measurement is performed based on adjacent keyframes and camera pose, and a loss function is used for training to obtain the depth map of the current keyframe;

[0009] TSDF voxel mesh is obtained through the TSDF model, the depth map is fused into the TSDF voxel mesh, and the 3D model of the scene is reconstructed to form a localization and mapping model.

[0010] Collect and preprocess image datasets, construct training datasets, and train localization and mapping models;

[0011] Aviation scene localization and mapping are performed based on the trained localization and mapping model.

[0012] Preferably, a monocular video stream and a TSDF model are acquired, keyframes are extracted from the monocular video stream, and dense depth values ​​are obtained by rendering the keyframes in combination with the TSDF model, and the camera pose is estimated. Specifically:

[0013] Visual odometry is obtained by extracting keyframes from the monocular video stream using the DSO-SLAM keyframe selection method and rendering the corresponding dense depth values ​​based on the TSDF model. The backend of the sliding window in the direct method sparse visual odometry is optimized, and the dense depth values ​​are used to perform front-end registration of the keyframes to solve the camera pose.

[0014] Preferably, unsupervised dense scene measurement is performed based on adjacent keyframes and camera pose, and a loss function is used for training to obtain the depth map of the current keyframe, including:

[0015] Obtain the baseline model and construct a multi-view cost volume based on adjacent keyframes and camera pose.

[0016] The cost volumes for each viewpoint are aggregated to obtain a fused cost volume. The fused cost volume is then preprocessed, and the preprocessed fused cost volume is processed by activation functions and regression operations to obtain a predicted depth map. Finally, the predicted depth map is optimized by combining a loss function to determine the depth map.

[0017] Preferably, constructing a multi-view cost volume includes:

[0018] Pre-correction is performed based on each keyframe to determine the corrected image for each frame;

[0019] A benchmark model is used to perform multi-scale encoding and decoding on the corrected image;

[0020] By setting a preset sampling depth, the camera poses of adjacent keyframes are mapped to the reference viewpoint through a forward parallel plane to establish a multi-view cost volume.

[0021] Preferably, the cost volumes based on each viewpoint are aggregated to obtain a fused cost volume, and the fused cost volume is preprocessed. The preprocessed fused cost volume is then processed using activation functions and regression operations to obtain a predicted depth map, including:

[0022] A variance-based mechanism is used to aggregate the cost volumes of each viewpoint to obtain a fused cost volume;

[0023] Smoothness constraints are introduced, and a multi-stage cost volume is constructed using a cascading approach. The smoothness constraints are then used to process the cost volume at each stage.

[0024] Based on the preset sampling depth range, the predicted depth map is obtained by regression along the depth direction of the processed fusion cost volume using the Softmax activation function and the Soft argmin operation, combined with the sampling depth range.

[0025] Preferably, the loss function includes basic constraints for model training, dynamic photometric consistency constraints based on uncertainty graphs, sparse optical flow constraints based on ORB feature matching, and rendering consistency constraints.

[0026] Preferably, the basic constraints for model training are as follows:

[0027] Based on the current keyframe and adjacent keyframes, and combined with the corresponding camera pose and predicted depth map, adjacent keyframes are mapped to the current keyframe to generate pseudo-images.

[0028] The distance and structural loss for mapping the keyframe image to the reference viewpoint are determined based on the current keyframe and the pseudo-image, respectively. The smoothness loss for maintaining gradient consistency at the image boundary is determined based on the current keyframe and the predicted depth map.

[0029] Preferably, the dynamic photometric consistency constraint based on the uncertainty graph is as follows:

[0030] Obtain the pseudo-image synthesized by mapping the predicted depth map of the current keyframe, and define the uncertainty map in combination with the current keyframe;

[0031] By combining distance, structural loss, and uncertainty graph, a dynamic photometric consistency constraint based on the uncertainty graph is obtained.

[0032] Preferably, the sparse optical flow constraint based on ORB feature matching is specifically as follows:

[0033] Obtain the ORB operator, and perform feature matching based on the ORB operator to obtain sparse optical flow values;

[0034] The optical flow values ​​of sparse points are obtained by using a benchmark model to predict the depth map, and then the sparse optical flow values ​​are combined to obtain sparse optical flow constraints based on ORB feature matching.

[0035] Preferably, the rendering consistency constraint includes:

[0036] An implicit neural cost volume is constructed and processed using a 3D UNet module to obtain appearance information. The appearance information is then analyzed to determine the luminance value loss.

[0037] The current keyframe is rendered to obtain sparse rendering depth values, and the depth value loss is determined by combining the predicted depth map and the uncertainty map.

[0038] The feature points extracted by the ORB operator are used as the sampling points for rendering. Sparse optical flow is obtained based on the camera pose and sparse rendering depth value. The optical flow loss is determined in conjunction with the sparse optical flow value.

[0039] The rendering consistency loss is obtained by combining the loss of brightness, depth and optical flow values.

[0040] The present invention has the following beneficial effects:

[0041] This application focuses on real-time localization and mapping in intelligent flight cockpits to achieve dense perception of the cockpit scene. First, it analyzes a monocular video stream to determine the camera attitude. Then, through unsupervised dense scene measurement, a predicted depth map is obtained to reduce the cost of collecting real measurement data labels. The supervised dense scene measurement process is trained using a loss function to improve the accuracy of depth map acquisition and the robustness of attitude estimation, thus obtaining the designed localization and mapping model. The localization and mapping model is trained using a training dataset, and the trained model is used for localization and dense mapping to improve the accuracy of localization and reconstruction, thereby enhancing the perception capability of the flight cockpit. It features lightweight deployment, high algorithm efficiency, and high accuracy, and can be applied to real-time localization and dense perception in intelligent flight cockpit scenarios. Attached Figure Description

[0042] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating the steps of an aviation scene localization and mapping method based on implicit neural rendering and optical flow assistance, provided in an embodiment of the present invention.

[0044] Figure 2 This is a schematic diagram illustrating the operational framework of an aviation scene localization and mapping method based on implicit neural rendering and optical flow assistance, provided in one embodiment of the present invention. Detailed Implementation

[0045] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of an aviation scene localization and mapping method based on implicit neural rendering and optical flow assistance proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0047] The following description, in conjunction with the accompanying drawings, details a specific scheme for an aviation scene localization and mapping method based on implicit neural rendering and optical flow assistance provided by this invention.

[0048] Please combine Figure 1 and Figure 2 It illustrates a flowchart and operational framework diagram of an aviation scene localization and mapping method based on implicit neural rendering and optical flow assistance provided in the first embodiment of the present invention. The method includes:

[0049] Step S1: Obtain the monocular video stream and TSDF model, extract keyframes based on the monocular video stream, render the dense depth values ​​by combining the keyframes with the TSDF model, and estimate the camera pose.

[0050] Step S2: Perform unsupervised dense scene measurement based on adjacent keyframes and camera pose, and train using a loss function to obtain the depth map of the current keyframe;

[0051] Step S3: Obtain the TSDF voxel mesh through the TSDF model, fuse the depth map into the TSDF voxel mesh, reconstruct the 3D model of the scene, and form a localization and mapping model;

[0052] Step S4: Collect and preprocess the image dataset, construct the training dataset, and train the localization and mapping model;

[0053] Step S5: Perform aviation scene localization and mapping based on the trained localization and mapping model.

[0054] To better illustrate the challenges faced by traditional aerial photogrammetry or SLAM (Simultaneous Localization and Mapping) techniques in handling complex, dynamic, or drastically changing aerial scenes, such as the difficulty of feature matching due to numerous occlusions and texture loss in high-resolution image sequences, or the need to generate high-precision 3D models in real time to meet rapid response requirements, a method for aerial scene localization and mapping based on implicit neural rendering and optical flow is proposed to achieve consistency between localization and mapping.

[0055] Furthermore, in step S1, specifically:

[0056] Visual odometry is obtained by extracting keyframes from the monocular video stream using the DSO-SLAM keyframe selection method and rendering the corresponding dense depth values ​​based on the TSDF model. The backend of the sliding window in the direct method sparse visual odometry is optimized, and the dense depth values ​​are used to perform front-end registration of the keyframes to solve the camera pose.

[0057] As an optional implementation, the monocular video stream is the Replica dataset, which is a large-scale, multi-sensor indoor scene 3D reconstruction dataset containing 18 scenes from the real world, such as offices, apartments, and libraries. Each scene has been finely reconstructed in 3D through laser scanning and multi-view stereo matching, and is equipped with multi-sensor data such as RGBD (Red Green Blue Depth Image, which combines traditional RGB color image information with depth information) images. Preferably, in this embodiment, 55,000 camera trajectories and images from the Replica dataset are used to train and evaluate the proposed dense SLAM framework.

[0058] To better illustrate, visual odometry is a technique that estimates camera motion by analyzing a continuous sequence of images. It mainly utilizes feature matching and geometric relationships between images taken by the camera at different time points to calculate the camera's pose changes in translation and rotation in three-dimensional space, thereby estimating the motion trajectory.

[0059] DSO-SLAM (Direct Sparse Odometry - Semi-Direct Monocular SLAM) is used for keyframe extraction. This means that the keyframe is determined based on a threshold of the motion primitive information matrix. When the amount of motion between the current frame and the nearest keyframe exceeds a preset threshold, or when the information matrix of the current frame is significantly higher than that of the historical keyframes, the frame is considered a keyframe. This ensures that while maintaining map sparsity, it effectively covers geometrically changing areas of the scene.

[0060] The TSDF (Truncated Signed Distance Function) model is mainly used to describe the shape and surface information of objects in three-dimensional space. It assigns a value to each point in three-dimensional space to represent the distance from that point to the nearest object surface, and distinguishes whether the point is inside or outside the object based on the sign of the distance. A positive value indicates that the point is outside the object, a negative value indicates that the point is inside the object, and a zero value corresponds to the precise location on the object surface.

[0061] Specifically, for monocular video streams, dynamic selection of information-rich and representative frames as keyframes is performed to reduce the computational load of subsequent processing and ensure positioning accuracy. Next, visual odometry is used to initially estimate the camera pose to construct an initial point cloud model. This point cloud model is then converted into a voxel mesh, and distance values ​​are calculated and truncated for each voxel using a TSDF model to generate dense depth values ​​containing complete geometric information of the scene. This dense depth value is obtained through TSDF model rendering. Finally, following the visual odometry method in Tandem, a direct sparse visual odometry approach is employed. The optimization backend of the direct sparse sliding window described in the paper maintains a sliding window containing the most recent keyframes and their corresponding pose estimates. Then, nonlinear least squares optimization is used to jointly optimize the poses of all keyframes and the 3D coordinates of map points within the window to eliminate accumulated errors and improve the robustness of pose estimation. Dense depth values ​​are used for front-end registration of keyframes. That is, dense depth values ​​are introduced as constraints during feature matching or photometric error calculation to enhance the accuracy of feature correspondences and more accurately solve the relative pose of the camera between keyframes, thereby achieving high-precision real-time localization and mapping.

[0062] Further, step S2 includes:

[0063] Step S21: Obtain the baseline model and construct the multi-view cost volume based on adjacent keyframes and camera pose.

[0064] Optionally, in this embodiment, the baseline model is the general MVS model, namely Cas-MVSNet (Cost Volume Aggregation for Multi-View Stereo Network), which is a deep learning-based multi-view stereo matching network that achieves 3D reconstruction tasks by constructing and aggregating cost volumes.

[0065] Further, in step S21, the multi-view cost volume is constructed, including:

[0066] Step S211: Perform pre-correction based on each keyframe to determine the corrected image for each frame; wherein, the correction methods include, but are not limited to, color balance adjustment, brightness and contrast optimization, geometric distortion correction, and motion blur elimination, in order to eliminate possible shooting defects or environmental interference factors and ensure that each corrected image meets the expected standard.

[0067] Step S212: Multi-scale encoding and decoding of the corrected image is performed using a baseline model. In this step, a shared-weight feature extraction network, namely the MVS model, is used to perform multi-scale encoding and decoding of the corrected image. The shared-weight mechanism can effectively reduce the number of model parameters, improve the efficiency and generalization ability of feature extraction, and ensure that the corrected image from different perspectives maintains a consistent representation space during feature learning. The multi-scale encoding and decoding structure captures the global context information and high-level semantic features of the image during the encoding stage through progressive downsampling and upsampling operations. During the decoding stage, feature fusion and upsampling are performed by combining low-level edge, texture and other detailed information to restore the fine structure of the image and obtain more refined high-level semantic features and low-level features.

[0068] Step S213: Preset sampling depth, map the camera pose of adjacent keyframes to the reference viewpoint through the forward parallel plane, and establish a multi-view cost volume.

[0069] Specifically, the preset sampling depth is denoted as... The cross-viewpoint differentiable matrix transformation representation is constructed, and the corresponding calculation formula is:

[0070]

[0071] in, Indicates the sampling depth The corresponding number Cross-view differentiable matrix transformation representation of a camera with multiple viewpoints; , , They represent the first Intrinsic parameters, rotation parameters, and translation parameters of a single-view camera; Indicates the main axis of the reference camera; , , These represent the intrinsic parameters, rotation parameters, and translation parameters of the reference camera, respectively. Indicates the corrected image; This indicates transpose.

[0072] It can be explained that the multi-view cost volume is obtained through cross-view differentiable matrix transformation, that is, by precisely aligning and fusing image features or depth information acquired from different viewpoints; through differentiable matrix transformation, a series of affine transformations or projection transformations such as rotation, translation, and scaling are learned and applied to map the corresponding information from each viewpoint to a unified cost volume coordinate space, thereby establishing the multi-view cost volume. Among them, the... Each viewpoint represents the viewpoint of any analysis, while the reference camera represents the viewpoint under the keyframe image of the current analysis.

[0073] Step S22: Aggregate the cost volumes based on each viewpoint to obtain a fused cost volume, and preprocess the fused cost volume. Process the preprocessed fused cost volume through activation function and regression operation to obtain a predicted depth map, and optimize the predicted depth map by combining the loss function to determine the depth map.

[0074] The method involves obtaining a predicted depth map to predict the dense depth of a scene using an unsupervised multi-view cost volume, reducing reliance on real depth label data and lowering the cost of collecting real measurement data labels.

[0075] Further, in step S22, the cost volumes based on each viewpoint are aggregated to obtain a fused cost volume. The fused cost volume is then preprocessed using activation functions and regression operations to obtain a predicted depth map, including:

[0076] Step S221: Use a variance-based mechanism to aggregate the cost volume of each viewpoint to obtain the fused cost volume.

[0077] Specifically, based on the cost volume obtained in step S21 for each viewpoint, a variance-based mechanism is used to aggregate multi-view cost volumes to suppress noise. That is, for each voxel point in 3D space, its cost volumes for all relevant viewpoints are collected, and the variance is calculated. When the variance is small, it indicates that the matching cost consistency under different viewpoints is high; conversely, when the variance is large, there may be matching errors and noise interference. Thus, the fusion cost volume is determined, and the corresponding calculation formula is:

[0078]

[0079] in, Indicates the cost of fusion; Indicates the first The cost of each perspective; This represents the average cost volume across all perspectives; Indicates the number of viewpoints.

[0080] Step S222: Introduce smoothness constraints and construct a multi-stage cost body using a cascading approach, and process the cost body of each stage by comprehensively applying smoothness constraints.

[0081] To clarify, in real-world scenarios, interference factors such as object occlusion and lighting can easily lead to the formation of non-Lambertian surfaces. The fusion cost volume is often contaminated by noise, thus introducing smoothness constraints on the fusion cost volume. This is achieved by using a 3D convolution module to regularize the fusion cost volume to further filter noise. However, since using a 3D convolution module is not only time-consuming but also consumes a lot of video memory, a cascaded approach is used to construct a multi-stage cost volume. That is, as the cost volume resolution increases, the search range of depth values ​​is reduced. For example, in the initial stage, to ensure the accuracy of depth value estimation, a wider depth search range is used to generate a high-resolution cost volume with relatively coarse depth information. Subsequently, in later stages, by reducing the depth search range and simultaneously increasing the spatial resolution of the cost volume, fine-grained processing of depth details is achieved.

[0082] Step S223: Based on the sampling depth, a preset sampling depth range is used. Along the depth direction of the processed fusion cost volume, the Softmax activation function and Soft argmin operation are used to regress and obtain the predicted depth map.

[0083] Specifically, the sampling depth range is denoted as , Indicates the lower limit of sampling depth; This represents the upper limit of the sampling depth. The predicted depth map is obtained by using the Softmax activation function and Softargmin operation along the depth direction of the cost volume processed in step S222. This transforms the processed cost volume into a normalized probability distribution to enhance the smoothness and discriminative power of the probability distribution along the depth direction. Softargmin is used to calculate the sum of the products of the probability value of each depth plane and its corresponding depth value to obtain continuous depth estimates, thus outputting a smoother predicted depth map and effectively alleviating the depth discontinuity problem caused by discrete depth sampling. The corresponding calculation formula is:

[0084]

[0085] in, Represents the predicted depth map; Indicates the sampling depth; This represents the Softmax activation function operation; This represents the fusion cost body after processing.

[0086] The explanation is that, based on the predicted depth map, optimization is performed using a loss function to determine the depth map. In other words, the loss function is used as the optimization objective to guide parameter adjustments in order to improve the accuracy of the predicted depth map.

[0087] Furthermore, the loss function includes basic constraints for model training, dynamic photometric consistency constraints based on uncertainty graphs, sparse optical flow constraints based on ORB feature matching, and rendering consistency constraints.

[0088] It can be explained that the basic constraints of model training refer to a series of basic restrictions and rules set during the training of machine learning or deep learning models in order to ensure that the models can learn effectively, have good generalization ability, and meet the needs of practical applications.

[0089] Furthermore, the fundamental constraints for model training are as follows:

[0090] Step Sa1: Based on the current keyframe and adjacent keyframes, and combining the corresponding camera pose and predicted depth map, map the adjacent keyframes to the current keyframe to generate a pseudo image.

[0091] Specifically, given the current keyframe and adjacent keyframes, denoted as follows: and And obtain the camera's intrinsic and extrinsic parameters, which are denoted as follows: and Using any pixel coordinate point in the current keyframe To clarify, the pixel coordinates correspond to the following 3D space point in the camera coordinate system: ,in, This represents the predicted depth map corresponding to the pixel coordinates, i.e., the predicted depth value; furthermore, it maps this 3D spatial point to the coordinates of adjacent keyframes. Therefore, based on the predicted depth map, adjacent keyframe images are mapped to the current keyframe viewpoint to generate pseudo-images. The corresponding calculation formula is:

[0092]

[0093] in, This represents the pseudo-image corresponding to the current keyframe.

[0094] Step Sa2: Determine the distance and structural loss for mapping the keyframe image to the reference viewpoint based on the current keyframe and the pseudo-image, respectively. Determine the smoothness loss for maintaining gradient consistency at the image boundaries based on the current keyframe and the predicted depth map.

[0095] Specifically, the difference between the current keyframe image and the pseudo-image is minimized by introducing a photometric consistency constraint. This involves comprehensively considering the distance loss and structural similarity loss between the reference image and the synthesized image, as well as gradient loss. First, the distance and structural loss from the source view image to the reference view are mapped, and the corresponding calculation formulas are as follows:

[0096]

[0097] in, Represents distance and structural loss; Indicates the threshold parameter; Represents structural similarity operations; Indicates the current keyframe; This represents the pseudo-image corresponding to the current keyframe; This indicates the effective mask area mapped from the current keyframe viewpoint to the viewpoint of the adjacent keyframe viewpoint.

[0098] It can be explained that the effective mask region refers to the image region of the current keyframe that can be accurately corresponded to and effectively utilized from the perspective of adjacent keyframes by analyzing the viewpoint differences and spatial relationships between the current keyframe and adjacent keyframes.

[0099] Next, the smoothness loss used to constrain the depth map and the current frame image to maintain gradient consistency at the image boundaries is determined, and the corresponding calculation formula is as follows:

[0100]

[0101] in, Indicates smoothness loss; Indicates the number of keyframes; Represents the pixels in a keyframe; Indicates the first Gradient changes in keyframe images; Indicates the first The gradient changes of the predicted depth map corresponding to each keyframe image.

[0102] What can be explained is the number of keyframes This refers to the viewpoint projected onto the current keyframe from other viewpoints. By analyzing the differences between the two frames, the smoothness loss is determined.

[0103] Understandably, although the baseline model can predict dense depth maps, limitations such as network model complexity and uneven illumination make it unreasonable to use only photometric consistency loss to measure the prediction error of the depth map within the mask. Therefore, a dynamic photometric consistency constraint based on an uncertainty map is proposed. Theoretically, pseudo-images generated from the predicted depth map... With the current keyframe image The greater the difference, the more likely the image contains undefined regions such as specular reflection, leading to ambiguity in feature matching and reducing the reliability of the predicted depth map.

[0104] Furthermore, the dynamic photometric consistency constraint based on the uncertainty graph is specifically as follows:

[0105] Step Sb1: Obtain the pseudo-image synthesized by mapping the predicted depth map of the current keyframe, and define the uncertainty map in combination with the current keyframe.

[0106] The explanation is as follows: The projection error determined by the predicted depth map, i.e., the pseudo-image synthesized by mapping the current keyframe to the predicted depth map, is used to evaluate the accuracy of depth prediction. An uncertainty map is constructed to evaluate the quality of the predicted depth map, which is used to dynamically guide the photometric loss in the unsupervised process. Therefore, the uncertainty map is defined, and the corresponding calculation formula is as follows:

[0107]

[0108] in, Represents an uncertainty graph; This represents the current keyframe image; This indicates selecting from multiple source perspectives. The smallest absolute value of the projection error; Indicates the first A pseudo-image synthesized by mapping keyframes corresponding to each viewpoint through a predicted depth map.

[0109] It should be noted that the source viewpoint refers to the multiple viewpoints corresponding to the generated keyframe images. That is, for the currently used monocular camera, the camera is in motion, and there are multiple viewpoints. If only the viewpoint with the smallest error is selected, there may be data errors. Therefore, the source viewpoint is chosen from among them. The minimum absolute value of the projection error is used to balance the error; generally, Values When uncertainty graph The larger the value, the lower the reliability of the current predicted depth map.

[0110] Step Sb2: Combine distance, structural loss, and uncertainty graph to obtain dynamic photometric consistency constraints based on uncertainty graph.

[0111] Provide an explanation when uncertainty diagrams are drawn. The larger the value, the higher the weight of the current pixel's contribution to the photometric loss, thus obtaining the dynamic photometric consistency constraint. The corresponding calculation formula is:

[0112]

[0113] in, This represents a dynamic photometric consistency constraint based on an uncertainty graph; Represents distance and structural loss; This represents an uncertainty graph.

[0114] It can be explained that the sparse optical flow constraint based on ORB feature matching extracts key points in the image sequence using the ORB (Oriented FAST and Rotated BRIEF) feature detection and matching algorithm. That is, it combines the Oriented FAST (Features from Accelerated Segment Test) key point detection method and the BRIEF (Binary Robust Independent Elementary Features) feature descriptor to constrain the optical flow field through the motion information between key points. The sparse optical flow constraint based on ORB feature matching is used to minimize the optical flow difference of the corresponding key points in the benchmark model.

[0115] Furthermore, the sparse optical flow constraint based on ORB feature matching is specifically as follows:

[0116] Step Sc1: Obtain the ORB operator and perform feature matching based on the ORB operator to obtain sparse optical flow values.

[0117] The explanation involves extracting ORB feature points from pairs of adjacent keyframe images to detect representative feature points. Then, binary descriptors generated by the ORB operator are used to match the feature points in the two frames, finding corresponding feature point pairs, which constitute sparse optical flow values. The spatial position change corresponding to each feature point pair can be approximately represented as the motion information of that region, denoted as . .

[0118] Step Sc2: Obtain the optical flow values ​​of sparse points when predicting the depth map using the benchmark model, and combine the sparse optical flow values ​​to obtain sparse optical flow constraints based on ORB feature matching.

[0119] To clarify, since the ORB operator's descriptor matching relies on the appearance similarity of keypoints, mismatches are prone to occur in regions with repetitive image textures. Therefore, it is necessary to optimize the initial matching points of the obtained adjacent keyframe images. Specifically, the optical flow values ​​of sparse points are obtained from the baseline model when the predicted depth map is transformed. These optical flow values ​​reflect the motion vectors of sparse points at different times or viewpoints, denoted as... This leads to the sparse optical flow constraint based on ORB feature matching, and the corresponding calculation formula is:

[0120]

[0121] in, This represents a sparse optical flow constraint based on ORB feature matching; This represents the sparse optical flow value corresponding to the ORB operator, i.e., the sparse point optical flow value. This represents the optical flow value of sparse points when the baseline model obtains the predicted depth map, i.e., the sparse optical flow value output by the baseline model. This represents the threshold used to measure the difference between the sparse point optical flow value and the sparse optical flow value output by the baseline model.

[0122] In particular, threshold This represents hyperparameters that, in addition to the sparse optical flow constraint based on ORB feature matching, satisfy a threshold condition in the rendering consistency constraint. The ORB sparse points are randomly selected as sampling points for Nerf to guide the model in learning the 3D structure and radiation field parameters of the scene, and the sparse optical flow values ​​are used as additional information to supervise the training of the rendering process.

[0123] Understandably, regarding rendering consistency constraints, the same object can appear very differently from different viewpoints due to uneven lighting intensity and occlusion. Existing unsupervised multi-view measurement models are trained based on the photometric consistency assumption, which can easily lead to local optima and mismatches. Therefore, to address the problem of projection distortion between adjacent keyframe images, implicit neural radiation field learning is used to generate the current keyframe image from the perspective of adjacent keyframes, thereby improving the accuracy of the predicted depth map through rendering consistency constraints. Rendering consistency constraints refer to the technical requirements and specifications that ensure the visual performance of the same scene or object remains consistent when rendered under different times and conditions.

[0124] Furthermore, rendering consistency constraints include:

[0125] Step Sd1: Construct the implicit neural cost volume and process it using the 3D UNet module to obtain appearance information. Analyze the appearance information and determine the luminance value loss.

[0126] Specifically, a latent neural cost body is constructed using variance fusion, denoted as... This involves performing statistical variance analysis on cost information from multiple adjacent keyframe perspectives and weighted fusion of cost features with smaller variances to form a more robust cost representation for the current keyframe image. This effectively suppresses noise interference from single-view cost information and improves the overall stability of the cost volume. Since the current keyframe image is generated using only information from adjacent keyframe perspectives, the reference view cost volume is not considered during the mapping of the implicit neural cost volume. This simplifies computational complexity through efficient fusion of local view information and ensures the real-time performance and adaptability of the implicit neural cost volume in dynamic scene changes.

[0127] Next, in order to generate meaningful neural code, the implicit neural cost volume is processed using a 3D UNet module to obtain the appearance information of the propagable scene. The corresponding calculation formula is as follows: The 3D UNet module, through its unique encoder-decoder structure, can effectively capture spatial features at different scales in a scene. Specifically, in the encoder part, deep semantic information and detailed features in the implicit neural cost volume are gradually extracted through multi-layer convolution and pooling operations. In the decoder part, the features extracted by the encoder are combined with low-level detailed information through upsampling and skip connection mechanisms to achieve accurate modeling of the scene's appearance information. While preserving the scene's geometric structure information, appearance attributes such as material and lighting are integrated, enabling the generated neural code volume to more comprehensively represent the scene's visual characteristics.

[0128] Then, in order to restore the appearance and geometry of the scene, given arbitrary 3D spatial points... Original keyframe image The pixel color is The launch direction is determined by the direction pointing towards the center of the camera, denoted as . The MLP (Multi-Layer Perceptron) is used to analyze neural code. The volume density and associated radiance are obtained through regression. Specifically, by extracting features layer by layer through fully connected layers in a multilayer perceptron and performing nonlinear transformations, the latent features encoding the volume are learned to represent volume density and radiance. Volume density represents the opacity or concentration of matter at each point in the scene; radiance represents the intensity and color information of light emitted or reflected at each point in the scene. The corresponding calculation formula is as follows:

[0129]

[0130] in, Indicates volume density; Indicates emissivity; Indicates multilayer sensor operation; Represents a point in 3D space; Indicates the launch direction; Indicates appearance information; Represents the neural coding body; This represents the pixel information corresponding to the keyframe image.

[0131] Differentiable volume rendering is used to obtain the color value of the current keyframe image, i.e., the synthesized luminance field color output. The corresponding calculation formula is:

[0132]

[0133]

[0134] in, This represents the color value of the current keyframe image; This represents the cumulative transmittance. , They represent the first Zhang Hedi Volume density of a keyframe image; Indicates the first The radioactivity of a keyframe image; Indicates An exponential function with base 0.

[0135] The difference between the color value of the current keyframe image and the true color value of the current keyframe is measured by mean square error. The luminance loss is determined by the following formula:

[0136]

[0137] in, This indicates a loss of brightness. Represents the actual color value of the current keyframe.

[0138] Step Sd2: Render the current keyframe to obtain sparse rendering depth values, and combine the predicted depth map and the uncertainty map to determine the depth value loss.

[0139] It can be explained that, for candidate points sampled by Gaussian, density integration is performed along the ray to determine the sparse rendering depth value corresponding to the viewpoint of the current keyframe, denoted as . To constrain the geometric features of the object's surface, i.e., to ensure that the depth predicted by the baseline model matches the rendered depth, the Smooth-L1 loss is used to measure the difference between the sparse rendered depth value and the depth map predicted by the baseline model. The corresponding calculation formula is as follows:

[0140]

[0141] in, Indicates depth value loss; This represents the Smooth-L1 loss metric. This represents the sparse rendering depth value corresponding to the viewpoint of the current keyframe; Represents the predicted depth map; Represents an uncertainty graph; This represents the threshold.

[0142] It can be noted that the Smooth-L1 loss metric uses a squared term when the error is small, providing a smoother gradient and accelerating model convergence; while when the error is large, it uses a linear term to avoid the problem of excessively large gradients in L2 loss under large errors, thus enhancing the model's robustness to outliers; threshold This represents a manually set threshold. When the uncertainty map corresponding to a sampling point is lower than the threshold, theoretically the depth value predicted by the benchmark model for that sampling point is more accurate, and therefore the rendering loss does not take this into account. Conversely, if the uncertainty map is higher than the threshold, rendering consistency loss is introduced.

[0143] Step Sd3: Use the feature points extracted by the ORB operator as the sampling points for rendering. Obtain sparse optical flow based on the camera pose and sparse rendering depth value, and determine the optical flow loss in conjunction with the sparse optical flow value.

[0144] It can be explained that by randomly selecting feature points extracted by the ORB operator as sampling points for rendering, the optical flow calculated based on camera intrinsic and extrinsic parameters and rendering depth values, i.e., sparse optical flow, is restricted to be consistent with the sparse optical flow value obtained through precise matching using the ORB operator, thereby improving rendering quality. The corresponding calculation formula is as follows:

[0145]

[0146] in, Indicates optical flow loss; This represents the optical flow of the current keyframe calculated based on camera intrinsic and extrinsic parameters and rendering depth values. This represents the sparse optical flow value corresponding to the ORB operator.

[0147] Step Sd4: Combine the loss of brightness value, depth value, and optical flow value to obtain the rendering consistency loss.

[0148] Specifically, the formula for calculating the rendering consistency loss is as follows:

[0149]

[0150] in, This indicates a loss of rendering consistency. This indicates a loss of brightness. Indicates depth value loss; This indicates optical flow loss.

[0151] The explanation is as follows: The loss function is determined, i.e., the total training loss of the baseline model is determined, and thus the depth map is obtained. The corresponding calculation formula is:

[0152]

[0153] in, Indicates the total training loss; This represents a dynamic photometric consistency constraint based on an uncertainty graph; Indicates smoothness loss; This represents a sparse optical flow constraint based on ORB feature matching; This indicates a loss of rendering consistency.

[0154] It can be explained that in step S3, a TSDF voxel mesh is obtained through the TSDF model, the depth map is fused into the TSDF voxel mesh, and the 3D model of the scene is reconstructed to form a localization and mapping model.

[0155] The TSDF model is characterized by its ability to effectively fuse depth information from multiple perspectives to achieve 3D reconstruction. This involves converting the depth map into a corresponding TSDF voxel mesh, with each voxel storing the TSDF value at that location. By weighted fusion of TSDF voxel meshes from multiple perspectives, the model's representation details are gradually optimized and refined, resulting in a dense and accurate 3D point cloud model that forms the localization and mapping model.

[0156] It can be explained that in step S4, image datasets are collected and preprocessed, training datasets are constructed, and localization and mapping models are trained.

[0157] Preferably, in this embodiment, the DTU dataset is used for comparison with mainstream methods. The DTU dataset is a dataset commonly used for experimental comparison in the MVS field. It consists of 124 scenes, each of which is acquired by a camera with a fixed trajectory under 7 different lighting conditions. In order to ensure fair comparison, the official training set and test set division method is used. The EuRoC (European Robotics Challenges Dataset) dataset is used to evaluate the generalization performance of the tracking and dense 3D reconstruction model. EuRoC is a visual-inertial dataset collected on a micro air vehicle (MAV). It mainly includes image sequences, IMU (Inertial Measurement Unit) data, camera parameters, robot pose and motion trajectory. It provides 11 scenes from slow flight under good visual conditions to dynamic flight with motion blur and poor lighting. It is often used for evaluation research of localization and SLAM algorithms.

[0158] Specifically, during training, it was performed on a single NVIDIA RTX 3090 GPU, with a batch size of 4 and an input image size of [missing information]. The number of source views was 3, the initial learning rate was 0.0001, and the number of samples for the three stages of depth map training (coarse to fine) were 48, 32, and 8, respectively, for a total of 15 epochs. During the testing phase, the number of source views was 4, and the input size was... .

[0159] During the training phase of the Replica dataset, CVA-MVSNet was used as the baseline model, and the input image size was [missing information]. The training process consisted of 10 epochs with a learning rate of 0.001. The number of samples in the cascaded depth ranges for the three stages were 48, 32, and 8, respectively. Notably, unlike CVA-MVSNet which uses an adaptive view aggregation module to fuse the feature cost volume, a simple variance aggregation mechanism was used to fuse the feature volume for ease of ablation experiments. During the inference phase, dense tracking and bundle adjustment were executed in parallel on the CPU (Central Processing Unit), and the inference of the unsupervised deep model was performed in C++ using TorchScript, asynchronously and in parallel with the TSDF model on the GPU (Graphics Processing Unit).

[0160] Next, evaluation metrics were determined. For the DTU dataset, following RC-MVSNet, the absolute error under a fixed threshold was used to evaluate the depth map quality. For the Replica and EuRoC datasets, following Tandem, visual odometry was run once to generate pose and keyframe windows, and the depth map prediction error under a set threshold was mainly used to evaluate the depth map quality. The corresponding calculation formula is as follows:

[0161]

[0162] in, This indicates that a threshold value has been set. Indicates the index function; This indicates the number of images in the dataset; This represents the predicted depth value of the output. This represents the actual depth value.

[0163] It can be explained that setting a threshold This represents the percentage of pixels whose predicted depth value is within 10% of the true value. The thresholds are set at 1% and 0.1% of the true value, respectively, as evaluation metrics. Then, the mean absolute error and the metrics proposed by Eigen et al. are used for evaluation. The corresponding calculation formula is as follows:

[0164]

[0165]

[0166] in, Indicates the mean absolute error; Indicates the evaluation indicators; This indicates taking the maximum value.

[0167] Finally, in step S5, aviation scene localization and mapping are performed based on the trained localization and mapping model.

[0168] Understandably, this application focuses on real-time localization and mapping for intelligent cockpits in aviation, aiming to achieve dense perception of intelligent cockpit scenes. First, it analyzes monocular video streams to determine camera attitude. Then, through unsupervised dense scene measurement, a predicted depth map is obtained to reduce the cost of collecting real measurement data labels. The supervised dense scene measurement process is trained using a loss function to improve the accuracy of depth map acquisition and the robustness of attitude estimation, thereby obtaining the designed localization and mapping model. The localization and mapping model is trained using a training dataset, and the trained model is used for localization and dense mapping to improve the accuracy of localization and reconstruction, thus enhancing the perception capability of the aviation cockpit. It features lightweight deployment, high algorithm efficiency, and high accuracy, and can be applied to real-time localization and dense perception in intelligent flight cockpit scenes.

[0169] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0170] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A method for aviation scene localization and mapping based on implicit neural rendering and optical flow assistance, characterized in that, The method includes: Acquire monocular video stream and TSDF model, extract keyframes based on monocular video stream, render dense depth values ​​by combining keyframes with TSDF model, and estimate camera pose; Unsupervised dense scene measurements are performed based on adjacent keyframes and camera pose, and a loss function is used for training to obtain the depth map of the current keyframe. TSDF voxel mesh is obtained through the TSDF model, the depth map is fused into the TSDF voxel mesh, and the 3D model of the scene is reconstructed to form a localization and mapping model. Collect and preprocess image datasets, construct training datasets, and train localization and mapping models; Aviation scene localization and mapping are performed based on the trained localization and mapping model.

2. The method for aviation scene localization and mapping based on implicit neural rendering and optical flow assistance according to claim 1, characterized in that, The process involves acquiring a monocular video stream and a TSDF model, extracting keyframes from the monocular video stream, rendering dense depth values ​​using the keyframes combined with the TSDF model, and estimating the camera pose. Specifically: Visual odometry is obtained by extracting keyframes from the monocular video stream using the DSO-SLAM keyframe selection method and rendering the corresponding dense depth values ​​based on the TSDF model. The backend of the sliding window in the direct method sparse visual odometry is optimized, and the dense depth values ​​are used to perform front-end registration of the keyframes to solve the camera pose.

3. The method for aviation scene localization and mapping based on implicit neural rendering and optical flow assistance according to claim 1, characterized in that, Unsupervised dense scene measurements are performed based on adjacent keyframes and camera pose, and a loss function is used for training to obtain the depth map of the current keyframe, including: Obtain the baseline model and construct a multi-view cost volume based on adjacent keyframes and camera pose. The cost volumes for each viewpoint are aggregated to obtain a fused cost volume. The fused cost volume is then preprocessed, and the preprocessed fused cost volume is processed by activation functions and regression operations to obtain a predicted depth map. Finally, the predicted depth map is optimized by combining a loss function to determine the depth map.

4. The method for aviation scene localization and mapping based on implicit neural rendering and optical flow assistance according to claim 3, characterized in that, Constructing a multi-perspective cost volume includes: Pre-correction is performed based on each keyframe to determine the corrected image for each frame; A benchmark model is used to perform multi-scale encoding and decoding on the corrected image; By setting a preset sampling depth, the camera poses of adjacent keyframes are mapped to the reference viewpoint through a forward parallel plane to establish a multi-view cost volume.

5. The method for aviation scene localization and mapping based on implicit neural rendering and optical flow assistance according to claim 4, characterized in that, The cost volumes from each viewpoint are aggregated to obtain a fused cost volume. This fused cost volume is then preprocessed using activation functions and regression operations to obtain a predicted depth map, including: A variance-based mechanism is used to aggregate the cost volumes of each viewpoint to obtain a fused cost volume; Smoothness constraints are introduced, and a multi-stage cost volume is constructed using a cascading approach. The smoothness constraints are then used to process the cost volume at each stage. Based on the preset sampling depth range, the predicted depth map is obtained by regression along the depth direction of the processed fusion cost volume using the Softmax activation function and the Soft argmin operation, combined with the sampling depth range.

6. The method for aviation scene localization and mapping based on implicit neural rendering and optical flow assistance according to claim 5, characterized in that, The loss function includes basic constraints for model training, dynamic photometric consistency constraints based on uncertainty graphs, sparse optical flow constraints based on ORB feature matching, and rendering consistency constraints.

7. The method for aviation scene localization and mapping based on implicit neural rendering and optical flow assistance according to claim 6, characterized in that, The fundamental constraints for model training are as follows: Based on the current keyframe and adjacent keyframes, and combined with the corresponding camera pose and predicted depth map, adjacent keyframes are mapped to the current keyframe to generate pseudo-images. The distance and structural loss for mapping the keyframe image to the reference viewpoint are determined based on the current keyframe and the pseudo-image, respectively. The smoothness loss for maintaining gradient consistency at the image boundary is determined based on the current keyframe and the predicted depth map.

8. The method for aviation scene localization and mapping based on implicit neural rendering and optical flow assistance according to claim 7, characterized in that, The dynamic photometric consistency constraint based on the uncertainty graph is as follows: Obtain the pseudo-image synthesized by mapping the predicted depth map of the current keyframe, and define the uncertainty map in combination with the current keyframe; By combining distance, structural loss, and uncertainty graph, a dynamic photometric consistency constraint based on the uncertainty graph is obtained.

9. The method for aviation scene localization and mapping based on implicit neural rendering and optical flow assistance according to claim 8, characterized in that, The sparse optical flow constraint based on ORB feature matching is as follows: Obtain the ORB operator, and perform feature matching based on the ORB operator to obtain sparse optical flow values; The optical flow values ​​of sparse points are obtained by using a benchmark model to predict the depth map, and then the sparse optical flow values ​​are combined to obtain sparse optical flow constraints based on ORB feature matching.

10. The method for aviation scene localization and mapping based on implicit neural rendering and optical flow assistance according to claim 9, characterized in that, Rendering consistency constraints include: An implicit neural cost volume is constructed and processed using a 3D UNet module to obtain appearance information. The appearance information is then analyzed to determine the luminance value loss. The current keyframe is rendered to obtain sparse rendering depth values, and the depth value loss is determined by combining the predicted depth map and the uncertainty map. The feature points extracted by the ORB operator are used as the sampling points for rendering. Sparse optical flow is obtained based on the camera pose and sparse rendering depth value. The optical flow loss is determined in conjunction with the sparse optical flow value. The rendering consistency loss is obtained by combining the loss of brightness, depth and optical flow values.

Images