A method, system and device for depth prediction and visual-inertial coupled localization and mapping

Abstract

The application relates to the technical field of simultaneous localization and mapping, in particular to a depth prediction and visual-inertial coupling positioning and mapping method, system and device. The application comprises the following steps: constructing and training a depth prediction network, quickly adapting a pre-trained model by setting a virtual camera focal length, and outputting a high-precision dense depth map; obtaining the depth information of feature points in a sliding window by using the depth prediction network, and completing the initialization of the gyroscope zero offset, the gravity vector and the speed; taking the depth prediction value as the optimization initial value of the inverse depth of the feature points; jointly inputting the projection depth residual error, the pre-integration residual error and the visual re-projection residual error into a sliding window optimizer for joint nonlinear optimization to estimate the system state; and using a fast voxel traversal algorithm to construct and update a truncated signed distance function map in real time. The application solves the problems of slow initialization, low positioning precision and weak environmental perception capability in the prior art by constructing a depth prediction network.

Description

Technical Field

[0001] This invention relates to the field of simultaneous localization and mapping technology, and in particular to a method, system and apparatus for depth prediction and visual-inertial coupled localization and mapping. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a core technology for achieving autonomous robot navigation. Among these technologies, visual-inertial systems (VIS) consisting of a monocular camera and an IMU (Inertial Measurement Unit) are widely used due to their low cost and small size. However, traditional monocular visual-inertial systems have the following inherent drawbacks: (1) Monocular cameras cannot directly obtain depth information of the scene. They must extract image feature points and perform triangulation to estimate the depth, which leads to insufficient stability during long-term global positioning and may cause scale drift. Furthermore, they cannot generate dense point cloud maps of the scene, resulting in weak perception capabilities.

[0003] (2) For visual inertial positioning systems, the accuracy of the constraints provided by the visual module largely depends on the texture features and light distribution of the surrounding scene. When the visual information is weak or chaotic, it is easy to fail to track. The IMU module is not affected by external visual information, but the inherent zero bias error and Gaussian noise of the accelerometer cause its long-term positioning divergence. Especially on platforms such as robot dogs where the body swings violently, commercial-grade IMUs relying solely on accelerometer integration will generate meter-level positioning errors within seconds.

[0004] In recent years, significant progress has been made in monocular depth estimation based on deep learning. However, how to efficiently and robustly couple depth prediction networks with the traditional visual inertial odometry (VIO) framework, and leverage this to improve positioning accuracy, accelerate initialization, and generate dense maps, remains a technical challenge for those skilled in the art. In existing technologies, some solutions only use depth prediction for initialization or as a loose post-processing step, failing to achieve system-level depth fusion and thus offering limited improvement in positioning accuracy. Furthermore, general depth models lack generalization ability in specific scenarios, making them difficult to apply directly. Summary of the Invention

[0005] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention provides a depth prediction and visual-inertial coupled localization mapping method, system, and apparatus, thereby solving the problems of slow initialization, low positioning accuracy, and weak environmental perception capabilities in the prior art.

[0006] This invention provides a depth prediction and visual-inertial coupled localization mapping method, comprising the following steps: S1: Collect RGB images from the environment, construct and train a depth prediction network based on the ViT encoder and ConvGRU decoder based on the RGB images, normalize all training and test images to a unified intrinsic parameter space by setting the virtual camera focal length, use the LoRA fine-tuning strategy to quickly adapt the pre-trained model, and output a high-precision dense depth map. S2: The depth prediction network is used to obtain the depth information of feature points within the sliding window. The PnP algorithm is used to solve the initial pose of the camera frame. The gyroscope zero bias, gravity vector and velocity are initialized by combining the IMU pre-integration data to obtain the depth prediction value. S3: Use the predicted depth value as the initial value for optimizing the inverse depth of the feature point; at the same time, construct the projection depth residual; incorporate the projection depth residual, IMU pre-integration residual, and visual reprojection residual into a sliding window optimizer for joint nonlinear optimization to estimate the system state and obtain the estimated optimization result; S4: Utilize the optimized precise pose and dense depth map of each frame to construct a point cloud map based on an octree data structure in real time, and to construct and update a truncated symbolic distance function map in real time using a VDB (Voxel Database) data structure and a fast voxel traversal algorithm.

[0007] According to the depth prediction and visual-inertial coupled localization mapping method provided by the present invention, the ViT encoder adopts ViT-Base, which segments the input image into image blocks of fixed size and captures the global context through a multi-head self-attention mechanism; the ConvGRU decoder upsamples the feature map output by the encoder multiple times, gradually recovers the high-resolution depth map by iteratively fusing multi-scale features, and finally outputs the depth probability distribution through the Softmax function and calculates the expected depth value.

[0008] According to the depth prediction and visual-inertial coupled localization mapping method provided by the present invention, the setting of the virtual camera focal length is implemented by: setting a virtual focal length. For any original camera focal length Calculate the scaling factor During training, the ground truth depth map is multiplied by the scaling factor. During inference, the depth map output by the network is divided by the scaling factor. This is to eliminate scale ambiguity caused by different camera intrinsic parameters.

[0009] According to the depth prediction and visual-inertial coupled localization mapping method provided by the present invention, the LoRA fine-tuning strategy is implemented as follows: freeze all the original weights of the pre-trained model, add the first low-rank update matrix and the second low-rank update matrix only in the bypass of the query, key, value, output linear layer of the Transformer encoder and the output linear layer of the ConvGRU, and train on a small dataset of the target scene.

[0010] According to the depth prediction and visual-inertial coupled localization mapping method provided by the present invention, the LoRA fine-tuned training loss function Smoothing L1 loss Virtual normal loss and knowledge distillation loss The weighted summation is calculated using the following formula: in, As the first coefficient, As the second coefficient, It is the third coefficient.

[0011] According to the depth prediction and visual-inertial coupled localization mapping method provided by the present invention, the projection depth residual is constructed as follows: for the first... The first observation of the frame and the depth prediction value are Feature points When it was the first When a frame is observed, it is projected onto the first frame through a pose transformation. In the frame camera coordinate system, the projection depth is obtained. Then the projection depth residual ,in, For this feature point at the th The direct depth prediction value of the frame.

[0012] According to the depth prediction and visual-inertial coupled positioning mapping method provided by the present invention, the projection depth residual is set to a covariance matrix, the value of which is a preset multiple of the visual reprojection error covariance matrix, and a robust kernel function is introduced. When the absolute value of the projection depth residual is greater than a set threshold, it is regarded as an outlier and treated as such, so as to suppress the interference of depth prediction outliers on optimization.

[0013] According to the depth prediction and visual-inertial coupled localization mapping method provided by the present invention, the construction and updating of the truncated symbolic distance function map is as follows: Define a truncated distance; for each ray originating from the camera center, use a fast voxel traversal algorithm to obtain all voxels along the path; calculate the distance from the voxel center to the camera. and pixel depth value Calculate the symbolic distance function value : If and only if If the value is less than the cutoff distance, the cutoff sign distance function value and weight of the voxel are updated by weighted average; otherwise, no processing is performed.

[0014] The present invention also provides a depth prediction and visual-inertial coupled localization and mapping system, comprising: Scene-adaptive monocular depth prediction module: Acquires RGB images in the environment, constructs and trains a depth prediction network based on ViT encoder and ConvGRU decoder based on the RGB images, normalizes all training and test images to a unified intrinsic parameter space by setting the virtual camera focal length, and uses LoRA fine-tuning strategy to quickly adapt the pre-trained model and output a high-precision dense depth map. Fast initialization module: The depth prediction network is used to obtain the depth information of feature points within the sliding window, the PnP algorithm is used to solve the initial pose of the camera frame, and the gyroscope zero bias, gravity vector and velocity are initialized by combining IMU pre-integration data to obtain the depth prediction value. Tightly coupled nonlinear optimization module: The depth prediction value is used as the initial value for optimizing the inverse depth of the feature point; at the same time, a projection depth residual is constructed, which measures the difference between the projected depth of the feature point when it is projected from the reference frame to other observation frames and the depth prediction value output by the depth prediction network in the corresponding frame; the projection depth residual, IMU pre-integration residual, and visual reprojection residual are jointly incorporated into a sliding window optimizer for joint nonlinear optimization to estimate the system state; Dense Mapping Module: Utilizing optimized precise poses and dense depth maps for each frame, it constructs point cloud maps in real time based on octree data structures, and truncated symbolic distance function maps in real time using VDB data structures and fast voxel traversal algorithms.

[0015] The present invention also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor is used to execute a computer program to implement the steps of a depth prediction and visual-inertial coupled localization mapping method as described above.

[0016] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: This solution achieves a qualitative leap in the initialization phase. Traditional visual-inertial systems rely on sufficient translational motion to triangulate feature points and restore absolute scale. This process is not only time-consuming but also prone to failure in static or purely rotating scenes. This solution innovatively utilizes a scene-adaptive depth prediction network to directly assign reliable depth priors to feature points within the sliding window. Combined with the PnP algorithm, it quickly solves the initial pose for each frame. Subsequently, it combines IMU pre-integration to complete the calibration of gyroscope zero bias, gravity vector, and velocity in one go, compressing the initialization time to the second level. Moreover, it eliminates the need for deliberate and complex initialization motions, significantly improving the real-time performance and robustness of system startup.

[0017] In terms of positioning accuracy, the core breakthrough of this scheme lies in proposing and tightly coupling the "projection depth residual". Unlike existing methods that only treat depth prediction as a loosely coupled post-processing or simple initial value, this scheme constructs a novel geometric constraint term by subtracting the projected depth of feature points when projected from the reference frame to the observation frame from the depth value directly output by the depth prediction network in that frame. This residual directly constrains the camera translation vector and the absolute depth of feature points, and is incorporated together with the IMU pre-integration residual and the visual reprojection residual into the Ceres optimizer for joint nonlinear optimization. This effectively compensates for the lack of constraints in pure vision in weak texture and fast-moving scenes, and fundamentally suppresses the scale drift problem during long-term operation.

[0018] In terms of system robustness design, this scheme constructs a multi-layered fault tolerance and acceleration mechanism. First, the depth prediction value is directly used as the initial value for the inverse depth optimization of feature points, providing the nonlinear optimizer with a more accurate convergence starting point than random initialization or triangulation, significantly reducing the number of iterations. Second, a differentiated information matrix strategy and Huber robust kernel function are introduced into the projected depth residual. When the residual abnormally increases, it automatically reduces the weight or truncates, effectively suppressing the interference of occasional outliers in the depth prediction network on the overall optimization, ensuring the system operates stably for a long time in complex and ever-changing environments.

[0019] In terms of environmental perception and mapping capabilities, this solution achieves a leap from traditional sparse perception to dense reconstruction. Thanks to the ability to obtain a high-precision dense depth map in each frame, the system runs an independent dense mapping module in parallel outside the localization thread, constructing two types of high-quality 3D maps in real time: one is a dense point cloud map managed based on an octree data structure, which efficiently supports large-scale spatial indexing; the other is a truncated symbolic distance function (TSDF) map updated in real time based on a VDB sparse voxel structure and a fast voxel traversal algorithm, balancing high spatial resolution and low memory usage. It can directly extract fine 3D meshes using the MarchingCubes algorithm, providing rich environmental information for robot navigation, obstacle avoidance, and scene understanding.

[0020] In terms of engineering practicality and scenario generalization, this solution achieves low-cost and high-efficiency scene adaptation through LoRA fine-tuning strategy and virtual camera normalization. The pre-trained large model only needs to collect a small number of representative images of the target scene, freeze most of the original weights, and train only the low-rank matrix parameters added by the side path to quickly converge to high accuracy. The training cost is much lower than full fine-tuning, and the prediction ability of general scenes is effectively preserved through knowledge distillation loss. At the same time, the virtual camera focal length setting eliminates the depth scale ambiguity caused by different camera intrinsic parameters, so that the same model can be seamlessly adapted to different hardware platforms. The entire solution only requires a monocular camera and a commercial-grade IMU to achieve high-precision localization and dense mapping. The hardware cost is low, the size is small, and the power consumption is controllable, making it particularly suitable for mobile robot platforms that are sensitive to payload and cost, such as robot dogs and drones.

[0021] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a depth prediction and visual-inertial coupled localization mapping method provided by the present invention.

[0024] Figure 2 This is a schematic diagram of the structure of a depth prediction and visual-inertial coupled positioning and mapping system provided by the present invention.

[0025] Figure 3 This is a schematic diagram of the electronic device provided by the present invention.

[0026] Figure label: 101. Scene-adaptive monocular depth prediction module; 102. Fast initialization module; 103. Tightly coupled nonlinear optimization module; 104. Dense mapping module; 810. Processor; 820. Communication interface; 830. Memory; 840. Communication bus. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but cannot be used to limit the scope of this invention.

[0028] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0029] Example The following is combined Figures 1 to 3 This invention is described.

[0030] like Figure 1 As shown, Figure 1 This is a flowchart illustrating a depth prediction and visual-inertial coupled localization mapping method provided by the present invention. It includes the following steps: S1: Collect RGB images from the environment, construct and train a depth prediction network based on the ViT encoder and ConvGRU decoder based on the RGB images, normalize all training and test images to a unified intrinsic parameter space by setting the virtual camera focal length, use the LoRA fine-tuning strategy to quickly adapt the pre-trained model, and output a high-precision dense depth map. S2: The depth prediction network is used to obtain the depth information of feature points within the sliding window. The PnP algorithm is used to solve the initial pose of the camera frame. The gyroscope zero bias, gravity vector and velocity are initialized by combining the IMU pre-integration data to obtain the depth prediction value. S3: Use the depth prediction value as the initial value for optimizing the inverse depth of the feature point; at the same time, construct the projection depth residual, which measures the difference between the projected depth of the feature point when it is projected from the reference frame to other observation frames and the depth prediction value output by the depth prediction network in the corresponding frame; incorporate the projection depth residual, IMU pre-integration residual, and visual reprojection residual into a sliding window optimizer for joint nonlinear optimization to estimate the system state; S4: Utilize the optimized precise pose and dense depth map of each frame to construct a point cloud map based on an octree data structure in real time, and construct and update a truncated symbolic distance function map in real time using the VDB data structure and fast voxel traversal algorithm.

[0031] Specifically, the ViT encoder uses ViT-Base to segment the input image into fixed-size image blocks and captures the global context through a multi-head self-attention mechanism; the ConvGRU decoder upsamples the feature map output by the encoder multiple times, gradually recovers the high-resolution depth map by iteratively fusing multi-scale features, and finally outputs the depth probability distribution through the Softmax function and calculates the expected depth value.

[0032] Specifically, this invention uses ViT-Base as the encoder. The input RGB image is first segmented into 14x14 image blocks, linearly projected, and then embedded with learnable positional codes before being fed into a 12-layer Transformer encoder. This encoder utilizes a multi-head self-attention mechanism to capture the global contextual information of the image, ultimately outputting a high-dimensional feature map. The decoder employs a ConvGRU (Convolutional Gated Recurrent Unit) structure. ConvGRU combines the spatial feature capture capability of convolutional operations with the sequence information modeling capability of GRU, resulting in better recovery of depth details compared to traditional CNN decoders. Specifically, the feature map output by the encoder is upsampled three times through deconvolution, generating three feature maps with different resolutions. These multi-scale features are iteratively fused through multiple ConvGRU units, gradually refining the depth prediction features. After repeating this iterative process multiple times (e.g., 4 times), a linear layer and a Softmax function are used to transform the depth prediction features into a probability distribution uniformly distributed across a pre-defined logarithmic space depth interval. Finally, by calculating the expected value of the probability distribution and the corresponding depth value, a dense depth map with a resolution of 1 / 4 of the input image is output.

[0033] Specifically, to address the issue of image scale ambiguity caused by different camera focal lengths, which affects the accuracy of depth prediction, this invention incorporates a virtual camera in the data preprocessing and post-processing stages of the depth prediction module. This virtual camera is connected in series before the image input network and after the network outputs the depth map. Its input consists of images and their corresponding real depth maps from different data sources (such as various public datasets and self-collected data), and its output is a depth map unified in a virtual intrinsic parameter space. Specifically, a unified virtual focal length is set. For any raw data, based on its original focal length... Calculate the scaling factor During training, the ground truth depth map is multiplied by... The depth map is scaled to align with the depth scale in the virtual intrinsic space, serving as a monitoring signal. During inference, the network outputs a depth map in the virtual intrinsic space, therefore it needs to be divided by... To restore it to its true scale for subsequent positioning and mapping.

[0034] Specifically, to enable a pre-trained large model to adapt to specific scenarios quickly and cost-effectively, this invention employs a LoRA (Low-Rank Adaptation) fine-tuning strategy. Based on a Metric3D model jointly pre-trained on multiple datasets, a small dataset of the target scene (e.g., 5000 representative images selected from the CEAR dataset) is loaded. During fine-tuning, all original weights of the pre-trained model are frozen, and training is performed only by adding a first low-rank matrix and a second low-rank matrix with rank 8 to the bypass of the Q, K, V, O linear layers of the Transformer encoder and the output linear layer of the ConvGRU. The system's total loss function... It consists of three parts: in, The first coefficient is 0.5 in this embodiment of the invention. The second coefficient is 0.3 in this embodiment of the invention. The third coefficient is 0.2 in this embodiment of the invention.

[0035] in, To smooth out the L1 loss and ensure the numerical accuracy of the predicted depth; For virtual normal loss, the constraints on scene geometry are enhanced by comparing the differences in local surface normals generated by the predicted depth and the true depth. The knowledge distillation loss is used to constrain the predicted probability distribution of the fine-tuned student model to not deviate too far from the pre-trained teacher model, thus maintaining generalization ability. The AdamW optimizer is used with a learning rate of 2e-5, trained for 50 epochs. After training, the LoRA parameters are merged with the original model parameters to obtain a high-precision scene-adaptive deep prediction model.

[0036] Specifically, the projection depth residual is constructed as follows: for the first... The first observation of the frame and the depth prediction value are Feature points When it was the first When a frame is observed, it is projected onto the first frame through a pose transformation. In the frame camera coordinate system, the projection depth is obtained. The formula for calculating the projection depth residual is: : in, For this feature point at the th The direct depth prediction value of the frame.

[0037] Select 5 consecutive frames within the sliding window. For the first frame (reference frame) The Shi-Tomasi corners extracted from the image are used to obtain their depth values ​​in the coordinate system of the frame through a fine-tuned depth prediction network. Using the camera intrinsic parameter matrix This feature point is restored to a three-dimensional spatial point. : in, Let x be the horizontal pixel coordinate of the feature point in the first frame of the image. Let T be the vertical pixel coordinate of the feature point in the first frame of the image, and the superscript T indicates vector transpose.

[0038] By utilizing the feature matching relationship between frame 1 and frame 2, a PnP optimization problem is constructed. To optimize the results, the calculation formula is as follows: in, For the first The actual pixel coordinates of each feature point in the second frame. For homogeneous coordinate normalization operation, To construct the rotation matrix of the pose transformation matrix of the second frame relative to the first frame, Let be the translation vector that forms the pose transformation matrix of the second frame relative to the first frame. Due to depth priors, this problem has a unique solution and can be solved iteratively using the Gauss-Newton method. Repeat this process to solve for the pose of all frames within the window relative to the reference frame.

[0039] The projection depth residual is set to a covariance matrix, the value of which is a preset multiple of the visual reprojection error covariance matrix. A robust kernel function is introduced, and when the absolute value of the projection depth residual is greater than a set threshold, it is treated as an outlier to suppress the interference of depth prediction outliers on optimization.

[0040] Specifically, the construction and updating method of the truncated symbol distance function map is as follows: Define a truncated distance; for each ray originating from the camera center, use a fast voxel traversal algorithm to obtain all voxels along the path; calculate the distance from the voxel center to the camera. and pixel depth value Calculate the symbolic distance function value : If and only if If the value is less than the cutoff distance, the cutoff sign distance function value and weight of the voxel are updated by weighted average; otherwise, no processing is performed.

[0041] like Figure 2 As shown, the following describes a depth prediction and visual-inertial coupled localization mapping system provided by the present invention. The depth prediction and visual-inertial coupled localization mapping system described below can be referred to in correspondence with the depth prediction and visual-inertial coupled localization mapping method described above.

[0042] Scene-adaptive monocular depth prediction module 101: Acquires RGB images in the environment, constructs and trains a depth prediction network based on the ViT encoder and ConvGRU decoder based on the RGB images, sets the virtual camera focal length, normalizes all training and test images to a unified intrinsic parameter space, uses LoRA fine-tuning strategy to quickly adapt the pre-trained model, and outputs a high-precision dense depth map. Fast initialization module 102: It uses the depth prediction network to obtain the depth information of feature points within the sliding window, uses the PnP algorithm to solve the initial pose of the camera frame, and combines the IMU pre-integration data to complete the initialization of gyroscope zero bias, gravity vector and velocity to obtain the depth prediction value. Tightly coupled nonlinear optimization module 103: uses the depth prediction value as the initial value for optimizing the inverse depth of the feature point; at the same time, it constructs a projection depth residual, which measures the difference between the projected depth of the feature point when it is projected from the reference frame to other observation frames and the depth prediction value output by the depth prediction network in the corresponding frame; the projection depth residual, IMU pre-integration residual, and visual reprojection residual are jointly incorporated into a sliding window optimizer for joint nonlinear optimization to estimate the system state; Dense Mapping Module 104: Utilizes optimized precise pose and dense depth map of each frame to construct point cloud map based on octree data structure in real time, and truncated symbolic distance function map to construct and update in real time using VDB data structure and fast voxel traversal algorithm.

[0043] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a depth prediction and visual-inertial coupled localization mapping method, which includes: S1: Collect RGB images from the environment, construct and train a depth prediction network based on the ViT encoder and ConvGRU decoder based on the RGB images, normalize all training and test images to a unified intrinsic parameter space by setting the virtual camera focal length, use the LoRA fine-tuning strategy to quickly adapt the pre-trained model, and output a high-precision dense depth map. S2: The depth prediction network is used to obtain the depth information of feature points within the sliding window. The PnP algorithm is used to solve the initial pose of the camera frame. The gyroscope zero bias, gravity vector and velocity are initialized by combining the IMU pre-integration data to obtain the depth prediction value. S3: Use the depth prediction value as the initial value for optimizing the inverse depth of the feature point; at the same time, construct the projection depth residual, which measures the difference between the projected depth of the feature point when it is projected from the reference frame to other observation frames and the depth prediction value output by the depth prediction network in the corresponding frame; incorporate the projection depth residual, IMU pre-integration residual, and visual reprojection residual into a sliding window optimizer for joint nonlinear optimization to estimate the system state; S4: Utilize the optimized precise pose and dense depth map of each frame to construct a point cloud map based on an octree data structure in real time, and construct and update a truncated symbolic distance function map in real time using the VDB data structure and fast voxel traversal algorithm.

[0044] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0045] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0047] It should be noted that the embodiments of this disclosure can be implemented using hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described apparatus and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a programmable memory or a data carrier such as an optical or electronic signal carrier.

[0048] Furthermore, although the operation of the methods of this disclosure is described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps. It should also be noted that the features and functions of two or more devices according to this disclosure may be embodied in one device. Conversely, the features and functions of one device described above may be further divided and embodied by multiple devices.

[0049] While this disclosure has been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the specific embodiments disclosed. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A depth prediction and visual-inertial coupled localization mapping method, characterized in that, Includes the following steps: S1: Collect RGB images from the environment, construct and train a depth prediction network based on the ViT encoder and ConvGRU decoder based on the RGB images, normalize all training and test images to a unified intrinsic parameter space by setting the virtual camera focal length, use the LoRA fine-tuning strategy to quickly adapt the pre-trained model, and output a high-precision dense depth map. S2: The depth prediction network is used to obtain the depth information of feature points within the sliding window. The PnP algorithm is used to solve the initial pose of the camera frame. The gyroscope zero bias, gravity vector and velocity are initialized by combining the IMU pre-integration data to obtain the depth prediction value. S3: Use the predicted depth value as the initial value for optimizing the inverse depth of the feature point; at the same time, construct the projection depth residual; The projection depth residual, IMU pre-integration residual, and visual reprojection residual are jointly incorporated into a sliding window optimizer for joint nonlinear optimization to estimate the system state and obtain the estimated optimization result. S4: Utilize the optimized precise pose and dense depth map of each frame to construct a point cloud map based on an octree data structure in real time, and construct and update a truncated symbolic distance function map in real time using the VDB data structure and fast voxel traversal algorithm.

2. The depth prediction and visual-inertial coupled localization mapping method according to claim 1, characterized in that, The ViT encoder uses ViT-Base to segment the input image into fixed-size image blocks and captures the global context through a multi-head self-attention mechanism. The ConvGRU decoder upsamples the feature map output by the encoder multiple times, gradually recovers the high-resolution depth map by iteratively fusing multi-scale features, and finally outputs the depth probability distribution through the Softmax function and calculates the expected depth value.

3. The depth prediction and visual-inertial coupled localization mapping method according to claim 1, characterized in that, The virtual camera focal length is set as follows: a virtual focal length is set. For any original camera focal length Calculate the scaling factor , During training, the ground truth depth map is multiplied by the scaling factor. During inference, the depth map output by the network is divided by the scaling factor. This is to eliminate scale ambiguity caused by different camera intrinsic parameters.

4. The depth prediction and visual-inertial coupled localization mapping method according to claim 1, characterized in that, The LoRA fine-tuning strategy is implemented as follows: freeze all the original weights of the pre-trained model, add the first low-rank update matrix and the second low-rank update matrix only in the bypass of the query, key, value, and output linear layers of the Transformer encoder and the output linear layer of the ConvGRU, and train on a small dataset of the target scene.

5. The depth prediction and visual-inertial coupled localization mapping method according to claim 1, characterized in that, The LoRA-tuned training loss function Smoothing L1 loss Virtual normal loss and knowledge distillation loss The weighted summation is calculated using the following formula: in, As the first coefficient, As the second coefficient, It is the third coefficient.

6. The depth prediction and visual-inertial coupled localization mapping method according to claim 1, characterized in that, The projection depth residual is constructed as follows: for the first... The first observation of the frame and the depth prediction value are Feature points When it was the first When a frame is observed, it is projected onto the first frame through a pose transformation. In the frame camera coordinate system, the projection depth is obtained. Then the projection depth residual ,in, For this feature point at the th The direct depth prediction value of the frame.

7. The depth prediction and visual-inertial coupled localization mapping method according to claim 1, characterized in that, The projection depth residual is set to a covariance matrix, the value of which is a preset multiple of the visual reprojection error covariance matrix. A robust kernel function is introduced, and when the absolute value of the projection depth residual is greater than a set threshold, it is treated as an outlier to suppress the interference of depth prediction outliers on optimization.

8. The depth prediction and visual-inertial coupled localization mapping method according to claim 1, characterized in that, The method for constructing and updating the truncated symbolic distance function map is as follows: Define a truncated distance; for each ray originating from the camera center, use a fast voxel traversal algorithm to obtain all voxels along the path; calculate the distance from the voxel center to the camera. and pixel depth value Calculate the symbolic distance function value : If and only if If the value is less than the cutoff distance, the cutoff sign distance function value and weight of the voxel are updated by weighted average; otherwise, no processing is performed.

9. A depth prediction and visual-inertial coupled localization and mapping system, characterized in that, Includes the following modules: Scene-adaptive monocular depth prediction module: Acquires RGB images in the environment, constructs and trains a depth prediction network based on ViT encoder and ConvGRU decoder based on the RGB images, normalizes all training and test images to a unified intrinsic parameter space by setting the virtual camera focal length, and uses LoRA fine-tuning strategy to quickly adapt the pre-trained model and output a high-precision dense depth map. Fast initialization module: The depth prediction network is used to obtain the depth information of feature points within the sliding window, the PnP algorithm is used to solve the initial pose of the camera frame, and the gyroscope zero bias, gravity vector and velocity are initialized by combining IMU pre-integration data to obtain the depth prediction value. Tightly coupled nonlinear optimization module: uses the predicted depth value as the initial optimization value for the inverse depth of the feature point; Simultaneously, a projection depth residual is constructed, which measures the difference between the projection depth of a feature point when it is projected from a reference frame to other observation frames and the depth prediction value output by the depth prediction network in the corresponding frame. The projection depth residual, IMU pre-integration residual, and visual reprojection residual are jointly incorporated into a sliding window optimizer for joint nonlinear optimization to estimate the system state. Dense Mapping Module: Utilizing optimized precise poses and dense depth maps for each frame, it constructs point cloud maps in real time based on octree data structures, and truncated symbolic distance function maps in real time using VDB data structures and fast voxel traversal algorithms.

10. An electronic device comprising a processor, a communication interface, a memory, and a communication bus, characterized in that, When the processor executes a computer program, it implements the steps of the depth prediction and visual-inertial coupled localization mapping method as described in any one of claims 1 to 8.

Images