Fast semantic segmentation and resource awareness fused instant localization and mapping

Abstract

The invention discloses instant localization and mapping integrating rapid semantic segmentation and resource awareness, and belongs to the technical field of visual synchronous localization and mapping (SLAM). The method comprises the following steps: firstly, inputting an RGB-D image sequence, and preprocessing an original image by using a lightweight image segmentation model FastSAM to extract low-complexity and high-quality image features; secondly, the extracted image features are input into a sparse image optimization module, and the system resource occupation is effectively reduced by dynamically adjusting the feature extraction resolution and the depth estimation frequency; then, an attention mechanism and a sparse feature guiding strategy are introduced into a local reconstruction module, so that the space consistency and the time coherence of sparse point clouds are improved, and the precision and the robustness of front-end tracking are improved; then, real-time reconstruction and distribution are carried out on calculation tasks through a task awareness scheduler, and self-adaptive reasoning and energy consumption control under different hardware platforms are achieved; finally, the optimized feature information is input into a DROID-SLAM rear-end module, and high-precision pose estimation and map reconstruction are completed. According to the method, on the premise of ensuring the positioning precision, the computing resource demand of the model is remarkably reduced, and the adaptability and practicability of the SLAM system on a resource-constrained platform are improved.

Description

Technical Field

[0001] The present invention relates to a visual simultaneous localization and mapping (SLAM) optimization technology, in particular to an instant localization and mapping technology that integrates fast semantic segmentation and resource perception, and belongs to the field of computer vision and robot perception technology. Background Art

[0002] Traditional deep learning-assisted SLAM systems, such as DROID-SLAM, often face the problems of high computing resource consumption and insufficient real-time performance while achieving high-precision pose estimation and dense mapping, making them difficult to deploy on resource-constrained edge devices or mobile platforms.

[0003] In recent years, the development of lightweight deep networks, attention mechanisms, and task-aware scheduling technologies has provided new insights into resource optimization for SLAM systems. SPAQ, a SLAM architecture designed for resource-aware optimization, can compress deep learning computational loads through dynamic scheduling and resolution adjustment, improving system adaptability and operational efficiency.

[0004] However, integrating the SPAQ concept into the existing DROID-SLAM framework still faces challenges such as complex structure, high module coupling, and loose scheduling granularity. Therefore, a systematic approach is urgently needed to modularize and optimize DROID-SLAM at multiple levels. Summary of the Invention

[0005] The purpose of the present invention is to provide a real-time positioning and mapping method that integrates fast semantic segmentation and resource perception to solve the problems of high resource usage, large inference delay and reduced accuracy of traditional SLAM systems.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] Step 1: Input RGB-D image sequence, use lightweight image segmentation model FastSAM to preprocess the original image, extract image features, and obtain low-complexity image features;

[0008] Step 2: Input the image features into the sparse graph optimization module;

[0009] Step 3: Dynamically adjust the feature extraction resolution and depth estimation frequency based on image texture complexity and inter-frame similarity to reduce system resource consumption;

[0010] Step 4: Introduce the attention mechanism and sparse feature guidance strategy in the local reconstruction module to enhance the spatial consistency and temporal coherence of the sparse point cloud;

[0011] Step 5: Dynamically allocate and schedule system tasks through the task-aware scheduler to achieve adaptive reasoning and resource optimization;

[0012] Step 6: Input the optimized feature information into the DROID-SLAM backend module;

[0013] Step 7: Complete high-precision pose estimation and dense map reconstruction.

[0014] Furthermore, the step one is specifically as follows: in order to reduce the computational complexity of the front-end feature extraction module, the present invention adopts FastSam and the cropped MobileNetV2 as the front-end encoder network, and its backbone structure adopts depthwise separable convolution (DepthwiseSeparableConvolution) and pointwise convolution (PointwiseConvolution) to reduce the amount of computation and maintain expressiveness.

[0015] Furthermore, the step 2 is specifically as follows: The encoding process can be expressed as:

[0016] F i =σ(BN(Conv 1×1 (DWConv 3×3 (I i ))))

[0017] Among them, DWConv 3×3 Represents a 3×3 depth-wise separable convolution. This module outputs a multi-scale feature map {F1,F2,...F n}, for subsequent module processing.

[0018] Furthermore, the step three is specifically as follows: the hierarchical feature maps are sequentially input into the multi-scale image optimization module, the key frames are processed with the original resolution, and the non-key frames are compressed to the original resolution. Figure 1 The size of the image is / 4 to save computing resources. Whether the image is a key frame is determined by the image entropy and the SSIM similarity between frames:

[0019]

[0020] Furthermore, the aforementioned step 4 is specifically as follows: multi-scale features are input into the ResGAM attention enhancement module for processing to enhance the feature response of important regions. The ResGAM module consists of a channel attention branch and a spatial attention branch. The former uses the SE attention mechanism to dynamically adjust the channel, while the latter uses a 7×7 convolution combined with a Sigmoid function to generate a spatial mask. The final fused and enhanced feature map is:

[0021] F′=F+Mc (F)·F+M s (F)·F

[0022] Furthermore, the step 5 is specifically as follows: constructing a task-aware scheduler module to obtain real-time indicators such as GPU occupancy, system latency, and mapping error, and deciding the current frame processing strategy based on the scheduling strategy network, including but not limited to: reducing resolution, skipping frames, pausing enhancement operations, etc. The final objective function is expressed as:

[0023]

[0024] Furthermore, the scheduler strategy network is specifically: using a lightweight multi-layer perceptron structure, inputting the system state table triple Output the decision action label θ to control resource allocation and reasoning execution path.

[0025] Furthermore, the step 6 is specifically as follows: inputting the feature map after attention enhancement into the GRU module of the DROID-SLAM backend to optimize the pose continuity and build depth consistency. The backend state update formula at each time t is:

[0026] h t =GRU(h t-1 ,[F′ t ,D t ,P t ]) The beneficial effects of the present invention are as follows:

[0027] The present invention introduces the lightweight image segmentation model FastSAM into the DROID-SLAM architecture and combines it with the dynamic resolution control and frequency scheduling strategy proposed by the SPAQ architecture to reduce the computing resource usage in the image processing and depth estimation processes. It focuses on optimizing the image feature extraction network, multi-scale sparse graph optimization strategy, attention enhancement module and task perception scheduling mechanism, effectively reducing the system's computing resource consumption, improving the accuracy of front-end tracking and back-end mapping, achieving efficient deployment and stable operation on resource-constrained devices, and significantly enhancing the real-time, robustness and adaptability of the SLAM system in complex environments. It has broad application prospects and practical engineering value. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1It is a schematic diagram of the process of the present invention;

[0030] Figure 2 Schematic diagram of the FastSam model structure of the present invention;

[0031] Figure 3 This is a schematic diagram of the SPAQ core model structure of the present invention; DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0033] See also Figure 1 As shown, the instant positioning and mapping method of the present invention that integrates fast semantic segmentation and resource perception includes the following steps:

[0034] S101: Input RGB-D image sequence, use lightweight image segmentation model FastSAM to preprocess the original image, extract image features, and obtain low-complexity image features;

[0035] S102: Inputting the image features into a sparse graph optimization module;

[0036] S103: Dynamically adjusting the resolution of feature extraction and the frequency of depth estimation according to image texture complexity and inter-frame similarity, thereby reducing system resource consumption;

[0037] S104: Introducing the attention mechanism and sparse feature guidance strategy into the local reconstruction module to enhance the spatial consistency and temporal coherence of sparse point clouds;

[0038] S105: Dynamically allocate and schedule system tasks through a task-aware scheduler to achieve adaptive reasoning and resource optimization.

[0039] S106: Inputting the optimized feature information into the DROID-SLAM backend module;

[0040] S107: Complete high-precision pose estimation and dense map reconstruction.

[0041] As an optimization solution for the above embodiment, step 1 specifically involves inputting an RGB-D image sequence, preprocessing the original image using the lightweight image segmentation model FastSAM, extracting features from the image, and generating low-complexity image features. This network streamlines the model structure, reduces computing resource consumption, and ensures efficient image processing. The lightweight network design aims to accelerate computation and improve system response speed by reducing the number of network layers and parameters.

[0042] As an optimization solution for the above embodiment, step 2 specifically involves inputting the image features extracted by the lightweight neural network into the sparse graph optimization module. This module further processes and optimizes the image features by minimizing the optimization objective function, thereby obtaining a more accurate image description. The sparse graph optimization module adjusts based on the sparse graph structure and image features to improve computational efficiency and reduce unnecessary computational overhead.

[0043] As an optimization solution of the above embodiment, the lightweight method is as follows: the above hierarchical feature maps are sequentially input into the multi-scale image optimization module, the key frames are processed with the original resolution, and the non-key frames are compressed to the original resolution. Figure 1 The size of the image is / 4 to save computing resources. Whether the image is a key frame is determined by the image entropy and the SSIM similarity between frames:

[0044]

[0045] As an optimization solution for the above embodiment, step 4 is specifically as follows: multi-scale features are input into the ResGAM attention enhancement module for processing to improve the feature response of important regions. The ResGAM module consists of a channel attention branch and a spatial attention branch. The former uses the SE attention mechanism to dynamically adjust the channel, and the latter uses a 7×7 convolution with a Sigmoid function to generate a spatial mask. The final fused and enhanced feature map is:

[0046] F′=F+M c (F)·F+M s (F)·F

[0047] As an optimization solution for the above embodiment, step five is specifically as follows: constructing a task-aware scheduler module to obtain real-time indicators such as GPU occupancy, system latency, and mapping error, and deciding the current frame processing strategy based on the scheduling strategy network, including but not limited to: reducing resolution, skipping frames, pausing enhancement operations, etc. The final objective function is expressed as:

[0048]

[0049] As an optimization solution of the above embodiment, the scheduler strategy network is specifically: using a lightweight multi-layer perceptron structure, inputting the system state table triple Output the decision action label θ to control resource allocation and reasoning execution path.

[0050] Furthermore, the step 6 is specifically as follows: inputting the feature map after attention enhancement into the GRU module of the DROID-SLAM backend to optimize the pose continuity and build depth consistency. The backend state update formula at each time t is:

[0051] h t =GRU(h t-1 , [F′ t , D t , P t ])

[0052] Furthermore, the step seven is specifically as follows: completing high-precision pose estimation and dense map reconstruction based on the point cloud.

[0053] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without the need for creative work are still within the scope of protection of the present invention.

Claims

1. A real-time localization and mapping system that integrates fast semantic segmentation and resource awareness, characterized by: The steps include: Step 1: Input RGB-D image sequence and extract features from the original image using the lightweight image segmentation model FastSAM to obtain low-complexity image features; Step 2: Input the image features into the sparse graph optimization module; Step 3: Dynamically adjust the feature extraction resolution and depth estimation frequency based on image texture complexity and inter-frame similarity to reduce system resource consumption; Step 4: Introduce the attention mechanism and sparse feature guidance strategy in the local reconstruction module to enhance the spatial consistency and temporal coherence of the sparse point cloud; Step 5: Dynamically allocate and schedule system tasks through the task-aware scheduler to achieve adaptive reasoning and resource optimization; Step 6: Input the optimized feature information into the DROID-SLAM backend module; Step 7: Complete high-precision pose estimation and dense map reconstruction.

2. The method according to claim 1, wherein The lightweight feature extraction network is a lightweight image segmentation model based on the Transformer architecture. Its feature extraction process is as follows: F t =ε(I t ;θ),F′ t =F t ·σ(W1·AvgPool(F t )+W2·MaxPol(F t )) Among them, ε represents the encoder, I t is the image input of time frame t, θ is the network weight, σ is the activation function, W1 and W2 are learnable parameters.

3. The method according to claim 1, wherein The sparse graph optimization module constructs the optimization objective function based on the graph structure as follows: Where ε is the edge set in the graph, ω ij is the edge weight coefficient based on the image edge gradient and similarity, and ρ(·) is the robust loss function used to suppress the influence of abnormal point pairs.

4. The method according to claim 1, wherein The task-aware scheduler module is based on the system resource status: The channel attention mechanism extracts features through global average pooling and maximum pooling, and generates a channel weight vector M c , to highlight important channels; The spatial attention mechanism generates a spatial weight map M based on the saliency distribution of the feature map. s , in order to strengthen the spatial characteristics of key areas, its expression is: M s =σ(Conv 7×7 ([F avg ;F max ])) F″ t =M s ·F′ t Among them, σ represents the Sigmoid activation function, [F agv ; F max ] represents the concatenation result of average pooling and maximum pooling in the channel dimension.

5. The method according to claim 1, wherein The task-aware scheduler module is based on the system resource status R t With the target frame rate f target , by optimizing P t The path achieves resource control, and its objective function is: Among them, f(P t ) represents the inference frame rate under this path, Power(P t ) represents the power consumption evaluation, λ is the control coefficient, and the scheduler can estimate the current delay tolerance of the system online and automatically adjust the inference branch and accuracy level.

6. The method according to claim 1, wherein The DROID-SLAM backend module uses a joint optimization strategy to accurately estimate the pose, and its objective function is: Among them, T i , T j is the pose matrix of frame i, j, is the estimated relative pose, is the reconstructed image based on the back-projection of the depth map, and α is the image reconstruction error weight, which is used to improve dense consistency.

7. The method according to claim 6, wherein The map reconstruction module adopts a multi-scale depth fusion mechanism to improve the stability of depth estimation. Its fusion function is expressed as: in, represents the (l)th level result of depth estimation, f merge It is a cross-layer fusion function, which can be a weighted average or residual fusion method to further enhance the consistency and accuracy of multi-layer depth prediction.

Images