Three-dimensional mapping method, apparatus, device, medium, and program product

Abstract

The application provides a three-dimensional mapping method, device, equipment, medium and program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring multiple image frames collected by a patrol robot, performing feature extraction and image segment tokenization processing on each image frame to obtain a corresponding image token sequence; acquiring a geometric perception map corresponding to a single image frame based on the single image frame and the image token sequence, performing merging on the image token sequence based on the geometric perception map, and generating a corresponding merging index, wherein the geometric perception map represents the geometric importance distribution corresponding to each image frame; taking multiple merged token sequences corresponding to multiple image frames as input, performing global self-attention calculation processing, and obtaining global features; performing restoration processing on the image token sequence based on the merging index and the global features, and performing three-dimensional mapping based on the token sequence after the restoration processing. The method provided by the application can improve the real-time mapping performance on the terminal side.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a three-dimensional mapping method, apparatus, device, medium, and program product. Background Technology

[0002] In industrial inspection scenarios (such as data center server rooms, substations, and automated warehouses), inspection robots need to perceive the environment and build 3D maps in real time during long-distance movement for navigation, obstacle avoidance, and asset inventory. Related technologies involve multi-frame inspection 3D mapping, with the network repeatedly calculating token merging indexes layer by layer, resulting in severe computational redundancy and low real-time mapping performance on the edge. Summary of the Invention

[0003] This application provides a three-dimensional mapping method and apparatus that can solve the problems of repeated calculation of token merging indexes layer by layer in the network, resulting in serious computational redundancy and low real-time mapping performance on the terminal side.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a three-dimensional mapping method, the method comprising: The inspection robot collects multiple frames of images, and performs feature extraction and image fragment tokenization on each frame in sequence to obtain the image token sequence corresponding to each frame. Based on a single frame image and the image token sequence, a geometric perception map corresponding to the single frame image is obtained. Based on the geometric perception map, token merging is performed on the image token sequence, and a corresponding merging index is generated. The geometric perception map represents the geometric importance distribution corresponding to each frame image. The multiple merged token sequences corresponding to the multi-frame images are used as input to perform global self-attention calculation to obtain global features. The image token sequence is restored based on the merged index and the global features, and a 3D map is constructed based on the restored token sequence.

[0005] Secondly, embodiments of this application provide a three-dimensional mapping apparatus, the apparatus comprising: The acquisition module is used to acquire multiple frames of images collected by the inspection robot, and to perform feature extraction and image fragment tokenization processing on each frame of image in sequence to obtain the image token sequence corresponding to each frame of image. The merging module is used to obtain the geometric perception map corresponding to the single frame image based on the single frame image and the image token sequence, perform token merging on the image token sequence based on the geometric perception map, and generate a corresponding merging index. The geometric perception map represents the geometric importance distribution corresponding to each frame image. The processing module is used to take the multiple merged token sequences corresponding to the multi-frame images as input, perform global self-attention calculation processing, and obtain global features; The restoration module is used to perform restoration processing on the image token sequence based on the merged index and the global features, and to perform 3D mapping based on the restored token sequence.

[0006] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and when the program or instructions are executed by the processor, they implement the steps of the three-dimensional mapping method as described in the first aspect.

[0007] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the three-dimensional mapping method as described in the first aspect.

[0008] In this embodiment, multiple frames of images collected by the inspection robot are acquired. Feature extraction and image segment tokenization are performed sequentially on each frame to obtain an image token sequence corresponding to each frame. A geometric perception map (GPA) corresponding to each single-frame image is obtained based on the single-frame image and the image token sequence. Token merging is performed on the image token sequence based on the GPA, and a corresponding merging index is generated. The GPA represents the geometric importance distribution corresponding to each frame image. Multiple merged token sequences corresponding to the multiple frames are used as input for global self-attention calculation to obtain global features. The image token sequence is then restored based on the merging index and the global features, and a 3D map is constructed based on the restored token sequence. Thus, by guiding the hierarchical simplification and merging of single-frame tokens through the GPA, combined with the fusion calculation of global spatiotemporal features from multiple frames, the computational power consumption for 3D mapping on the inspection robot's edge is reduced, while the core geometric structure of the scene is fully preserved. This effectively improves the accuracy of 3D reconstruction of large-scale scenes across multiple frames, the topological coherence between frames, and the real-time mapping performance on the edge. Attached Figure Description

[0009] Figure 1 One of the flowcharts of the three-dimensional mapping method provided in the embodiments of this application; Figure 2This is a second schematic flowchart of the three-dimensional mapping method provided in the embodiments of this application; Figure 3 A diagram of the three-dimensional reconstruction network architecture provided in the embodiments of this application; Figure 4 A flowchart for obtaining a geometry-sensing map (GA map) provided for embodiments of this application; Figure 5 This is a schematic diagram of the structure of the three-dimensional mapping device provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] For ease of understanding, the following describes some aspects of the embodiments of this application: In industrial inspection scenarios (such as data center computer rooms, substations, and automated warehouses), inspection robots need to perceive the environment in real time and build a 3D map during long-distance movement in order to navigate, avoid obstacles, and conduct asset inventory.

[0012] Traditional Simultaneous Localization and Mapping (SLAM): Traditional SLAM based on feature point method or direct method is prone to losing tracking when dealing with repetitive textures (such as server racks in a computer room) or scenes with changing lighting, and it is difficult to generate dense 3D reconstruction results.

[0013] Light Detection and Ranging (LiDAR) SLAM: Currently, LiDAR is widely used in the industry for mapping. It offers high ranging accuracy and is unaffected by lighting conditions, but the resulting point cloud maps lack rich visual semantics and texture information, making it difficult to directly meet the needs of refined tasks such as equipment status identification and meter reading in industrial inspections. Furthermore, high-beam LiDAR hardware is extremely expensive.

[0014] Multi-sensor fusion / integrated navigation mapping: This approach combines real-time kinematic (RTK), inertial measurement unit (IMU), wheeled odometer, lidar, and vision sensors. While robust, this solution suffers from a bulky system architecture, complex multi-sensor spatiotemporal calibration, and high computational resource consumption, resulting in high overall and long-term maintenance costs. This makes large-scale deployment on lightweight, low-cost end-side inspection devices (such as quadrupedal robotic dogs) difficult.

[0015] End-to-end 3D reconstruction based on Transformer networks: In recent years, models such as the Visual Geometry Global Transformer (VGGT) have achieved high-quality pose estimation and dense reconstruction by tokenizing image sequences and utilizing a global attention mechanism. However, in long-sequence scenarios of industrial inspection (such as a single inspection covering thousands of images), the computational complexity and memory usage of Transformers increase quadratically with the sequence length, causing the robot's edge computing unit to be unable to process the data in real time, often requiring uploading to the cloud and resulting in high latency.

[0016] In related technologies, a 3D robot inspection system is disclosed. This system constructs a 3D path model in virtual space and generates an inspection plan, converting the virtual 3D path into a physical motion path around the actual asset. It then controls the robot to collect sensor data in the target area and generate inspection records. Its core lies in the planning and execution control of inspection tasks based on the 3D model, emphasizing the robot's autonomous movement and task management in space. However, the above solutions focus on system-level path planning and task execution control, guiding robot movement and data acquisition through a pre-built 3D path model. They do not address the model structure optimization problem for end-to-end 3D reconstruction of large-scale multi-view images during the inspection process. Especially in long-distance continuous inspection scenarios, the robot typically needs to process thousands of frames of image data. The 3D mapping implementation in this application also targets industrial inspection robot applications, but its technical focus is on solving the problem of "long-sequence multi-view" issues during inspection. Figure 3 The real-time computational challenge of "3D mapping" is addressed by introducing a geometric perception token merging mechanism and a cached index reuse strategy. This reduces the computational load and memory usage of the model while ensuring the accuracy of pose estimation and 3D reconstruction, enabling 3D mapping to run in real time on the robot's end-device. This reduces reliance on cloud computing power and lowers system latency.

[0017] In related technologies, a SLAM-based mapping and autonomous navigation method for inspection robots has been disclosed, which achieves robot localization and map construction in the underground environment through visual perception, odometry calibration, and graph optimization algorithms. Its core lies in establishing pose constraints through feature matching and graph optimization to obtain a globally consistent map, thereby supporting the execution of inspection tasks. However, the above scheme belongs to the traditional SLAM technology route, relying on feature extraction and backend graph optimization for mapping, resulting in low processing efficiency, especially when long sequence image inputs are easily limited by computational complexity and storage bottlenecks. Traditional SLAM usually solves the mapping efficiency problem by optimizing the graph structure, but does not propose a structural-level solution to the bottleneck of the global self-attention computational complexity in deep neural networks increasing quadratically with sequence length. In contrast, the three-dimensional mapping method of this application can adopt the VGGT architecture, based on the Transformer model for end-to-end multi-view mapping. Figure 3 3D reconstruction can significantly improve processing speed while maintaining high accuracy. By introducing geometric-aware merging and cross-layer caching mechanisms, computational redundancy is effectively reduced, solving the efficiency problem of traditional SLAM when processing large-scale image sequences, and achieving real-time inference capabilities with high accuracy and low latency.

[0018] In other related technologies, a token pruning module is introduced into the Swing Transformer architecture to perform actions such as removing, packing, or merging tokens, thereby reducing computational overhead and improving inference efficiency by reducing the number of tokens. However, the above solutions are mainly aimed at general visual recognition / classification tasks and are not designed for the special geometric constraints of 3D reconstruction in industrial inspection. Their pruning strategies are usually based on semantics or activation values, without considering the geometric structure in 3D space (such as cabinet edges and equipment textures). In 3D mapping tasks, blind pruning can lead to the loss of key geometric features, causing map construction "faults" or decreased accuracy. There are also methods that select specific blocks in multiple Transformer blocks of the visual Transformer model to perform token merging operations, merging them by calculating the similarity between adjacent tokens to reduce computation and maintain recognition accuracy. However, these methods are not specifically designed for the Transformer model, and their token merging is mainly based on the feature similarity between tokens, without combining the geometric constraint information in the 3D reconstruction task for importance evaluation, and without designing cross-layer index caching and reuse mechanisms for multi-view long sequence input scenarios. The merging strategy of the previous method belongs to single-layer local optimization, while the three-dimensional mapping method of the present application can not only introduce geometric perception to guide the merging decision, but also use cross-layer token similarity stability to realize index cache reuse, thereby achieving system-level acceleration from the overall architecture level, and the technical solution is more systematic.

[0019] In other related technologies, dynamic sparse masks are generated to perform sparsification computations during VGGT model inference, and the sparsity rate is dynamically adjusted according to image complexity to reduce computational load and GPU memory usage while maintaining accuracy. However, using dynamic sparse masks to control the computation path is a sparse computation optimization strategy and does not involve token merging rules based on geometric importance evaluation, nor does it propose a cross-layer merging index cache reuse mechanism. Its core lies in sparse matrix computation, while the 3D mapping method in this application reduces the global attention sequence length through structural token compression and combines a caching mechanism to reduce repeated matching overhead, belonging to a different model acceleration path.

[0020] The related technologies have the following drawbacks: 1. Traditional visual SLAM is inefficient and produces sparse point clouds: Traditional SLAM relies on feature extraction and backend graph optimization for mapping, resulting in slow processing speed. When faced with long sequence image inputs, it is easily limited by computational complexity and storage bottlenecks, leading to memory overflow or latency. The generated sparse or semi-dense maps cannot provide sufficient geometric details.

[0021] 2. LiDAR / integrated navigation mapping is costly, semantically fragmented, and bloated: LiDAR SLAM lacks visual texture, resulting in the separation of mapping and visual inspection tasks; integrated navigation requires handling cumbersome sensor calibration and fusion, which places extremely high demands on computing power and hardware, making it unsuitable for lightweight industrial inspection robots with limited computing power.

[0022] 3. In long sequence scenarios, the inference complexity and memory overhead increase twice: Taking the scheme of "concatenating all frame tokens and then performing global self-attention calculation" as an example, as the number of input frames increases, Global Attention dominates the overall inference latency, and the memory increases rapidly and is prone to overflow, making it difficult to support a single forward reconstruction at the level of 1000 frames.

[0023] 4. General token merging ignores 3D geometry binding: In VGGT-like models, patch tokens have a clear correspondence with local geometry. Existing random or fixed-step merging methods tend to merge high-information tokens such as edges and textures with low-information tokens, violating geometric constraints and leading to a decrease in pose estimation and point cloud reconstruction quality.

[0024] The technical problem to be solved is that traditional SLAM methods often cannot run efficiently when processing long sequence image inputs due to computational complexity and memory bottlenecks, resulting in sparse point clouds and difficulty in meeting the needs of long-term inspection tasks.

[0025] To address the shortcomings of traditional SLAM in large-scale data processing during robotic inspection, the following technical issues need to be resolved: 1. Without significantly reducing the accuracy of camera pose estimation and point cloud / depth density prediction, the computational load and memory usage of the model can be reduced. This enables the model to stably handle long input sequences of up to 1000 frames, while simultaneously significantly reducing overall inference latency.

[0026] 2. During the token compression process, explicit protection is provided for high-information tokens such as edges and textures that are more critical to geometric reasoning, thereby reducing the loss of geometric details and improving the stability and accuracy of pose estimation and 3D reconstruction results.

[0027] The following description, in conjunction with the accompanying drawings, further illustrates the three-dimensional mapping method, apparatus, device, and storage medium proposed in the embodiments of this application.

[0028] Please see Figure 1 , Figure 1 As shown in the figure, one of the flowcharts of a three-dimensional mapping method provided in this application embodiment includes: Step 110: Acquire multiple frames of images collected by the inspection robot, and perform feature extraction and image fragment tokenization processing on each frame of image in sequence to obtain the image token sequence corresponding to each frame of image. In this step, the multi-frame images collected by the inspection robot are continuous images reflecting the work scene, continuously collected by the robot as it moves within the work area. These multi-frame images can be collected at fixed time intervals or at preset distance intervals, and can completely cover the scene information of the inspection area, providing basic image data for subsequent 3D mapping. Feature extraction and image fragment tokenization are performed on each frame sequentially to obtain the image token sequence corresponding to each frame. Feature extraction refers to extracting key information that can characterize the content of an image from a single frame. This feature extraction process can use a convolutional neural network model or a visual Transformer model. These models extract features such as texture, edge, and color from the image to obtain the image's feature information. Image segment tokenization involves segmenting the image after feature extraction and converting each image segment into a corresponding token. The image segmentation can use a fixed-size sliding window segmentation method or an adaptive segmentation algorithm that dynamically adjusts the segmentation size according to the complexity of the image content. The token corresponding to each image segment can represent the feature information of that segment. The image token sequence is a sequence formed by arranging the tokens corresponding to each segmented image segment in a preset order after image segment tokenization. Each frame of the image corresponds to a unique image token sequence, which can completely represent the segment feature distribution of that frame of the image.

[0029] For example, an industrial inspection robot continuously acquires multi-view image sequences during its movement. These image sequences originate from inspection video streams or continuously captured images, comprising N frames (N≥2). The images have uniform resolution and are arranged in chronological order to form a long input sequence. In practical industrial inspection applications, a single inspection process may cover thousands of image frames. The system is deployed on the edge computing unit of the inspection robot, continuously processing the long image sequence under limited computing power and memory resources, providing input data for subsequent camera pose estimation and 3D scene reconstruction.

[0030] Furthermore, features can be extracted from each frame of the image using a self-supervised visual backbone network (DINOV2) through the computing unit, and corresponding image token sequences (patch tokens) can be generated. At the same time, several special tokens (cameratoken, register token) are introduced to carry and aggregate camera and global geometric information, for example: The input can be a sequence of multi-view images containing N frames: ; Features for each frame can be extracted using a self-supervised visual backbone network (DINOV2), denoted as... Convert to patchtoken: , , n refers to the number of patch tokens converted from each frame of image, and d is the feature dimension of the token.

[0031] To unify the inference of multiple 3D attributes, several special tokens (one camera token and four register tokens) are added to the patch token of each frame image to aggregate various 3D attributes (camera parameters, scene geometry). Specifically, a special set of tokens can be set for the first reference frame. Other frames share a different set of special tokens. The final token registration process can be represented as follows: , The final result It can be fed into the VGGT backbone for attention calculation.

[0032] Step 120: Obtain the geometric perception map corresponding to the single frame image based on the single frame image and the image token sequence; perform token merging on the image token sequence based on the geometric perception map and generate a corresponding merging index; the geometric perception map represents the geometric importance distribution corresponding to each frame image. In this step, the aforementioned geometric perception map is a feature map representing the geometric importance distribution of each frame of the image. It reflects the geometric criticality of each region in a single frame of the image. Its acquisition process can combine the geometric features of a single frame of the image with the feature information of the image token sequence. Specifically, by extracting geometric information such as edges and contours from a single frame of the image and combining it with the feature distribution of each token in the image token sequence, the geometric importance score of each region in each frame of the image is comprehensively determined, thus forming the geometric perception map. This geometric perception map can be used to distinguish between geometrically critical regions that need to be retained and non-critical regions that can be simplified. Token merging, based on the geometric importance distribution represented by the geometric perception map, classifies and merges the tokens in the image token sequence. Tokens with higher geometric importance can be retained, while tokens with lower geometric importance can be merged to reduce the number of tokens in the token sequence and decrease subsequent computational overhead. This token merging process can prioritize tokens based on their geometric importance scores, prioritizing the retention of tokens with higher scores and merging tokens with lower scores into a single token according to certain rules. The merge index records the mapping information of the ownership relationship of each token during the token merging process. It can clearly identify the merged token corresponding to each merged token in the image token sequence. It can be generated in the form of a sequence mapping table or recorded in the form of an index array. Subsequently, the position and ownership relationship of each token before merging can be traced back through this merge index, which can provide basic support for the subsequent restoration processing of the image token sequence.

[0033] For example, a geometric importance score can be calculated for each frame and its patch token to generate a geometry-aware map (GA map) that is aligned one-to-one with the token grid.

[0034] Step 130: Take the multiple merged token sequences corresponding to the multi-frame images as input and perform global self-attention calculation to obtain global features; In this step, global self-attention computation refers to performing feature association and fusion computation on the token sequence after merging multiple frames globally. This global self-attention computation can adopt a multi-layer stacked network structure or an improved self-attention network structure. Through the above network structure, the spatiotemporal correlation features between the token sequences of multiple frames can be mined, and the global fusion of features of multiple frames can be achieved. The global feature is the comprehensive feature that can represent the global features of the entire inspection scene by fusing the feature information of the token sequence after merging multiple frames through the above global self-attention computation. The global feature can reflect the overall geometric structure and spatial distribution information of the inspection scene.

[0035] Step 140: Perform restoration processing on the image token sequence based on the merged index and the global features, and perform 3D mapping based on the restored token sequence.

[0036] In this step, the above-mentioned restoration process involves determining the ownership relationship of each token based on the merge index, backfilling the global features to the corresponding token positions, restoring the original length and feature dimension of the image token sequence, and ensuring that the token sequence can completely represent the original image information. The 3D mapping process involves constructing a 3D model of the inspection scene based on the restored token sequence, combined with the image's feature information and global features. This 3D mapping process can use dense point cloud reconstruction algorithms or surface reconstruction algorithms. Through these algorithms, 2D image information can be transformed into a 3D spatial model, achieving accurate 3D restoration of the inspection scene and meeting the needs of industrial inspection for 3D scene modeling.

[0037] For example, after completing the global self-attention calculation, the merged dst token can be copied and restored to the original token layout based on the cached merge index, restoring the token sequence length consistent with the input. The restored token can be input into the decoding module, which outputs camera pose, depth map, point cloud, and other 3D reconstruction-related attributes.

[0038] After acquiring the camera pose trajectory and 3D point cloud data, the inspection robot can use the pose trajectory as a continuous estimate of its own motion state to construct a motion path model in a global coordinate system. It then combines this with environmental structure information formed by the 3D point cloud to generate a real-time 3D map. The pose trajectory is used to achieve precise positioning and path tracking control of the robot within the inspection area; the 3D point cloud is used to characterize the spatial distribution of obstacles and the geometric structure of equipment, thereby supporting dynamic obstacle avoidance decisions and path replanning.

[0039] For example, you can refer to Figure 2 In the 3D mapping method of this application, an acceleration scheme of "Geometry-aware Cached Token Merging" is proposed. The system can be deployed on the edge computing unit of an industrial inspection robot. The input is a sequence of multi-view images continuously acquired during robot movement (such as a video stream of a machine room inspection), and the output can be the camera pose trajectory and dense scene point cloud for 3D mapping. This is used to improve the performance of long multi-view sequences. Figure 3 The inference efficiency and memory utilization of the 3D reconstruction model (VGGT architecture) are due to: (1) In multi-view Figure 3In the dimensional reconstruction task, tokens corresponding to the same or adjacent spatial regions in different frames have high similarity in feature representation. There is a lot of repeated computation in the cross-frame global feature association process, which leads to significant redundancy in global attention computation.

[0040] (2) In the three-dimensional reconstruction scenario, the token has a clear one-to-one correspondence with the local spatial location and geometric structure in the image. Geometric consistency constraints can be introduced during the token merging process to effectively retain key geometric information while reducing the number of tokens, thereby accelerating reasoning while maintaining reconstruction accuracy.

[0041] (3) In the multi-layer network structure, the token similarity relationship and attention pattern between adjacent layers have strong stability, and the token merging decision can be reused across layers. By caching the merging index, repeated calculations can be avoided, further reducing the overall inference overhead.

[0042] (4) Based on the above structural-level optimization, in some optional implementations, engineering optimization techniques such as model fine-tuning and FP8 quantization can be combined to systematically compress the computation and storage process, thereby maximizing the improvement of long sequence multi-view. Figure 3 The reasoning speed and memory efficiency of dimensional reconstruction.

[0043] Based on the above ideas, in some optional implementations, a token merging and restoration process can be introduced before and after each Global Attention module: before entering Global Attention, the tokens are merged to reduce the scale of participating in global self-attention calculation; after leaving Global Attention, the tokens are restored to the original layout according to the cache index, thereby ensuring the consistency of input for subsequent intra-frame feature processing and dense prediction modules.

[0044] In some other alternative implementations, see [reference]. Figure 3 The flowchart of the entire process of geometry-aware token merging multi-frame visual SLAM 3D scene reconstruction can be divided into two main modules: the upper layer is the end-to-end complete mapping link, and the lower layer is a detailed breakdown of the core steps of token merging.

[0045] The upper layer can include the entire multi-frame 3D mapping process: Multiple frames of original images are input during the movement of the inspection robot. The continuous time-series multi-angle images of the same scene are collected and can be used as the original data source for the entire 3D mapping.

[0046] Image tokenization divides a single frame image into segments, generating a patch token for each small image block. All tokens are arranged spatially to form a regular token grid, and the ordered arrangement yields an image token sequence, which corresponds to the image segment tokenization process.

[0047] Geometry-aware Token Merging Core Module GA-merge: Geometric perception token merging, which uses a geometric perception graph to classify, group, and merge tokens, reducing the number of tokens. G-Attn×24: A 24-layer stacked geometric self-attention network that can correspond to multi-layer stacked networks + cross-layer cache reuse and merging indexes, without having to repeatedly calculate token ownership relationships in each layer; unmerge: Token unmerge and restoration, relying on the merge index to restore the original complete token sequence; F-Attn inter-frame global self-attention merges multiple frames and simplifies the token input to a unified input. It integrates inter-frame temporal and spatial global correlation features and outputs global 3D features of the scene. It can correspond to the global self-attention multi-frame fusion operation. For example, a dual-task output header can be used: Camera Head (Camera Pose Branch): Calculates the robot's camera pose and motion trajectory for each frame, enabling SLAM (Simultaneous Localization and Mapping). Depth Head: Scene Depth Branch - Predicts depth information for each region of the image. After multi-frame alignment and fusion, the final result can output a dense 3D point cloud model of the inspection scene, completing the 3D mapping of a large-scale scene.

[0048] In some alternative implementations, the lower layer may include GA-merge token merging step-by-step details: Partition token importance partitioning, based on a geometric perception graph, can assign importance labels to all raw tokens to distinguish their levels, for example: Red Token: Geometric Perception Token, used in high-importance areas such as the edges of scene structures, and does not participate in any merging process. Light orange token: Target anchor token, serving as a retention carrier for feature aggregation. Light gray tokens: Redundant tokens to be merged, representing low-importance areas in a flat scenario; these can be merged and compressed. Grouping tokens is based on the similarity of features between tokens. All tokens to be merged are matched and assigned to their corresponding anchor tokens, and a permanent merging index is generated simultaneously to record the token correspondence. Merging aggregates and combines features of all tokens in the same group to be merged into the corresponding anchor token, significantly reducing the total number of tokens and lowering the computational and memory overhead of subsequent Transformers. G-Attn geometric self-attention computation uses a simplified number of tokens to efficiently compute deep geometric features of a scene. Unmerging tokens restores the previously saved merge index and backfills the global features of the anchor tokens to the positions of all original tokens to be merged; the positions and features of the red key geometric tokens remain unchanged throughout the process, and finally the initial complete token grid and sequence length are restored 1:1.

[0049] Furthermore, existing token merging strategies only consider token similarity, leading to the loss of crucial geometric information such as edges and textures in 3D reconstruction tasks. This application proposes a geometry-aware token partitioning and merging strategy. In the 3D mapping method of this application, through geometry-aware GA map, token partitioning and merging under geometric constraints, and a mechanism for preserving key geometric details, the computational load can be reduced and inference accelerated while effectively preserving key geometric information such as edges and textures, maintaining high-precision 3D reconstruction results.

[0050] In the 3D mapping method of this application, the single-frame token is simplified and merged in a hierarchical manner by guiding the geometric perception map, and combined with the global spatiotemporal feature fusion calculation of multiple frame sequences, which reduces the computing power consumption of the inspection robot's edge 3D mapping, while completely preserving the core geometric structure of the scene, effectively improving the accuracy of multi-frame large-scale scene 3D reconstruction, the topological coherence between frames, and the edge real-time mapping performance.

[0051] Optionally, the geometric perception map is generated by a weighted fusion of an image gradient map and a token feature variance map; The image gradient map is obtained by performing edge gradient calculations based on the single-frame image using an edge detection operator. The token feature variance map is obtained based on the image token sequence through local variance statistics of token features.

[0052] In this embodiment, the aforementioned geometric perception map is generated by weighted fusion of the image gradient map and the token feature variance map. Weighted fusion means assigning corresponding weights to the image gradient map and the token feature variance map respectively, and then fusing the two to obtain the final geometric perception map. The weight setting can be flexibly adjusted according to the actual needs of the inspection scenario, or it can be determined adaptively through model training, so that the fused geometric perception map can more accurately represent the geometric importance distribution of a single frame image.

[0053] The aforementioned image gradient map can be obtained by performing edge gradient calculations on the single-frame image using an edge detection operator. The image gradient map is a feature map that reflects the degree of edge change in each region of a single-frame image. It can capture key geometric information such as edges and contours in the image, providing a basis for judging geometric importance. The aforementioned edge gradient calculation can be understood as calculating the difference in grayscale values ​​between adjacent pixels in a single-frame image, thereby quantifying the clarity and trend of image edges. The edge detection operator can be either the Sobel operator or the Canny operator. Performing these operations on the single-frame image effectively extracts edge features from the image, avoids interference from irrelevant noise on the edge detection results, and ensures the accuracy of the image gradient map.

[0054] The aforementioned token feature variance map can be obtained based on the image token sequence through local variance statistics of token features. This token feature variance map reflects the dispersion of each token feature in the image token sequence, and can help determine the feature complexity of each region of the image, thereby supporting the assessment of geometric importance. The aforementioned local variance statistics of token features can be understood as calculating the variance of the feature values ​​of each token and its neighboring tokens in the image token sequence. The magnitude of the variance represents the consistency of the token features in that region. The larger the variance, the more drastic the feature changes in that region, which may correspond to a geometrically critical region in the image. The smaller the variance, the smoother the feature changes in that region, which can be appropriately simplified. The range of local variance statistics can be set to a fixed number of neighboring tokens, or the statistical range can be adaptively adjusted according to the distribution density of the tokens to adapt to inspection scene images of different complexities, ensuring that the token feature variance map can truly reflect the feature distribution of the image.

[0055] For example, you can refer to Figure 4 The construction of a geometry-aware map (GA map) can include: Input: The original RGB image for each frame and the patch token grid for that frame.

[0056] Output: Geometric importance score map at the same resolution as the token grid. .

[0057] Furthermore, the gradient map can be obtained in the following way: The input original RGB image of the i-th frame is: in , These represent the height and width of the image, respectively.

[0058] Convert each frame to grayscale: ; The horizontal / vertical gradient magnitudes are obtained using the Sobel operator: ; * indicates convolution. Represents the Sobel operators for the horizontal and vertical directions.

[0059] The resulting pixel-level gradient map is obtained through fusion. ; Next, perform pixel-to-token downsampling to obtain the gradient map: in These represent the height and width of the token, respectively.

[0060] Furthermore, the calculation of the variance map includes: The patch token grid corresponding to the i-th frame of the input image is calculated as follows: ; In 3 Within a window of 3, calculate the local variance of adjacent region tokens: ; ; This indicates element-wise multiplication.

[0061] The variance map (VAR map) of the image is obtained by summing the variances of all channels to form a single-channel "variance intensity map". ; Furthermore, geometry-sensing map (GA map) computation can include: gradient map Variance plot All The weighted fusion outputs the final geometric importance score map. : ; Let α and β represent normalization, and β be the weighting coefficients. Extensive experimental evidence demonstrates that the improvement in model reconstruction accuracy is insensitive to the two coefficients α and β, and... For optimal results, the GA map is used to indicate tokens that are more critical to geometric inference (typically corresponding to regions with high gradients / high variance).

[0062] Furthermore, in some alternative implementations, the aforementioned geometric importance index can be replaced by feature entropy, which determines the region complexity by statistically analyzing the local feature entropy values ​​of the image token sequence; or the Laplacian operator can be used to calculate the second derivative of the image to identify regions with drastic texture changes. The aforementioned geometric importance indices can be substituted for each other or used in combination to adapt to different inspection environment lighting variations or scene complexity; that is, different geometric importance indices (such as Canny edges, Laplacian, feature entropy, etc.) can be used. This application does not specifically limit the specific use of these indices.

[0063] In the 3D mapping method of this application, the geometric perception map obtained by weighted fusion of the image gradient map and the token feature variance map generated in the above manner can combine the edge geometric features of the image and the feature distribution features of the token to more comprehensively and accurately characterize the geometric importance distribution of a single frame image. This can provide a more reliable judgment basis for subsequent token merging based on the geometric perception map and ensure the accuracy of subsequent 3D mapping.

[0064] Optionally, the step of performing token merging on the image token sequence based on the geometric perception map and generating a corresponding merge index includes: The tokens in the image token sequence are divided into geometric perception tokens, target anchor tokens, and tokens to be merged. The tokens to be merged are matched to the target anchor tokens based on feature similarity for feature aggregation. The geometric perception token is a preset proportion image token sequence with the highest geometric importance score corresponding to the geometric perception map, and the geometric perception token does not participate in token merging.

[0065] In this implementation, when processing Token sequence merging transactions according to the geometric perception map, all image Tokens can be categorized into geometric perception Tokens, target anchor point Tokens, and Tokens to be merged. Geometric perception Tokens are image Tokens with high geometric importance scores; these Tokens correspond to the core structural locations of the inspection scene and will not participate in the Token merging calculation. The proportion of these important Tokens can be dynamically adapted to narrow, enclosed inspection environments, or a fixed range can be used for open factory inspection scenarios.

[0066] The aforementioned target anchor token can be understood as the retained token responsible for receiving and summarizing feature information in the token merging process, serving as the carrier unit for regional feature fusion. The tokens to be merged correspond to areas with smooth and redundant scene features and low geometric structural criticality, making them suitable for classification, fusion, and simplification.

[0067] Various tokens to be merged are matched with corresponding target anchor tokens based on the similarity of their own characteristics, thereby completing cross-token feature aggregation processing. The similarity between tokens can be determined by using cosine distance or Euclidean distance to accurately determine the matching relationship between tokens. Feature aggregation involves summarizing and integrating the scene features of multiple tokens to be merged into the corresponding target anchor token, reducing the total number of tokens while fully preserving the overall feature information of the region.

[0068] For example, the token partitioning and anchor selection strategy may include: GA token selection: All tokens corresponding to each frame of the GA map can be sorted from highest to lowest score, and the top p% of the scores can be selected as the GA token (e.g., p=10%). The GA token is "frozen" during the merging process and does not participate in the merging, in order to protect critical geometric details such as edges and textures.

[0069] dst token (anchor token) selection: To balance cross-frame consistency and spatial balance, the dst token can be selected according to the following rules: (1) First frame: Retain all tokens in the first frame (including 5 special tokens) as dst (as world coordinates / global anchoring to enhance cross-frame consistency). (2) Other frames: Divide the token grid into 2×2, and select the token with the lowest GA score in each 2×2 grid as dst. Prioritize smooth, low-information regions to maximize merging benefits; The source token (token to be merged) has been determined: All tokens other than the GA token and the dst token are src tokens, which are merged into the dst token as redundant tokens.

[0070] Furthermore, similarity matching and merging methods may include: Matching index calculation (src→dst): For each src token, calculate its cosine similarity with the features of all dst tokens, select the dst with the highest similarity as its anchor point, and obtain the mapping index. .

[0071] Feature aggregation (merging) can sort all source tokens from highest to lowest similarity based on the above similarity, and select the top 75% of source tokens to participate in the merging with the corresponding destination token. For each destination token, the source set assigned to it is collected. And update the dst features as follows: The merged sequence of GAtoken and updated dst token is used to enter Global Attention, thereby significantly reducing the sequence length and quadratic complexity of global self-attention.

[0072] In the 3D mapping method of this application, the above-mentioned hierarchical token processing method can avoid the loss of key geometric features of the scene, while effectively reducing the scale of computational data, making subsequent global attention computation more efficient, and meeting the needs of lightweight real-time 3D mapping for inspection robots.

[0073] Optionally, the global self-attention calculation is an operation performed using a multi-layer stacked network. After performing token merging on the image token sequence based on the geometric perception map and generating the corresponding merge index, the method further includes: The merge index is cached according to a preset layer interval in the multi-layer stacked network, and the merge index is reused in the corresponding network layer.

[0074] In this implementation, since the similarity between tokens in adjacent layers and the attention pattern are observed to be stable, the same set of src→dst matching indices can be shared in multiple adjacent layers, thereby avoiding the repeated calculation of a large-scale similarity matrix in each layer.

[0075] The aforementioned multi-layer stacked network is a deep neural network structure formed by sequentially arranging multiple feature operation layers. The total number of network layers can be reasonably selected according to the size of the inspection scenario, or flexibly adapted according to the computing power conditions of the robot's embedded hardware.

[0076] After completing the token merging process and obtaining the corresponding merge index, the merge index can be temporarily stored according to a pre-defined layer distance within the multi-layer stacked network. Caching can be understood as temporarily storing the ownership correspondence information between tokens in a high-speed access area, facilitating rapid retrieval and use by subsequent network layers. The aforementioned pre-defined layer interval represents the number of network layers between two adjacent recalculations of the merge index. The interval rule can be executed cyclically at a fixed interval or dynamically adjusted adaptively based on the magnitude of feature changes between consecutive frames. In subsequent corresponding network operation layers, the cached merge index can be directly called to participate in feature calculations, eliminating the need to recalculate token matching and merging relationships at each network layer.

[0077] In the three-dimensional mapping method of this application, index reuse can effectively reduce redundant and repetitive calculations, reduce overall computing power consumption, and at the same time ensure the stable and unified fusion of multi-frame spatiotemporal global features, effectively improving the real-time processing efficiency of three-dimensional mapping in inspection scenarios.

[0078] For example, the model has a total of 24 stacked Global Attention layers. The matching index is recalculated every L layers (e.g., L=6), and the cached index is reused in the remaining layers. That is, the index is calculated only in layers {1, 1+L, 1+2L, 1+3L}, a total of 4 times, and the remaining 20 layers reuse the index. In other words, the matching index of src→dst is cached within the preset network layer intervals and reused in subsequent layers to reduce the additional overhead caused by repeated similarity calculations.

[0079] In related technologies, the feature distribution characteristics between model layers are not fully explored, leading to repeated calculations of merging indexes, introducing a large amount of redundant computation, and increasing time. In the implementation of this application, cross-layer token similarity stability is discovered, and an innovative caching reuse strategy for merging indexes is proposed. Through cross-layer token similarity stability and caching reuse of merging indexes, approximately 25% inference speedup can be achieved without sacrificing reconstruction accuracy.

[0080] Optionally, the step of restoring the image token sequence based on the merged index and the global features includes: Assign the global features obtained from the global self-attention operation to each target anchor Token; The target anchor token corresponding to each original token to be merged is determined based on the merge index. The global features carried by the target anchor token are mapped back to the position of the corresponding original token to be merged, and the geometrically perceived token retains its original position and features to restore the original length and feature dimension of the image token sequence.

[0081] In this implementation, after completing the global self-attention operation, the calculated global scene features can be assigned to each target anchor token, so that the retained anchor tokens possess complete and coherent multi-frame scene global spatial information. The belonging anchor point corresponding to each original token to be merged can be determined by relying on the association mapping relationship of the merged index records. The token belonging lookup method can be a sequential traversal matching method or an array-based fast indexing location method.

[0082] Furthermore, the global features carried by the target anchor token can be mapped and restored to the corresponding points of each original token to be merged according to the original positional relationship. Throughout the entire sequence restoration process, the position and original feature parameters of the geometrically aware token remain fixed and do not participate in feature adjustment or positional changes.

[0083] In the three-dimensional mapping method of this application, the original data length and feature dimension specifications of the image token sequence can be restored through the above-mentioned ordered processing, so that the simplified token sequence can be restored to its initial complete form, ensuring the integrity of the scene's geometric structure information, and thus ensuring the accuracy and completeness of the subsequent three-dimensional scene reconstruction.

[0084] In some optional implementations, to support dense predictions such as depth maps / point clouds, the original sequence length can be restored before entering the prediction head. Specifically, for each dst token d, the merged features... Copy back to the original set of tokens it overwrites The position of the GA token remains unchanged. After restoration, the model can continue to perform intra-frame attention to recover local geometric details and ensure that the input dimension of the output head is consistent with the original model.

[0085] Optionally, the aggregation process of matching the tokens to be merged to the target anchor tokens based on feature similarity further includes: Only floating-point operations during the aggregation process are handled with lightweight 8-bit floating-point precision.

[0086] In this implementation, 8-bit floating-point precision lightweight quantization (FP8) is used to reduce computational overhead and memory usage during the model inference stage. Specifically, the mixed-precision computing framework provided by the NVIDIA Transformer Engine is used to accelerate some computation modules in the model through FP8 quantization.

[0087] To avoid a significant impact of quantization on reconstruction accuracy, the embodiments of this application only introduce FP8 quantization into the calculation process within the aggregator module, while the prediction head modules for camera pose, depth, or point clouds maintain high-precision data representation. Through this approach, the computational complexity and resource consumption during inference are effectively reduced while ensuring that the model output accuracy remains largely unaffected.

[0088] In related technologies, FP8 quantization is not performed on the 3D reconstruction feedforward model. The implementation method of this application can explore system FP8 quantization on VGGT; FP8 quantization is performed on the model backbone while preserving the high accuracy of the prediction head. While maintaining the accuracy, FP8 quantization and the 3D reconstruction feedforward model can further reduce memory overhead by about 20% and inference latency by about 25%.

[0089] In some other alternative implementations, after introducing token merging, the aggregator and prediction head (such as camera head, depth head) can be fine-tuned to compensate for accuracy loss; the technology in this application uses a mixture of multiple public datasets to fine-tune the model, with in-batch sampling of 4–48 frames and 20K training epochs.

[0090] It should be noted that the embodiments of this application have the following advantages compared with related technologies: 1. Reduced complexity and video memory usage, supporting thousands of frames per second input.

[0091] Technical means: (1) Reducing the number of tokens entering Global Attention before token merging can reduce complexity; (2) Caching and reusing indexes reduces redundant calculations; (3) System FP8 quantization and low-bit inference can reduce GPU memory usage.

[0092] 2. End-to-end inference latency is significantly reduced.

[0093] Technical means: (1) Reduce the number of tokens entering Global Attention before token merging, thus reducing the amount of computation; (2) Cache and reuse indexes to reduce redundant computation; (3) FP8 quantization of the system and low bit inference can reduce the amount of video memory.

[0094] 3. Geometric details are better preserved, and reconstruction accuracy remains the same.

[0095] Technical means: (1) GA map explicitly identifies key areas such as edges / textures and protects them as GA tokens; (2) dst selects areas with low information to avoid erroneous merging of key tokens; (3) Fine-tuning of big data model.

[0096] Please see Figure 5 , Figure 5This is one of the structural schematic diagrams of a three-dimensional mapping device 500 provided in an embodiment of this application. As shown in the figure, the three-dimensional mapping device 500 includes: The acquisition module 510 is used to acquire multiple frames of images collected by the inspection robot, and to perform feature extraction and image fragment tokenization processing on each frame of image in sequence to obtain the image token sequence corresponding to each frame of image. The merging module 520 is used to obtain the geometric perception map corresponding to the single frame image based on the single frame image and the image token sequence, perform token merging on the image token sequence based on the geometric perception map, and generate a corresponding merging index. The geometric perception map represents the geometric importance distribution corresponding to each frame image. Processing module 530 is used to take the multiple merged token sequences corresponding to the multi-frame images as input, perform global self-attention calculation processing, and obtain global features; The restoration module 540 is used to perform restoration processing on the image token sequence based on the merged index and the global features, and to perform three-dimensional mapping based on the restored token sequence.

[0097] Optionally, the geometric perception map is generated by a weighted fusion of an image gradient map and a token feature variance map; The image gradient map is obtained by performing edge gradient calculations based on the single-frame image using an edge detection operator. The token feature variance map is obtained based on the image token sequence through local variance statistics of token features.

[0098] Optionally, the merging module 520 is specifically used for: The tokens in the image token sequence are divided into geometric perception tokens, target anchor tokens, and tokens to be merged. The tokens to be merged are matched to the target anchor tokens based on feature similarity for feature aggregation. The geometric perception token is a preset proportion image token sequence with the highest geometric importance score corresponding to the geometric perception map, and the geometric perception token does not participate in token merging.

[0099] Optionally, the global self-attention calculation is an operation performed using a multi-layer stacked network, and the 3D mapping device 500 can also be used for: The merge index is cached according to a preset layer interval in the multi-layer stacked network, and the merge index is reused in the corresponding network layer.

[0100] Optionally, the merging module 520 is specifically used for: Assign the global features obtained from the global self-attention operation to each target anchor Token; The target anchor token corresponding to each original token to be merged is determined based on the merge index. The global features carried by the target anchor token are mapped back to the position of the corresponding original token to be merged, and the geometrically perceived token retains its original position and features to restore the original length and feature dimension of the image token sequence.

[0101] Optionally, the merging module 520 can also be used for: Only floating-point operations during the aggregation process are handled with lightweight 8-bit floating-point precision.

[0102] For details, see Figure 6 As shown in the figure, this application embodiment also provides an electronic device, including a bus 601, a transceiver 602, an antenna 603, a bus interface 604, a processor 605, and a memory 606.

[0103] Processor 605, used for: The inspection robot collects multiple frames of images, and performs feature extraction and image fragment tokenization on each frame in sequence to obtain the image token sequence corresponding to each frame. Based on a single frame image and the image token sequence, a geometric perception map corresponding to the single frame image is obtained. Based on the geometric perception map, token merging is performed on the image token sequence, and a corresponding merging index is generated. The geometric perception map represents the geometric importance distribution corresponding to each frame image. The multiple merged token sequences corresponding to the multi-frame images are used as input to perform global self-attention calculation to obtain global features. The image token sequence is restored based on the merged index and the global features, and a 3D map is constructed based on the restored token sequence.

[0104] exist Figure 6In this document, a bus architecture (represented by bus 601) is used. Bus 601 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 605 and memory represented by memory 606. Bus 601 may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 604 provides an interface between bus 601 and transceiver 602. Transceiver 602 may be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 605 is transmitted over a wireless medium via antenna 603, which further receives data and transmits data to processor 605.

[0105] Processor 605 manages bus 601 and general processing, and also provides various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 606 can be used to store data used by processor 605 during operation.

[0106] Alternatively, the processor 605 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0107] Optionally, the geometric perception map is generated by a weighted fusion of an image gradient map and a token feature variance map; The image gradient map is obtained by performing edge gradient calculations based on the single-frame image using an edge detection operator. The token feature variance map is obtained based on the image token sequence through local variance statistics of token features.

[0108] Optionally, the processor 605 is specifically used for: The tokens in the image token sequence are divided into geometric perception tokens, target anchor tokens, and tokens to be merged. The tokens to be merged are matched to the target anchor tokens based on feature similarity for feature aggregation. The geometric perception token is a preset proportion image token sequence with the highest geometric importance score corresponding to the geometric perception map, and the geometric perception token does not participate in token merging.

[0109] Optionally, the processor 605 is specifically used for: The merge index is cached according to a preset layer interval in the multi-layer stacked network, and the merge index is reused in the corresponding network layer.

[0110] Optionally, the processor 605 is specifically used for: Assign the global features obtained from the global self-attention operation to each target anchor Token; The target anchor token corresponding to each original token to be merged is determined based on the merge index. The global features carried by the target anchor token are mapped back to the position of the corresponding original token to be merged, and the geometrically perceived token retains its original position and features to restore the original length and feature dimension of the image token sequence.

[0111] Optionally, the processor 605 is specifically used for: Only floating-point operations during the aggregation process are handled with lightweight 8-bit floating-point precision.

[0112] It should be noted that the electronic device provided in this application embodiment is a device capable of executing the above-described three-dimensional mapping method. Therefore, all implementation methods in the above-described three-dimensional mapping method embodiments are applicable to this electronic device and can achieve the same or similar beneficial effects. To avoid repetition, this embodiment will not elaborate further.

[0113] This application also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described three-dimensional mapping method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0114] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described three-dimensional mapping method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0115] This application also provides a computer program product, including computer instructions. When executed by a processor, these computer instructions implement the various processes of the above-described three-dimensional mapping method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0116] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0118] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A three-dimensional mapping method, characterized in that, The method includes: The inspection robot collects multiple frames of images, and performs feature extraction and image fragment tokenization on each frame in sequence to obtain the image token sequence corresponding to each frame. Based on a single frame image and the image token sequence, a geometric perception map corresponding to the single frame image is obtained. Based on the geometric perception map, token merging is performed on the image token sequence, and a corresponding merging index is generated. The geometric perception map represents the geometric importance distribution corresponding to each frame image. The multiple merged token sequences corresponding to the multi-frame images are used as input to perform global self-attention calculation to obtain global features. The image token sequence is restored based on the merged index and the global features, and a 3D map is constructed based on the restored token sequence.

2. The method according to claim 1, characterized in that, The geometric perception map is generated by a weighted fusion of the image gradient map and the token feature variance map; The image gradient map is obtained by performing edge gradient calculations based on the single-frame image using an edge detection operator. The token feature variance map is obtained based on the image token sequence through local variance statistics of token features.

3. The method according to claim 1, characterized in that, The step of performing token merging on the image token sequence based on the geometric perception map and generating a corresponding merge index includes: The tokens in the image token sequence are divided into geometric perception tokens, target anchor tokens, and tokens to be merged. The tokens to be merged are matched to the target anchor tokens based on feature similarity for feature aggregation. The geometric perception token is a preset proportion image token sequence with the highest geometric importance score corresponding to the geometric perception map, and the geometric perception token does not participate in token merging.

4. The method according to any one of claims 1 to 3, characterized in that, The global self-attention calculation is performed using a multi-layer stacked network. After performing token merging on the image token sequence based on the geometric perception map and generating the corresponding merge index, the calculation further includes: The merge index is cached according to a preset layer interval in the multi-layer stacked network, and the merge index is reused in the corresponding network layer.

5. The method according to claim 3, characterized in that, The process of restoring the image token sequence based on the merged index and the global features includes: Assign the global features obtained from the global self-attention operation to each target anchor Token; The target anchor token corresponding to each original token to be merged is determined based on the merge index. The global features carried by the target anchor token are mapped back to the position of the corresponding original token to be merged, and the geometrically perceived token retains its original position and features to restore the original length and feature dimension of the image token sequence.

6. The method according to claim 3, characterized in that, The aggregation process of matching the tokens to be merged to the target anchor tokens based on feature similarity further includes: Only floating-point operations during the aggregation process are handled with lightweight 8-bit floating-point precision.

7. A three-dimensional mapping device, characterized in that, include: The acquisition module is used to acquire multiple frames of images collected by the inspection robot, and to perform feature extraction and image fragment tokenization processing on each frame of image in sequence to obtain the image token sequence corresponding to each frame of image. The merging module is used to obtain the geometric perception map corresponding to the single frame image based on the single frame image and the image token sequence, perform token merging on the image token sequence based on the geometric perception map, and generate a corresponding merging index. The geometric perception map represents the geometric importance distribution corresponding to each frame image. The processing module is used to take the multiple merged token sequences corresponding to the multi-frame images as input, perform global self-attention calculation processing, and obtain global features; The restoration module is used to perform restoration processing on the image token sequence based on the merged index and the global features, and to perform 3D mapping based on the restored token sequence.

8. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the three-dimensional mapping method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the three-dimensional mapping method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the three-dimensional mapping method as described in any one of claims 1 to 6.

Images