A feedforward multi-frame lidar reconstruction method based on cross-frame geometric inference

Abstract

The application discloses a kind of feedforward multi-frame laser radar reconstruction methods based on cross-frame geometric inference, it is related to laser radar data processing technical field, by obtaining continuous multi-frame LiDAR point cloud sequence and executing spherical projection, generate the two-dimensional range image of the scanning line structure reservation;Range image is tokenized to block in line direction and adds angular position embedding, while introducing the learnable camera token of each frame, each frame token is spliced into uniform sequence input transformer encoder, local geometric modeling and cross-frame geometric correlation are realized by the alternative calculation of intra-frame self-attention and global self-attention;Camera token and spatial token are extracted from the updated token feature respectively, output each frame pose by pose prediction head, output dense depth map and confidence map with the same resolution as input by reconstruction prediction head.The application completes global pose estimation and dense geometric reconstruction in single forward propagation, without iterative optimization or explicit completion module.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, 3D reconstruction, lidar data processing and deep learning, and particularly relates to a feedforward multi-frame lidar reconstruction method based on cross-frame geometric inference. Background Technology

[0002] 3D scene reconstruction is one of the core technologies in fields such as autonomous driving environmental perception, high-precision map construction, and robot navigation. LiDAR (Light Detection and Ranging) has become the mainstream environmental perception sensor in this field due to its ability to directly acquire high-precision spatial geometric information. Currently, LiDAR-based 3D reconstruction methods can be mainly divided into two categories: one is the traditional multi-stage processing method based on explicit geometric optimization; the other is the data-driven method based on learning.

[0003] Traditional methods are typically constructed as multi-stage serial processing pipelines. A typical process is as follows: First, consecutive frames are paired-registered using the Iterative Closest Point (ICP) algorithm and its variants to estimate the relative pose between frames; then, based on the estimated pose, the point clouds of each frame are gradually fused into a unified coordinate system using strategies such as the Truncated Signed Distance Function (TSDF) or Surfel fusion, ultimately generating a grid or point cloud map for downstream tasks. Such methods have several inherent limitations in practical applications: (1) low computational efficiency, requiring multiple iterations for each frame registration, making it difficult to efficiently process large-scale long-sequence data; (2) significant error accumulation, where small errors generated by front-end registration irreversibly propagate to the back-end fusion stage, leading to global errors. Figure 1 (3) The task modules are isolated from each other. Pose estimation and dense reconstruction are two independent links, lacking information feedback and joint optimization mechanisms; (4) The ability to handle occlusion and long-distance sparse areas is insufficient, and the final reconstruction results often retain a large number of incomplete structural holes.

[0004] In recent years, deep learning-based feedforward reconstruction methods have made significant progress in the field of RGB images. For example, models such as VGGT and DUSt3R can directly regress camera pose and dense depth maps from multiple frames of RGB images end-to-end. However, such methods rely heavily on the rich texture information and photometric consistency assumptions in the images, and their technical approach is difficult to directly transfer to LiDAR data processing tasks. LiDAR point cloud data and RGB images have fundamental differences in data characteristics, specifically: (1) data sparsity, only sporadic point cloud sampling can be obtained from the surface of distant targets; (2) lack of photometric information, unable to provide visual cues such as color or texture for feature matching; (3) non-uniform sampling mode, the vertical scan lines are relatively sparse, while the horizontal lines are relatively dense; (4) weak cross-frame correspondence, due to changes in viewing angle and occlusion, the observation of the same physical point in consecutive frames has natural instability, making matching difficult.

[0005] For LiDAR data, most existing learning methods focus on processing single-frame point clouds. For example, models such as TULIP and LiDAR-SR are mainly used for completing or upsampling single-frame point clouds, limiting their processing scope to within a single frame and failing to effectively utilize the rich geometric correlation information contained between multiple frames of data. In addition, some research has attempted to apply the Transformer architecture to LiDAR odometry estimation, but the output of these methods is usually limited to the relative pose between frames, lacking the ability to perform dense geometric reconstruction of the entire scene.

[0006] Therefore, how to simultaneously recover globally consistent sensor pose and complete scene dense geometry from multiple LiDAR sequences in a non-iterative and highly efficient feedforward manner within a unified framework remains a key technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides a feedforward multi-frame lidar reconstruction method based on cross-frame geometric inference. By establishing cross-frame geometric associations in a unified latent space through Transformer, end-to-end pose estimation and dense reconstruction are achieved, thereby overcoming the computational complexity and error accumulation problems of traditional iterative optimization methods and solving the technical obstacle that existing learning methods cannot effectively utilize multi-frame geometric information.

[0008] This invention provides a feedforward multi-frame lidar reconstruction method based on cross-frame geometric inference, comprising: Acquire a continuous sequence of multiple frames of LiDAR point cloud, perform spherical projection on each frame of point cloud, and generate a two-dimensional range image; Each frame of the range image is divided into rows and tokenized to obtain a token sequence, and an angular position embedding is added to each token; A learnable camera token is introduced for each frame, and the camera tokens of each frame are concatenated with the spatial tokens of the corresponding frames to form a unified token sequence for multiple frames. The unified token sequence is input into the Transformer encoder. The intra-frame self-attention layer in the Transformer encoder performs self-attention calculation between tokens in the same frame, and the global self-attention layer in the Transformer encoder performs self-attention calculation between all tokens in all frames, and outputs the updated token features for each frame. Extract the camera token corresponding to each frame from the updated token features of each frame, input the pose prediction head, and output the translation vector and rotation attitude of each frame in the world coordinate system. Extract the spatial tokens corresponding to each frame from the updated token features of each frame, input them into the reconstruction prediction head, and output a dense depth map and a point-by-point confidence map with the same resolution as the input range image.

[0009] Optionally, the spherical projection includes: The vertical resolution of the range image is determined based on the number of vertical scan lines of the lidar, and the horizontal resolution of the range image is determined based on the number of horizontal samples. For each 3D point in each frame of the point cloud, the pixel coordinates of the 3D point in the range image are calculated based on the azimuth angle, pitch angle, vertical field of view boundary and horizontal resolution of the LiDAR. The value of each pixel is set to the distance from the three-dimensional point in the corresponding spatial direction to the origin of the lidar coordinate system. For pixel positions where no laser return is received, the value of the pixel is set to zero.

[0010] Optionally, when performing the row-oriented block tokenization, the block size is one pixel in the vertical direction and the horizontal direction is a preset aggregation width, while the vertical direction remains non-aggregated, so that each token covers multiple consecutive horizontal pixels on a single scan line; and each block is mapped to a token vector through a linear embedding layer.

[0011] Optionally, during the process of adding angular position embeddings for each token, a learnable parameter vector is used as the angular position embedding, and each angular position embedding is associated with the vertical index and horizontal index of the corresponding token, wherein the vertical index represents the scan line number and the horizontal index represents the azimuth block number.

[0012] Optionally, concatenating the camera token of each frame with the spatial token of the corresponding frame includes: The camera token of each frame is placed before all spatial tokens in that frame to form the intra-frame token sequence of that frame; The intra-frame token sequences of all frames are concatenated in chronological order to form a unified token sequence across multiple frames.

[0013] Optionally, during the process of inputting the unified token sequence into the Transformer encoder for processing, the Transformer encoder is constructed by stacking multiple layers of alternating intra-frame self-attention layers and global self-attention layers. The number of layers of the Transformer encoder is set, and multi-head self-attention mechanism and feedforward network are used for calculation in each self-attention layer, as well as application layer normalization and residual connection.

[0014] Optionally, during the output of the translation vector and rotation attitude in the world coordinate system for each frame: Using the world coordinate system of the first frame in the continuous multi-frame lidar point cloud sequence as the reference coordinate system, the pose of the first frame is set to unit translation and zero rotation. A multilayer perceptron structure is used as the pose prediction head; The camera tokens corresponding to each frame are input into the pose prediction head, and the pose prediction head regresses the translation vectors and rotation quaternions of each frame in the world coordinate system from the camera tokens.

[0015] Optionally, in the process of outputting a dense depth map and a point-by-point confidence map with the same resolution as the input range image, a DPT-style upsampling decoder is used as the reconstruction prediction head. The reconstruction prediction head decodes the dense depth map and the point-by-point confidence map with the same resolution as the input range image from the spatial tokens corresponding to each frame. The value of each pixel in the confidence map represents the uncertainty of the depth prediction at the corresponding location.

[0016] Optionally, during the training phase, the system further includes dynamically generating an observation region mask, a complete region mask, and an ignored region mask based on the comparison between the input range image and the ground value range image, and constructing a joint loss function for end-to-end training.

[0017] Optionally, during the construction of the joint loss function: The region in the input range image where the pixel values ​​are valid and the corresponding pixel values ​​in the true value range image are valid is marked as the observation region; Regions in the input range image with invalid pixel values ​​and corresponding valid pixel values ​​in the true value range image are marked as regions that can be completed. Regions in the input range image with invalid pixel values ​​and corresponding invalid pixel values ​​in the true value range image are marked as ignored regions; Based on the observed region, the completeable region, and the ignored region, pose loss, depth regression loss, confidence loss, multi-scale gradient loss, and cross-frame geometric consistency loss are constructed respectively. The pose loss, depth regression loss, confidence loss, multi-scale gradient loss, and cross-frame geometric consistency loss are weighted and combined to construct a joint loss function.

[0018] Compared with the prior art, the present invention has the following advantages and technical effects: This invention transforms a sequence of consecutive multi-frame LiDAR point clouds into a two-dimensional image through spherical projection, converting the disordered point cloud into a regular grid representation that retains the original scan line structure, thus providing a suitable input format for Transformer networks. Furthermore, by using row-oriented block tokenization and angular position embedding, each token encodes the prior physical scan direction while maintaining the independent scan line geometry, ensuring that the tokenization process does not compromise the inherent sampling characteristics of LiDAR.

[0019] Furthermore, this invention introduces learnable camera tokens for each frame and concatenates them with spatial tokens to form a unified sequence, enabling the Transformer to jointly encode multi-frame data in the same feature space. By alternating intra-frame self-attention layers and global self-attention layers, intra-frame self-attention models local geometry between tokens in a single frame, while global self-attention establishes cross-frame geometric relationships between all tokens across all frames. This allows observation information of the same spatial region from different frames to be automatically aggregated through the attention mechanism. Therefore, this invention can achieve deep fusion of multi-frame features in a single forward propagation without iterative optimization, fundamentally avoiding the accumulation problem caused by the propagation of front-end registration errors to the back-end in traditional methods.

[0020] Building upon this foundation, camera tokens and spatial tokens are extracted from the updated token features. The pose prediction head regresses the translation vectors and rotational poses of each frame in the world coordinate system, while the reconstruction prediction head outputs a dense depth map with the same resolution as the input. This achieves joint output of global pose estimation and dense geometric reconstruction within a unified framework. Pose estimation provides a cross-frame aligned coordinate reference for depth reconstruction, while depth reconstruction provides rich geometric constraints for pose estimation. The two mutually reinforce each other, overcoming the isolation between pose estimation and reconstruction tasks in traditional methods. The entire method operates in a fully feedforward manner, eliminating the need for explicit completion modules. It achieves depth recovery of occluded and sparse regions through cross-frame information aggregation via a global self-attention mechanism, significantly improving the efficiency and completeness of geometric reconstruction of multi-frame LiDAR sequences. Attached Figure Description

[0021] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1A schematic diagram comparing the conventional reconstruction method provided in the embodiments of the present invention with the feedforward reconstruction method of this application; Figure 2 This is a schematic diagram of the architecture of the feedforward multi-frame lidar reconstruction method based on cross-frame geometric inference provided in an embodiment of the present invention. Detailed Implementation

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0024] Example 1 like Figures 1-2 As shown, this embodiment provides a feedforward multi-frame lidar reconstruction method based on cross-frame geometric inference, the method including the following steps: Step S1: Obtain a continuous multi-frame LiDAR point cloud sequence, perform spherical projection on each frame of point cloud to generate a two-dimensional range image, wherein each pixel value in the range image represents the distance in the corresponding spatial direction; Step S2: Perform row-oriented block tokenization on each frame range image, where each token covers a single scan line pixel in the vertical direction and multiple consecutive pixels in the horizontal direction. Map each block to a token vector through a linear embedding layer and add angular position embedding to each token. Step S3: Introduce a learnable camera token for each frame, and concatenate the camera token with all spatial tokens of that frame to form a unified token sequence for multiple frames; Step S4: Input the unified token sequence into the Transformer encoder, which is composed of multiple alternating intra-frame self-attention layers and global self-attention layers stacked together; intra-frame self-attention is calculated between tokens in the same frame to model local geometry; global self-attention is calculated between all tokens in all frames to establish cross-frame geometric relationships. Step S5: Extract the camera tokens corresponding to each frame from the features output by the Transformer, input the pose prediction head, and output the translation vector and rotation quaternion of the frame in the world coordinate system. Step S6: Extract the spatial tokens corresponding to each frame from the features output by the Transformer, input them into the reconstruction prediction head, and output a dense depth map and a point-by-point confidence map with the same resolution as the input range image. Step S7: During the training phase, based on the comparison between the input range image and the ground truth image, an observation region mask, a complete region mask, and an ignored region mask are generated, and a joint loss function is constructed. The joint loss function includes pose loss, depth regression loss, confidence loss, multi-scale gradient loss, and cross-frame geometric consistency loss.

[0025] As a feasible implementation method, the specific steps include: Step S1: Spherical projection of multi-frame LiDAR point clouds: Acquire a continuous sequence of multiple frames of LiDAR point cloud data. Perform spherical projection on each frame of the point cloud to map the 3D points onto a 2D grid composed of azimuth and elevation angles, generating a 2D range image. Each pixel value in the range image represents the distance in the corresponding spatial direction.

[0026] Specifically, the vertical resolution of the spherical projection is equal to the number of LiDAR scan lines H, and the horizontal resolution is equal to the number of horizontal samples W. For any 3D point p=(x,y,z) in each frame of the point cloud, calculate the azimuth and elevation angles of that point, and then map the 3D point to the corresponding pixel coordinates in the range image according to the following mapping relationship: ; ; Where H is the number of LiDAR scan lines (e.g., 128) and W is the horizontal resolution (e.g., 1024). and The vertical field of view boundary is defined by r, where r is the distance from the 3D point to the origin of the lidar coordinate system. The value of each pixel is set to the distance from the 3D point in that direction to the origin of the lidar coordinate system. For pixel positions where no laser return is received, the value of that pixel is set to zero.

[0027] Through the above processing, the original disordered and unstructured lidar point cloud is converted into a regular two-dimensional grid representation, thereby adapting to the input format of the neural network, while preserving the physical structure of the sensor sampling, that is, the vertical direction corresponds to the laser scan line, and the horizontal direction corresponds to the continuous change of azimuth angle, laying the foundation for subsequent geometric perception tokenization.

[0028] Step S2: Row-wise block tokenization and corner-wise position embedding: Each frame of the image is segmented into blocks and tokenized. The block size is one pixel vertically and a preset aggregation width *p* horizontally, with no vertical aggregation to preserve the independent geometry of each scan line. Each token covers *p* consecutive horizontal pixels on a single scan line, where *p* is typically 16. A learnable ViT encoder is introduced to map each block to a token vector.

[0029] An angular position embedding is added to each token. The angular position embedding is a learnable parameter vector, and each angular position embedding is associated with the vertical index and horizontal index of the corresponding token, where the vertical index represents the scan line number and the horizontal index represents the azimuth block number, which are used to encode the physical scan direction of the LiDAR.

[0030] Through the above processing, the range image is segmented into geometric semantic units (tokens), and a spatial orientation prior is injected into each token. Traditional ViT uses square blocks, but the vertical and horizontal directions in LiDAR range images have distinctly different geometric statistical characteristics—the vertical direction has sparse information and each scan line is independent, while the horizontal direction has dense and continuous information. The row-oriented block design of this invention is specifically designed to address this characteristic, ensuring that the tokenization process does not disrupt the physical scanning structure of LiDAR.

[0031] Step S3: Camera token and sequence construction: A learnable camera token is introduced for each frame, which is identical to the spatial token in terms of feature dimensions. The camera token for each frame is placed before all spatial tokens for that frame to form an intra-frame token sequence. Then, the intra-frame token sequences of all frames are concatenated in chronological order to form a unified token sequence across multiple frames.

[0032] The camera token serves as a global aggregation node for each frame. During subsequent Transformer processing, it aggregates information from all spatial tokens within that frame through a self-attention mechanism. This results in the final output camera token carrying the global geometric features of that frame, making it suitable for pose regression. Simultaneously, tokens from multiple frames are concatenated into a unified sequence, enabling the Transformer to perform cross-frame information exchange within the same feature space.

[0033] Step S4: Alternating cross-frame geometric inference Transformer: A unified token sequence is input into the Transformer encoder. The Transformer encoder consists of multiple layers of alternating intra-frame self-attention layers and global self-attention layers stacked together, with the number of layers set to L (L=24). Each layer contains two sequentially executed self-attention sub-layers, each followed by a feedforward network and layer normalization, and residual connections are used.

[0034] Intra-frame self-attention layers compute attention only between tokens within the same frame, modeling the local geometry within a single frame, including depth continuity along scan lines and the perpendicular relationship between adjacent scan lines. Global self-attention layers compute attention across all tokens across all frames, establishing cross-frame geometric correspondences. Through global self-attention, any token can pay attention to tokens in any other frame. When a spatial location in a frame is missing due to occlusion, valid observations of that location from other frames can be aggregated to the corresponding token in the current frame using high attention weights. Through multi-layer stacking, the model gradually fuses geometric information from multiple frames, achieving implicit completion of missing regions.

[0035] Intra-frame self-attention and global self-attention have clearly defined roles: the former is responsible for perceiving the structure within a single frame, while the latter is responsible for cross-frame inference. The two are executed alternately, so that each layer performs global fusion immediately after local modeling, progressively building a complete geometric understanding of the scene.

[0036] Step S5: Pose Prediction From the updated token features output by the Transformer, the camera tokens corresponding to each frame are extracted and input into the pose prediction head. The pose prediction head adopts a multilayer perceptron structure. Using the world coordinate system of the first frame in a continuous multi-frame LiDAR point cloud sequence as the reference coordinate system, the pose of the first frame is set to unit translation and zero rotation. The pose prediction head regresses the translation vector and rotation quaternion of each frame in the world coordinate system from the camera tokens corresponding to each frame, and outputs a 7-dimensional pose vector containing xyz translation and rotation quaternions.

[0037] Step S6: Dense Reconstruction From the updated token features output by the Transformer, spatial tokens corresponding to each frame are extracted and input into the reconstruction prediction head. The reconstruction prediction head employs a DPT-style upsampling decoder to decode dense depth maps and point-by-point confidence maps with the same resolution as the input range image from the spatial tokens corresponding to each frame. The value of each pixel in the confidence map represents the uncertainty of the model's depth prediction at that location.

[0038] Step S7: Region-aware supervision strategy during the training phase: During the training phase, based on the comparison between the input range image and the ground truth image, the observed region mask, the complete region mask, and the ignored region mask are dynamically generated, and a joint loss function is constructed for end-to-end training.

[0039] Specifically, in each training iteration, three types of masks are dynamically calculated and generated based on the current batch of data. Regions with valid pixel values ​​in both the input range image and the ground truth image are marked as observation regions; regions with invalid pixel values ​​in both the input range image and the ground truth image are marked as regions that can be completed; and regions with invalid pixel values ​​in both the input range image and the ground truth image are marked as ignored regions.

[0040] The joint loss function is expressed as follows: ; This includes five loss items with clearly defined roles: This represents the pose loss, which supervises the translation vector and rotation quaternions obtained by regressing the camera tokens output by the Transformer through the pose prediction head. Its function is to establish a stable global coordinate frame, ensuring that the point clouds of all frames are aligned in the same reference frame. Only when the pose is accurate can cross-frame geometric aggregation and consistency constraints work effectively.

[0041] The depth regression loss, whose core function is to drive the model to learn accurate depth values, calculates the difference between the predicted depth map and the ground truth depth map in both the observed and completable regions. Ignored regions are not included in this loss calculation. In the observed region, this loss forces the model to preserve the true geometry in the input; in the completable region, this loss guides the model to use cross-frame information to fill in missing geometric structures.

[0042] The confidence loss is used to supervise the confidence map output by the model, enabling the model to express the uncertainty of its predictions. In the observed region, the model should output high confidence, constrained by a binary cross-entropy loss; in the ignored region, the model should output low confidence, also constrained by a binary cross-entropy loss; in the implausible region, an uncertainty-weighted loss is used, ensuring that the confidence is lower at locations with larger prediction depth errors, while a logarithmic regularization term prevents the confidence from degenerating to zero. This design allows the model to automatically reduce confidence in regions that are difficult to implausibly, providing a reference for downstream tasks.

[0043] This represents the multi-scale gradient loss, which calculates the gradient differences between the predicted depth map and the ground truth depth map in the horizontal and vertical directions over the observed and completable regions, and accumulates them across multiple downsampling scales. Its purpose is to enhance the model's sensitivity to depth discontinuities (i.e., object edges), preventing overly smooth reconstruction results and thus maintaining clear geometric boundaries.

[0044] The cross-frame geometric consistency loss is a key supervisory signal for the success of implicit incomplete. Its calculation process is as follows: First, the predicted depth maps of each frame are back-projected into a 3D point cloud; then, the point cloud is transformed to the reference coordinate system of the first frame using the predicted poses of each frame; finally, the Chamfer distance between the transformed point clouds of other frames and the point cloud of the first frame is calculated. This loss forces the model to output geometrically consistent multi-frame reconstruction results—that is, predictions of the same physical surface from different viewpoints must remain consistent in 3D space. It is this constraint that forces the Transformer's global self-attention layer to learn the correct cross-frame geometric correspondence, enabling the model to automatically recover the geometry of occluded and sparse regions from multi-frame observations without an explicit incomplete module.

[0045] In terms of training strategy, considering the strong coupling between pose estimation and depth reconstruction, this invention adopts a two-stage training approach. The first stage is the pose warm-up stage, where the reconstruction prediction head is frozen, and only the pose loss is optimized, i.e., setting... =1, with the remaining loss weights set to 0, enabling the Transformer encoder to quickly learn inter-frame geometric correspondences and establish a stable coordinate system. The second stage is the joint optimization stage, where all modules are unfrozen, all loss terms are enabled, and depth reconstruction and pose estimation mutually promote each other and converge jointly.

[0046] Compared with existing technologies, this invention converts multi-frame LiDAR point clouds into structured range images and performs geometrically aware tokenization representation. Then, it utilizes a Transformer network for cross-frame feature modeling, simultaneously predicting pose and reconstruction results within a unified framework. Furthermore, by introducing occupancy information modeling, the model can distinguish between "unobserved" and "real empty space," thereby avoiding erroneous predictions in invalid regions.

[0047] Specifically, this invention extends the feedforward 3D reconstruction method from RGB images to multi-frame LiDAR sequences for the first time. By unifying the network structure, it simultaneously outputs pose and reconstruction results, avoiding the iterative optimization and multi-stage processing in traditional methods, thereby significantly improving computational efficiency and reducing system complexity.

[0048] Existing LiDAR completion methods all employ explicit design, either using an encoder-decoder structure to fill holes in a single-frame point cloud or using a generative model for surface prediction. These methods require dedicated completion modules and loss functions and often lack geometric realism constraints. This invention does not use any explicit completion module. Instead, it leverages the global self-attention mechanism of the Transformer to allow the model to autonomously learn cross-frame geometric correspondences. Specifically, the global self-attention layer allows tokens from any frame to directly attend to all tokens in other frames. When a region in a frame is missing due to occlusion or sparse sampling, observations of that region in other frames (from different viewpoints) can be aggregated into the corresponding token in the current frame using high attention weights. After processing through multiple Transformer layers, the features of the missing region are automatically completed, thus decoding a complete depth map. The geometric realism of this implicit completion is guaranteed by cross-frame geometric consistency loss, forcing the reconstruction of the same physical surface in different frames to remain consistent in a unified coordinate system.

[0049] Unlike RGB images where every pixel can be supervised, LiDAR-range images contain a large number of invalid pixels, which can be further divided into two distinct categories: "true empty space" and "unobserved surfaces." Indiscriminate supervision can lead to incorrect predictions in empty spaces or forced zero values ​​in areas that should be filled in. This invention employs a dynamic masking mechanism to calculate online whether each pixel belongs to the observed region (input has a value), the fillable region (input has no value but the ground truth has a value), or the ignored region (both input and ground truth have no value). Different loss functions are applied to different regions—the observed region retains the original input, the fillable region learns cross-frame completion, and the ignored region is not included in training. Furthermore, cross-frame geometric consistency loss is crucial to this training strategy. The predicted depth of each frame is back-projected into 3D, and the predicted pose is transformed to the coordinate system of the first frame to calculate the point cloud distance. This forces the model to learn geometrically consistent cross-frame correspondences, which is the core supervision signal for successful implicit completion.

[0050] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A feedforward multi-frame lidar reconstruction method based on cross-frame geometric inference, characterized in that, include: Acquire a continuous sequence of multiple frames of LiDAR point cloud, perform spherical projection on each frame of point cloud, and generate a two-dimensional range image. Each frame of the range image is divided into rows and tokenized to obtain a token sequence, and an angular position embedding is added to each token; A learnable camera token is introduced for each frame, and the camera tokens of each frame are concatenated with the spatial tokens of the corresponding frames to form a unified token sequence for multiple frames. The unified token sequence is input into the Transformer encoder. The intra-frame self-attention layer in the Transformer encoder performs self-attention calculation between tokens in the same frame, and the global self-attention layer in the Transformer encoder performs self-attention calculation between all tokens in all frames, and outputs the updated token features for each frame. Extract the camera token corresponding to each frame from the updated token features of each frame, input the pose prediction head, and output the translation vector and rotation attitude of each frame in the world coordinate system. Extract the spatial tokens corresponding to each frame from the updated token features of each frame, input them into the reconstruction prediction head, and output a dense depth map and a point-by-point confidence map with the same resolution as the input range image.

2. The method according to claim 1, characterized in that, The spherical projection includes: The vertical resolution of the range image is determined based on the number of vertical scan lines of the lidar, and the horizontal resolution of the range image is determined based on the number of horizontal samples. For each 3D point in each frame of the point cloud, the pixel coordinates of the 3D point in the range image are calculated based on the azimuth angle, pitch angle, vertical field of view boundary and horizontal resolution of the LiDAR. The value of each pixel is set to the distance from the three-dimensional point in the corresponding spatial direction to the origin of the lidar coordinate system. For pixel positions where no laser return is received, the value of the pixel is set to zero.

3. The method according to claim 1, characterized in that, When performing the row-oriented block tokenization, the block size is one pixel in the vertical direction and the horizontal direction is a preset aggregation width. The vertical direction is kept non-aggregated, so that each token covers multiple consecutive horizontal pixels on a single scan line. Each block is mapped to a token vector through a linear embedding layer.

4. The method according to claim 1, characterized in that, In the process of adding angular position embeddings for each token, a learnable parameter vector is used as the angular position embedding. Each angular position embedding is associated with the vertical index and horizontal index of the corresponding token. The vertical index represents the scan line number and the horizontal index represents the azimuth block number.

5. The method according to claim 1, characterized in that, Concatenating the camera tokens of each frame with the corresponding spatial tokens includes: The camera token of each frame is placed before all spatial tokens in that frame to form the intra-frame token sequence of that frame; The intra-frame token sequences of all frames are concatenated in chronological order to form a unified token sequence across multiple frames.

6. The method according to claim 1, characterized in that, During the process of inputting the unified token sequence into the Transformer encoder, the Transformer encoder is constructed by stacking multiple layers of alternating intra-frame self-attention layers and global self-attention layers. The number of layers of the Transformer encoder is set, and multi-head self-attention mechanism and feedforward network are used for calculation in each self-attention layer, as well as application layer normalization and residual connection.

7. The method according to claim 1, characterized in that, In the process of outputting the translation vector and rotation attitude of each frame in the world coordinate system: Using the world coordinate system of the first frame in the continuous multi-frame lidar point cloud sequence as the reference coordinate system, the pose of the first frame is set to unit translation and zero rotation. A multilayer perceptron structure is used as the pose prediction head; The camera tokens corresponding to each frame are input into the pose prediction head, and the pose prediction head regresses the translation vectors and rotation quaternions of each frame in the world coordinate system from the camera tokens.

8. The method according to claim 1, characterized in that, In the process of outputting a dense depth map and a point-by-point confidence map with the same resolution as the input range image, a DPT-style upsampling decoder is used as the reconstruction prediction head. The reconstruction prediction head decodes the dense depth map and the point-by-point confidence map with the same resolution as the input range image from the spatial tokens corresponding to each frame. The value of each pixel in the confidence map represents the uncertainty of the depth prediction at the corresponding location.

9. The method according to claim 1, characterized in that, It also includes dynamically generating observation region masks, complete region masks, and ignore region masks based on the comparison between the input range image and the ground value range image during the training phase, and constructing a joint loss function, and using the joint loss function for end-to-end training.

10. The method according to claim 9, characterized in that, During the construction of the joint loss function: The region in the input range image where the pixel values ​​are valid and the corresponding pixel values ​​in the true value range image are valid is marked as the observation region; Regions in the input range image with invalid pixel values ​​and corresponding valid pixel values ​​in the true value range image are marked as regions that can be completed. Regions in the input range image with invalid pixel values ​​and corresponding invalid pixel values ​​in the true value range image are marked as ignored regions; Based on the observed region, the completeable region, and the ignored region, pose loss, depth regression loss, confidence loss, multi-scale gradient loss, and cross-frame geometric consistency loss are constructed respectively. The pose loss, depth regression loss, confidence loss, multi-scale gradient loss, and cross-frame geometric consistency loss are weighted and combined to construct a joint loss function.

Images