A Transformer-Based LiDAR Point Cloud Analysis Method

Abstract

This invention provides a transformer-based method for LiDAR point cloud analysis, relating to the field of LiDAR point cloud segmentation technology. The invention involves simultaneously acquiring LiDAR point clouds and RGB images of a scene using both LiDAR and a camera, and then filtering the LiDAR point clouds. A sparse depth map of the RGB images is constructed using the LiDAR point clouds. A pre-trained depth estimation model, conditioned on camera intrinsics, the relative pose of the camera acquiring the RGB images, and the sparse depth map, converts the RGB images of the scene into a dense depth map, ensuring alignment between the generated dense depth map and the LiDAR point clouds. The matched dense depth map and LiDAR point clouds are then provided to a Unet architecture point cloud segmentation model based on Transformer blocks. The Unet architecture point cloud segmentation model supplements the LiDAR point cloud features with the dense depth map features to achieve LiDAR point cloud segmentation.

Description

Technical Field

[0001] This invention relates to the field of lidar point cloud segmentation technology, and in particular to a lidar point cloud analysis method based on transformer. Background Technology

[0002] LiDAR point cloud segmentation is a core technology in autonomous driving, robot perception, and 3D scene understanding. Its goal is to divide unordered point cloud data into regions with semantic or instance significance.

[0003] Traditional methods include: LiDAR point cloud segmentation based on geometric features: Point clouds are segmented using geometric features such as normal vectors, curvature, and density. Examples include RANSAC (Random Sample Consensus) for planar segmentation and region growing algorithms. While geometric feature-based segmentation is fast, it is sensitive to noise and struggles with complex scenes. LiDAR point cloud segmentation based on clustering: Euclidean clustering (such as DBSCAN) is based on inter-point distance and is suitable for separating physically independent objects (such as vehicles and pedestrians). However, in complex scenes, target stacking leads to the clustering of different target point clouds, making it difficult to handle complex scenes. With the development of deep learning, methods have evolved to project point clouds onto 2D images (such as spherical projection or top-down views) and process them using CNNs, such as SqueezeSeg and RangeNet++. However, projection leads to the loss of geometric information. Subsequent developments have used native point cloud networks such as PointNet, PointNet++, PointCNN, and KPConv to directly process unordered point clouds while preserving geometric information. Another approach is to convert point clouds into regular voxel meshes and process them using 3D convolutions, such as VoxelNet and SECOND. To address the issue of sparse point clouds in LiDAR, especially at long ranges, subsequent approaches have proposed combining RGB images, LiDAR data, or semantic information to improve segmentation accuracy. For example, this involves fusing camera images with point clouds (e.g., PV-RCNN) and combining semantic segmentation with instance differentiation, such as PointGroup and GICN. This multimodal approach effectively improves the performance of LiDAR point cloud segmentation, but modal alignment issues still exist, leading to segmentation errors. Summary of the Invention

[0004] To solve the above-mentioned technical problems, or at least partially solve them, the present invention provides a LiDAR point cloud analysis method based on transformer.

[0005] In a first aspect, the present invention provides a LiDAR point cloud analysis method based on transformer, comprising:

[0006] The S100 simultaneously acquires scene lidar point clouds and RGB images through lidar and camera, and performs filtering processing on the lidar point clouds.

[0007] S200 uses LiDAR point clouds to construct sparse depth maps of RGB images;

[0008] S300, the pre-trained depth estimation model converts the RGB image of the scene into a dense depth map based on camera intrinsics, the relative pose of the camera that acquired the RGB image, and the sparse depth map.

[0009] After acquiring the dense depth map, S400 provides the matched dense depth map and LiDAR point cloud to the Unet architecture point cloud segmentation model based on Transformer blocks. The Unet architecture point cloud segmentation model supplements the dense depth map features into the LiDAR point cloud features to achieve LiDAR point cloud segmentation.

[0010] Furthermore, the construction of a sparse depth map of an RGB image using lidar point clouds includes: S201, obtaining the spatial transformation relationship between the matching lidar point cloud and the RGB image based on the lidar and camera poses;

[0011] S202, transform the LiDAR point cloud to the coordinate system of the RGB image according to the spatial transformation relationship;

[0012] S203 uses camera intrinsic parameters to project the coordinate-transformed LiDAR point cloud onto the RGB image space to obtain the corresponding pixel coordinates, thereby establishing the relationship between pixel coordinates and LiDAR point cloud.

[0013] S204: The projected LiDAR point cloud is given partial pixel coordinate depth information of the RGB image to obtain a sparse depth map.

[0014] Furthermore, the depth estimation model includes a camera intrinsic encoder, a sparse depth map encoder, a relative pose encoder, and a DUSt3R backbone network; the camera intrinsic encoder combines camera intrinsics with RGB image I1 and RGB image I2 to achieve camera intrinsic encoding; the sparse depth map encoder achieves sparse depth map encoding; and the relative pose encoder achieves relative pose encoding.

[0015] Furthermore, the encoding process of the camera intrinsic encoder is as follows:

[0016] Calculate the normalized ray direction for each pixel in each RGB image based on camera intrinsic parameters:

[0017]

[0018] Where u1, v1 are the coordinates of any pixel in RGB image I1, and K1 is the camera intrinsic parameter of the camera that acquires RGB image I1; u2, v2 are the coordinates of any pixel in RGB image I2, and K2 is the camera intrinsic parameter of the camera that acquires RGB image I2.

[0019] The normalized ray directions of all pixels form a ray coding map, which is aligned with the RGB image. The ray coding map is then embedded to obtain a camera intrinsic token. The camera intrinsic token is then mapped through a linear layer to the visual Transformer encoder of the DUSt3R backbone network and combined with the image token of the RGB image. The image token comes from the image embedding of the DUSt3R backbone network.

[0020] Furthermore, the sparse depth map encoder obtains a sparse depth token by embedding the sparse depth map. The sparse depth token is then mapped to the visual Transformer encoder of the DUSt3R backbone network through a linear layer and combined with the image token of the RGB image. The image token comes from the image embedding of the DUSt3R backbone network.

[0021] Furthermore, the relative pose encoder flattens the rotation matrix of the relative pose and concatenates it with the normalized translation matrix to obtain the relative pose vector. After mapping the relative pose vector through a multilayer perceptron, it is incorporated into the global context query in the decoding process of the visual Transformer decoder of the DUSt3R backbone network.

[0022] Furthermore, the loss function for training the depth estimation model is the sum of the confidence-weighted distance and confidence entropy between the predicted dense depth map and the true depth map. The parameters of the depth estimation model are adjusted with the goal of minimizing the loss function.

[0023] Furthermore, the Unet architecture point cloud segmentation model replaces the original convolutional blocks with Transformer blocks in its encoder and decoder. Cross-attention is set on the jump chains between the encoder and decoder. The semantic features of the dense depth map are fused into the jump chains through the cross-attention. The decoder of the Unet architecture point cloud segmentation model decodes the lidar point cloud features that combine the semantic features of the dense depth map. Finally, the lidar point cloud segmentation mask is obtained through convolutional block mapping, thus realizing lidar point cloud segmentation.

[0024] Furthermore, the Transformer block includes a multi-scale deformable sliding window self-attention network and a multi-scale feedforward network;

[0025] The working principle of the self-attention mechanism of the multi-scale deformable sliding window is as follows:

[0026] The multi-scale deformable sliding window self-attention uses a multi-scale deformable offset network to predict the deformable offset of each position in the local feature map within the region defined by each scale sliding window.

[0027] Adding the corresponding deformable offset to each location in the local feature map yields the target sampling location in the global feature map;

[0028] Features are sampled from all target sampling locations; the sampled features are then stitched together according to their locations to form a new local feature map;

[0029] By introducing deformable offsets, the self-attention support of multi-scale deformable sliding windows adaptively adjusts the receptive field of each scale of the sliding window, allowing attention to be focused on relevant areas outside the sliding window regions of various scales;

[0030] The multi-scale deformable sliding window self-attention constructs a corresponding multi-scale attention head for each scale sliding window, and any multi-scale attention head performs local context modeling on the deformable local feature map of its corresponding scale receptive field.

[0031] Furthermore, the global attention results, which are modeled by local contexts through different receptive fields, are concatenated, normalized, and provided to the multi-scale feedforward network. The multi-scale feedforward network maps the feature maps through linear layers and then divides them into three parts. The three parts of the feature maps are processed by two convolutional layers with different kernel sizes and a dilated layer in parallel. Finally, the three sets of feature maps are fused through linear layer mapping.

[0032] Secondly, the present invention provides a transformer-based lidar point cloud analysis device, comprising: at least one processing unit, wherein the processing unit, storage unit and acquisition unit are interconnected via a bus unit, the storage unit stores a computer program and data acquired by the acquisition unit, and the computer program is executed by the processing unit to implement the transformer-based lidar point cloud analysis method.

[0033] Thirdly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned transformer-based lidar point cloud analysis method.

[0034] The technical solutions provided in the embodiments of the present invention have the following advantages compared with the prior art:

[0035] This invention simultaneously acquires LiDAR point clouds and RGB images of a scene using LiDAR and a camera, and filters the LiDAR point clouds. A sparse depth map of the RGB images is constructed using the LiDAR point clouds. A pre-trained depth estimation model, based on camera intrinsics, the relative pose of the camera acquiring the RGB images, and the sparse depth map, converts the RGB images of the scene into a dense depth map. With the assistance of the LiDAR point clouds, a dense depth map is generated, ensuring alignment between the dense depth map and the LiDAR point clouds, thus better guiding the analysis of the LiDAR point clouds.

[0036] After acquiring the dense depth map, the matched dense depth maps and LiDAR point clouds are provided to the Unet architecture point cloud segmentation model based on Transformer blocks. The Unet architecture point cloud segmentation model supplements the LiDAR point cloud features with the dense depth map features to achieve LiDAR point cloud segmentation. The depth estimation model implements multi-condition encoding, using a dedicated encoder to process camera intrinsics to generate a ray-coded map, embedding the sparse depth map into sparse depth tokens, and vectorizing the relative pose and mapping it through a multilayer perceptron. These vectors are then used as conditional inputs to the DUSt3R backbone network. In the visual Transformer encoder of the DUSt3R backbone network, image tokens and conditional information (camera intrinsics, depth tokens) are fused. During decoding, pose conditions are introduced into the global query, and relative positions are modeled through cross-attention to generate dense depth maps and confidence scores. Furthermore, training combines confidence-weighted depth error and entropy minimization to optimize the reliability and accuracy of depth estimation.

[0037] In this application, the multi-scale deformable attention mechanism of the Transformer block in the Unet architecture point cloud segmentation model supports adaptive receptive field: the sampling position of sliding windows at each scale is dynamically adjusted through a deformable offset network to capture local and cross-regional context. The computational efficiency of the multi-scale deformable attention mechanism is optimized: local attention calculation is performed using multi-scale sliding windows, reducing the complexity of the Transformer and making it suitable for large-scale point cloud processing. Different scale attention heads in the multi-scale deformable attention mechanism support parallel multi-scale processing. To adapt to the concatenation of attention head results from different scales in the multi-scale deformable attention mechanism, a multi-scale feedforward network (3×3 convolution, dilated convolution, and 5×5 convolution in parallel) is combined to improve feature diversity. The Unet architecture point cloud segmentation model extracts Laplacian pyramid features from dense depth maps, encodes them via convolution, and injects them into point cloud features through cross-attention to supplement semantic information and achieve feature enhancement. The Unet architecture point cloud segmentation model replaces traditional convolution with Transformer blocks to adapt to the characteristics of point clouds, and the encoder and decoder are connected via cross-attention skipping to introduce the semantics of dense depth maps to enhance multi-level features, thereby better achieving LiDAR point cloud segmentation.

[0038] This application significantly improves the point cloud segmentation accuracy in complex scenes by deeply fusing LiDAR and camera data and combining them with an improved Transformer architecture, making it suitable for fields such as autonomous driving and 3D reconstruction. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A flowchart illustrating a LiDAR point cloud analysis method based on transformer, provided for an embodiment of the present invention;

[0042] Figure 2 This is a flowchart for obtaining a sparse depth map provided in an embodiment of the present invention;

[0043] Figure 3 A flowchart for camera intrinsic parameter encoding provided in an embodiment of the present invention;

[0044] Figure 4 An architecture diagram of the Unet architecture point cloud segmentation model provided in an embodiment of the present invention;

[0045] Figure 5 An architecture diagram of the Unet architecture point cloud segmentation model provided in an embodiment of the present invention;

[0046] Figure 6 A flowchart of the Transformer block in the Unet architecture point cloud segmentation model provided in this embodiment of the invention;

[0047] Figure 7 This is a schematic diagram of a transformer-based lidar point cloud analysis device provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0050] Example 1

[0051] like Figure 1 As shown, the present invention provides a transformer-based method for analyzing lidar point clouds. The invention comprises the following steps:

[0052] The S100 simultaneously acquires scene lidar point clouds and RGB images through lidar and camera, and performs filtering processing on the lidar point clouds to remove noise or irrelevant data.

[0053] The S200 uses LiDAR point clouds to construct sparse depth maps of RGB images.

[0054] like Figure 2 As shown, the process of step S200 includes: S201, obtaining the spatial transformation relationship between the matching LiDAR point cloud and the RGB image based on the LiDAR and camera pose;

[0055] In the specific implementation process, the extrinsic parameters between the lidar and the camera are obtained through calibration, including: rotation matrix R and translation vector t, and the spatial transformation relationship matrix T between the lidar and the camera is defined.

[0056]

[0057] S202, transform the LiDAR point cloud to the coordinate system of the RGB image according to the spatial transformation relationship;

[0058] The homogeneous coordinates of a point cloud in the lidar coordinate system are represented as P LiD =[x,y,z,1] T Its corresponding camera coordinate system coordinates are: P Cam =TP LiD =R[x,y,z] T +t.

[0059] S203 uses camera intrinsic parameters to project the coordinate-transformed LiDAR point cloud onto the RGB image space to obtain the corresponding pixel coordinates, thereby establishing the relationship between pixel coordinates and LiDAR point cloud.

[0060] Camera intrinsic parameter matrix:

[0061]

[0062] Among them, f xN ,f yN For the camera capturing RGB images, c represents the focal length along the x and y axes. xN ,c yN The coordinates of the main point.

[0063] Using camera intrinsic parameters, the lidar point P in the camera coordinate system is... Cam Projected onto RGB image pixel coordinates (u N ,v N ):

[0064]

[0065] Among them, P Cam,X ,P Cam,Y ,P Cam,Z These are the lidar points P in the camera coordinate system. Cam The xyz coordinates.

[0066] S204: The projected LiDAR point cloud is given partial pixel coordinate depth information of the RGB image to obtain a sparse depth map.

[0067] By using sensor calibration parameters (camera intrinsics, relative attitude) to project the point cloud onto the image coordinate system, sparse depth information is given to the RGB image, and a pixel-to-LiDAR point cloud association is established to form a sparse depth map.

[0068] S300, a pre-trained depth estimation model, converts the RGB image of a scene into a dense depth map based on camera intrinsics, the relative pose of the camera acquiring the RGB image, and a sparse depth map.

[0069] In specific implementation, the depth estimation model receives RGB images I1 and I2 from two adjacent frames of a scene from any dynamic camera, or RGB images I1 and I2 from any two relatively fixed cameras at the same moment of the scene. The depth estimation model receives conditional information. For any dynamic camera, the conditional information includes camera intrinsic parameters corresponding to the two adjacent RGB images I1 and I2, the relative pose of the two adjacent RGB images I1 and I2, and the sparse depth map corresponding to the two adjacent RGB images I1 and I2. For any two relatively fixed cameras, the conditional information includes camera intrinsic parameters of the two cameras, the relative pose of the two cameras, and the sparse depth map corresponding to the RGB images I1 and I2 captured by the two cameras.

[0070] In the specific implementation process, such as Figure 4 As shown, the depth estimation model includes a camera intrinsic encoder, a sparse depth map encoder, a relative pose encoder, and a DUSt3R backbone network. The working principles of each component are as follows:

[0071] The camera intrinsic encoder combines camera intrinsics with RGB image I1 and RGB image I2 to achieve camera intrinsic encoding, such as... Figure 3 As shown, the specific encoding process is as follows:

[0072] The camera intrinsic encoder calculates the normalized ray direction for each pixel in each RGB image based on the camera intrinsics:

[0073]

[0074] Where u1, v1 are the coordinates of any pixel in RGB image I1, and K1 is the camera intrinsic parameter of the camera that acquired RGB image I1. u2, v2 are the coordinates of any pixel in RGB image I2, and K2 is the camera intrinsic parameter of the camera that acquired RGB image I2.

[0075] The normalized ray directions of all pixels form a ray coding map, which is aligned with the RGB image. This ray coding map is then embedded to obtain a camera intrinsic token. This camera intrinsic token is then mapped through a linear layer to the visual Transformer encoder of the DUSt3R backbone network, where it is combined with the image token of the RGB image. The image token originates from the image embedding of the DUSt3R backbone network.

[0076] The sparse depth map encoder obtains a sparse depth token by embedding the sparse depth map. The sparse depth token is then mapped through a linear layer to the visual Transformer encoder of the DUSt3R backbone network and combined with the image token of the RGB image.

[0077] The relative pose encoder flattens the rotation matrix of the relative pose and concatenates it with the normalized translation matrix to obtain the relative pose vector. After mapping the relative pose vector through a multilayer perceptron, it is incorporated into the global context query in the decoding process of the visual Transformer decoder of the DUSt3R backbone network.

[0078] In the process of encoding RGB images, the visual Transformer encoder of the DUSt3R backbone network introduces camera intrinsic tokens and sparse depth tokens as guiding conditions. The guiding conditions provide spatial information missing in the RGB images to the image encoding process, so that the image encoding features generated by the visual Transformer encoder of the DUSt3R backbone network are fused with spatial information that conforms to the actual situation.

[0079] A learnable global context query is constructed for all embedded image codes. This global context query learns the global associations between all embeddings. The image codes and the global context query are combined and input into the visual Transformer decoder of the DUST3R backbone network. During the decoding of the image codes, the relative pose vectors are combined with the global context query through the mapping result of the multilayer perceptron. The mapping result of the relative pose vectors is introduced as a global condition into the global context query, ensuring that the generated dense depth map can incorporate the camera's relative pose information from the relative pose vectors. This guides the visual Transformer decoder of the DUST3R backbone network to more accurately model relative position information when performing cross-attention.

[0080] The downstream task head of the DUSt3R backbone network generates corresponding dense depth maps and confidence maps based on the decoding features provided by the visual Transformer decoder.

[0081] The loss function for training the depth estimation model is the sum of the confidence-weighted distance and confidence entropy between the predicted dense depth map and the true depth map. The parameters of the depth estimation model are adjusted with the goal of minimizing the loss function.

[0082] The depth estimation model implements multi-condition encoding, utilizing a dedicated encoder to process camera intrinsics and generate a ray-coded map. The sparse depth map is embedded as a sparse depth token, and the relative pose vector is vectorized and mapped to conditions via a multilayer perceptron. The image token and conditional information (camera intrinsics, depth token) are fused in the visual Transformer encoder of the DUST3R backbone network. During decoding, pose conditions are introduced into the global query, and relative position is modeled through cross-attention to generate a dense depth map and confidence score. Furthermore, training combines confidence-weighted depth error with entropy minimization to optimize the reliability and accuracy of depth estimation.

[0083] After obtaining the dense depth map, S400 provides the matched dense depth map and LiDAR point cloud to the pre-trained Unet architecture point cloud segmentation model based on Transformer blocks. The Unet architecture point cloud segmentation model supplements the dense depth map features into the LiDAR point cloud features to achieve LiDAR point cloud segmentation.

[0084] like Figure 5 As shown, the Unet architecture point cloud segmentation model uses Transformer blocks to replace the original convolutional blocks in its encoder and decoder, and sets cross-attention on the jump chains between the encoder and decoder.

[0085] The laser radar point cloud is embedded and encoded, and then used as the input to the encoder of the Unet architecture point cloud segmentation model.

[0086] The Transformer block includes a multi-scale deformable sliding window self-attention network and a multi-scale feedforward network. For example... Figure 6 As shown, the working principle of the multi-scale deformable sliding window self-attention mechanism, designed to adapt to the characteristics of lidar point clouds, is as follows:

[0087] The multi-scale deformable sliding window self-attention method uses a multi-scale deformable offset network to predict the deformable offset of each position of the local feature map within the region defined by each scale sliding window; the deformable offset network is a convolutional network.

[0088] Adding the corresponding deformable offset to each location in the local feature map yields the target sampling location in the global feature map;

[0089] Features are sampled from all target sampling locations; the sampled features are then stitched together according to their locations to form a new local feature map;

[0090] By introducing deformable offsets, multi-scale deformable sliding window self-attention support adaptively adjusts the receptive field of the sliding window at each scale, allowing attention to be focused on relevant areas outside the sliding window regions at various scales.

[0091] The multi-scale deformable sliding window self-attention constructs a corresponding multi-scale attention head for each scale sliding window, and any multi-scale attention head performs local context modeling on the deformable local feature map of its corresponding scale receptive field.

[0092] The global attention results, which are modeled by local contexts through different receptive fields, are concatenated, normalized, and then provided to the multi-scale feedforward network. The multi-scale feedforward network maps the feature maps through linear layers and then divides them into three parts. The three parts of the feature maps are processed by parallel 3×3 convolutional layers, 3×3 dilated convolutional layers, and 5×5 convolutional layers, respectively. Finally, the three sets of feature maps are fused through linear layer mapping.

[0093] To achieve the effect of enriching the LiDAR point cloud features with dense depth map features, the Laplacian pyramid of the dense depth map is obtained. The pyramid feature map of the Laplacian pyramid is then used to extract semantic features through convolution. The semantic features are then combined into the skip chains of the corresponding layers through cross attention.

[0094] The Unet architecture point cloud segmentation model decodes and combines the semantic features of dense depth maps with the features of LiDAR point clouds. Finally, it obtains the LiDAR point cloud segmentation mask through convolutional block mapping, thus achieving LiDAR point cloud segmentation.

[0095] The loss function used to train the Unet architecture point cloud segmentation model is the sum of cross-entropy loss and cross-union ratio loss between the predicted LiDAR point cloud segmentation mask and the actual LiDAR point cloud segmentation results.

[0096] The Unet architecture point cloud segmentation model's multi-scale deformable attention mechanism supports adaptive receptive field: it dynamically adjusts the sampling position of sliding windows at each scale through a deformable offset network to capture local and cross-regional context. The computational efficiency of the multi-scale deformable attention mechanism is optimized: it utilizes multi-scale sliding windows for local attention computation, reducing Transformer complexity and making it suitable for large-scale point cloud processing. Different scale attention heads in the multi-scale deformable attention mechanism support parallel multi-scale processing. To accommodate the concatenation of attention head results from different scales, a multi-scale feedforward network (3×3 convolution, dilated convolution, and 5×5 convolution in parallel) is combined to enhance feature diversity. The Unet architecture point cloud segmentation model extracts Laplacian pyramid features from dense depth maps, encodes them via convolution, and then injects them into point cloud features through cross-attention to supplement semantic information and enhance features for better LiDAR point cloud segmentation.

[0097] Example 2

[0098] See Figure 7 As shown, this embodiment of the invention provides a transformer-based lidar point cloud analysis device, comprising: at least one processing unit, wherein the processing unit, storage unit, and acquisition unit are interconnected via a bus unit. The storage unit, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the software program, computer-executable program, and module corresponding to a transformer-based lidar point cloud analysis method in this embodiment of the invention. The processing unit implements the aforementioned transformer-based lidar point cloud analysis method by running the software program, computer-executable program, and module stored in the storage unit, including:

[0099] The S100 simultaneously acquires scene lidar point clouds and RGB images through lidar and camera, and performs filtering processing on the lidar point clouds.

[0100] S200 uses LiDAR point clouds to construct sparse depth maps of RGB images;

[0101] S300, the pre-trained depth estimation model converts the RGB image of the scene into a dense depth map based on camera intrinsics, the relative pose of the camera that acquired the RGB image, and the sparse depth map.

[0102] After acquiring the dense depth map, S400 provides the matched dense depth map and LiDAR point cloud to the Unet architecture point cloud segmentation model based on Transformer blocks. The Unet architecture point cloud segmentation model supplements the dense depth map features into the LiDAR point cloud features to achieve LiDAR point cloud segmentation.

[0103] The acquisition unit includes a lidar and a camera.

[0104] Of course, the computer program stored in the storage unit of the transformer-based lidar point cloud analysis device provided in the embodiments of the present invention is not limited to the method operation described above, but can also execute related operations in the transformer-based lidar point cloud analysis method provided in any embodiment of the present invention.

[0105] Example 3

[0106] This invention provides a computer-readable storage medium storing a computer program. When executed, the computer program implements the transformer-based lidar point cloud analysis method, including:

[0107] The S100 simultaneously acquires scene lidar point clouds and RGB images through lidar and camera, and performs filtering processing on the lidar point clouds.

[0108] S200 uses LiDAR point clouds to construct sparse depth maps of RGB images;

[0109] S300, the pre-trained depth estimation model converts the RGB image of the scene into a dense depth map based on camera intrinsics, the relative pose of the camera that acquired the RGB image, and the sparse depth map.

[0110] After acquiring the dense depth map, S400 provides the matched dense depth map and LiDAR point cloud to the Unet architecture point cloud segmentation model based on Transformer blocks. The Unet architecture point cloud segmentation model supplements the dense depth map features into the LiDAR point cloud features to achieve LiDAR point cloud segmentation.

[0111] The computer-readable storage medium provided in the embodiments of the present invention stores a computer program that is not limited to the method operation described above, but can also execute related operations in the LiDAR point cloud analysis method based on transformer provided in any embodiment of the present invention.

[0112] In the embodiments provided by this invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, structures, or units, and may be electrical, mechanical, or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0115] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A transformer-based lidar point cloud analysis method, characterized in that, The method comprises the following steps: S100, synchronously collecting scene laser radar point cloud and RGB image by laser radar and camera, and filtering the laser radar point cloud; S200, constructing a sparse depth map of the RGB image by using the laser radar point cloud; S300, pre-training a depth estimation model to convert the RGB image of the scene into a dense depth map under the condition of camera intrinsic parameters, relative pose of the camera for collecting the RGB image and the sparse depth map; The depth estimation model comprises a camera intrinsic encoder, a sparse depth map encoder, a relative pose encoder and a DUSt3R backbone network; the camera intrinsic encoder combines camera intrinsic parameters with an RGB image and an RGB image to realize camera intrinsic encoding; the sparse depth map encoder realizes sparse depth map encoding; The relative pose encoder realizes relative pose encoding; The camera intrinsic parameter encoder encodes as follows: Calculate the normalized light direction corresponding to each pixel of each RGB image according to the camera intrinsic parameters: ， ； in, RGB image The coordinates of any pixel position in the image. To acquire RGB images The camera's internal parameters; RGB image The coordinates of any pixel position in the image. To acquire RGB images The camera's internal parameters; The normalized light directions of all pixels form a light encoding map, which is aligned with the RGB image; the light encoding map is embedded and encoded to obtain a camera intrinsic parameter token; the camera intrinsic parameter token is mapped to the visual Transformer encoder of the DUSt3R backbone network through a linear layer, and is combined with the image token of the RGB image, which is obtained from the image embedding of the DUSt3R backbone network; The sparse depth map encoder embeds and encodes the sparse depth map to obtain a sparse depth token, which is mapped to the visual Transformer encoder of the DUSt3R backbone network through a linear layer, and is combined with the image token of the RGB image, which is obtained from the image embedding of the DUSt3R backbone network; The relative pose encoder flattens and normalizes the rotation matrix of the relative pose, and splices the translation matrix to obtain a relative pose vector, which is mapped through a multi-layer perceptron and combined into the global context query in the decoding process of the visual Transformer decoder of the DUSt3R backbone network; S400, after obtaining the dense depth map, the matched dense depth map and laser radar point cloud are provided to a point cloud segmentation model based on the Unet architecture of the Transformer block, and the Unet architecture of the point cloud segmentation model supplements the features of the dense depth map into the features of the laser radar point cloud to realize laser radar point cloud segmentation.

2. The transformer-based lidar point cloud analysis method of claim 1, wherein, The method for constructing a sparse depth map of an RGB image by using a laser radar point cloud comprises the following steps: S201, obtaining the spatial conversion relationship between the matched laser radar point cloud and the RGB image according to the laser radar and the camera pose; S202, converting the laser radar point cloud to the coordinate system of the RGB image according to the spatial conversion relationship; S203, projecting the laser radar point cloud converted by the coordinate into the RGB image space by using the camera intrinsic parameters to obtain corresponding pixel coordinates, thereby establishing the relationship between the pixel coordinates and the laser radar point cloud; 3. The transformer-based lidar point cloud analysis method of claim 1, wherein, S204, assigning the laser radar point cloud after projection with the depth information of part of the pixel coordinates of the RGB image to obtain a sparse depth map. The loss function for training the depth estimation model is the sum of the confidence weighted distance and the confidence entropy between the predicted dense depth map and the real depth map, and the parameters of the depth estimation model are adjusted to minimize the loss function.

4. The transformer-based lidar point cloud analysis method of claim 1, wherein, The encoder-decoder of the Unet architecture point cloud segmentation model replaces the original convolution block with a Transformer block, and a cross attention is arranged on the skip link between the encoder-decoder, which fuses the semantic features of the dense depth map into the skip link through the cross attention, and the decoder of the Unet architecture point cloud segmentation model decodes the laser radar point cloud features combined with the semantic features of the dense depth map, finally maps the laser radar point cloud segmentation mask through the convolution block to realize the laser radar point cloud segmentation.

5. The transformer-based lidar point cloud analysis method of claim 4, wherein, The Transformer block comprises a multi-scale deformable sliding window self-attention and a multi-scale feedforward network; The multi-scale deformable sliding window self-attention works as follows: The multi-scale deformable sliding window self-attention predicts the deformable offset of each position of the local feature map in the region defined by each scale sliding window by using a multi-scale deformable offset network; Each position of the local feature map plus its corresponding deformable offset obtains a target sampling position in the global feature map; Features sampled from all target sampling positions are spliced into a new local feature map according to positions; By introducing the deformable offset, the multi-scale deformable sliding window self-attention supports adaptively adjusting the receptive field of each scale sliding window, allowing attention to be focused on related regions outside the sliding window region of each scale; The multi-scale deformable sliding window self-attention constructs a corresponding multi-scale attention head for each scale sliding window, and any multi-scale attention head locally models the context of the deformable local feature map in its corresponding scale receptive field.

6. The transformer-based lidar point cloud analysis method of claim 5, wherein, The global attention results locally modeled by different receptive fields are spliced and normalized to provide the multi-scale feedforward network, which is divided into three parts after mapping the feature map through a linear layer, and the three parts of features are processed through two convolution layers with different kernel sizes and expansion layer convolutions in parallel, and then the three groups of features are fused through a linear layer mapping.

Images