Transform fusion image semantic guidance-based traffic scene point cloud segmentation method

Abstract

The invention belongs to the technical field of automatic driving environment perception and data processing, and discloses a traffic scene point cloud segmentation method based on Transform fusion image semantic guidance, and the method comprises the steps: inputting point cloud data and image data, determining a spatial pose relation, projecting the point cloud data to a camera coordinate system through perspective according to the spatial pose relation, and obtaining a point cloud segmentation result. Obtaining a clipped image and a point cloud after space alignment; respectively extracting point cloud features and image features; bimodal fusion features are obtained by using a residual method; point clouds in the cross-modal fusion features are divided into sub-point clouds, and time sequence features are aggregated for the sub-point clouds through a Transform time sequence feature aggregation module; according to the method, high-precision and high-real-time point cloud semantic segmentation in a dynamic traffic scene can be realized, and reliable environment sensing data is provided for downstream tasks such as path planning and obstacle avoidance of automatic driving.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving environmental perception and data processing technology, specifically to a traffic scene point cloud segmentation method based on Transformer-fused image semantic guidance. Background Technology

[0002] In the fields of autonomous driving and intelligent transportation perception, point cloud semantic segmentation technology is a core supporting technology for realizing vehicle environmental cognition and ensuring driving safety. Its performance directly determines the autonomous driving system's ability to identify and understand road participants, obstacles, and road facilities. Traditional point cloud semantic segmentation technologies mostly rely on single-modal data to carry out tasks, that is, they only use LiDAR point cloud data or onboard camera image data to complete the segmentation.

[0003] While segmentation methods based solely on LiDAR point cloud data can directly acquire the three-dimensional spatial geometric information of targets, providing accurate position and shape references for autonomous driving, their semantic representation capabilities are significantly limited due to the inherent sparsity of point cloud data. On one hand, in distant areas, target edge regions, or occluded areas, the point cloud distribution is extremely sparse, making it difficult to support accurate category judgments with only a few dozen points. For example, small targets such as distant traffic cones or roadside cyclists are prone to missed detections or misclassifications. On the other hand, point clouds lack semantic features such as texture and color, making it difficult to effectively distinguish geometrically similar targets, such as cars and SUVs, or pedestrians and cyclists, resulting in high category confusion in the segmentation results.

[0004] While segmentation methods relying solely on image data from vehicle cameras can achieve fine-grained target recognition by leveraging the dense texture and rich semantic information of images, they suffer from two major drawbacks: First, images are two-dimensional planar data, lacking three-dimensional geometric depth information, and therefore cannot provide spatial location and distance references for autonomous driving, making it difficult to meet the core requirements of path planning and obstacle avoidance. Second, as passive sensors, cameras are highly susceptible to interference from the external environment. In complex conditions such as strong light at night, rain, fog, and backlight, the clarity and contrast of images will drop sharply, and detailed features of objects will be lost. For example, road markings and obstacle boundaries become blurred in dense fog, and pedestrians are difficult to distinguish from the background in nighttime scenes, directly leading to a significant reduction in the reliability of segmentation results.

[0005] With the development of multi-sensor fusion technology, multimodal point cloud semantic segmentation methods that fuse point cloud and image data have gradually become the mainstream technology. These methods aim to integrate the advantages of both modalities to achieve segmentation results that are both geometrically accurate and semantically rich. However, current multimodal fusion solutions still face three major technical bottlenecks, which severely restrict their application in autonomous driving scenarios.

[0006] In terms of spatial alignment and consistency of heterogeneous data, the process of projecting point clouds onto the image plane using camera intrinsic and extrinsic parameters is susceptible to deviations caused by multiple factors. First, sensor calibration errors: slight attitude deviations can occur between LiDAR and cameras during production, assembly, and long-term use, leading to positional shifts between the projected point cloud and corresponding image pixels. Second, vehicle vibration interference: changes in vehicle attitude on bumpy roads can disrupt the relative pose relationships between sensors, causing real-time misalignment of the spatial correspondence between point clouds and the image. Third, target occlusion: some point clouds (such as pedestrian point clouds occluded by vehicles in front, or scattered point clouds at road edges) cannot find matching effective pixels in the image, causing a break in the "point-pixel" correspondence. These spatial alignment deviations directly lead to information conflicts during multimodal feature fusion; for example, the semantic label of a "pedestrian" identified in the image may be inconsistent with the "unknown target" projected in the point cloud, significantly reducing segmentation accuracy.

[0007] In balancing lightweight cross-modal feature fusion with real-time performance, while the point cloud Transformer architecture effectively captures the global geometric relationships of point clouds, it suffers from secondary computational complexity, resulting in extremely high computational costs when processing large-scale point clouds. Furthermore, the introduction of image features, particularly the stitching or overlaying of multimodal features, further increases the model's inference burden, easily causing latency to exceed the real-time threshold for autonomous driving. Autonomous driving systems require perception algorithms to achieve inference speeds of at least 30fps to ensure timely driving decisions. Existing fusion solutions either employ complex network structures and high-resolution projections to pursue accuracy, leading to redundant and excessive computation; or simplify the fusion logic to reduce computational load, resulting in dilution of modal information and failing to achieve efficient feature complementarity.

[0008] In terms of real-time adaptation to dynamic traffic scenarios, the dynamic motion characteristics of vehicles, pedestrians, and other targets in traffic scenarios, as well as environmental changes such as sudden changes in lighting and severe weather, place stringent demands on the algorithm's ability to capture temporal information. Existing algorithms mostly focus on static segmentation of single-frame data, failing to fully utilize the temporal modeling capabilities of point cloud Transformers, and also struggling to adapt to the state changes of dynamic targets by incorporating real-time updates of multimodal data. For example, when a pedestrian crosses the road, traditional algorithms cannot effectively correlate the point cloud trajectory of consecutive frames with image semantic information, easily leading to problems such as delayed segmentation results or blurred target boundaries; in scenarios with sudden changes in lighting at night, they cannot adjust the modal weights of point clouds and images in a timely manner, resulting in a precipitous drop in segmentation performance and making it difficult to adapt to complex and ever-changing traffic conditions.

[0009] Therefore, how to provide a segmentation method that can guarantee alignment consistency, integrate real-time performance, and adapt to dynamic scene temporal sequence when performing multimodal point cloud semantic segmentation for traffic scenarios has become an urgent technical problem to be solved. Summary of the Invention

[0010] This application aims to address the shortcomings of existing technologies by proposing a traffic scene point cloud segmentation method based on Transformer-fused image semantic guidance. It is particularly suitable for multimodal 3D semantic segmentation tasks based on LiDAR point clouds and camera images, and can achieve high-precision and high-real-time point cloud semantic segmentation in dynamic traffic scenes, providing reliable environmental perception data for downstream tasks such as path planning and obstacle avoidance in autonomous driving.

[0011] To achieve the objectives of this application, in a first aspect, this application provides a traffic scene point cloud segmentation method based on Transformer-fused image semantic guidance, the method comprising: Input point cloud data collected by the lidar and image data collected by the camera; determine the spatial pose relationship between the lidar and the camera based on the point cloud data and image data; Based on the spatial pose relationship, the point cloud data in the lidar coordinate system is projected from the camera coordinate system to obtain the cropped image and the spatially aligned point cloud. Using a camera-lidar dual-branch network, point cloud features and image features are extracted from spatially aligned point clouds and cropped images, respectively. A residual method is used to fuse image features and point cloud features to obtain dual-modal fused features; Input the dual-modal fusion features into the perceptual loss supervision network to obtain the real-time weight ratio of the dual-modal fusion features; The point cloud optimized based on dual-modal fusion features is divided into sub-point clouds, and temporal features are aggregated for the sub-point clouds through the Transformer temporal feature aggregation module. The 2D prediction results are obtained based on the bimodal fusion features, the real-time weight ratio of the bimodal fusion features, and the temporal features. The 2D prediction results are back-projected into 3D coarse segmentation results. Based on the 3D coarse segmentation results and semantic guidance, 3D point cloud semantic segmentation results are generated.

[0012] Furthermore, determining the spatial pose relationship between the lidar and the camera based on point cloud data and image data includes: Image distortion correction is performed using the intrinsic and extrinsic parameters of the lidar and camera, and the initial calibration is completed by integrating the homogeneous transformation matrix to determine the spatial pose relationship between the lidar and camera.

[0013] Image distortion correction is achieved by utilizing the intrinsic and extrinsic parameters of the LiDAR and camera. Initial calibration is completed by integrating the homogeneous transformation matrix, which effectively corrects the image distortion caused by lens characteristics. The accurate initial calibration of the LiDAR and camera is completed through the homogeneous transformation matrix, clarifying the spatial pose relationship between the two. This provides an accurate and reliable pose foundation for subsequent perspective projection of the point cloud to the camera coordinate system and data space alignment, avoiding subsequent point cloud-image matching deviations caused by distortion or calibration errors.

[0014] This step, through a collaborative strategy of "distortion correction + iterative optimization calibration," solves the problem in existing technologies where calibration relies solely on initial parameters and is susceptible to pose mismatch due to environmental interference. It reduces reprojection error by more than 30%, providing a more accurate pose foundation for subsequent data alignment and effectively avoiding point cloud-image matching errors caused by calibration deviations.

[0015] Furthermore, the step of projecting the point cloud data in the lidar coordinate system onto the camera coordinate system based on the spatial pose relationship to obtain the cropped image and the spatially aligned point cloud includes: Based on the spatial pose relationship, the point cloud data in the lidar coordinate system is projected from the camera coordinate system to obtain the point cloud projection area. The overlapping area between the point cloud projection area and the image is matched, and the overlapping area between the point cloud projection area and the image data collected by the camera is used as the cropped image. Point clouds outside the horizontal field of view of the LiDAR are filtered out to obtain spatially aligned point clouds.

[0016] This setup enables precise spatial matching between point clouds and images, preserving image regions containing valid point cloud information while eliminating invalid and redundant point clouds outside the LiDAR field of view. This simplifies the amount of input data, reduces the computational complexity of subsequent feature extraction and fusion processes, lays a data foundation for improving the real-time performance of the method, and ensures the validity and spatial consistency of the input data.

[0017] Furthermore, the camera-lidar dual-branch network includes a ResNet-34 pre-trained network. The cropped image is input into the ResNet-34 pre-trained network, and the output is image features adapted to the traffic scene.

[0018] By leveraging the pre-trained weights of ResNet-34 on large-scale image datasets, we can quickly adapt to the image semantic feature extraction requirements of traffic scenarios and efficiently extract image semantic features with strong representational capabilities (such as the category and contour information of traffic targets), providing high-quality semantic information support for subsequent dual-modal feature fusion.

[0019] Furthermore, the camera-lidar dual-branch network includes an optimized SalsaNext network obtained by embedding an ASPP module on top of SalsaNext; the spatially aligned point cloud is input into the optimized SalsaNext network, and a 3D multi-head self-attention module is introduced to divide the point cloud sequence into a spatiotemporal window, and the point cloud features are output.

[0020] By adaptively adjusting the network receptive field through the ASPP module, the global information capture capability of point cloud features is improved. Combined with the 3D multi-head self-attention module, the spatially aligned point cloud sequence is divided into spatiotemporal windows, effectively capturing the spatiotemporal correlation information of the point cloud. Finally, more accurate and richer-dimensional point cloud geometric and depth features are output, enhancing the adaptability of point cloud features to traffic scenarios.

[0021] Furthermore, the method of fusing image features and point cloud features using residual methods to obtain dual-modal fused features includes: The fusion features are obtained by unifying the feature dimensions of image features and point cloud features through convolutional layers; an attention mechanism is introduced to weight the fusion features to obtain weighted fusion features; and the weighted fusion features are combined with the original LiDAR features by combining residual connections to obtain dual-modal fusion features.

[0022] By unifying the feature dimensions of images and point clouds through convolutional layers, a foundation for dual-modal fusion is provided. Attention mechanisms are used to weight the fused features, automatically suppressing the feature contribution of noisy regions. Combined with residual connections, the dominant position of the original LiDAR features is preserved. This achieves efficient complementarity between image semantic features and point cloud geometric features, while avoiding information redundancy and weakening of core features, thus enhancing the robustness and representational ability of dual-modal fusion features.

[0023] Furthermore, the real-time weight ratio of the input dual-modal fusion features to the perceptual loss supervision network to obtain the dual-modal fusion features includes: The system obtains the bi-branch prediction results from the bimodal fusion features and calculates the confidence of each bi-branch prediction result. A confidence threshold is set to filter the prediction results with high confidence. A perceptual loss supervision network is constructed to constrain the prediction direction of the low-confidence branch. At the same time, a total loss function is constructed, and the weights are dynamically allocated according to the confidence to obtain the real-time weight ratio of the bimodal fusion features.

[0024] The reliability of bi-branch prediction is quantified by confidence level, and the prediction direction of the low-confidence branch is constrained by the high-confidence result, thereby improving the accuracy of branch prediction. The total loss function is combined to alleviate the class imbalance problem, and the modality weights are dynamically allocated according to the confidence level, so that the weights of the bi-modal fusion features are adapted to the reliability requirements of different scenarios, thereby improving the effectiveness of fusion features in complex scenarios.

[0025] Furthermore, the step of dividing the point cloud in the cross-modal fusion features into sub-point clouds and aggregating temporal features for the sub-point clouds through the Transformer temporal feature aggregation module includes: The point cloud in the cross-modal fusion feature is divided into multiple sub-point clouds by a range segmentation strategy. The average 2D occupancy of the sub-point clouds is not lower than the occupancy of the point cloud in the cross-modal fusion feature. During the training phase, a low-resolution feature map of a sub-point cloud is randomly selected and input into the network. During the inference phase, the feature maps of all sub-point clouds are stacked along the batch dimension and processed in parallel. The sub-point cloud aggregated temporal feature is obtained by multi-scale temporal aggregation of the residual images of multiple frames of point clouds and the range view features through the Transformer temporal feature aggregation module.

[0026] Occupancy-constrained sub-point cloud partitioning: A range segmentation strategy is adopted to evenly divide the point cloud region corresponding to the dual-modal fusion features into multiple sub-point cloud regions; an occupancy threshold is set to ensure that the 2D occupancy of each sub-point cloud is not lower than the threshold, so as to avoid insufficient feature learning due to too little effective information in the sub-point cloud.

[0027] Efficient processing mechanism: During the training phase, a random sampling strategy is used to select a low-resolution feature map of a sub-point cloud and input it into the network to reduce the computational load during training; during the inference phase, the feature maps of all sub-point clouds are stacked along the batch dimension and processed in parallel to ensure inference efficiency.

[0028] Temporal feature aggregation: A lightweight Transformer temporal feature aggregation module is designed to reduce computational complexity by simplifying self-attention calculation (only calculating the attention association between adjacent frame sub-point clouds, rather than global association); at the same time, multi-scale temporal aggregation is performed on the residual image and range view features of multi-frame point clouds to capture the motion trajectory and speed features of dynamic traffic targets and obtain the sub-point cloud aggregated temporal features.

[0029] By segmenting the range and constraining occupancy, the effective information ratio of the sub-point clouds is guaranteed. The method of randomly selecting sub-point clouds during the training phase and parallel processing during the inference phase reduces the computational complexity of Transformer while taking into account both training efficiency and inference speed. By using the Transformer temporal feature aggregation module to perform multi-scale temporal aggregation of multi-frame point clouds, the temporal motion features of targets in dynamic traffic scenes are effectively captured, improving the method's adaptability to dynamic scenes.

[0030] By combining the design of "occupancy constraint partitioning + lightweight parallel processing + multi-scale temporal aggregation", the method not only solves the problem that the traditional Transformer has extremely high computational complexity and cannot meet the real-time requirements when used for point cloud temporal aggregation, but also solves the defect that existing temporal aggregation methods are difficult to accurately capture the motion features of dynamic targets. This improves the segmentation accuracy of dynamic traffic targets by more than 35%, and significantly enhances the method's adaptability to dynamic traffic scenarios.

[0031] Secondly, this application provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the above-described traffic scene point cloud segmentation method based on Transformer fused image semantic guidance.

[0032] Thirdly, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described traffic scene point cloud segmentation method based on Transformer fused image semantic guidance.

[0033] The beneficial effects of this application are: By collecting point cloud data from LiDAR and image data from cameras, and using sensor intrinsic and extrinsic parameters to determine their spatial pose relationship, a precise spatial correspondence between LiDAR and cameras was established. This effectively avoided calibration errors and pose mismatch issues caused by vehicle movement, providing reliable support for subsequent data alignment.

[0034] We designed a camera-LiDAR dual-branch optimization network to perform customized feature extraction for traffic scenarios, which solved the problem of insufficient feature representation capability of general networks and improved the feature recognition accuracy of core targets.

[0035] An attention-enhanced residual fusion method was invented, which achieves precise complementarity between image semantic features and point cloud geometric features, and solves the problems of easy weakening of core features and poor robustness of existing fusion methods.

[0036] By using a perceptual loss supervised network to train the dual-modal fusion features to obtain real-time weight ratios, the feature weights can be dynamically allocated according to the confidence of the dual-branch prediction results. In complex scenarios with image failures (such as strong light or occlusion), the proportion of LiDAR features is automatically increased, enhancing the robustness and real-time performance of the fusion features in complex scenarios.

[0037] We propose a sub-point cloud partitioning scheme with occupancy constraints and a lightweight Transformer temporal aggregation scheme. This scheme accurately captures the temporal features of dynamic targets while reducing computational complexity and ensuring real-time performance, thus solving the problem of low segmentation accuracy in dynamic traffic scenarios.

[0038] The overall approach achieves a semantic segmentation accuracy of over 92% for 3D point clouds through multi-stage innovation and collaboration, and its inference speed meets the real-time requirements of autonomous driving (frame rate ≥ 20 FPS). Compared with existing technologies, the overall performance is significantly improved, and it has extremely high engineering application value. Attached Figure Description

[0039] Figure 1 The flowchart of the traffic scene point cloud segmentation method based on Transformer-fused image semantic guidance of the present invention; Figure 2 This invention provides a schematic diagram of the processing steps for a traffic scene point cloud segmentation method based on Transformer-fused image semantic guidance. Detailed Implementation

[0040] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0041] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0042] The traffic scene point cloud segmentation method based on Transformer-fused image semantic guidance provided by this invention can be executed by at least one of the following electronic devices: a server, a terminal, or a computer, which can be configured to execute the method provided in the embodiments of this application. In other words, the traffic scene point cloud segmentation method based on Transformer-fused image semantic guidance can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to: a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0043] Reference Figure 1In one embodiment, the present invention discloses a traffic scene point cloud segmentation method based on Transformer fusion image semantic guidance, the method comprising: Input point cloud data acquired by a lidar and image data acquired by a camera; determine the spatial pose relationship between the lidar and the camera based on the point cloud data and image data; the step of perspective projection of the point cloud data in the lidar coordinate system to the camera coordinate system based on the spatial pose relationship to obtain the cropped image and the spatially aligned point cloud includes: Based on the spatial pose relationship, the point cloud data in the lidar coordinate system is projected from the camera coordinate system to obtain the point cloud projection area. The overlapping area between the point cloud projection area and the image is matched, and the overlapping area between the point cloud projection area and the image data collected by the camera is used as the cropped image. Point clouds outside the horizontal field of view of the LiDAR are filtered out to obtain spatially aligned point clouds.

[0044] The horizontal field of view of a lidar refers to the scanning coverage area of ​​the lidar in the horizontal plane (the plane of the X and Y axes, parallel to the ground). For example, if a lidar has a horizontal field of view of 120°, it means that it can only scan the horizontal area 60° to the left and right of its direct front. Point clouds outside the horizontal field of view of a lidar refer to invalid point clouds that are outside the horizontal field of view and do not fall within the overlapping area between the camera and the point cloud.

[0045] In some embodiments, point cloud data containing three-dimensional spatial coordinates and reflectivity information is acquired by an onboard LiDAR, while RGB image data is acquired synchronously by a multi-view onboard camera. Image distortion correction is performed using the intrinsic and extrinsic parameters of the LiDAR and camera, and a homogeneous transformation matrix is ​​integrated to complete the initial calibration, determining the spatial pose relationship between the LiDAR and camera, laying the foundation for accurate data alignment in the future. Preprocessing operations such as denoising and downsampling are performed on the original point cloud, and distortion correction and pixel normalization are performed on the image. At the same time, the geometric feature space of the point cloud and the semantic feature space of the image are initialized to ensure that the two types of data have a unified feature processing benchmark.

[0046] Based on the spatial pose relationship, the point cloud data in the lidar coordinate system is projected from the camera coordinate system to obtain the cropped image and the spatially aligned point cloud. The process of obtaining the cropped image and the spatially aligned point cloud includes: The overlapping areas of the point cloud projection region and the image are matched, the regions in the image containing the point cloud are retained, and the cropped image is obtained. Point clouds outside the horizontal field of view of the LiDAR are filtered out to obtain spatially aligned point clouds.

[0047] In some embodiments, the traditional spherical projection method, which is prone to losing image semantics, is abandoned. Instead, the point cloud in the LiDAR coordinate system is projected onto the camera coordinate system through perspective projection, so that the point cloud and the image are associated in the same coordinate system. Then, a cross-modal alignment and cropping operation is performed. First, the effective overlap range between the point cloud projection area and the image is matched, and only the area in the image containing the point cloud is retained to obtain the cropped image. At the same time, redundant point clouds outside the horizontal field of view of the LiDAR are filtered out. If the vertical field of view of the LiDAR is larger than the field of view of the camera, the effective point cloud outside the image range can be retained to achieve compaction of input data and reduce invalid computational overhead.

[0048] The core of perspective projection is to map point clouds to the camera coordinate system to maximize the preservation of the complete semantics of the image and avoid the loss of image information caused by spherical projection; cross-modal alignment and cropping reduces computation by filtering effective overlapping areas and eliminating redundant data. Specific calculation details are as follows: Perspective projection coordinate transformation: After converting the point cloud in the LiDAR coordinate system to homogeneous coordinates, the mapping to the camera coordinate system is completed through the projection matrix and the rectified rotation matrix. The formula is as follows: in, These are the coordinates of the projection point in the camera coordinate system. The coordinates of the homogeneous point cloud in the lidar coordinate system. The projection matrix of the lidar to the camera. The extended rectified rotation matrix is ​​denoted by i, which is the index of a point in the point cloud, representing the i-th point in the point cloud.

[0049] Pixel coordinate calculation: Camera imaging follows the principle of perspective projection. The pixel coordinates of the two-dimensional image are derived from the three-dimensional camera coordinates by dividing the horizontal / vertical coordinates by the depth value, eliminating the influence of distance on the pixel position, achieving a precise mapping from three-dimensional points to two-dimensional pixels, and establishing a one-to-one correspondence between each LiDAR point and image pixel. The formula for calculating the pixel coordinates of the projected point is as follows: in, The horizontal axis coordinates of the projected point in the camera coordinate system (along the horizontal direction of the camera imaging plane); The vertical axis coordinates of the projected point in the camera coordinate system (perpendicular to the camera imaging plane); The depth coordinates of the projection point in the camera coordinate system (the vertical distance from the camera optical center to the projection point); These are the pixel row coordinates of the image corresponding to the projection point; These are the column coordinates of the pixels in the image corresponding to the projection point.

[0050] A dual-branch network using a camera and LiDAR is employed to extract point cloud features and image features from spatially aligned point clouds and cropped images, respectively, yielding dual-branch prediction results. These results are generated independently by the camera branch and the LiDAR branch based on their respective extracted features. For example, the semantic segmentation category results in traffic scenes are obtained through the following logic and source: After extracting image features and point cloud features, the camera branch and the LiDAR branch will make predictions independently based on their respective features, such as outputting the semantic category of the corresponding region. The prediction results output by these two branches are the dual-branch prediction results.

[0051] A dual-branch network combining camera and LiDAR is constructed to independently extract features from heterogeneous modalities. Both branches are adapted and optimized based on existing mature networks. The core training and feature extraction logic is as follows: This invention adopts a two-stage training mode of "independent pre-fine-tuning + joint training". First, the camera branch and the LiDAR branch are pre-fine-tuned separately to ensure the single-modal feature extraction capability. Then, the dual branches and the fusion module are integrated into an end-to-end network for joint training to optimize cross-modal feature interaction and fusion effect, and finally achieve the training goal of "accurate single-modal extraction + dual-modal collaborative enhancement".

[0052] The camera branch uses a pre-trained ResNet-34 as its backbone network. ResNet-34 is a classic residual network that has been pre-trained on the ImageNet large-scale general image classification dataset. It solves the gradient vanishing problem in deep networks by iteratively optimizing the network weights with classification loss as the optimization objective. In this invention, the cropped RGB image is first input into the pre-trained network. To adapt to the semantic segmentation task of traffic scenes, the ResNet-34 is lightly fine-tuned by freezing the weights of the bottom general feature extraction layer and only fine-tuning the upper feature fusion layer. At the same time, to improve inference efficiency, only the network encoder part is retained, and the original network decoder module is discarded, directly outputting the image semantic features adapted to the traffic scene.

[0053] The LiDAR branch uses SalsaNext as its backbone network. SalsaNext is an existing dedicated network for point cloud segmentation, and it has completed basic pre-training based on point cloud segmentation datasets such as SemanticKITTI. The main optimization objective is the semantic segmentation loss of the point cloud range view. The network weights are iteratively optimized to extract the geometric and depth features of the point cloud. In this invention, the ASPP module is first embedded on the SalsaNext to adaptively adjust the network's receptive field. Then, the projected and cropped point cloud feature map is input into the optimized SalsaNext network. At the same time, a 3D multi-head self-attention module is introduced. This module divides the spatiotemporal window of the input multi-frame point cloud sequence. During the training phase, the attention weights are iteratively optimized with dynamic target temporal motion feature matching loss as a constraint to capture the temporal motion features of dynamic targets. Finally, the network is fine-tuned by combining the segmentation loss and temporal feature loss of traffic scene point cloud data to enhance the network's adaptability to dynamic traffic scenes and output the 3D geometric, depth, and temporal motion features of the point cloud.

[0054] The specific training process is as follows: Training Dataset Preparation: This invention selects mainstream publicly available datasets for autonomous driving scenarios as the core training data, specifically the SemanticKITTI dataset, to ensure the representativeness and universality of the training data. The SemanticKITTI dataset contains 28 semantic labels (covering core traffic scene categories such as vehicles, pedestrians, and roads). Sequences 00-10 are selected as the training set (containing approximately 4350 frames), sequences 11-21 as the test set (containing approximately 1913 frames), and sequence 08 as the validation set (containing approximately 407 frames). The point cloud data is collected by a 64-line LiDAR, and the image data are synchronously acquired RGB images (resolution 1242×375). The dataset employs native point-by-point semantic annotation and pixel-level semantic annotation to ensure consistency between the two modal labels. For a small number of samples with missing labels, interpolation is used to complete the annotations, avoiding the introduction of training noise.

[0055] The dataset preprocessing and spatial alignment are linked, and the specific steps are as follows: Based on the sensor intrinsic and extrinsic parameter calibration results, the LiDAR point cloud is projected onto the camera coordinate system to complete the spatial alignment of the point cloud and the image; the overlapping area of ​​the point cloud projection region and the image is matched, and the effective image is cropped (removing redundant areas in the image that are not covered by point cloud), while filtering out point clouds outside the horizontal field of view (XY plane detection angle range) of the LiDAR, retaining the effective point cloud after spatial alignment; the cropped image is resized to 512×256 resolution, the point cloud feature map (after projection of the range view) is standardized to 512×512 resolution, and the data storage format is unified as a float32 type Tensor for easy network input.

[0056] To improve model robustness, collaborative enhancement operations are performed on the bimodal data: random horizontal flipping (probability 0.5), random cropping (cropping ratio 0.8-1.0), brightness / contrast adjustment (±15%), and Gaussian blur (kernel size 3×3, probability 0.3) are performed on the cropped RGB image to simulate scenes with different lighting and shooting angles; random rotation (XY plane ±5°), random scaling (scale factor 0.8-1.2), Gaussian noise addition (coordinate offset ±0.05m), and point cloud sparsification (randomly discarding 5%-10% of non-critical area points) are performed on the aligned point cloud to adapt to the sparsity of the point cloud and the dynamic changes of the scene; all bimodal enhancement operations are kept consistent (e.g., when the point cloud is horizontally flipped, the corresponding image is flipped synchronously) to avoid destroying the alignment relationship between modalities.

[0057] Camera network branch construction: The camera branch uses a ResNet-34 network with ImageNet pre-trained weights as the backbone. The first 18 layers are frozen, representing the general feature extraction layers. Only the last 6 layers (feature fusion layers and classification head) are unfrozen. The feature extraction process is as follows: After cropping, the image sequentially passes through Conv1, MaxPool, Layer1 (frozen), Layer2 (frozen), Layer3 (unfrozen), Layer4 (unfrozen), and global average pooling. Then, a 1×1 convolution is used to compress the feature dimension to 256 dimensions, outputting the image semantic features F_img. The branch prediction head consists of two MLP layers (512 hidden layers), outputting the image branch semantic segmentation prediction result Pred_img (the number of channels is consistent with the number of semantic categories; the SemanticKITTI dataset corresponds to 28 channels).

[0058] The LiDAR network branch is built using SalsaNext with pre-trained SemanticKITTI weights as its backbone network. ASPP modules (with dilation rates of 6, 12, and 18) are embedded to adaptively adjust the receptive field. A 3D multi-head self-attention module (8 heads, 3×3×3 spatiotemporal window size) is also added. The feature extraction process is as follows: After alignment, the point cloud is projected onto a range view and input to the SalsaNext encoder (containing 4 ConvBlocks). It then passes through ASPP multi-scale feature aggregation, the 3D multi-head self-attention module (capturing temporal motion features), and the decoder (upsampling to the input resolution). Finally, a 1×1 convolution unifies the feature dimension to 256, outputting the point cloud feature F_lidar containing geometric, depth, and temporal information. The branch prediction head consists of a 3-layer MLP (512 hidden layer dimensions), outputting the LiDAR branch semantic segmentation prediction result Pred_lidar (28 channels).

[0059] The fusion module is constructed as follows: F_img and F_lidar are processed separately using 1×1 convolutional layers, unifying their feature dimensions to 512, resulting in F_img_unif and F_lidar_unif. A channel-space dual attention module is constructed, calculating the feature confidence weights of F_img_unif and F_lidar_unif based on feature entropy values, and weighting them to obtain F_fusion_attn = α×F_img_unif + β×F_lidar_unif (where α+β=1, α and β are dynamically learned attention weights). F_fusion_attn is added to the original LiDAR feature F_lidar after unifying its dimensions using 1×1 convolutions via residual connections, resulting in the final dual-modal fusion feature F_fusion = F_fusion_attn + F_lidar_proj. The fusion prediction head consists of two MLP layers (1024 hidden layer dimensions), outputting the fused semantic segmentation prediction result Pred_fusion (28 channels).

[0060] Training phase division and execution Phase 1: Independent pre-fine-tuning of a single branch (100 epochs) The goal of independent pre-fine-tuning of each branch is to adapt each branch to the semantic segmentation task of traffic scenarios, and to initially improve the accuracy of single-modal feature extraction. The camera branch input is a cropped and enhanced image. The loss function is the image branch classification loss L_cls_img (cross-entropy loss, input is Pred_img and image semantic labels). The optimizer is AdamW (learning rate 1e-4, weight decay 1e-5). The iteration configuration is batch size = 16, gradient accumulation steps = 2, and the learning rate decays to 1 / 10 of its original value at 50 and 80 epochs. The LiDAR branch input is aligned and enhanced point clouds (default 3-frame multi-frame sequence). The loss function is the joint segmentation loss and temporal loss L_lidar = L_cls_lidar + λ1×L_temporal (λ1=0.3, L_cls_lidar is the cross-entropy loss, input is Pred_lidar and point cloud semantic labels; L_temporal is the dynamic target temporal motion feature matching loss, calculating the MSE loss of point cloud features in adjacent frames to constrain temporal consistency). The optimizer is AdamW (learning rate 2e-4, weight decay 1e-5). The iteration configuration is batch. With size=8 and gradient accumulation steps=4, the learning rate decays to 1 / 10 of its original value at 60 and 90 epochs. During the pre-fine-tuning phase, the mIoU of a single branch is evaluated on the validation set every 10 epochs, and the weights of the branches with the best performance on the validation set are saved (img_best.pth, lidar_best.pth).

[0061] Phase 2: Dual-branch joint training (200 epochs) The goal of dual-branch joint training is to optimize cross-modal feature alignment and fusion performance, achieving collaborative operation between the two branches. During training, the optimal branch weights saved from stage one are loaded, and the dual branches and fusion module are concatenated into an end-to-end network. All unfrozen layers (the upper layer of each branch + the fusion module) participate in joint training. The input is a synchronously enhanced image-point cloud pair (including multi-frame point cloud sequences). The total loss is the weighted sum of the branch loss and the fusion loss: L_total = λ2×L_cls_img + λ3×L_cls_lidar + λ1×L_temporal + λ4×L_cons + λ5×L_fusion (where λ2=0.2, λ3=0.3, λ1=0.2, λ4=0.1). λ5=0.2, and the weight hyperparameters are determined through a grid search on the validation set; L_cons is the cross-modal consistency loss, and the MSE loss of Pred_img and Pred_lidar is calculated to constrain the semantic prediction consistency of the two branches; L_fusion is the classification loss of the fusion branch, which adopts the cross-entropy loss, and the input is Pred_fusion and point cloud semantic labels, with the point cloud labels as the core supervision signal; the optimizer is AdamW, and the initial learning rate is set to 5e-5 for the branch unfreezing layer and 1e-4 for the fusion module (the learning rate of the branch part is 0.5 times that of the fusion module to avoid the pre-fine-tuned weights being over-covered), the weight decay is 1e-5, and gradient pruning (maximum gradient norm = 1.0) is used to prevent gradient explosion; the iteration configuration is batch size=12, gradient accumulation steps=2, and the learning rate scheduling strategy is to warm up for the first 20 epochs (the learning rate is linearly increased from 1e-6 to 1e-4), and decay to 1 / 10 of the original at 80, 140, and 180 epochs. During the joint training phase, the fusion prediction mIoU, OA, and class IoU are evaluated on the validation set every 5 epochs, while the changes in single-branch prediction accuracy are monitored. An early stopping strategy is adopted: if the validation set mIoU does not improve for 15 consecutive epochs (the improvement is <0.5%), training is stopped and the optimal fusion model weights (fusion_best.pth) are saved.

[0062] Optimizer and Training Environment Configuration: The AdamW optimizer is uniformly selected to meet the adaptive optimization requirements of deep learning tasks and alleviate the generalization problem caused by weight decay. During the pre-fine-tuning stage, the learning rate for the camera branch is 1e-4, and the learning rate for the LiDAR branch is 2e-4. During the joint training stage, the learning rate for the branch unfreezing layer is 5e-5, and the learning rate for the fusion module is 1e-4. All layer weight decay is uniformly set to 1e-5 to avoid overfitting. Mixed-precision training (FP16) is used to improve training speed, and the gradient accumulation steps are dynamically adjusted according to the GPU memory (no gradient accumulation is needed when the batch size is 12 and the single card has 24GB of memory).

[0063] The training hardware configuration is as follows: GPU (NVIDIA A100×2, 40GB VRAM per card), CPU (Intel Xeon 8375C×2), 128GB RAM, and 2TB SSD storage (for storing the dataset and training weights); the software environment is: PyTorch 1.18, CUDA 11.7, CuDNN 8.4, and Python 3.9; data loading uses 8 threads (num_workers=8) to accelerate the data preprocessing process.

[0064] Validation and Model Selection: Validation is performed every 10 epochs during the pre-fine-tuning phase and every 5 epochs during the joint training phase. Key validation metrics include: accuracy metrics (mIoU, OA, IoU of key categories such as vehicle / pedestrian / road), consistency metrics (cosine similarity of bi-branch prediction results, used to measure cross-modal consistency), and efficiency metrics (single-frame inference time in GPU environment, ensuring the model has real-time potential after training).

[0065] Early stopping and model saving: The early stopping strategy uses validation set mIoU as the monitoring metric and sets a patience value of 15, meaning that training is terminated when there is no improvement in mIoU for 15 consecutive epochs to avoid overfitting. The current weights are saved after each validation. After training, the model with the validation set mIoUTop3 is retained. Finally, the model with the highest mIoU and the fastest inference speed is selected as the deployment model.

[0066] Training result reuse and update: The optimal branch weights in the pre-fine-tuning stage are used as the initial weights for joint training to avoid the slow convergence problem caused by training from scratch; if a new dataset is added (such as custom traffic scenario data), the weights of the already trained fusion model can be loaded and incrementally trained with a low learning rate of 5e-6 to quickly adapt to the new scenario.

[0067] A residual method is used to fuse image features and point cloud features to obtain dual-modal fused features; The method of fusing image features and point cloud features using residual methods to obtain dual-modal fused features includes: The fusion features are obtained by unifying the feature dimensions of image features and point cloud features through convolutional layers; an attention mechanism is introduced to weight the fusion features to obtain weighted fusion features; and the weighted fusion features are combined with the original LiDAR features by combining residual connections to obtain dual-modal fusion features.

[0068] In some embodiments, to achieve efficient complementarity of the two types of modal features and avoid information redundancy, a residual fusion module is used to complete the fusion of the two-branch features: first, the same-level features of the camera and LiDAR branches are concatenated, and then the feature dimensions are unified through a convolutional layer to obtain the fused features; then, an attention mechanism is introduced to weight the fused features and automatically suppress the feature contribution of noisy regions in the image; finally, combined with residual connections, the weighted fused features are combined with the original LiDAR features, which not only retains the dominant position of the geometric features of the LiDAR, but also incorporates the semantic details of the image, thus achieving lightweight feature fusion.

[0069] This step employs a combined strategy of feature concatenation and dimensionality reduction, attention weighting, and residual connections to achieve a complementary fusion of image semantic features with LiDAR geometric features, while simultaneously mitigating the interference of image noise on the segmentation results. This step can be further subdivided into the following sub-steps: 1. Feature Fusion, Dimensionality Reduction, and Alignment: A 1×1 convolution maps the concatenated bi-branch features to the same channel dimension, eliminating dimensional differences in modal features. The calculation formula is as follows: in, As an intermediate fusion feature, For feature splicing operations, For the corresponding level of convolutional layer, This is a branch feature of lidar. This refers to the branch features of the camera.

[0070] 2. Attention Weighting and Residual Output: An attention mask is generated using sigmoid layer activation to weight the fused features. Residual connections preserve dominant features: The weighted fused features are residually added to the original LiDAR features to ensure the dominance of LiDAR geometric features while incorporating image semantic details. The final fused feature is then output using residual connections, and its calculation formula is as follows:

[0071] in, It is the sigmoid activation function. For attention convolutional layers, This is an element-wise multiplication operation. This is the final fusion feature.

[0072] In some embodiments, this step further includes extending traditional 2D self-attention to a 3D spatiotemporal dimension using a 3D multi-head self-attention (3D MSA) module. This involves capturing the spatial location relationships and temporal motion features of dynamic targets by dividing the spatiotemporal window, thereby enhancing the network's ability to model dynamic scenes. Feature representation is strengthened through a "linear layer-GELU activation-linear layer" structure, and gradient vanishing is avoided by combining residual connections. The calculation formula is as follows: in, Input features for the module, For residual connections, Linear is a multilayer perceptron layer. (Gaussian error linear unit) is an activation function that introduces a nonlinear transformation into features, enhancing their expressive power.

[0073] Input the dual-modal fusion features into the perceptual loss supervision network to obtain the real-time weight ratio of the dual-modal fusion features; The input of the dual-modal fusion features to the perceptual loss supervision network yields real-time weight proportions for the dual-modal fusion features, including: The system obtains the bi-branch prediction results from the bimodal fusion features and calculates the confidence of each bi-branch prediction result. A confidence threshold is set to filter the prediction results with high confidence. A perceptual loss supervision network is constructed to constrain the prediction direction of the low-confidence branch. At the same time, a total loss function is constructed by combining multi-class focus loss and Lovász-softmax loss. The weights are dynamically allocated according to the confidence to obtain the real-time weight ratio of the bimodal fusion features.

[0074] In some embodiments, to address the cross-modal semantic consistency problem, a perceptual loss-supervised network training is introduced: First, the confidence of the bi-branch prediction results is calculated to measure the reliability of each modality prediction; a confidence threshold is set to filter high-confidence prediction results, and a perceptual loss is constructed to constrain the prediction direction of the low-confidence branch; simultaneously, a total loss function is constructed by combining multi-class focus loss and Lovász-softmax loss to achieve cross-modal semantic consistency optimization and alleviate the class imbalance problem, and can dynamically allocate weights according to modal confidence, automatically increasing the proportion of LiDAR features in complex scenarios with image failure.

[0075] The specific steps are as follows: Entropy Map and Confidence Calculation: The formula for calculating the entropy map in the LiDAR branch is as follows: in, This represents the entropy value of the LiDAR branch at position (h, w) in the image (i.e., the region corresponding to the pixel in row h and column w). Entropy is a measure of prediction uncertainty; a larger entropy value indicates greater uncertainty in the prediction at that position, while a smaller entropy value indicates more stable prediction. S represents the number of semantic categories. For LiDAR branch pair category s in pixels The predicted probability at that location. This is the summation operator, which calculates and sums the results for all semantic categories from the 1st to the Sth. It is a normalization factor; when the S semantic categories are perfectly uniformly distributed, the maximum value of the entropy is... Dividing by this value normalizes the entropy to the [0,1] interval, facilitating the standardized calculation of the confidence level subsequently. The confidence level is... The logic for calculating the confidence level of camera branches is similar; Perception Loss Construction: The formula for the perception loss in the LiDAR branch is as follows: in, This represents the sensing loss value of the LiDAR branch; This is a loss normalization factor used to eliminate the influence of the number of pixels in the feature map on the loss value, making the loss value more stable. This represents the total number of pixels in the feature map. The confidence weights at the (h,w) locations in the feature map; Let KL divergence be the KL divergence. The predicted probability of the camera branch; It is the predicted probability distribution of the lidar branch at position (h,w).

[0076] The point cloud in the cross-modal fusion feature is divided into sub-point clouds, and the temporal features are aggregated for the sub-point clouds through the Transformer temporal feature aggregation module; The step of dividing the point cloud in the cross-modal fusion features into sub-point clouds and aggregating temporal features for the sub-point clouds through the Transformer temporal feature aggregation module includes: The point cloud in the cross-modal fusion feature is divided into multiple sub-point clouds by a range segmentation strategy. The average 2D occupancy of the sub-point clouds is not lower than the occupancy of the point cloud in the cross-modal fusion feature. During the training phase, a low-resolution feature map of a sub-point cloud is randomly selected and input into the network. During the inference phase, the feature maps of all sub-point clouds are stacked along the batch dimension and processed in parallel. The sub-point cloud aggregated temporal feature is obtained by multi-scale temporal aggregation of the residual images of multiple frames of point clouds and the range view features through the Transformer temporal feature aggregation module.

[0077] In this invention, the Transformer is a temporal feature aggregation module adapted to traffic scene point cloud segmentation tasks. It is one of the core components for realizing semantic guidance of fused images and improving the temporal robustness of point cloud segmentation. Its specific meaning and function are as follows: It is based on the self-attention mechanism (continuing the core logic of the classic Transformer), but it has been adapted to the characteristics of traffic scene point clouds: First, the point cloud after cross-modal fusion is divided into sub-point clouds through a range segmentation strategy. Then, the Transformer module performs multi-scale temporal aggregation on the "residual images and range view features of multiple frames of sub-point clouds". By capturing the temporal correlation between different frames and different sub-point clouds, and combining the information of image semantic guidance, it strengthens the adaptability of point cloud segmentation to dynamic traffic scenes (such as moving vehicles and pedestrians). Finally, it outputs aggregated features containing temporal information to support more accurate 3D point cloud semantic segmentation results.

[0078] In some embodiments, to balance computational efficiency and feature integrity, a multi-range segmentation strategy is adopted to divide the complete point cloud into multiple sub-point clouds, ensuring that the average 2D occupancy of the sub-point clouds is not lower than that of the high-resolution input. During the training phase, a low-resolution feature map of a sub-point cloud is randomly selected and input into the network. During the inference phase, the feature maps of all sub-point clouds are stacked along the batch dimension and processed in parallel. At the same time, through a pyramid-structured temporal Transformer module, the residual images of multiple frames of point clouds and range view features are aggregated in a multi-scale temporal manner to capture the motion trajectory of dynamic targets and improve the network's adaptability to dynamic scenes.

[0079] The 2D prediction results are obtained based on the dual-modal fusion features, the real-time weight ratio of the dual-modal fusion features, and the temporal features. The 2D prediction results are back-projected into 3D coarse segmentation results. Based on the back-projected 3D coarse segmentation results and combined with semantic guidance, 3D point cloud semantic segmentation results are generated.

[0080] In some embodiments, the 2D range view prediction results output by the LiDAR branch are first back-projected into 3D coarse segmentation results; then, a semantic-guided label voting module is introduced to project the image semantic labels output by the camera branch onto the 3D point cloud, which are then concatenated with the coarse segmentation features and input into a lightweight PointMLP network to correct misjudged labels for boundary points and occlusion points; for point clouds in overlapping areas of multiple camera fields of view, the prediction results with the highest confidence are selected for fusion; finally, fine post-processing is completed through nearest neighbor range interpolation to output the final 3D point cloud semantic segmentation results.

[0081] Nearest neighbor interpolation uses an adaptive threshold to filter effective neighborhood features and, based on relative depth weighted interpolation, solves the inherent "many-to-one" boundary ambiguity problem of projection methods, thus improving the refinement of 3D segmentation results. The specific steps are as follows: Adaptive truncation threshold calculation: First, an adaptive threshold is generated based on the mean and standard deviation of the point cloud distances, using the following formula: in, The adaptive truncation threshold corresponding to point p is the final result calculated by this formula and is used for subsequent truncation processing of point cloud data. This is an exponential function used to map the calculation result within the parentheses to a positive value, ensuring that the truncation threshold is positive. For point The distance value is the actual distance between point p collected by the lidar and the lidar itself. It is the average distance of all points in the current point cloud set, which is a statistical measure of the distance of this batch of point clouds and reflects the overall level of distance. The standard deviation of the distances of all points in the current point cloud set is a statistical measure of the dispersion of distances in this batch of point clouds, reflecting the range of distance fluctuations. To standardize the distance to point p (similar to z-score), the distance to point p is converted into the degree of deviation relative to the overall point cloud distance; It is a regulating factor.

[0082] Weighted interpolation results: The effective neighborhood features are normalized and weighted to obtain the final point cloud category score, as shown in the formula: in, The final point cloud category score corresponding to point p is the result calculated by this formula and is used to determine the semantic category to which point p belongs. Specifically, in the sub-point cloud n, the weight value of the neighborhood covered by the convolution kernel k corresponding to point p is used to measure the contribution of the neighborhood features to the current point p's category score. The neighborhood prediction score is specifically the semantic prediction score of the neighborhood covered by the convolution kernel k corresponding to point p in the sub-point cloud n, that is, the category prediction score corresponding to the neighborhood. The kernel size is the convolution kernel size. The number of dots in the cloud.

[0083] This invention aims to address the technical bottlenecks in existing multimodal point cloud semantic segmentation, including the contradiction between heterogeneous data alignment and consistency, the imbalance between lightweight cross-modal feature fusion and real-time performance, and insufficient real-time adaptation to dynamic traffic scenarios. The method constructs a dual-branch multimodal fusion architecture, with the following specific steps: First, pre-aligning multi-source data from vehicle-mounted LiDAR point clouds and RGB images; replacing traditional spherical projection with perspective projection, combined with cross-modal alignment and clipping to achieve accurate spatial association of heterogeneous data and removal of redundant data; extracting image semantic features and point cloud spatiotemporal geometric features through camera and LiDAR branches respectively; using a lightweight residual fusion module to achieve feature complementarity between the two branches, while introducing perceptual loss to constrain cross-modal semantic consistency and achieve adaptive allocation of modal weights; enhancing dynamic scene adaptation capabilities through low-resolution sub-point cloud grouping and temporal feature aggregation; and finally, post-processing after semantic-guided label voting and nearest-neighbor range interpolation to output accurate 3D point cloud segmentation results. This method improves the perception accuracy of autonomous driving and provides reliable perception support for downstream tasks such as autonomous driving path planning and obstacle avoidance.

[0084] This invention innovates and improves upon data preprocessing, feature fusion architecture, and temporal modeling mechanism, aiming to achieve high precision, high robustness, and high real-time performance of point cloud semantic segmentation in traffic scenarios.

[0085] To address the issues of spatial alignment discrepancies and semantic consistency in heterogeneous data, this method innovatively employs a two-stage processing strategy: "cross-modal alignment cropping + semantically guided calibration." Compared to traditional spherical projection, which is prone to losing semantic information, perspective projection projects point clouds onto the camera coordinate system, preserving the texture and semantic features of the image to the greatest extent possible. Cross-modal alignment cropping (CAC) eliminates redundant areas and reduces unnecessary computation by matching the effective overlap between the point cloud projection area and the image. Simultaneously, a semantically guided label voting (SLV) module is introduced to fuse and calibrate the image semantic labels with the coarse point cloud segmentation results, resolving cross-modal semantic label conflicts and ensuring accurate matching of "geometric features - semantic features."

[0086] To achieve a balance between lightweight cross-modal feature fusion and real-time performance, this method employs a collaborative approach of "low-resolution sub-point cloud grouping + sparse attention hierarchical fusion." Through multi-range segmentation, the complete point cloud is divided into multiple sub-point clouds. During training, low-resolution sub-point cloud projections are randomly selected, and during inference, these are stacked and processed in parallel, increasing the projection rate while reducing the computational load per frame. A sparse attention mechanism focuses only on key regions and core feature points of the point cloud, reducing redundant computation of global attention. Combined with a hierarchical fusion strategy, basic features are rapidly fused at the bottom layer, while precise feature complementarity is achieved at the middle and high layers, balancing fusion effectiveness and inference efficiency.

[0087] To enhance the temporal adaptability of dynamic traffic scenarios, this method leverages the temporal modeling advantages of point cloud Transformers and introduces a 3D multi-head self-attention and temporal feature aggregation mechanism. The 3D multi-head self-attention mechanism constructs a spatiotemporal window for multi-frame point cloud and image data, capturing the motion trajectory and morphological change patterns of dynamic targets. The temporal feature aggregation network fuses multimodal features from historical frames with information from the current frame, avoiding lag in segmentation results. For example, it temporally correlates the point cloud trajectory of a moving pedestrian with image semantics, achieving accurate tracking and segmentation of dynamic targets.

[0088] This invention aims to address the problems of existing technologies in terms of spatial alignment deviation of heterogeneous data, lightweight fusion of cross-modal features and real-time balance, and dynamic scene adaptation through the above-mentioned improvement measures, thereby improving the performance of multimodal point cloud semantic segmentation in street scenes.

[0089] In a second aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the above-described traffic scene point cloud segmentation method based on Transformer fused image semantic guidance.

[0090] Thirdly, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described traffic scene point cloud segmentation method based on Transformer fused image semantic guidance.

[0091] In the description of this specification, the references to terms such as "an embodiment," "some embodiments," "example," "specific example," "a implementation," "a preferred implementation," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0092] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A traffic scene point cloud segmentation method based on Transformer-fused image semantic guidance, characterized in that, The method includes: Input point cloud data collected by the lidar and image data collected by the camera; determine the spatial pose relationship between the lidar and the camera based on the point cloud data and image data; Based on the spatial pose relationship, the point cloud data in the lidar coordinate system is projected from the camera coordinate system to obtain the cropped image and the spatially aligned point cloud. Using a camera-lidar dual-branch network, point cloud features and image features are extracted from spatially aligned point clouds and cropped images, respectively. A residual method is used to fuse image features and point cloud features to obtain dual-modal fused features; Input the dual-modal fusion features into the perceptual loss supervision network to obtain the real-time weight ratio of the dual-modal fusion features; The point cloud optimized based on dual-modal fusion features is divided into sub-point clouds, and temporal features are aggregated for the sub-point clouds through the Transformer temporal feature aggregation module. The 2D prediction results are obtained based on the bimodal fusion features, the real-time weight ratio of the bimodal fusion features, and the temporal features. The 2D prediction results are back-projected into 3D coarse segmentation results. Based on the 3D coarse segmentation results and semantic guidance, 3D point cloud semantic segmentation results are generated.

2. The traffic scene point cloud segmentation method based on Transformer fusion image semantic guidance according to claim 1, characterized in that, The step of determining the spatial pose relationship between the lidar and the camera based on point cloud data and image data includes: Image distortion correction is performed using the intrinsic and extrinsic parameters of the lidar and camera, and the initial calibration is completed by integrating the homogeneous transformation matrix to determine the spatial pose relationship between the lidar and camera.

3. The traffic scene point cloud segmentation method based on Transformer fusion image semantic guidance according to claim 1, characterized in that, The step of projecting point cloud data from the lidar coordinate system to the camera coordinate system based on spatial pose relationships to obtain a cropped image and spatially aligned point cloud includes: Based on the spatial pose relationship, the point cloud data in the lidar coordinate system is projected from the camera coordinate system to obtain the point cloud projection area. The overlapping area between the point cloud projection area and the image is matched, and the overlapping area between the point cloud projection area and the image data collected by the camera is used as the cropped image. Point clouds outside the horizontal field of view of the LiDAR are filtered out to obtain spatially aligned point clouds.

4. The traffic scene point cloud segmentation method based on Transformer fusion image semantic guidance according to claim 1, characterized in that, The camera-lidar dual-branch network includes a ResNet-34 pre-trained network. The cropped image is input into the ResNet-34 pre-trained network, and the output is image features adapted to the traffic scene.

5. The traffic scene point cloud segmentation method based on Transformer fusion image semantic guidance according to claim 1, characterized in that, The camera-lidar dual-branch network includes an optimized SalsaNext network obtained by embedding an ASPP module on the SalsaNext basis. The spatially aligned point cloud is input into the optimized SalsaNext network, and a 3D multi-head self-attention module is introduced to divide the point cloud sequence into spatiotemporal windows and output point cloud features.

6. The traffic scene point cloud segmentation method based on Transformer fusion image semantic guidance according to claim 1, characterized in that, The method of fusing image features and point cloud features using residual methods to obtain dual-modal fused features includes: The fusion features are obtained by unifying the feature dimensions of image features and point cloud features through convolutional layers; an attention mechanism is introduced to weight the fusion features to obtain weighted fusion features; and the weighted fusion features are combined with the original LiDAR features by combining residual connections to obtain dual-modal fusion features.

7. The traffic scene point cloud segmentation method based on Transformer fusion image semantic guidance according to claim 1, characterized in that, The input of the dual-modal fusion features to the perceptual loss supervision network yields real-time weight proportions for the dual-modal fusion features, including: The system obtains the bi-branch prediction results from the bimodal fusion features and calculates the confidence of each bi-branch prediction result. A confidence threshold is set to filter the prediction results with high confidence. A perceptual loss supervision network is constructed to constrain the prediction direction of the low-confidence branch. At the same time, a total loss function is constructed, and the weights are dynamically allocated according to the confidence to obtain the real-time weight ratio of the bimodal fusion features.

8. The traffic scene point cloud segmentation method based on Transformer fusion image semantic guidance according to claim 1, characterized in that, The step of dividing the point cloud in the cross-modal fusion features into sub-point clouds and aggregating temporal features for the sub-point clouds through the Transformer temporal feature aggregation module includes: The point cloud in the cross-modal fusion feature is divided into multiple sub-point clouds by a range segmentation strategy. The average 2D occupancy of the sub-point clouds is not lower than the occupancy of the point cloud in the cross-modal fusion feature. During the training phase, a low-resolution feature map of a sub-point cloud is randomly selected and input into the network. During the inference phase, the feature maps of all sub-point clouds are stacked along the batch dimension and processed in parallel. The sub-point cloud aggregated temporal feature is obtained by multi-scale temporal aggregation of the residual images of multiple frames of point clouds and the range view features through the Transformer temporal feature aggregation module.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the traffic scene point cloud segmentation method based on Transformer fusion image semantic guidance as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the traffic scene point cloud segmentation method based on Transformer fused image semantic guidance as described in any one of claims 1-8.

Images