A method and system for channel point cloud segmentation by radar and camera adaptive fusion

Abstract

The application discloses a waterway point cloud segmentation method and system based on adaptive fusion of radar and camera, and belongs to the technical field of image processing. Radar point cloud is converted into radar cylindrical BEV features through cylindrical voxel segmentation, and waterway camera images are converted into camera cylindrical BEV features through view transformation. Adaptive fusion weights are calculated by using a dynamic feature selection mechanism for the radar cylindrical BEV features and the camera cylindrical BEV features, and the cylindrical BEV features are fused by combining a cross-attention mechanism to generate fused cylindrical BEV features. The fused cylindrical BEV features are input into a segmentation head after sequentially passing through a 3D encoder and a sparse decoder to generate waterway point cloud segmentation results. The method significantly improves the accuracy, robustness and real-time performance of point cloud segmentation in a waterway scene through geometric adaptation, dynamic fusion and efficient architecture design.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a waterway point cloud segmentation method and system that uses radar and camera adaptive fusion. Background Technology

[0002] The rapid development of 3D scene perception technology has brought about tremendous changes to autonomous driving systems. However, research on waterway scene perception remains very limited. A simple idea is to directly transfer land-based autonomous driving methods to aquatic environments. Most on-road autonomous driving methods focus on LiDAR-based environmental perception, but water surfaces often encounter heavy fog and strong light reflection, making LiDAR less suitable for water surface applications. Currently, 3+1D millimeter-wave radar data is gradually attracting researchers' attention due to its cost-effectiveness and robustness, and is being fused with semantically rich RGB image data as a reliable option for perceiving waterway environments.

[0003] As a mainstream fusion paradigm, BEV-based methods effectively address the spatial registration problem between point cloud data and RGB image data to a certain extent, while providing robust representations of 3D spatial features. However, most BEV-based methods choose to first divide the 3D space into cubic voxels, which does not take into account the spatial distribution inconsistencies of the point cloud. To align radar BEV features based on cubic voxel segmentation, the camera branches must adopt the same segmentation strategy. However, this method is not suitable for image frustum points with non-uniform distribution in 3D space. For sparse millimeter-wave radar point clouds on the water surface, the uneven distribution of information within voxels caused by non-uniformity will degrade the performance of the segmentation model.

[0004] Visual sensors are highly susceptible to strong light and inclement weather, conditions that frequently occur on water surfaces. Many segmentation models perform well under normal weather conditions and with ideal sensor functionality; however, their performance often deteriorates sharply when faced with severe weather and sensor malfunctions. This performance degradation stems from the fusion module incorporating a large amount of invalid or even harmful information, thus affecting the model's robustness. Identifying this invalid information and mitigating its impact on fused features has become an attractive research focus. Summary of the Invention

[0005] The purpose of this invention is to overcome the problem of uneven information distribution within voxels caused by the non-uniform distribution of sparse millimeter-wave radar point clouds on the water surface, and to propose a waterway point cloud segmentation method and system that adaptively fuses radar and camera.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a waterway point cloud segmentation method based on adaptive fusion of radar and camera technologies, comprising the following steps: Radar point clouds are converted into radar cylindrical BEV features through cylindrical voxel segmentation, and waterway camera images are converted into camera cylindrical BEV features through view transformation. The cylindrical BEV features of radar and camera are used to calculate adaptive fusion weights using a dynamic feature selection mechanism. Combined with a cross-attention mechanism, the cylindrical BEV features are fused to generate fused cylindrical BEV features. The fused cylindrical BEV features are sequentially passed through a 3D encoder and a sparse decoder before being input into the segmentation head to produce waterway point cloud segmentation results.

[0007] Furthermore, the dynamic feature selection mechanism is as follows: The radar cylindrical BEV features are processed by a sparse decoder and a segmentation head to obtain radar segmentation results. The segmentation label is used to calculate the mIoU to obtain the radar mIoU. The radar mIoU is then used to assign adaptive fusion weights to the radar cylindrical BEV features. The camera's cylindrical BEV features are processed by a sparse decoder and a segmentation head to obtain image segmentation results. The segmentation labels are used to calculate the mIoU, resulting in the image mIoU. The image mIoU is then used to assign adaptive fusion weights to the camera's cylindrical BEV features.

[0008] Furthermore, the formula for the mIoU prediction head is as follows:

[0009]

[0010]

[0011] in, It is a BEV feature from a certain modality. The weights of BEV features in the spatial dimension.

[0012] Furthermore, the radar point cloud is converted into radar cylindrical BEV features through cylindrical voxel segmentation, specifically: Based on the RCS value and three-dimensional polar coordinates of the radar point cloud, the point cloud features are assigned to the polar coordinates and neighborhood of the BEV pixel to obtain cylindrical BEV features. The Gaussian weight map of the cylindrical BEV feature is calculated based on the RCS value. If a pixel has multiple weights, the maximum value among the multiple weights is taken as the final value of the Gaussian weight map. The cylindrical BEV features are concatenated with the final value of the Gaussian weight map and then fed into a multilayer perceptron to obtain cylindrical BEV features based on RCS prior. Convolutional blocks are used to fuse the cylindrical BEV features based on RCS priors and the cylindrical BEV features learned by the sparse encoder to obtain radar cylindrical BEV features.

[0013] Furthermore, the waterway camera image is converted into camera cylindrical BEV features through view transformation, specifically: Image pixels with depth information are estimated using a deep network. These pixels are then mapped to a Cartesian coordinate system. A coordinate transformation is performed to obtain the camera polar coordinate space, which is consistent with the cylindrical BEV feature of the radar. The cylindrical BEV feature of the camera is obtained by using BEVPooling to partition the image in a cylindrical shape within the 3D space.

[0014] Furthermore, the feature fusion of the cylindrical BEV is specifically as follows: The cylindrical BEV features of the radar and the cylindrical BEV features of the camera are combined with the weights assigned by the dynamic feature selection mechanism, and then updated camera features are obtained through a cross-attention mechanism. The updated camera features are concatenated with the radar cylindrical BEV features to obtain the concatenated features. Convolutional blocks are then used to compress the feature dimensions of the concatenated features to complete the cylindrical BEV feature fusion.

[0015] Furthermore, the 3D encoder and sparse decoder simultaneously process fused features and single-modal features. The segmentation loss of the 3D encoder and sparse decoder is:

[0016] in, , and These are the segmentation losses for fused features, camera cylindrical BEV features, and radar cylindrical BEV features, respectively. and It represents the loss weights for camera loss and radar loss.

[0017] Secondly, the present invention provides a waterway point cloud segmentation system that adaptively fuses radar and camera technologies, comprising: The cylindrical BEV feature conversion module is used to convert radar point clouds into radar cylindrical BEV features through cylindrical voxel segmentation, and to convert camera images into camera cylindrical BEV features through view transformation. The cylindrical BEV fusion module is used to calculate adaptive fusion weights from radar cylindrical BEV features and camera cylindrical BEV features using a dynamic feature selection mechanism. Combined with a cross-attention mechanism, the cylindrical BEV features are fused to generate fused cylindrical BEV features. The point cloud segmentation result generation module is used to fuse cylindrical BEV features, which are then passed through a 3D encoder and a sparse decoder and input into the segmentation head to generate waterway point cloud segmentation results.

[0018] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned radar and camera adaptive fusion waterway point cloud segmentation method.

[0019] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, describes a method for waterway point cloud segmentation through adaptive fusion of radar and camera.

[0020] Compared with the prior art, the present invention has the following beneficial technical effects: This invention proposes a radar and camera adaptive fusion method for waterway point cloud segmentation. It generates cylindrical BEV features from radar and camera data through cylindrical voxel segmentation and view transformation. Adaptive fusion is achieved using dynamic weight allocation and cross-attention, and end-to-end waterway point cloud segmentation is completed using a sparse convolutional encoder-decoder architecture. The cylindrical BEV coordinate system naturally adapts to the perception space of outdoor sensors, improving spatial alignment accuracy. The dynamic feature selection mechanism automatically adjusts modal weights according to the scene, and cross-attention enhances modal complementarity. Sparse convolution reduces computational cost, making it suitable for embedded deployment. The entire process is differentiable and optimized, and the fused features possess both geometric robustness and semantic richness, significantly improving small target detection rate and adaptability to adverse weather conditions. The output standardized BEV features can be directly integrated with downstream tasks such as unmanned surface vessel path planning, forming a perception-decision closed loop, demonstrating significant engineering value. Attached Figure Description

[0021] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of the invention in any way. Furthermore, the shapes and proportions of the components in the drawings are merely schematic to aid in understanding the invention and do not specifically limit the shapes and proportions of the components. In the drawings: Figure 1 This is a flowchart of a waterway point cloud segmentation method based on radar and camera adaptive fusion according to the present invention.

[0022] Figure 2 This is a structural diagram of a waterway point cloud segmentation system based on radar and camera adaptive fusion according to the present invention.

[0023] Figure 3 This is an electronic device diagram of a waterway point cloud segmentation method based on radar and camera adaptive fusion according to the present invention.

[0024] Figure 4 This is the architecture of the CylinderFusion model in this embodiment of the invention.

[0025] Figure 5 A comparison is made between the allocation method based on RCS prior proposed by RCBEVDet and the method proposed in this invention.

[0026] Figure 6 This is a schematic diagram of cylindrical BEV pooling.

[0027] Figure 7 Visualization results of different models on the WaterScenes dataset and its image augmentation dataset.

[0028] Figure 8 Comparison of mIoU for different methods within different distance ranges.

[0029] Figure 9 The mIoU for DFS-CylinderFusion across different distance ranges. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] Example 1 See Figure 1 A waterway point cloud segmentation method based on adaptive fusion of radar and camera methods includes the following steps: Radar point clouds are converted into radar cylindrical BEV features through cylindrical voxel segmentation, and waterway camera images are converted into camera cylindrical BEV features through view transformation. The cylindrical BEV features of radar and camera are used to calculate adaptive fusion weights using a dynamic feature selection mechanism. Combined with a cross-attention mechanism, the cylindrical BEV features are fused to generate fused cylindrical BEV features. The fused cylindrical BEV features are sequentially passed through a 3D encoder and a sparse decoder before being input into the segmentation head to produce waterway point cloud segmentation results.

[0032] The cylindrical BEV coordinate system is naturally suited to the perception space of outdoor sensors and the curved boundaries of waterways (such as bends and ports), reducing feature position offsets in Cartesian coordinates and improving the spatial alignment accuracy between camera images and radar point clouds. By dynamically calculating fusion weights (e.g., based on attention scores or uncertainty estimation), the model can automatically adjust the contribution ratio between radar and camera. A bidirectional feature interaction mechanism can capture long-distance dependencies between modalities, such as using the spatial continuity of radar point clouds to correct specular reflection noise in camera images. Sparse convolution only processes non-empty voxels, reducing the computational overhead of large blank areas (such as open water surfaces) in waterway scenes. The sparse decoder preserves details through upsampling and skip connections, avoiding feature blurring caused by upsampling in traditional decoders, making it particularly suitable for fine segmentation of waterway edges (such as shorelines). The adaptively fused BEV features combine the geometric robustness of radar (resistance to lighting / weather interference) with the semantic richness of cameras (such as color and texture). Under adverse conditions such as rain, fog, and nighttime, the model can automatically reduce camera weights, avoiding the precipitous performance drop caused by camera failure in traditional methods. By fusing data from multiple sources, this method reduces the risk of overfitting to a single sensor and improves the model's generalization ability across different scenarios (such as different water levels and seasons). Through geometric adaptation, dynamic fusion, and efficient architecture design, this method significantly improves the accuracy, robustness, and real-time performance of point cloud segmentation in waterway scenarios, providing key technical support for applications such as autonomous driving on water and environmental monitoring.

[0033] The dynamic feature selection mechanism is as follows: The radar cylindrical BEV features are processed by a sparse decoder and a segmentation head to obtain radar segmentation results. The segmentation label is used to calculate the mIoU to obtain the radar mIoU. The radar mIoU is then used to assign adaptive fusion weights to the radar cylindrical BEV features. Similarly, the camera cylindrical BEV features are processed by a sparse decoder and a segmentation head to obtain image segmentation results. The segmentation label is used to calculate the mIoU to obtain the image mIoU. The image mIoU is then used to assign adaptive fusion weights to the camera cylindrical BEV features.

[0034] The formula for the mIoU prediction head is shown below:

[0035]

[0036]

[0037] in, It is a BEV feature from a certain modality. The weights of BEV features in the spatial dimension.

[0038] The radar point cloud is converted into radar cylindrical BEV features through cylindrical voxel segmentation, specifically: Based on the RCS value and three-dimensional polar coordinates of the radar point cloud, the point cloud features are assigned to the polar coordinates and neighborhood of the BEV pixel to obtain cylindrical BEV features. The Gaussian weight map of the cylindrical BEV feature is calculated based on the RCS value. If a pixel has multiple weights, the maximum value among the multiple weights is taken as the final value of the Gaussian weight map. The cylindrical BEV features are concatenated with the final value of the Gaussian weight map and then fed into a multilayer perceptron to obtain cylindrical BEV features based on RCS prior. Convolutional blocks are used to fuse the cylindrical BEV features based on RCS priors and the cylindrical BEV features learned by the sparse encoder to obtain radar cylindrical BEV features.

[0039] Waterway camera images are converted into camera cylindrical BEV features through view transformation, specifically: Image pixels with depth information are estimated using a deep network. These pixels are then mapped to a Cartesian coordinate system. A coordinate transformation is performed to obtain the camera polar coordinate space, which is consistent with the cylindrical BEV feature of the radar. The cylindrical BEV feature of the camera is obtained by using BEVPooling to partition the image in a cylindrical shape within the 3D space.

[0040] The specific feature fusion of cylindrical BEVs is as follows: The cylindrical BEV features of radar and camera are combined with weights assigned by a dynamic feature selection mechanism, and then updated camera features are obtained through a cross-attention mechanism. The updated camera features are then concatenated with the cylindrical BEV features of radar to obtain concatenated features. Convolutional blocks are used to compress the feature dimensions of the concatenated features to complete the fusion of cylindrical BEV features.

[0041] The 3D encoder and sparse decoder simultaneously process fused features and single-modal features. The segmentation loss of the 3D encoder and sparse decoder is:

[0042] in, , and These are the segmentation losses for fused features, camera cylindrical BEV features, and radar cylindrical BEV features, respectively. and It represents the loss weights for camera loss and radar loss.

[0043] Example 2 See Figure 2 A radar and camera adaptive fusion waterway point cloud segmentation system, comprising: The cylindrical BEV feature conversion module is used to convert radar point clouds into radar cylindrical BEV features through cylindrical voxel segmentation, and to convert camera images into camera cylindrical BEV features through view transformation. The cylindrical BEV fusion module is used to calculate adaptive fusion weights from radar cylindrical BEV features and camera cylindrical BEV features using a dynamic feature selection mechanism. Combined with a cross-attention mechanism, the cylindrical BEV features are fused to generate fused cylindrical BEV features. The point cloud segmentation result generation module is used to fuse cylindrical BEV features, which are then passed through a 3D encoder and a sparse decoder and input into the segmentation head to generate waterway point cloud segmentation results.

[0044] Example 3 See Figure 3 An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned radar and camera adaptive fusion waterway point cloud segmentation method.

[0045] Example 4 A computer-readable storage medium storing a computer program, which, when executed by a processor, describes a radar and camera adaptive fusion method for waterway point cloud segmentation.

[0046] Example 5 The CylinderFusion proposed in this invention is built upon the BEVFusion architecture, such as... Figure 4 As shown, the point cloud and image are converted into cylindrical BEV features through cylindrical voxel segmentation and view transformation, respectively. Using a cross-attention mechanism, guided by adaptive weights calculated by the DFS module, fused cylindrical BEV features are generated. Subsequently, the 3D encoder refines the fused features, and the sparse decoder upsamples them. Finally, the segmentation head produces the waterway point cloud segmentation result.

[0047] Following the voxel segmentation method proposed by Cylinder3D, this invention converts the Cartesian coordinates of the point cloud to polar coordinates. The point cloud with polar coordinates is then assigned to its corresponding cylindrical voxels. After voxelization, the sparse cylindrical voxels are downsampled in voxel space by a sparse encoder, and a dense voxel representation is generated through a cylindrical voxel assignment module based on RCS prior. For the camera branch, features are extracted from the front view image by an image encoder, and then converted into cylindrical BEV features by a view converter. In the cylindrical fusion module, the DFS module assigns adaptive weights to the cylindrical camera and radar BEV features and performs feature fusion through a hybrid operation of cross-attention and convolution. Subsequently, the 3D encoder further refines the fused BEV features and re-sparses the voxel representations. The sparse decoder uses skip connections to fuse the multi-scale features of the sparse encoder and upsamples the features to reconstruct a fine-grained voxel space. Finally, the segmentation head generates the final point cloud segmentation.

[0048] 1. Cylinder assignment module based on RCS prior Previous methods for converting sparse radar point cloud features into dense BEV features often result in most BEV feature pixels being empty. RCBEVDet addresses this issue by allocating pixel features to their neighboring voxels based on the RCS value (which reflects object size). Similarly, this invention utilizes this property to fill cylindrical BEV features. A comparison between RCBEVDet's RCS-based prior allocation method and the method proposed in this invention is as follows: Figure 5 As shown.

[0049] RCS value based on radar point cloud and three-dimensional polar coordinates Point cloud features polar coordinates assigned to BEV pixels and its neighborhood . and The following requirements must be met:

[0050] (1) Through this process, the cylindrical BEV feature was finally obtained. .

[0051] This invention also calculates the Gaussian weight map of the cylindrical BEV feature based on the RCS value, and the calculation formula is as follows: (2) in and These are pixel polar coordinates. If If a pixel at a given location receives multiple weights, the maximum value is used as the final Gaussian weight map. The final value. Then... and They are concatenated and fed into a multilayer perceptron (MLP) to obtain cylindrical BEV features based on RCS priors. Finally, this invention uses convolutional block pairs. It is fused with the cylindrical BEV features learned by the sparse encoder.

[0052] 2. Cylindrical BEV Fusion Module In the radar branch, the input point cloud is converted using voxelization based on the polar coordinates of each point in the point cloud. Transformation into sparse cylindrical voxel characterization . and These represent the number of points in the point cloud and the number of effective voxels, respectively. This represents the feature dimension of the input point cloud. Represents the polar coordinates of the point cloud, where Indicates radius, Indicates angle, Indicates elevation. This represents the voxel resolution along each dimension of a three-dimensional cylindrical space.

[0053] To align with the cylindrical voxel space of the radar features, the image branch must acquire the 3D polar coordinates corresponding to each image pixel. Specifically, when converting camera viewpoint features to BEV features using LSS, image pixels with depth information (estimated by a deep network) are first mapped to a Cartesian coordinate system. Coordinate transformation yields a camera polar coordinate space consistent with the radar branch, and BEVPooling is used to obtain the final cylindrical BEV features of the camera, such as... Figure 6 As shown.

[0054] Due to the high noise in radar point cloud position measurements and inaccurate image depth estimation, radar BEV features and image BEV features may not be precisely aligned in the feature space. To alleviate feature misalignment, this invention uses a multi-head deformable cross-attention mechanism and a feedforward network (FFN) to fuse multimodal features. Given radar features... and camera features ,right Add learnable positional embeddings to obtain queries and reference points and will Consider them as Key and Value. Updated camera features are obtained through a cross-attention mechanism. Afterwards, and By concatenating the features, more efficient spatial information can be preserved, and convolutional blocks are used to compress the feature dimensions of the concatenated features. The fusion module can be expressed by the formula:

[0055] (3)

[0056] 3. Dynamic Feature Selection In real-world scenarios, especially on water, image data captured by cameras is often susceptible to environmental interference and limitations imposed by the cameras themselves, such as heavy fog, rain, snow, or physical damage to the camera. In such cases, the goal of this invention is to make the fusion module rely more heavily on the radar branch and discard inefficient information from the camera. To achieve this, this invention designs a Dynamic Feature Selection (DFS) mechanism to adaptively weight features from each modality.

[0057] In short, the weights assigned by the DFS module measure the importance of features at different locations within the BEV feature set; these weights reflect how much attention the model should allocate to the BEV features. Therefore, features from different modalities can be fused based on the assigned weights. (4) in As a feature of fusion, It comes from modality The weights of the BEV features, It is modal BEV characteristics.

[0058] BEV features will ultimately yield metrics that quantify their segmentation performance. Therefore, this invention designs a lightweight prediction head, called the shared dynamic feature selection module, to predict the mIoU of BEV features. The formula for the mIoU prediction head is as follows:

[0059]

[0060] (5) in These are BEV features derived from a specific modality. This invention uses a spatial attention mechanism to obtain the weights of the BEV features in the spatial dimension. ,and This is used in Equation (4). In order to obtain the Ground Truth (true label) of the mIoU prediction head, a sparse decoder and a segmentation head must be used to obtain the segmentation results of the BEV features of a single modality, and the mIoU must be calculated using the segmentation label.

[0061] 3D encoder and sparse decoder for fused features Training is then performed. However, due to the special design of dynamic feature selection mechanisms, they need to be able to process single-modal features simultaneously. Therefore, they can be regarded as shared modules for processing fused features and single-modal features, and the segmentation loss can be written in the following form: (6) in , and These are the segmentation losses for fused features, camera features, and radar features, respectively. and It represents the loss weights for camera loss and radar loss.

[0062] The advantages of this invention have been demonstrated through numerous experiments.

[0063] As shown in Table 1, the proposed DFS-CylinderFusion model achieves a mIoU of 66.8%, while CylinderFusion achieves 68.7% mIoU on the WaterScenes test set. DFS-CylinderFusion sacrifices some model performance but gains strong robustness, demonstrating a significant advantage over pure point cloud-based models. CylinderFusion even outperforms the baseline BEVFusion by 15.9 mIoU. This indicates that the proposed model effectively fuses radar and camera information. Table 1. Comparison of different model methods on the WaterScenes test set. This indicates retraining during the experimental process of this invention.

[0064] Figure 8 and Figure 9The variation of mIoU across different point distances is illustrated. Unlike BEVFusion and CylinderFusion (radar only), the proposed model does not exhibit extreme performance degradation with increasing distance. This demonstrates the superiority of the fusion method in the proposed model, and the multimodal fusion method further enhances its long-range sensing capability. Analysis of the segmentation performance between different categories at different distances reveals that the segmentation accuracy of small objects decreases significantly with increasing point distance. However, the segmentation accuracy of large objects is largely unaffected by point cloud distance. This indicates that the proposed model has advantages in long-range segmentation.

[0065] The advantages of each improvement have also been demonstrated through numerous ablation experiments.

[0066] Table 2 shows the results of different feature fusions. Compared with directly applying convolutions to radar and camera features, the combination of deformable cross-attention mechanism and convolutions shows stronger performance. Furthermore, fusing the original radar features with the results of deformable cross-attention further improves the network performance.

[0067] Table 2 shows the ablation experiments performed on different fusion components in the fusion module on the WaterScenes validation set.

[0068] Radar represents the original cylindrical BEV features of the radar branch. DCA Radar represents the deformable cross-attention result with radar BEV features as the query and camera BEV features as the key and value.

[0069] Table 3 shows the contributions of different modules in the model. The results indicate that the specific design for the 3D cylindrical voxel space effectively improves the model's performance. Since WaterScenes provides 2D semantic labels for images, this invention utilizes these labels to supervise the training of the image encoder, achieving better segmentation results.

[0070] Table 3 Ablation experiments on different network modules on the WaterScenes validation set.

[0071] Table 4 shows the contributions of different image backbones. The results indicate that different image backbones do not show significant performance differences under 2D supervision.

[0072] Table 4. Ablation experiments on different Backbone images with 2D supervision on the WaterScenes validation set.

[0073] Table 5 illustrates the impact of different BEV sizes on the model. Due to the special design of the sparse encoder, the elevation resolution in the mesh size varies with the BEV size; however, this invention argues that this variation has a minimal impact on the results. As shown in Table 5, excessively large or small BEV sizes are not optimal for segmentation tasks.

[0074] Table 5 Ablation experiments on different BEV sizes on the WaterScenes validation set.

[0075] To verify the effectiveness of the DFS mechanism, further experiments were conducted on the WaterScenes validation set after image enhancement. In addition to constructing the DFS module using spatial attention (as shown in Equation (5)), experiments were also conducted using a DFS module composed of multiple convolutional layers. The results are shown in Table 6, demonstrating the powerful performance and robustness of the DFS mechanism.

[0076] Table 6. Robust ablation experiments performed on the image-enhanced WaterScenes validation set.

[0077] This invention simulates autonomous vehicles operating under adverse weather conditions by adding fog to images. As shown in Table 6, the DFS module built from convolutional layers exhibits the best robustness, while the DFS module based on a spatial attention mechanism also performs well. Furthermore, this invention simulates sensor damage by adding noise to images. By changing the parameters... To control noise, an image affected by noise can be represented as: (7) in Indicates the noise level. For the original image, This is the image after noise has been added. The noise is Gaussian with a mean of 0. As shown in Table 6, the DFS module based on the spatial attention mechanism exhibits the highest robustness when processing images affected by noise. Furthermore, the DFS mechanism proposed in this invention ensures that the accuracy of Clutter category point cloud prediction remains stable, which is crucial for real-world unmanned watercraft applications. Figure 7 The visualizations show the results of different models on the WaterScenes dataset and its image augmentation dataset. Red indicates incorrect predictions, and cyan indicates correct predictions.

[0078] Experimental setup The proposed method was trained and evaluated on the WaterScenes dataset. WaterScenes is the largest water surface perception benchmark based on cameras and 3+1D millimeter-wave radar, and is also the only point cloud segmentation dataset based on cameras and 3+1D millimeter-wave radar. The WaterScenes dataset contains 54,120 frames and 202,807 objects, of which 37,884 frames were used for training, 10,824 frames for validation, and 5,412 frames for testing. WaterScenes has point-level annotations of 3+1D radar point clouds, including seven categories and a background noise category. To verify the robustness of the proposed model, image augmentation was used to simulate adverse weather conditions and sensor malfunctions encountered by the unmanned surface vehicle. For this purpose, fog and noise were artificially added to the images in the WaterScenes validation set, such as... Figure 7 As shown.

[0079] The evaluation metric is the same as for most 3D semantic segmentation tasks, using the mean intersection-union ratio (mIoU) as the main evaluation metric, which is the average IoU across all classes.

[0080] The implementation details use BEVFusion as the baseline. The radial coordinate range is limited to 0 to 200, and the angular coordinate range is limited to 0 to... Furthermore, the range of elevation coordinates is limited to -32 to 32. The 3D cylinder voxel space is divided into... The grid shape and size are determined. A sparse encoder is constructed using multiple sparse convolutional blocks, and a sparse decoder is constructed using sparse deconvolution. Additionally, a SECOND is used to construct the 3D encoder. For the camera branch, the image size is set to (544, 960). An image encoder is constructed using a Swin Transformer, and an LSS is used as the view transformer. For data augmentation, scaling, adding positional noise, and flipping operations are applied to the point cloud, while resizing, rotation, and flipping operations are applied to the image. AdamW is used as the optimizer, and CosineAnnealingLR is used as the learning policy. The learning rate, weight decay, and gradient clipping size are set to 0.0001, 0.01, and 35, respectively. The training epochs are set to 20, and the batch size is set to 4.

[0081] The loss function is shown in Equation (6), 3D segmentation loss Depend on , and Composition, will and All values ​​are set to 0.1. All loss components are the sum of cross-entropy loss and Lovasz-softmax loss, in a 1:1 ratio. Additionally, an extra cross-entropy loss is introduced to supervise the image encoder training. Meanwhile, the ground truth of the DFS module is obtained through model inference, enabling the use of L1 loss to supervise the training of the DFS module.

[0082] This invention proposes a novel semantic segmentation network, CylinderFusion, which fuses features within a 3D cylindrical voxel space. To generate more reasonable radar BEV features, a cylinder assignment module based on RCS prior is introduced, which assigns radar features to the cylindrical voxel space according to the RCS value. A dynamic feature selection mechanism is proposed, which improves the robustness of the model by adaptively weighting features from different modalities. The proposed model is validated and tested on the original WaterScenes dataset and an updated version with image enhancement. Extensive experiments demonstrate the superior performance and robustness of this invention.

[0083] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, read-only optical discs, optical storage, etc.) containing computer-usable program code.

[0084] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0085] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0086] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A waterway point cloud segmentation method based on adaptive fusion of radar and camera technologies, characterized in that... Includes the following steps: Radar point clouds are converted into radar cylindrical BEV features through cylindrical voxel segmentation, and waterway camera images are converted into camera cylindrical BEV features through view transformation. The cylindrical BEV features of radar and camera are used to calculate adaptive fusion weights using a dynamic feature selection mechanism. Combined with a cross-attention mechanism, the cylindrical BEV features are fused to generate fused cylindrical BEV features. The cylindrical BEV features are fused and then passed through a 3D encoder and a sparse decoder before being input into the segmentation head to produce waterway point cloud segmentation results. The dynamic feature selection mechanism is as follows: The radar cylindrical BEV features are processed by a sparse decoder and a segmentation head to obtain radar segmentation results. The segmentation label is used to calculate the mIoU to obtain the radar mIoU. The radar mIoU is then used to assign adaptive fusion weights to the radar cylindrical BEV features. The camera cylindrical BEV features are processed by a sparse decoder and a segmentation head to obtain image segmentation results. The segmentation labels are used to calculate the mIoU, and the image mIoU is obtained. The image mIoU is then used to assign adaptive fusion weights to the camera cylindrical BEV features. The formula for the mIoU prediction head is shown below: in, It is a BEV feature from a certain modality. The weights of BEV features in the spatial dimension.

2. The waterway point cloud segmentation method based on radar and camera adaptive fusion according to claim 1, characterized in that, The radar point cloud is converted into radar cylindrical BEV features through cylindrical voxel segmentation, specifically: Based on the RCS value and three-dimensional polar coordinates of the radar point cloud, the point cloud features are assigned to the polar coordinates and neighborhood of the BEV pixel to obtain cylindrical BEV features. The Gaussian weight map of the cylindrical BEV feature is calculated based on the RCS value. If a pixel has multiple weights, the maximum value among the multiple weights is taken as the final value of the Gaussian weight map. The cylindrical BEV features are concatenated with the final value of the Gaussian weight map and then fed into a multilayer perceptron to obtain cylindrical BEV features based on RCS prior. Convolutional blocks are used to fuse the cylindrical BEV features based on RCS priors and the cylindrical BEV features learned by the sparse encoder to obtain radar cylindrical BEV features.

3. The waterway point cloud segmentation method based on radar and camera adaptive fusion according to claim 1, characterized in that, Waterway camera images are converted into camera cylindrical BEV features through view transformation, specifically: Image pixels with depth information are estimated using a deep network. These pixels are then mapped to a Cartesian coordinate system. A coordinate transformation is performed to obtain the camera polar coordinate space, which is consistent with the cylindrical BEV feature of the radar. The cylindrical BEV feature of the camera is obtained by using BEVPooling to partition the image in a cylindrical shape within the 3D space.

4. The waterway point cloud segmentation method based on radar and camera adaptive fusion according to claim 1, characterized in that, The specific feature fusion of cylindrical BEVs is as follows: The cylindrical BEV features of the radar and the cylindrical BEV features of the camera are combined with the weights assigned by the dynamic feature selection mechanism, and then updated camera features are obtained through a cross-attention mechanism. The updated camera features are concatenated with the radar cylindrical BEV features to obtain the concatenated features. Convolutional blocks are then used to compress the feature dimensions of the concatenated features to complete the cylindrical BEV feature fusion.

5. The waterway point cloud segmentation method based on radar and camera adaptive fusion according to claim 1, characterized in that, The 3D encoder and sparse decoder simultaneously process fused features and single-modal features. The segmentation loss of the 3D encoder and sparse decoder is: in, , and These are the segmentation losses for fused features, camera cylindrical BEV features, and radar cylindrical BEV features, respectively. and It represents the loss weights for camera loss and radar loss.

6. A waterway point cloud segmentation system that adaptively fuses radar and camera technologies, characterized in that, The waterway point cloud segmentation method using radar and camera adaptive fusion as described in any one of claims 1-5 includes: The cylindrical BEV feature conversion module is used to convert radar point clouds into radar cylindrical BEV features through cylindrical voxel segmentation, and to convert camera images into camera cylindrical BEV features through view transformation. The cylindrical BEV fusion module is used to calculate adaptive fusion weights from radar cylindrical BEV features and camera cylindrical BEV features using a dynamic feature selection mechanism. Combined with a cross-attention mechanism, the cylindrical BEV features are fused to generate fused cylindrical BEV features. The point cloud segmentation result generation module is used to fuse cylindrical BEV features, which are then passed through a 3D encoder and a sparse decoder and input into the segmentation head to generate waterway point cloud segmentation results.

7. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements a waterway point cloud segmentation method based on radar and camera adaptive fusion as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements a waterway point cloud segmentation method based on radar and camera adaptive fusion as described in any one of claims 1-5.

Images