A water surface segmentation method and system based on cross-modal fusion distillation

Abstract

The present invention relates to a water surface segmentation method and system using cross-modal fusion distillation, which belongs to the field of computer vision technology. First, a point-pixel correspondence is established, and 2D features are extracted based on whether this relationship contains 2D pixels to obtain point-by-point 2D features; then the 3D feature map is calculated using the nearest neighbor interpolation method to obtain point-by-point 3D features; the two are fused to obtain fused features; water surface segmentation is predicted using point-by-point 2D features to obtain a 2D prediction score; water surface segmentation is predicted using the fused features to obtain a fused prediction score; the distillation loss between the 2D prediction score and the fused prediction score is calculated; finally, the 2D prediction network is adjusted using a method for reducing the distillation loss to obtain an adjusted 2D prediction network; and water surface prediction segmentation is performed using the adjusted 2D prediction network. The present invention utilizes a cross-modal fusion distillation method to transfer knowledge learned by a complex network, simplify the network, reduce the amount of calculation, and do not affect the segmentation effect.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a water surface segmentation method and system based on LiDAR point cloud-assisted image cross-modal fusion distillation. Background Art

[0002] Unmanned vehicles (UAVs) offer low risk and high flexibility, making them valuable for performing demanding and dangerous tasks on the water, such as maritime surveillance, emergency rescue, and disaster recovery. Accurate water surface segmentation provides a key guarantee for the autonomous and safe navigation of UAVs. To improve the safety of both vessels and personnel, research on surface segmentation methods for UAVs is essential.

[0003] The water surface segmentation task is suitable for understanding large-scale outdoor scenes. It aims to process images at the pixel level, segmenting the area where the unmanned vehicle is navigating into surface and non-surface areas. This effectively limits the scope of the detection algorithm, enabling better detection of both surface and non-surface objects, increasing detection accuracy, and ensuring safe navigation for the unmanned vehicle. In the past few years, research has focused on using camera images or lidar point clouds as input for water surface segmentation. However, due to the inherent limitations of each sensor, single-sensor approaches are susceptible to challenges in complex environments. Specifically, images captured by camera sensors provide rich information such as color and texture, but lack depth information and are susceptible to lighting and weather conditions. For example, strong light reflecting off the water surface can cause the camera to capture a lot of glare, affecting the water surface segmentation performance. Compared to image information, point clouds collected by lidar sensors provide excellent depth sensing and are robust to factors such as lighting and weather, but lack the rich information such as color and texture that images provide. Therefore, some research has proposed using lidar point clouds to supplement camera images, fusing the two to achieve environmental perception and improve the performance of segmentation algorithms.

[0004] However, existing fusion methods have very complex networks and heavy computational burdens in processing both images and point clouds. Therefore, there is an urgent need for a technical solution that is computationally efficient while ensuring segmentation accuracy. Summary of the Invention

[0005] The purpose of the present invention is to provide a water surface segmentation method and system based on cross-modal fusion distillation, which uses the cross-modal fusion distillation method to transfer the knowledge learned by the complex network, simplify the network, reduce the amount of calculation without affecting the segmentation effect, and effectively solve the problem in the existing technology that the fusion method network is very complex and the computational burden of processing images and point clouds at the same time is heavy.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A water surface segmentation method using cross-modal fusion distillation, comprising:

[0008] Acquire 2D image data and 3D point cloud data; wherein the 2D image data and the 3D point cloud data are within the same field of view;

[0009] Using a 2D network to obtain a 2D feature map of the 2D image data

[0010] Perspectively projecting the 3D point cloud data onto a 2D plane according to an intrinsic parameter matrix and an extrinsic parameter matrix of a camera that acquires the 2D image data to obtain 2D plane pixels;

[0011] Establishing a point-pixel correspondence relationship using the 2D plane pixels and the 3D point cloud data;

[0012] Find the 2D feature map contained in the point-pixel correspondence Pixels in, get the containing pixels;

[0013] Extract the 2D features corresponding to the pixels to obtain point-by-point 2D features

[0014] Use the 3D network to obtain the 3D feature map F of the 3D point cloud data 3D ;

[0015] Using the correspondence between points and voxels in the 3D feature map F 3D Perform the nearest neighbor interpolation method on it to obtain point-by-point 3D features

[0016] The point-by-point 2D features are transformed using the MLP network Convert to 3D learner features

[0017] The 3D learner features With the point-by-point 3D features Splicing and fusion to obtain fusion features

[0018] Using the 2D prediction network according to the point-by-point 2D features Perform water surface segmentation prediction and use the classifier to obtain 2D prediction scores;

[0019] Using the 3D prediction network according to the fusion features Perform water surface segmentation prediction and use the classifier to obtain the fusion prediction score;

[0020] Calculating the distillation loss between the 2D prediction score and the fusion prediction score using KL divergence;

[0021] Adjusting the 2D prediction network using a method for reducing the distillation loss to obtain an adjusted 2D prediction network;

[0022] The regulated 2D prediction network is used to perform water surface prediction and segmentation.

[0023] In some embodiments, the 2D feature map of the 2D image data is obtained by using a 2D network. Specifically include:

[0024] Extracting multi-scale features of the 2D image data using a 2D network;

[0025] Deconvolution is used to upsample the multi-scale features to the same resolution as the 2D image data to obtain a 2D feature map.

[0026] In some embodiments, the calculation formula for obtaining 2D plane pixels by perspective-projecting the 3D point cloud data onto a 2D plane based on the intrinsic parameter matrix and extrinsic parameter matrix of the camera that acquired the 2D image data is:

[0027]

[0028] Among them, (u i ,v i ) represents the 2D plane pixel coordinates, K is the camera's intrinsic parameter matrix, T is the camera's extrinsic parameter matrix, (x i ,y i ,z i ) represents the 3D point cloud coordinates.

[0029] In some embodiments, the point-pixel correspondence is: in, Indicates the floor operation, (u i ,v i ) represents the 2D plane pixel coordinate, O pixel Represents the point-pixel correspondence map, and N represents the number of 3D points.

[0030] In some embodiments, the 2D features corresponding to the pixels are extracted to obtain point-by-point 2D features. After that, it also includes:

[0031] The point-by-point 2D features The elements in are accumulated element by element.

[0032] In some embodiments, the 3D learner features With the point-by-point 3D features Splicing and fusion to obtain fusion features After that, it also includes:

[0033] The fusion feature The elements in are accumulated element by element.

[0034] In some embodiments, the calculation formula for calculating the distillation loss between the 2D prediction score and the fusion prediction score using KL divergence is:

[0035]

[0036] Where N represents the number of 3D points, n is the nth point, and D KL It refers to KL divergence, S 3D2D represents the fusion prediction score, S 2D Represents the 2D prediction score.

[0037] In some embodiments, the 2D network includes: a ResNet34 encoder and an FCN decoder.

[0038] In some embodiments, the 3D network includes: a PointNet++ encoder and a decoder.

[0039] The present invention also discloses a water surface segmentation system using cross-modal fusion distillation, the system comprising:

[0040] A data acquisition module, configured to acquire 2D image data and 3D point cloud data; the 2D image data and the 3D point cloud data are within the same field of view;

[0041] 2D feature map acquisition module, used to obtain the 2D feature map of the 2D image data using a 2D network

[0042] a projection module, configured to perspective-project the 3D point cloud data onto a 2D plane according to an intrinsic parameter matrix and an extrinsic parameter matrix of a camera that acquires the 2D image data, to obtain 2D plane pixels;

[0043] A point-pixel correspondence establishment module, configured to establish a point-pixel correspondence using the 2D plane pixels and the 3D point cloud data;

[0044] Contains a pixel search module for finding the 2D feature map contained in the point-pixel correspondence Pixels in, get the containing pixels;

[0045] The point-by-point 2D feature acquisition module is used to extract the 2D features corresponding to the pixels and obtain the point-by-point 2D features.

[0046] 3D feature map acquisition module, used to obtain the 3D feature map F of the 3D point cloud data using a 3D network 3D ;

[0047] Point-by-point 3D feature acquisition module, used to use the correspondence between points and voxels in the 3D feature map F 3D Perform the nearest neighbor interpolation method on it to obtain point-by-point 3D features

[0048] MLP conversion module is used to convert the point-by-point 2D features into Convert to 3D learner features

[0049] Feature splicing module, used to combine the 3D learner features With the point-by-point 3D features Splicing and fusion to obtain fusion features

[0050] 2D prediction module, used to use the 2D prediction network according to the point-by-point 2D features Perform water surface segmentation prediction and use the classifier to obtain 2D prediction scores;

[0051] A fusion prediction module is used to use a 3D prediction network to predict the fusion features Perform water surface segmentation prediction and use the classifier to obtain the fusion prediction score;

[0052] a distillation loss calculation module, configured to calculate the distillation loss between the 2D prediction score and the fusion prediction score using KL divergence;

[0053] a 2D prediction network adjustment module, configured to adjust the 2D prediction network using a method for reducing the distillation loss to obtain an adjusted 2D prediction network;

[0054] The water surface prediction and segmentation module is used to perform water surface prediction and segmentation using the adjusted 2D prediction network.

[0055] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0056] The present invention provides a water surface segmentation method and system based on cross-modal fusion distillation, firstly obtaining 2D image data and 3D point cloud data; using a 2D network to obtain a 2D feature map of the 2D image data; The 3D point cloud data is perspective-projected onto a 2D plane according to the camera's intrinsic and extrinsic matrix to obtain 2D plane pixels; a point-pixel correspondence is established between the 2D plane pixels and the 3D point cloud data; and the 3D point cloud data is converted into point-by-point 2D features according to the point-pixel correspondence. Use 3D network to obtain 3D feature map F of 3D point cloud data 3D ; Use the correspondence between points and voxels in the feature map F3D Perform the nearest neighbor interpolation method on it to obtain point-by-point 3D features The point-by-point 2D features are transformed using the MLP network Convert to 3D learner features The 3D learner features With point-by-point 3D features Splicing and fusion to obtain fusion features Using 2D prediction network based on point-by-point 2D features Perform water surface segmentation prediction and use the classifier to obtain 2D prediction scores; use the 3D prediction network to fusion features Water surface segmentation prediction is performed, and a fusion prediction score is obtained using a classifier; the distillation loss between the 2D prediction score and the fusion prediction score is calculated using KL divergence; the 2D prediction network is adjusted using a method to reduce the distillation loss to obtain an adjusted 2D prediction network; and the adjusted 2D prediction network is used to perform water surface prediction segmentation. The present invention utilizes a cross-modal fusion distillation method to transfer knowledge learned by a complex network, simplify the network, reduce computational complexity without affecting segmentation results, and fully utilize LiDAR point clouds with rich depth information during training to promote 2D image representation. This effectively solves the problem of the existing fusion method network being very complex and the heavy computational burden of processing both images and point clouds simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 Flowchart of the water surface segmentation method using cross-modal fusion distillation provided in Example 1 of the present invention.

[0059] Figure 2 This is an overall block diagram of the water surface segmentation method of cross-modal fusion distillation provided in Example 1 of the present invention.

[0060] Figure 3 This is a schematic diagram of the process of obtaining point-by-point 2D features provided in the first embodiment of the present invention.

[0061] Figure 4 This is a schematic diagram of the process of obtaining point-by-point 3D features provided in the first embodiment of the present invention.

[0062] Figure 5 Schematic diagram of the knowledge distillation process provided in Example 1 of the present invention.

[0063] Figure 6 A block diagram of the water surface segmentation system using cross-modal fusion distillation provided in Example 2 of the present invention.

[0064] Figure 7 Flowchart of the water surface segmentation method using cross-modal fusion distillation provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0065] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0066] The purpose of the present invention is to provide a water surface segmentation method and system based on cross-modal fusion distillation, which uses the cross-modal fusion distillation method to transfer the knowledge learned by the complex network, simplify the network, reduce the amount of calculation without affecting the segmentation effect, and effectively solve the problem in the existing technology that the fusion method network is very complex and the computational burden of processing images and point clouds at the same time is heavy.

[0067] Nowadays, most of the existing fusion methods based on deep learning establish a mapping between two-dimensional pixels and three-dimensional points, fuse the features of the image into the features of the point for processing, and improve the segmentation effect. However, considering the different fields of view of the camera and the lidar, the lidar collects point clouds with a 360° field of view, while the field of view of the camera is limited, and only a small part of the two fields of view overlap. To this end, based on the different characteristics of the external environment information collected by different shipborne sensors, a surface segmentation method based on images as the main component and point clouds as the auxiliary component is designed for unmanned boat surface segmentation. In view of the fact that the existing fusion method network is powerful and complex, and the computational burden of processing images and point clouds at the same time is heavy, the present invention designs a cross-modal fusion distillation method to transfer the knowledge learned by the complex network, simplify the network, reduce the amount of computation without affecting the segmentation effect. During the training process, this method makes full use of the LiDAR point cloud with rich depth information to promote the representation of 2D images. The network after knowledge distillation can achieve higher accuracy, which is more conducive to practical applications, especially in mobile devices, and promotes the high intelligence and real-time nature of unmanned boats.

[0068] The present invention provides a water surface segmentation method with cross-modal fusion and distillation. First, the image and point cloud are input into independent 2D and 3D encoders respectively, and the respective multi-scale features are extracted in parallel. Secondly, a point-pixel correspondence is established to transfer information between the two modes and generate paired features of the two modalities. Then, a multi-scale cross-modal fusion-then-distillation method is used to extract multimodal knowledge into a single image modality. The multi-scale information of the two modalities is first fused, and then knowledge distillation is performed, that is, the LiDAR point cloud spatial information is obtained from the multimodal data, and the complementary three-dimensional knowledge is effectively transferred to the two-dimensional network, while retaining the color and texture information of the two-dimensional image. Finally, the features of each scale of 2D and 3D are used to generate water surface segmentation predictions, which are supervised by 2D image labels.

[0069] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0070] Example 1:

[0071] like Figure 1 As shown, this embodiment provides a water surface segmentation method of cross-modal fusion distillation, and its overall block diagram is as follows Figure 2 As shown, the method includes:

[0072] S1. Acquire 2D image data and 3D point cloud data; the 2D image data and the 3D point cloud data are in the same field of view.

[0073] S2. Obtaining a 2D feature map of the 2D image data using a 2D network

[0074] First, a 2D network is used to extract the multi-scale features of the 2D image data; then, deconvolution is used to upsample the multi-scale features to the same resolution as the 2D image data to obtain a 2D feature map.

[0075] In this embodiment, the encoder of the two-dimensional network uses ResNet34 with 2D convolution to extract multi-scale features of the image; and FCN is used as the decoder. The feature map of the first decoder is first jump-connected to the last encoder layer, and then the features of the previous encoder layer are gradually up-sampled to obtain a 2D feature map.

[0076] S3. Perspectively project the 3D point cloud data onto a 2D plane according to the intrinsic parameter matrix and extrinsic parameter matrix of the camera of the acquired 2D image data to obtain 2D plane pixels.

[0077] Similar to most multi-sensor methods, this embodiment uses perspective projection and calculates the point-to-pixel mapping between the point cloud and the image, based on the camera intrinsic parameter matrix calculated by camera calibration. and the external parameter matrix Given a 3D point cloud Each point of the 3D point cloud Get 2D plane pixels through perspective projection The specific formula is as follows:

[0078]

[0079] Among them, (u i ,v i ) represents the 2D plane pixel coordinates, K is the camera's intrinsic parameter matrix, T is the camera's extrinsic parameter matrix, (x i ,y i ,z i ) represents the 3D point cloud coordinates.

[0080] S4. Establishing a point-pixel correspondence relationship using the 2D plane pixels and the 3D point cloud data.

[0081] After projection, the point-pixel correspondence between the 3D point cloud and 2D image modes is calculated using the following formula;

[0082]

[0083] in, Indicates the floor operation, (u i ,v i ) represents the 2D plane pixel coordinate, O pixel Represents the point-pixel correspondence mapping diagram. If the point-pixel correspondence O pixel If any pixel on the 2D feature map is included in Corresponding to extracting a point-by-point 2D feature

[0084] S5. Find the 2D feature map contained in the point-pixel correspondence , get the pixels that contain .

[0085] S6: Extract the 2D features corresponding to the pixels to obtain point-by-point 2D features. The steps for obtaining point-by-point 2D features are as follows: Figure 3 shown.

[0086] Then, the point-by-point 2D features The elements in are accumulated element by element.

[0087] S7, using a 3D network to obtain a 3D feature map F of the 3D point cloud data 3D .

[0088] This example uses PointNet++ as the encoder for the 3D network. The decoder upsamples features of different scales to their original size and concatenates them, which can better learn hierarchical information.

[0089] S8, using the correspondence between points and voxels in the feature map F 3D Perform the nearest neighbor interpolation method on it to obtain point-by-point 3D features

[0090] Given a point cloud Each point The point-voxel correspondence is obtained by the following formula:

[0091]

[0092] Where r is the resolution of the voxelization of the lth layer, (x i ,y i ,z i ) is the coordinate of the i-th point cloud, and N represents the number of 3D points. It is known that the original 3D features from the 3D point cloud segmentation network PointNet++ Then according to the point-voxel correspondence O voxel In the original feature map F 3D Perform the nearest neighbor interpolation method to obtain point-by-point 3D features See also Figure 4 .

[0093] S9, using the MLP network to transform the point-by-point 2D features Convert to 3D learner features

[0094] Since the backbone networks used in 2D and 3D are different, the generated features will be somewhat different, and it is not appropriate to directly fuse the two. Convert to 3D learner features The gap between the two features is narrowed by MLP.

[0095] S10, the 3D learner features With the point-by-point 3D features Splicing and fusion to obtain fusion features Then the fusion feature The elements in are accumulated element by element.

[0096] S11, using the 2D prediction network according to the point-by-point 2D features Perform water surface segmentation prediction and use the classifier to obtain 2D prediction scores.

[0097] S12, using the 3D prediction network according to the fusion features Perform water surface segmentation prediction and use the classifier to obtain the fusion prediction score.

[0098] like Figure 5 As shown, in this embodiment, one branch transforms the 3D learner features With the point-by-point 3D features Splicing and fusion to obtain fusion features Enhanced by element-wise accumulation Make predictions; another branch generates 2D features Element-wise accumulation to generate enhanced 2D features Make predictions.

[0099] Figure 5 The cross-modal fusion part (dashed line) in is only used during training. After the model is trained, the 3D related branches are discarded and the model can run almost without adding additional computational cost.

[0100] S13. Calculate the distillation loss between the 2D prediction score and the fusion prediction score using KL divergence.

[0101] S14. Optimizing the 2D prediction network using a method for reducing the distillation loss to obtain an adjusted 2D prediction network. This step specifically involves optimizing the parameters of the 2D prediction network to reduce the distillation loss, thereby making the prediction results of the 2D prediction network closer to the fusion prediction results, ultimately obtaining an adjusted 2D prediction network.

[0102] S15. Utilize the adjusted 2D prediction network to perform water surface prediction and segmentation.

[0103] In this embodiment, two independent classifiers are used to obtain the fusion prediction score S 3D2D and 2D prediction score S 2D , (the predicted score outputs the semantic score, i.e., the predicted score, through the classifier (fully connected layer) at the end of the network) and uses KL divergence to calculate the distillation loss L between the two to reduce the difference between the two predicted values. The calculation formula is as follows:

[0104]

[0105] Where N represents the number of 3D points, n is the nth point, and D KL It refers to KL divergence. S is calculated based on KL divergence. 3D2D and S2D The distillation loss L aims to convert the prediction score S of pure 2D features into 2D The prediction score S of 2D and 3D fusion features 3D2D More similar, thereby improving S 2D , which plays a role in improving the two-dimensional water surface segmentation. Intuitively, it means making the output of the student (2D prediction result) as close as possible to the output probability of the teacher (fusion prediction result) to achieve one-way distillation. The advantages of this knowledge distillation framework are: first, the student (2D prediction result) learns the rich 3D spatial depth information of the teacher (fusion prediction result) to enhance its own 2D feature learning without losing the specific 2D modal information; second, the fusion branch is only used in the training phase. After the model is trained, the computational burden is greatly reduced in the actual prediction process. Knowledge distillation is to train a complex teacher network and then let the simple student network learn the teacher's results, so as to achieve the goal of not reducing the prediction results while reducing the computational burden.

[0106] This example implements 3D point cloud-assisted water surface segmentation from 2D images by utilizing a multimodal fusion model to distill knowledge. During training, 2D and 3D information is fused and then distilled into a pure 2D network. During actual prediction, the 2D network is directly used for segmentation, resulting in water surface segmentation results.

[0107] In general, this embodiment takes into account the complementarity of images and point clouds, and transfers the complementary 3D point cloud information to the 2D network through a multimodal fusion and distillation module, making full use of 3D knowledge and retaining 2D modal information.

[0108] The cross-modal fusion distillation water surface segmentation method provided in this embodiment first obtains 2D image data and 3D point cloud data; uses a 2D network to obtain a 2D feature map of the 2D image data The 3D point cloud data is perspective-projected onto a 2D plane according to the camera's intrinsic and extrinsic matrix to obtain 2D plane pixels; a point-pixel correspondence is established between the 2D plane pixels and the 3D point cloud data; and the 3D point cloud data is converted into point-by-point 2D features according to the point-pixel correspondence. Use 3D network to obtain 3D feature map F of 3D point cloud data 3D ; Use the correspondence between points and voxels in the feature map F 3D Perform the nearest neighbor interpolation method on it to obtain point-by-point 3D features The point-by-point 2D features are transformed using the MLP network Convert to 3D learner features The 3D learner features With point-by-point 3D features Splicing and fusion to obtain fusion features Using 2D prediction network based on point-by-point 2D features Perform water surface segmentation prediction and use the classifier to obtain 2D prediction scores; use the 3D prediction network to fusion features Water surface segmentation prediction is performed, and a fusion prediction score is obtained using a classifier; the distillation loss between the 2D prediction score and the fusion prediction score is calculated using KL divergence; the 2D prediction network is adjusted using a method to reduce the distillation loss to obtain an adjusted 2D prediction network; and the adjusted 2D prediction network is used to perform water surface prediction segmentation. The present invention utilizes a cross-modal fusion distillation method to transfer knowledge learned by a complex network, simplify the network, reduce computational complexity without affecting segmentation results, and fully utilize LiDAR point clouds with rich depth information during training to promote 2D image representation. This effectively solves the problem of the existing fusion method network being very complex and the heavy computational burden of processing both images and point clouds simultaneously.

[0109] Example 2:

[0110] like Figure 6 As shown, this embodiment provides a water surface segmentation system with cross-modal fusion distillation, the system comprising:

[0111] The data acquisition module M1 is used to acquire 2D image data and 3D point cloud data; the 2D image data and the 3D point cloud data are in the same field of view;

[0112] 2D feature map acquisition module M2, used to obtain the 2D feature map of the 2D image data using a 2D network

[0113] A projection module M3 is configured to perspective-project the 3D point cloud data onto a 2D plane according to an intrinsic parameter matrix and an extrinsic parameter matrix of a camera that acquires the 2D image data, to obtain 2D plane pixels;

[0114] A point-pixel correspondence establishing module M4 is configured to establish a point-pixel correspondence using the 2D plane pixels and the 3D point cloud data;

[0115] It includes a pixel search module M5 for searching the 2D feature map contained in the point-pixel correspondence relationship. Pixels in, get the containing pixels;

[0116] Point-by-point 2D feature acquisition module M6 is used to extract the 2D features corresponding to the pixels and obtain point-by-point 2D features.

[0117] 3D feature map acquisition module M7, used to obtain the 3D feature map F of the 3D point cloud data using a 3D network 3D ;

[0118] Point-by-point 3D feature acquisition module M8 is used to use the correspondence between points and voxels in the feature map F 3D Perform the nearest neighbor interpolation method on it to obtain point-by-point 3D features

[0119] MLP conversion module M9 is used to convert the point-by-point 2D features into Convert to 3D learner features

[0120] Feature splicing module M10, used to combine the 3D learner features With the point-by-point 3D features Splicing and fusion to obtain fusion features

[0121] 2D prediction module M11, used to use the 2D prediction network according to the point-by-point 2D features Perform water surface segmentation prediction and use the classifier to obtain 2D prediction scores;

[0122] Fusion prediction module M12, for using 3D prediction network according to the fusion features Perform water surface segmentation prediction and use the classifier to obtain the fusion prediction score;

[0123] A distillation loss calculation module M13 is used to calculate the distillation loss between the 2D prediction score and the fusion prediction score using KL divergence;

[0124] a 2D prediction network adjustment module M14, configured to adjust the 2D prediction network using a method for reducing the distillation loss to obtain an adjusted 2D prediction network;

[0125] The water surface prediction and segmentation module M15 is used to perform water surface prediction and segmentation using the adjusted 2D prediction network.

[0126] Example 3:

[0127] This embodiment provides a water surface segmentation method based on cross-modal fusion distillation. The overall process is as follows: Figure 7 As shown, the specific steps include:

[0128] Step 1: Use ResNet34 with 2D convolution as the encoder for the 2D network, and use FCN as the decoder to gradually upsample the features of the previous encoder layer to obtain feature maps of different scales. Finally, the feature map provided by the decoder is input into a linear classifier for classification, resulting in a 2D water surface segmentation prediction. Using FCN as the decoder, the last three layers of the CNN network are converted to 1x1 convolutions, making the entire network model composed entirely of convolutional layers. FCN classifies each pixel in the input image and divides the pixels into two categories: water surface and non-water surface, which is the 2D image water surface segmentation prediction result.

[0129] Step 2: Extract the point cloud that overlaps with the image field of view from the original point cloud as 3D input, and use PointNet++ as the encoder for the 3D network. Instead of using U-Net, the decoder upsamples features at different scales to their original size and concatenates them before inputting them into the classifier. This architecture better learns hierarchical information and achieves more effective 3D water surface segmentation predictions. Using the PointNet++ network, the initial point cloud, which incorporates more hierarchical feature information, is passed through a fully connected layer to classify each point in the point cloud, resulting in a 3D point cloud segmentation result.

[0130] It should be noted that steps 1 and 2 in this embodiment are a brief introduction to the composition of the 2D network and the 3D network. Step 3 is to establish a point-pixel correspondence. The purpose is to use this point-to-pixel correspondence to generate paired features of the two modes for further knowledge distillation.

[0131] Step 3: Establish point-pixel correspondence, generate paired features of 2D and 3D modalities, and perform knowledge distillation.

[0132] The process of 2D feature generation is as follows: given the input monocular camera image Through a 2D network (the 2D network described in step 1, ResNet34 as encoder, FCN as decoder), multi-scale features are extracted from hidden layers of different resolutions Use deconvolution to transform the two-dimensional feature F 2D Upsample to the same resolution as the original image to obtain the feature map Similar to most multi-sensor methods, perspective projection is used and point-to-pixel mapping between point cloud and image is calculated. The camera intrinsic parameter matrix is ​​calculated based on camera calibration. and the external parameter matrix Given a 3D point cloud P, each point Get 2D plane pixels through perspective projection The specific formula is as follows:

[0133]

[0134] (u i ,v i ) refers to the 2D plane pixel coordinates, K is the camera internal parameter matrix, T is the external parameter matrix, (x i ,y i ,z i ) are the 3D point coordinates.

[0135] After projection, the point-pixel correspondence between the 3D point cloud and 2D image modes is expressed as follows:

[0136]

[0137] in, Indicates the rounding down operation. According to the point-pixel correspondence mapping diagram O pixel , from the feature map Extract point-by-point 2D features

[0138] Step 4: The process of 3D feature generation is as follows:

[0139] Given a point cloud Each point The point-voxel correspondence is expressed as follows:

[0140]

[0141] Among them, r is the resolution of the voxel in the lth layer, (x i ,y i ,z i ) is the coordinate of each i point.

[0142] Get the original 3D features from the 3D network PointNet++ Then according to the point-voxel correspondence O voxel , in the original feature map F 3D Perform the nearest neighbor interpolation method to obtain point-by-point 3D features Similar to point-wise 2D features, here the nearest neighbor interpolation method is calculated on the original feature map to obtain point-wise 3D features

[0143] Step 5: Use a multi-scale cross-modal fusion followed by distillation module, as follows:

[0144] The structure of this module includes 2D and 3D modality fusion and 2D modality-preserving knowledge distillation.

[0145] Since the backbone networks used in 2D and 3D are different, the generated features will be somewhat different, and it is not appropriate to directly fuse the two. To this end, first use MLP to generate the generated 2D features. Convert to Inspired by the “reset gate” in the recurrent neural network GRU model, the 2D features are transformed into Convert to It can be called a 3D learner; the 3D learner narrows the gap between the two features. The generated 2D features Element-wise accumulation to generate enhanced 2D features That is, the 2D part is enhanced by features and combined with the classifier (fully connected layer) to obtain the 2D prediction score S 2D , and do distillation at the result level (see step 6).

[0146] The cross-modal fusion part is only used during training. During the model inference process, the 3D-related branches are discarded to reduce the additional computational burden.

[0147] Step 6: Knowledge distillation with 2D modality preservation. and Use two independent classifiers to obtain the fusion prediction score S 3D2D and 2D prediction score S 2D and use KL divergence to calculate the distillation loss L between the two:

[0148] L=D KL (S 3D2D ||S 2D ),

[0149] And knowledge distillation provides rich spatial depth information to enhance 2D feature learning without losing any specific modality information of 2D.

[0150] Step 7: Calculate the fusion prediction score S 3D2D and 2D prediction score S 2D , is to obtain the KL divergence between the fusion prediction and the 2D prediction through the formula in step 6, which is used to estimate the difference between the two prediction results to achieve the evaluation function.

[0151] Step 8. The present invention uses 3D point clouds to assist 2D images in multimodal fusion into a single knowledge distillation module, obtains 3D information from multimodal data, and then refines it into a pure 2D network for two-dimensional semantic segmentation to obtain water surface segmentation results.

[0152] According to the above steps, a water surface segmentation method based on LiDAR point cloud-assisted image cross-modal fusion distillation is obtained.

[0153] It is worth noting that steps 5 and 6 of this embodiment introduce the fusion of multimodal information into a single knowledge distillation module. This method mainly uses this module to fuse and distill 2D and 3D multimodal information, and then feeds it into a 2D network for water surface segmentation, thus enabling the 3D point cloud to assist the 2D image.

[0154] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0155] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A water surface segmentation method based on cross-modal fusion distillation, characterized in that: The method comprises: Acquire 2D image data and 3D point cloud data; wherein the 2D image data and the 3D point cloud data are within the same field of view; Using a 2D network to obtain a 2D feature map of the 2D image data Perspectively projecting the 3D point cloud data onto a 2D plane according to an intrinsic parameter matrix and an extrinsic parameter matrix of a camera that acquires the 2D image data to obtain 2D plane pixels; Establishing a point-pixel correspondence relationship using the 2D plane pixels and the 3D point cloud data; Find the 2D feature map contained in the point-pixel correspondence Pixels in, get the containing pixels; Extract the 2D features corresponding to the pixels to obtain point-by-point 2D features Use the 3D network to obtain the 3D feature map F of the 3D point cloud data 3D ; Using the correspondence between points and voxels in the 3D feature map F 3D Perform the nearest neighbor interpolation method on it to obtain point-by-point 3D features The point-by-point 2D features are transformed using the MLP network Convert to 3D learner features The 3D learner features With the point-by-point 3D features Splicing and fusion to obtain fusion features Using the 2D prediction network according to the point-by-point 2D features Perform water surface segmentation prediction and use the classifier to obtain 2D prediction scores; Using the 3D prediction network according to the fusion features Perform water surface segmentation prediction and use the classifier to obtain the fusion prediction score; Calculating the distillation loss between the 2D prediction score and the fusion prediction score using KL divergence; Adjusting the 2D prediction network using a method for reducing the distillation loss to obtain an adjusted 2D prediction network; The regulated 2D prediction network is used to perform water surface prediction and segmentation.

2. The water surface segmentation method of cross-modal fusion distillation according to claim 1 is characterized in that The 2D feature map of the 2D image data is obtained by using a 2D network. Specifically include: Extracting multi-scale features of the 2D image data using a 2D network; Deconvolution is used to upsample the multi-scale features to the same resolution as the 2D image data to obtain a 2D feature map.

3. The water surface segmentation method of cross-modal fusion distillation according to claim 1 is characterized in that The calculation formula for obtaining 2D plane pixels by perspective-projecting the 3D point cloud data onto a 2D plane based on the intrinsic parameter matrix and extrinsic parameter matrix of the camera that acquires the 2D image data is: Among them, (u i ,v i ) represents the 2D plane pixel coordinates, K is the camera's intrinsic parameter matrix, T is the camera's extrinsic parameter matrix, (x i ,y i ,z i ) represents the 3D point cloud coordinates.

4. The water surface segmentation method of cross-modal fusion distillation according to claim 1 is characterized in that The point-pixel correspondence is: in, Indicates the floor operation, (u i ,v i ) represents the 2D plane pixel coordinate, O pixel Represents the point-pixel correspondence map, and N represents the number of 3D points.

5. The water surface segmentation method of cross-modal fusion distillation according to claim 1 is characterized in that The 2D features corresponding to the pixels are extracted to obtain point-by-point 2D features. After that, it also includes: The point-by-point 2D features The elements in are accumulated element by element.

6. The water surface segmentation method of cross-modal fusion distillation according to claim 1 is characterized in that In the 3D learner feature With the point-by-point 3D features Splicing and fusion to obtain fusion features After that, it also includes: The fusion feature The elements in are accumulated element by element.

7. The water surface segmentation method of cross-modal fusion distillation according to claim 1 is characterized in that The calculation formula for calculating the distillation loss between the 2D prediction score and the fusion prediction score using KL divergence is: Where N represents the number of 3D points, n is the nth point, and D KL It refers to KL divergence, S 3D2D represents the fusion prediction score, S 2D Represents the 2D prediction score.

8. The water surface segmentation method of cross-modal fusion distillation according to claim 1 is characterized in that The 2D network includes: a ResNet34 encoder and an FCN decoder.

9. The water surface segmentation method of cross-modal fusion distillation according to claim 1, characterized in that The 3D network includes: a PointNet++ encoder and a decoder.

10. A water surface segmentation system based on cross-modal fusion distillation, characterized in that: The system comprises: A data acquisition module, configured to acquire 2D image data and 3D point cloud data; the 2D image data and the 3D point cloud data are within the same field of view; 2D feature map acquisition module, used to obtain the 2D feature map of the 2D image data using a 2D network a projection module, configured to perspective-project the 3D point cloud data onto a 2D plane according to an intrinsic parameter matrix and an extrinsic parameter matrix of a camera that acquires the 2D image data, to obtain 2D plane pixels; A point-pixel correspondence establishment module, configured to establish a point-pixel correspondence using the 2D plane pixels and the 3D point cloud data; Contains a pixel search module for finding the 2D feature map contained in the point-pixel correspondence Pixels in, get the containing pixels; The point-by-point 2D feature acquisition module is used to extract the 2D features corresponding to the pixels and obtain the point-by-point 2D features. 3D feature map acquisition module, used to obtain the 3D feature map F of the 3D point cloud data using a 3D network 3D ; Point-by-point 3D feature acquisition module, used to use the correspondence between points and voxels in the 3D feature map F 3D Perform the nearest neighbor interpolation method on it to obtain point-by-point 3D features MLP conversion module is used to convert the point-by-point 2D features into Convert to 3D learner features Feature splicing module, used to combine the 3D learner features With the point-by-point 3D features Splicing and fusion to obtain fusion features 2D prediction module, used to use the 2D prediction network according to the point-by-point 2D features Perform water surface segmentation prediction and use the classifier to obtain 2D prediction scores; A fusion prediction module is used to use a 3D prediction network to predict the fusion features Perform water surface segmentation prediction and use the classifier to obtain the fusion prediction score; a distillation loss calculation module, configured to calculate the distillation loss between the 2D prediction score and the fusion prediction score using KL divergence; a 2D prediction network adjustment module, configured to adjust the 2D prediction network using a method for reducing the distillation loss to obtain an adjusted 2D prediction network; The water surface prediction and segmentation module is used to perform water surface prediction and segmentation using the adjusted 2D prediction network.

Images