A 3D point cloud enhancement method and system

Abstract

The application discloses a 3D point cloud enhancement method and system. The method comprises the following steps: acquiring an image data set and a corresponding 3D original point cloud set; training a target visual model based on the image data set; inputting the image data set into the frozen target visual model for prediction; converting the image data set based on the prediction result; fusing the predicted 3D point cloud set obtained through conversion and the 3D original point cloud set; training a target PointNet++ model based on the fused 3D point cloud set; acquiring an image to be recognized and a corresponding 3D original point cloud, and inputting the image to be recognized and the corresponding 3D original point cloud into the target visual model and the target PointNet++ model for prediction to obtain a target 3D point cloud. The method generates point clouds based on the target visual model prediction and fuses the point clouds with the original point clouds, effectively alleviating the point cloud sparsity problem; supervision is simultaneously performed in two-dimensional and three-dimensional spaces, so that the model has spatial perception and visual understanding capabilities, and the generalization performance is improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and point cloud processing technology, and more specifically, to a 3D point cloud enhancement method and system. Background Technology

[0002] When UAVs perform target monitoring tasks in complex environments, they often use lidar to scan the environment and generate 3D point clouds. However, due to the limited field of view, interference from obstructions, and limited scanning frequency of lidar, the collected point clouds often suffer from sparseness, missing data, or noise. This sparsity of point clouds makes it difficult for traditional point cloud-based segmentation methods to guarantee accuracy and robustness.

[0003] To address the issue of scarce point cloud data, researchers have proposed various multimodal fusion methods in existing technologies. For example, some methods attempt to fuse 2D image features with 3D point cloud features to supplement the insufficient point cloud information. Specifically, this involves extracting image features using convolutional neural networks and concatenating them with point cloud features to improve detection performance. However, this method still suffers from the following drawbacks:

[0004] 1. Strong dependence on point cloud generation: Most existing methods directly use the original point cloud, which does not fundamentally solve the problem of sparse point clouds;

[0005] 2. Simple feature fusion: It often uses feature splicing or weighted averaging, failing to fully utilize the spatial and semantic information of two-dimensional images;

[0006] 3. Limited supervision: Supervision is only conducted in three-dimensional space, lacking guidance from two-dimensional visual features, resulting in insufficient generalization ability of the model.

[0007] Therefore, there is an urgent need for a method that can fully utilize two-dimensional images to generate supplementary point clouds and achieve multimodal deep fusion during feature extraction to improve the overall performance of UAV target segmentation. Summary of the Invention

[0008] This invention provides a 3D point cloud enhancement method to solve the problem of sparse point clouds in the prior art.

[0009] To achieve the above objectives, this invention provides a 3D point cloud enhancement method, comprising: S1, acquiring an image dataset and the corresponding original 3D point cloud set; inputting the image dataset into the backbone network of the original visual model for feature extraction; inputting the extracted feature maps into a depth value decoder, a focal length offset decoder, and a segmentation decoder for prediction; S2, calculating a first total loss value based on the first prediction result; updating the original visual model in reverse according to the first total loss value; repeating S1-S2 until the first total loss value fluctuates within a first preset range, thereby obtaining a target visual model; S3, freezing the target visual model and inputting the image dataset into the frozen target visual model for prediction; and converting the image dataset into a target visual model based on the second prediction result. S4. Predict the 3D point cloud; Fuse the predicted 3D point cloud with the original 3D point cloud; Extract features from the fused 3D point cloud using the original PointNet++ model, and introduce a two-dimensional segmentation vector during the feature extraction process. Upsample the extracted feature map to obtain the third prediction result; S5. Calculate the second total loss value based on the third prediction result; Update the original PointNet++ model in reverse according to the second total loss value; Repeat S4~S5 until the second total loss value fluctuates within a second preset range, and obtain the target PointNet++ model; S6. Obtain the image to be identified and the corresponding original 3D point cloud, and input them into the target visual model and the target PointNet++ model for prediction to obtain the target 3D point cloud.

[0010] Optionally, S1 includes: acquiring an image dataset of the ship via a UAV and acquiring a corresponding 3D original point cloud of the ship via LiDAR; classifying each pixel in the image dataset to obtain the classified category of each pixel; inputting the classified image dataset into the backbone network of the original visual model for feature extraction; inputting the extracted feature maps into a depth value decoder, a focal length offset decoder, and a segmentation decoder for prediction, respectively, to obtain the first predicted depth, the first predicted camera intrinsic parameter, and the first segmentation feature map for each pixel; and generating the predicted category probability of each pixel in each layer of the first segmentation feature map based on logistic regression.

[0011] Optionally, S2 includes: calculating a segmentation loss value based on the predicted class probability of each pixel in each layer of the first segmentation feature map and the labeled class of each pixel; calculating a camera intrinsic loss value based on the predicted first predicted camera intrinsics and the real camera intrinsics; calculating a depth loss value based on the predicted first predicted depth of each pixel and the real depth of each pixel; calculating a first total loss value based on the segmentation loss value, the camera intrinsic loss value, and the depth loss value; updating the original visual model in reverse based on the first total loss value; repeating S1~S2 until the first total loss value fluctuates within a first preset range to stop, thereby obtaining the target visual model; wherein, the real depth of each pixel is calculated based on the 3D original point cloud and the real camera intrinsics.

[0012] Optionally, S3 includes: mapping the labeled category of each pixel in the image dataset to the 3D original point cloud to obtain the labeled category of each point in the 3D original point cloud; freezing the target visual model and inputting the image dataset corresponding to the 3D original point cloud into the frozen target visual model for prediction to obtain the second prediction depth, second prediction camera intrinsic parameters, and second segmentation feature map of each pixel; and transforming the image dataset according to the second prediction depth and second prediction camera intrinsic parameters of each pixel to obtain the predicted 3D point cloud.

[0013] Optionally, S4 includes: fusing the predicted 3D point cloud with the original 3D point cloud; performing multi-stage feature extraction on the fused 3D point cloud using the original PointNet++ model, and introducing a two-dimensional segmentation vector during the multi-stage feature extraction process to obtain 3D point cloud features with two-dimensional segmentation features; wherein the two-dimensional segmentation vector is generated by a self-attention mechanism from the second segmentation feature map; and upsampling the 3D point cloud features with two-dimensional segmentation features to obtain the predicted class probability of each point in the fused 3D point cloud.

[0014] Optionally, S5 includes: mapping the predicted class probability of each point in the fused 3D point cloud to the image dataset to obtain the mapped predicted class probability of each pixel; calculating the point cloud loss value based on the predicted class probability of each point in the fused 3D point cloud and the labeled class of each point in the original 3D point cloud; calculating the image loss value based on the mapped predicted class probability of each pixel and the labeled class of each pixel; calculating the second total loss value based on the point cloud loss value and the image loss value; updating the original PointNet++ model in reverse based on the second total loss value; repeating S4~S5 until the second total loss value fluctuates within a second preset range, thereby obtaining the target PointNet++ model.

[0015] Optionally, the step of performing multi-stage feature extraction on the fused 3D point cloud using the original PointNet++ model, and introducing a two-dimensional segmentation vector during the multi-stage feature extraction process to obtain 3D point cloud features with two-dimensional segmentation features, includes: generating a two-dimensional segmentation vector by applying a self-attention mechanism to the second segmentation feature map; performing initial feature extraction on the fused 3D point cloud using the original PointNet++ model to obtain fused 3D point cloud features; and querying the two-dimensional segmentation vector in the fused 3D point cloud features to obtain 3D point cloud features that match the two-dimensional feature information. The process involves: 1) Adding the 3D point cloud features that match the 2D feature information to the fused 3D point cloud features before querying, resulting in a first-stage 3D point cloud feature; 2) Using the first-stage 3D point cloud features as the fused 3D point cloud features, repeating the query and addition process to obtain a second-stage 3D point cloud feature; 3) Using the second-stage 3D point cloud features as the fused 3D point cloud features, repeating the query and addition process to obtain a third-stage 3D point cloud feature; and 4) Using the third-stage 3D point cloud features as the fused 3D point cloud features, repeating the query and addition process to obtain the 3D point cloud feature with 2D segmentation features.

[0016] On the other hand, the present invention provides a 3D point cloud enhancement system, which includes: a first prediction unit, used to acquire an image dataset and a corresponding 3D original point cloud set, input the image dataset into the backbone network of the original visual model for feature extraction; input the extracted feature maps into a depth value decoder, a focal length offset decoder, and a segmentation decoder for prediction; a first reverse update unit, used to calculate a first total loss value based on the first prediction result; reverse update the original visual model according to the first total loss value; repeat S1~S2 until the first total loss value fluctuates within a first preset range and stops, to obtain a target visual model; a second prediction and conversion unit, used to freeze the target visual model and input the image dataset into the frozen target visual model for prediction; convert the image dataset into a predicted value based on the second prediction result. The system comprises: a 3D point cloud; a third prediction unit, used to fuse the predicted 3D point cloud with the original 3D point cloud; the fused 3D point cloud is used for feature extraction using the original PointNet++ model, and a two-dimensional segmentation vector is introduced during the feature extraction process; the extracted feature map is upsampled to obtain the third prediction result; a second reverse update unit, used to calculate the second total loss value based on the third prediction result; the original PointNet++ model is updated in reverse according to the second total loss value; S4~S5 are repeated until the second total loss value fluctuates within a second preset range, and the target PointNet++ model is obtained; and a generation unit, used to acquire the image to be identified and the corresponding original 3D point cloud, input them into the target visual model and the target PointNet++ model for prediction, and obtain the target 3D point cloud.

[0017] Optionally, the first prediction unit includes: an acquisition subunit, used to acquire an image dataset of the ship via a UAV and a corresponding 3D original point cloud of the ship via a LiDAR; a calibration subunit, used to classify each pixel in the image dataset to obtain the calibrated class of each pixel; a feature extraction subunit, used to input the calibrated image dataset into the backbone network of the original visual model for feature extraction; a first prediction subunit, used to input the extracted feature map into a depth value decoder, a focal length offset decoder, and a segmentation decoder for prediction, obtaining a first predicted depth, a first predicted camera intrinsic parameter, and a first segmentation feature map for each pixel; and a generation subunit, used to generate the predicted class probability of each pixel in each layer of the first segmentation feature map based on logistic regression.

[0018] Optionally, the first reverse update unit includes: a first calculation subunit, used to calculate a segmentation loss value based on the predicted class probability of each pixel in each layer of the first segmentation feature map and the labeled class of each pixel; a second calculation subunit, used to calculate a camera intrinsic loss value based on the predicted first predicted camera intrinsic parameters and the real camera intrinsic parameters; a third calculation subunit, used to calculate a depth loss value based on the predicted first predicted depth of each pixel and the real depth of each pixel; a fourth calculation subunit, used to calculate the first total loss value based on the segmentation loss value, the camera intrinsic loss value, and the depth loss value; a first reverse update subunit, used to reverse update the original visual model based on the first total loss value; repeating the first prediction unit and the first reverse update unit until the first total loss value fluctuates within a first preset range to stop, thereby obtaining the target visual model; wherein, the real depth of each pixel is calculated based on the 3D original point cloud and the real camera intrinsic parameters.

[0019] The beneficial effects of this invention are:

[0020] This invention provides a 3D point cloud enhancement method and system. The method obtains a target visual model through training on a two-dimensional image, predicts and generates a point cloud based on the target visual model, and merges it with the original point cloud to effectively alleviate the point cloud sparsity problem. Two-dimensional segmentation features are introduced in the feature extraction stage to realize the interactive fusion of two-dimensional visual features and three-dimensional point cloud features, thereby enhancing feature expression ability and model segmentation accuracy. Supervision is performed simultaneously in two-dimensional and three-dimensional spaces, enabling the model to have both spatial perception and visual understanding capabilities, thus improving generalization performance. Attached Figure Description

[0021] Figure 1 This is a flowchart of a 3D point cloud enhancement method provided in an embodiment of the present invention;

[0022] Figure 2This is a flowchart illustrating the prediction of an image dataset provided in an embodiment of the present invention;

[0023] Figure 3 This is a flowchart of generating a target visual model provided in an embodiment of the present invention;

[0024] Figure 4 This is a flowchart of generating a predicted 3D point cloud provided in an embodiment of the present invention;

[0025] Figure 5 This is a flowchart of the predicted class probability of each point in the fused 3D point cloud provided by an embodiment of the present invention;

[0026] Figure 6 This is a flowchart of generating the target PointNet++ model provided in an embodiment of the present invention;

[0027] Figure 7 This is a flowchart of generating 3D point cloud features for two-dimensional segmentation, provided in an embodiment of the present invention.

[0028] Figure 8 This is a schematic diagram of the structure of a 3D point cloud enhancement system provided in an embodiment of the present invention;

[0029] Figure 9 This is a schematic diagram of the structure of the first prediction unit provided in an embodiment of the present invention;

[0030] Figure 10 This is a schematic diagram of the structure of the first reverse update unit provided in an embodiment of the present invention;

[0031] Figure 11 This is a schematic diagram of the structure of the second prediction and conversion unit provided in an embodiment of the present invention;

[0032] Figure 12 This is a schematic diagram of the structure of the third prediction unit provided in an embodiment of the present invention;

[0033] Figure 13 This is a schematic diagram of the structure of the second reverse update unit provided in an embodiment of the present invention;

[0034] Figure 14 This is a schematic diagram of the extraction subunit provided in an embodiment of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0036] Figure 1 This is a flowchart of a 3D point cloud enhancement method provided in an embodiment of the present invention, such as... Figure 1 As shown, the method includes:

[0037] S1. Obtain the image dataset and the corresponding 3D original point cloud set. Input the image dataset into the backbone network of the original visual model for feature extraction. Input the extracted feature maps into the depth value decoder, focal length offset decoder, and segmentation decoder for prediction.

[0038] In one optional implementation, Figure 2 This is a flowchart of the image dataset prediction provided in the embodiments of the present invention, such as... Figure 2 As shown, S1 includes:

[0039] S11. Obtain image datasets of ships through drones, and obtain 3D original point clouds of ships through lidar;

[0040] In specific application scenarios, such as ship monitoring tasks, drones can acquire image datasets of ships, while LiDAR can simultaneously acquire raw 3D point clouds of the ships. Alternatively, publicly available datasets containing synchronized 2D images and 3D point clouds, such as the KITTI dataset, can be used directly.

[0041] S12. Classify each pixel in the image dataset to obtain the labeled class of each pixel;

[0042] Specifically, each pixel in each image of the image dataset is manually categorized to obtain the categorized category of each pixel in each image of the image dataset (i.e., which type of ship the current pixel belongs to).

[0043] S13. Input the labeled image dataset into the backbone network of the original visual model for feature extraction;

[0044] Specifically, the backbone network of the original visual model in this application preferably uses ViT-small, which is based on the Transformer architecture and has the ability to model global features.

[0045] Taking a two-dimensional image from the image dataset as an example: the two-dimensional image after being labeled with the category is input into the backbone network of ViT-small for feature extraction to obtain the image feature map; the same method is used to obtain the image feature map corresponding to each two-dimensional image for all two-dimensional images (i.e., the image dataset).

[0046] S14. Input the extracted feature maps into the depth value decoder, focal length offset decoder, and segmentation decoder respectively for prediction, and obtain the first predicted depth, first predicted camera intrinsic parameters, and first segmentation feature map for each pixel in a one-to-one correspondence.

[0047] Taking the image feature map corresponding to a two-dimensional image as an example: the image feature map is input into the depth value decoder for prediction to obtain the first depth map;

[0048] The depth decoder consists of three upsampling layers (such as the Unet decoder) and a 1*1*3 convolution; the first depth map consists of three layers: the first layer represents the relative distance (which gives the relative distance of each pixel), the second layer represents the scaling scale (which gives the scaling scale of each pixel), and the third layer represents the offset scale (which gives the offset scale of each pixel).

[0049] Taking a single pixel as an example: the first predicted depth of the current pixel can be calculated based on the relative distance, scaling scale, and offset scale, using the following formula:

[0050] ;

[0051] in, The distance is relative. To scale, For the offset scale, The first predicted depth for the current pixel. The relative distance to the current pixel. The scaling factor for the current pixel. This represents the offset scale of the current pixel.

[0052] Taking the image feature map corresponding to a two-dimensional image as an example: the image feature map is input into the focal length offset decoder for prediction to obtain the first predicted camera intrinsic parameters;

[0053] The focal length offset decoder consists of 3 convolutional kernels and one fully convolutional kernel (1*1*4); the first predictive camera intrinsic parameters are: predicted focal length x, predicted focal length y, predicted offset x, and predicted offset y.

[0054] Taking the image feature map corresponding to a two-dimensional image as an example: the image feature map is input into the segmentation decoder for prediction to obtain the first segmentation feature map;

[0055] The segmentation decoder consists of 3 upsampling layers (such as the Unet decoder) and a 1*1*N convolution, where N represents the total number of categories; the first segmentation feature map consists of N layers, with each layer representing one category.

[0056] S15. Generate the predicted class probability of each pixel in each layer of the first segmentation feature map based on logistic regression.

[0057] Specifically, let's take the first segmentation feature map corresponding to a two-dimensional image as an example:

[0058] There are two categories: A and B. The first segmentation feature map consists of two layers: the first layer of the first segmentation feature map represents class A, and the second layer represents class B. The probability of each pixel being predicted as class A is obtained by using a logistic regression algorithm on the first layer of the first segmentation feature map; the probability of each pixel being predicted as class B is obtained by using a logistic regression algorithm on the second layer of the first segmentation feature map.

[0059] S2. Calculate the first total loss value based on the first result obtained from the prediction; update the original visual model in reverse according to the first total loss value; repeat S1~S2 until the first total loss value fluctuates within a first preset range and then stop to obtain the target visual model.

[0060] In one optional implementation, Figure 3 This is a flowchart of generating a target visual model provided in an embodiment of the present invention; as shown below. Figure 3 As shown, S2 includes:

[0061] S21. Based on the predicted class probability of each pixel in each layer of the first segmentation feature map and the labeled class of each pixel, the segmentation loss value is calculated.

[0062] Specifically, taking the first layer of the segmentation feature map corresponding to a two-dimensional image as an example, this layer belongs to class A:

[0063] The loss value of the current layer is calculated based on the predicted class probability of each pixel in the current layer of the first segmentation feature map and the labeled class of each pixel; the formula is as follows:

[0064]

[0065] in, This represents the loss value for the current layer. The width of the first segmentation feature map; The height of the first segmentation feature map; Let the pixel in row i and column j be categorized; if it is background, then... =0, if it is class A, then =1, if it is another class, then =0; The predicted class probability of the pixel in the i-th row and j-th column in the current layer of the first segmentation feature map is the probability of being predicted as class A.

[0066] The loss values ​​of all layers are obtained using the method described above. These loss values ​​are then summed and averaged to obtain the segmentation loss value for the current 2D image. Finally, the segmentation loss values ​​of all 2D images are summed and averaged to obtain the segmentation loss value for the image dataset. .

[0067] S22. Calculate the camera intrinsic loss value based on the first predicted camera intrinsic parameters and the real camera intrinsic parameters.

[0068] Specifically, the camera intrinsic loss value is calculated based on the predicted first camera intrinsic parameters (first predicted focal length x, first predicted focal length y, first predicted offset x, first predicted offset y) and the actual camera intrinsic parameters (actual focal length x, actual focal length y, actual offset x, actual offset y); the specific formula is as follows:

[0069] ;

[0070] in, For camera internal parameters, Focal length For offset, This represents the camera intrinsic parameter loss value. Let x be the first predicted focal length. For the true focal length x, Let y be the first predicted focal length. For the true focal length y, Let x be the first predicted offset. For the actual offset x, Let y be the first predicted offset. This represents the actual offset y.

[0071] S23. Calculate the depth loss value based on the first predicted depth of each pixel and the true depth of each pixel;

[0072] The true depth of each pixel is calculated based on the 3D original point cloud and the intrinsic parameters of the real camera.

[0073] Specifically, a two-dimensional image corresponds to a 3D original point cloud; taking a point in the 3D original point cloud (the current point) as an example, it is mapped to a pixel in the two-dimensional image;

[0074] The specific formula is as follows:

[0075]

[0076]

[0077]

[0078] Where x is the x-value of the current point, y is the y-value of the current point, and z is the z-value of the current point. For the true focal length x, For the actual offset x, For the true focal length y, For the true offset y, Let x be the pixel x in the 2D image corresponding to the current point x. Let y be the pixel value in the 2D image corresponding to the current point y. This represents the true depth of the pixel in the 2D image corresponding to the current point.

[0079] Taking a 2D image corresponding to a 3D original point cloud as an example:

[0080] The depth loss value of the current 2D image is calculated based on the true depth of each pixel in the current 2D image corresponding to the current original 3D point cloud, and the first predicted depth of each pixel in the current 2D image. The formula is as follows:

[0081]

[0082] in, This represents the depth loss value of the current 2D image. The width of the current 2D image. The height of the current two-dimensional image, Let this be the true depth of the pixel in the i-th row and j-th column of the current 2D image. The first predicted depth of the pixel in the i-th row and j-th column of the current two-dimensional image.

[0083] The depth loss value of the image dataset is obtained by summing and averaging the depth loss values ​​of all 2D images. .

[0084] S24. Calculate the first total loss value based on the segmentation loss value, camera intrinsic parameter loss value, and depth loss value;

[0085] Specifically, the segmentation loss value of the image dataset Camera intrinsic parameter loss value Depth loss values ​​of image datasets Summing and averaging yields the first total loss value.

[0086] S25. Update the original visual model in reverse according to the first total loss value; repeat S1~S2 until the first total loss value fluctuates within a preset range and stop to obtain the target visual model.

[0087] The backbone network, depth value decoder, focal length offset decoder, and segmentation decoder of the original visual model are updated in reverse according to the first total loss value; S1~S2 are repeated until the first total loss value fluctuates within a first preset range (±0.1%), and the target visual model is obtained.

[0088] S3. Freeze the target visual model and input the image dataset into the frozen target visual model for prediction; based on the second result obtained from the prediction, convert the image dataset into a predicted 3D point cloud;

[0089] In one optional implementation, Figure 4 This is a flowchart of the generation and prediction of a 3D point cloud provided in an embodiment of the present invention; as shown below. Figure 4 As shown, S3 includes:

[0090] S31. Map the labeled category of each pixel in the image dataset to the 3D original point cloud set to obtain the labeled category of each point in the 3D original point cloud set;

[0091] Specifically, in S21, the category of each pixel in the image dataset has been labeled to obtain the labeled category of each pixel; based on this, the labeled category of each pixel in the image dataset is mapped to the 3D original point cloud to obtain the labeled category of each point in the 3D original point cloud; that is, the current pixel in the image dataset corresponds to the current point in the 3D original point cloud, so the labeled category of the current pixel is also the labeled category of the current point.

[0092] S32. Freeze the target visual model and input the image dataset corresponding to the 3D original point cloud into the frozen target visual model for prediction to obtain the second prediction depth, the second prediction camera intrinsic parameters, and the second segmentation feature map for each pixel.

[0093] Specifically, the target visual model is frozen, and the image dataset corresponding to the 3D original point cloud is input into the frozen target visual model for prediction. That is, prediction is made based on the depth value decoder, focal length offset decoder, and segmentation decoder in the target visual model to obtain the second predicted depth, the second predicted camera intrinsic parameters, and the second segmentation feature map for each pixel.

[0094] S33. The image dataset is transformed according to the second predicted depth and the second predicted camera intrinsic parameters of each pixel to obtain the predicted 3D point cloud.

[0095] Specifically, taking the conversion of a single pixel in an image dataset as an example, the conversion formula is as follows:

[0096]

[0097]

[0098]

[0099] in, The x-value of the current pixel. The y-value of the current pixel. For the second prediction offset x, For the second predicted focal length x, For the second prediction offset y, For the second predicted focal length y, The second predicted depth for the current pixel. The x-values ​​of the predicted 3D points obtained through transformation. The y-values ​​of the predicted 3D points obtained through transformation. The z-value of the predicted 3D point obtained through transformation.

[0100] After converting all pixels in the image dataset, the predicted 3D point cloud is obtained.

[0101] S4. Fuse the predicted 3D point cloud with the original 3D point cloud; use the original PointNet++ model to extract features from the fused 3D point cloud, and introduce a two-dimensional segmentation vector during the feature extraction process. Upsample the extracted feature map to obtain the third predicted result.

[0102] In one optional implementation, Figure 5 This is a flowchart illustrating the predicted class probability of each point in the fused 3D point cloud provided by an embodiment of the present invention; for example... Figure 5 As shown, S4 includes:

[0103] S41. Merge the predicted 3D point cloud with the original 3D point cloud;

[0104] The predicted 3D point cloud is fused with the original 3D point cloud to make the original 3D point cloud denser.

[0105] S42. The fused 3D point cloud is subjected to multi-stage feature extraction using the original PointNet++ model, and a two-dimensional segmentation vector is introduced during the multi-stage feature extraction process to obtain 3D point cloud features with two-dimensional segmentation features; wherein, the two-dimensional segmentation vector is generated by the self-attention mechanism of the second segmentation feature map.

[0106] In one optional implementation, Figure 7 This is a flowchart of generating 3D point cloud features for two-dimensional segmentation, provided in an embodiment of the present invention; such as... Figure 7 As shown, S42 includes:

[0107] S421. Generate a two-dimensional segmentation vector by applying a self-attention mechanism to the second segmentation feature map;

[0108] S422. The fused 3D point cloud set is used to perform the first feature extraction using the original PointNet++ model to obtain the fused 3D point cloud features.

[0109] S423. Query the two-dimensional segmentation vector in the fused 3D point cloud features to obtain the 3D point cloud features that match the two-dimensional feature information; add the 3D point cloud features that match the two-dimensional feature information to the fused 3D point cloud features before querying to obtain the first-stage 3D point cloud features.

[0110] S424. The 3D point cloud features of the first stage are used as the fused 3D point cloud features. The query and addition are repeated to obtain the 3D point cloud features of the second stage.

[0111] Specifically, the two-dimensional segmentation vector is queried in the first-stage 3D point cloud features to obtain the first-stage 3D point cloud features that match the two-dimensional feature information; the first-stage 3D point cloud features that match the two-dimensional feature information are added to the first-stage 3D point cloud features before the query to obtain the second-stage 3D point cloud features.

[0112] S425. The two-stage 3D point cloud features are used as the fused 3D point cloud features. The query and addition are repeated to obtain the three-stage 3D point cloud features.

[0113] Specifically, the two-dimensional segmentation vector is queried in the two-stage 3D point cloud features to obtain the two-stage 3D point cloud features that match the two-dimensional feature information; the two-stage 3D point cloud features that match the two-dimensional feature information are added to the two-stage 3D point cloud features before the query to obtain the three-stage 3D point cloud features.

[0114] S426. The three-stage 3D point cloud features are used as the fused 3D point cloud features. The query and addition are repeated to obtain the 3D point cloud features of the two-dimensional segmentation features.

[0115] Specifically, the two-dimensional segmentation vector is queried in the three-stage 3D point cloud features to obtain the three-stage 3D point cloud features that match the two-dimensional feature information; the three-stage 3D point cloud features that match the two-dimensional feature information are added to the three-stage 3D point cloud features before the query to obtain the four-stage 3D point cloud features, which are the 3D point cloud features of the two-dimensional segmentation features.

[0116] S43. Upsample the 3D point cloud features of the two-dimensional segmentation features to obtain the predicted class probability of each point in the fused 3D point cloud set.

[0117] Specifically, the 3D point cloud features of the 2D segmentation features are upsampled to obtain an updated 3D point cloud feature map (i.e., width*height*channel*N); the updated 3D point cloud feature map consists of N layers, with each layer representing a category.

[0118] Specifically, assuming there are two categories: category A and category B, the updated 3D point cloud feature map consists of two layers. The first layer of the updated 3D point cloud feature map represents category A, and the second layer represents category B. The first layer of the updated 3D point cloud feature map is used to obtain the probability of each point in the fused 3D point cloud set being predicted as category A through a logistic regression algorithm. The second layer of the updated 3D point cloud feature map is used to obtain the probability of each point in the fused 3D point cloud set being predicted as category B through a logistic regression algorithm.

[0119] S5. Calculate the second total loss value based on the predicted third result; update the original PointNet++ model in reverse according to the second total loss value; repeat S4~S5 until the second total loss value fluctuates within a second preset range, and obtain the target PointNet++ model;

[0120] In one optional implementation, Figure 6 This is a flowchart of generating the target PointNet++ model provided in an embodiment of the present invention; as shown below. Figure 6 As shown, S5 includes:

[0121] S51. Map the predicted class probability of each point in the fused 3D point cloud to the image dataset to obtain the mapped predicted class probability of each pixel.

[0122] S52. Calculate the point cloud loss value based on the predicted class probability of each point in the fused 3D point cloud set and the labeled class of each point in the original 3D point cloud set.

[0123] Specifically, taking a layer that updates the 3D point cloud feature map as an example, this layer belongs to category A:

[0124] The point cloud loss value for the current layer is calculated based on the predicted class probability of each point in the current layer of the updated 3D point cloud feature map and the labeled class of each point; the formula is as follows:

[0125]

[0126] in, This represents the point cloud loss value for the current layer. To update the width, To update the height of the 3D point cloud feature map To update the channels of the 3D point cloud feature map; Let the label of the point in the i-th row, j-th column, and z-th highest point be defined. If it is the background, then... =0, if it is class A, then =1, if it is another class, then =0; The probability of predicting the class of the point in the i-th row, j-th column, and z-th highest point in the current layer of the updated 3D point cloud feature map is the probability of predicting it as class A.

[0127] The point cloud loss values ​​for all layers are obtained using the above method. The total point cloud loss value is obtained by summing and averaging the point cloud loss values ​​for all layers.

[0128] S53. Calculate the image loss value based on the predicted class probability of each pixel and the labeled class of each pixel.

[0129] Specifically, taking the mapping of a layer of the 3D point cloud feature map in the image dataset as an example, this layer belongs to class A:

[0130] The predicted class probability of each point in the current layer of the updated 3D point cloud feature map is mapped to the image dataset, resulting in the mapped predicted class probability for each pixel. The image loss value for the current layer is calculated based on the mapped predicted class probability and the labeled class of each pixel; the formula is as follows:

[0131]

[0132] in, This represents the image loss value for the current layer. For each image in the image dataset, the width, The height of each image in the image dataset. Let the pixel in row i and column j be categorized; if it is background, then... =0, if it is class A, then =1, if it is another class, then =0; The mapping predicts the class probability of the pixel in the i-th row and j-th column, that is, the probability of predicting it as class A.

[0133] The image loss values ​​of all layers are obtained by the above method. The total image loss value is obtained by summing and averaging the image loss values ​​of all layers.

[0134] S54. Calculate the second total loss value based on the point cloud loss value and the image loss value;

[0135] The second total loss value is obtained by multiplying the total point cloud loss value by 'a' and the total image loss value by 'b', and then adding them together.

[0136] S55. Update the original PointNet++ model in reverse according to the second total loss value; repeat S4~S5 until the second total loss value fluctuates within the second preset range (±0.1%), and then stop to obtain the target PointNet++ model.

[0137] S6. Obtain the image to be identified and its corresponding 3D original point cloud, and input them into the target visual model and the target PointNet++ model for prediction to obtain the target 3D point cloud.

[0138] Specifically, the image to be identified and its corresponding 3D raw point cloud are acquired and input into the target visual model and the target PointNet++ model for prediction. This yields the target 3D point cloud and the predicted class probability of each point in the target 3D point cloud. The target 3D point cloud is richer in information than the 3D raw point cloud. Based on the predicted class probability of each point in the target 3D point cloud, the target (ship) can be accurately segmented.

[0139] Figure 8 This is a schematic diagram of the structure of a 3D point cloud enhancement system provided in an embodiment of the present invention; as shown below. Figure 8 As shown, the system includes:

[0140] The first prediction unit 201 is used to acquire the image dataset and the corresponding 3D original point cloud, input the image dataset into the backbone network of the original visual model for feature extraction, and input the extracted feature maps into the depth value decoder, focal length offset decoder and segmentation decoder for prediction.

[0141] In one optional implementation, Figure 9 This is a schematic diagram of the structure of the first prediction unit provided in an embodiment of the present invention; as shown below. Figure 9 As shown, the first prediction unit 201 includes:

[0142] The acquisition subunit 2011 is used to acquire image datasets of ships via UAVs and to acquire 3D raw point clouds of ships via LiDAR.

[0143] The calibration subunit 2012 is used to classify each pixel in the image dataset to obtain the calibrated class of each pixel;

[0144] The feature extraction subunit 2013 is used to input the labeled image dataset into the backbone network of the original visual model for feature extraction;

[0145] The first prediction subunit 2014 is used to input the extracted feature map into the depth value decoder, the focal length offset decoder, and the segmentation decoder for prediction, and obtain the first predicted depth, the first predicted camera intrinsic parameter, and the first segmentation feature map for each pixel in a one-to-one correspondence.

[0146] The generation subunit 2015 is used to generate the predicted class probability of each pixel in each layer of the first segmentation feature map based on logistic regression.

[0147] The first reverse update unit 202 is used to calculate the first total loss value based on the first result obtained from the prediction; to reverse update the original visual model according to the first total loss value; to repeat S1~S2 until the first total loss value fluctuates within a first preset range and then stop, thereby obtaining the target visual model;

[0148] In one optional implementation, Figure 10 This is a schematic diagram of the structure of the first reverse update unit provided in an embodiment of the present invention; as shown below. Figure 10 As shown, the first reverse update unit 202 includes:

[0149] The first computational subunit 2021 is used to calculate the segmentation loss value based on the predicted class probability of each pixel in each layer of the first segmentation feature map and the labeled class of each pixel.

[0150] The second calculation subunit 2022 is used to calculate the camera intrinsic loss value based on the first predicted camera intrinsic parameters and the real camera intrinsic parameters obtained from the prediction.

[0151] The third calculation subunit 2023 is used to calculate the depth loss value based on the first predicted depth of each pixel and the true depth of each pixel.

[0152] The fourth calculation subunit 2024 is used to calculate the first total loss value based on the segmentation loss value, the camera intrinsic parameter loss value, and the depth loss value;

[0153] The first reverse update subunit 2025 is used to reverse update the original visual model according to the first total loss value; repeat the first prediction unit 201 and the first reverse update unit 202 until the first total loss value fluctuates within a first preset range and then stop to obtain the target visual model.

[0154] The true depth of each pixel is calculated based on the 3D original point cloud and the intrinsic parameters of the real camera.

[0155] The second prediction and conversion unit 203 is used to freeze the target visual model and input the image dataset into the frozen target visual model for prediction; based on the second result obtained from the prediction, the image dataset is converted into a predicted 3D point cloud;

[0156] In one optional implementation, Figure 11 This is a schematic diagram of the structure of the second prediction and conversion unit provided in an embodiment of the present invention; as shown below. Figure 11As shown, the second prediction and conversion unit 203 includes:

[0157] The corresponding subunit 2031 is used to map the labeled category of each pixel in the image dataset to the 3D original point cloud set, so as to obtain the labeled category of each point in the 3D original point cloud set;

[0158] The second prediction subunit 2032 is used to freeze the target visual model and input the image dataset corresponding to the 3D original point cloud into the frozen target visual model for prediction, so as to obtain the second prediction depth, the second prediction camera intrinsic parameters, and the second segmentation feature map for each pixel.

[0159] The transformation subunit 2033 is used to transform the image dataset according to the second predicted depth and the second predicted camera intrinsic parameters of each pixel to obtain the predicted 3D point cloud.

[0160] The third prediction unit 204 is used to fuse the predicted 3D point cloud with the original 3D point cloud; the fused 3D point cloud is used to extract features using the original PointNet++ model, and a two-dimensional segmentation vector is introduced during the feature extraction process. The extracted feature map is upsampled to obtain the third prediction result.

[0161] In one optional implementation, Figure 12 This is a schematic diagram of the structure of the third prediction unit provided in an embodiment of the present invention; as shown below. Figure 12 As shown, the third prediction unit 204 includes:

[0162] The fusion subunit 2041 is used to fuse the predicted 3D point cloud with the original 3D point cloud;

[0163] Extraction subunit 2042 is used to perform multi-stage feature extraction on the fused 3D point cloud using the original PointNet++ model, and introduces a two-dimensional segmentation vector in the multi-stage feature extraction process to obtain 3D point cloud features with two-dimensional segmentation features; wherein, the two-dimensional segmentation vector is generated by the self-attention mechanism of the second segmentation feature map;

[0164] In one optional implementation, Figure 14 This is a schematic diagram of the extraction subunit provided in an embodiment of the present invention, as shown below. Figure 14 As shown, the extraction subunit 2042 includes:

[0165] The generation module 20421 is used to generate a two-dimensional segmentation vector from the second segmentation feature map using a self-attention mechanism;

[0166] Extraction module 20422 is used to perform initial feature extraction on the fused 3D point cloud using the original PointNet++ model to obtain the fused 3D point cloud features;

[0167] The first query module 20423 is used to query the two-dimensional segmentation vector in the fused 3D point cloud features to obtain the 3D point cloud features that match the two-dimensional feature information; and to add the 3D point cloud features that match the two-dimensional feature information to the fused 3D point cloud features before querying to obtain the first-stage 3D point cloud features.

[0168] The second query module 20424 is used to repeatedly query and add the 3D point cloud features of the first stage as the fused 3D point cloud features to obtain the 3D point cloud features of the second stage.

[0169] The third query module 20425 is used to repeatedly query and add the 3D point cloud features of the second stage as the fused 3D point cloud features to obtain the 3D point cloud features of the third stage.

[0170] The fourth query module 20426 is used to repeatedly query and add the three-stage 3D point cloud features as the fused 3D point cloud features to obtain the 3D point cloud features of the two-dimensional segmentation features.

[0171] The upsampling subunit 2043 is used to upsample the 3D point cloud features of the two-dimensional segmentation features to obtain the predicted class probability of each point in the fused 3D point cloud set.

[0172] The second reverse update unit 205 is used to calculate the second total loss value based on the predicted third result; to reverse update the original PointNet++ model according to the second total loss value; to repeat S4~S5 until the second total loss value fluctuates within a second preset range and then stop, thus obtaining the target PointNet++ model;

[0173] In one optional implementation, Figure 13 This is a schematic diagram of the structure of the second reverse update unit provided in an embodiment of the present invention; as shown below. Figure 13 As shown, the second reverse update unit 205 includes:

[0174] The mapping subunit 2051 is used to map the predicted class probability of each point in the fused 3D point cloud to the image dataset to obtain the mapped predicted class probability of each pixel.

[0175] The fifth calculation subunit 2052 is used to calculate the point cloud loss value based on the predicted class probability of each point in the fused 3D point cloud set and the labeled class of each point in the original 3D point cloud set.

[0176] The sixth calculation subunit 2053 is used to calculate the image loss value based on the mapping prediction class probability of each pixel and the labeled class of each pixel;

[0177] The seventh calculation subunit 2054 is used to calculate the second total loss value based on the point cloud loss value and the image loss value;

[0178] The second reverse update subunit 2055 is used to reverse update the original PointNet++ model according to the second total loss value; the third prediction unit 204 and the second reverse update unit 205 are repeated until the second total loss value fluctuates within a second preset range, and the target PointNet++ model is obtained.

[0179] The generation unit 206 is used to acquire the image to be identified and the corresponding 3D original point cloud, and input it into the target visual model and the target PointNet++ model for prediction to obtain the target 3D point cloud.

[0180] The system of the present invention corresponds to the method described above, and the specific implementation of the system will not be repeated here.

[0181] The beneficial effects of this invention are:

[0182] This invention provides a 3D point cloud enhancement method and system. The method trains a target visual model using a 2D image, predicts and generates a point cloud based on the target visual model, and fuses it with the original sparse point cloud, substantially increasing the density and coverage of the point cloud. It innovatively introduces a cross-attention mechanism, enabling the semantic features of the 2D image to actively guide and enhance the learning of 3D point cloud features. This fusion method is no longer a simple feature superposition but establishes an adaptive association between semantics and geometry, significantly improving the quality of feature representation and the accuracy of segmentation boundaries. By simultaneously using 3D point cloud loss and 2D image loss to supervise the model during the training phase of the target PointNet++ model, the model is forced to learn not only the structural information of 3D space but also the semantic information of the 2D image. This dual supervision greatly improves the model's generalization ability and robustness, making it more stable in complex scenes.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 3D point cloud enhancement method, characterized in that, The method comprises the following steps: S1, obtaining an image data set and a corresponding 3D original point cloud set, and inputting the image data set into a backbone network of an original visual model for feature extraction; inputting the extracted feature maps into a depth value decoder, a focal length offset decoder and a segmentation decoder respectively for prediction; S2, calculating a first total loss value based on the first result obtained by prediction; updating the original visual model in reverse according to the first total loss value; repeating S1-S2 until the first total loss value fluctuates within a first preset range, and stopping to obtain a target visual model; S3, freezing the target visual model, inputting the image data set into the frozen target visual model for prediction, and converting the image data set based on the second result obtained by prediction to obtain a predicted 3D point cloud set; S4, fusing the predicted 3D point cloud set with the 3D original point cloud set, extracting features of the fused 3D point cloud set by using an original PointNet++ model, introducing a two-dimensional segmentation vector in the feature extraction process, upsampling the extracted feature maps to obtain a third prediction result; S5, calculating a second total loss value based on the third prediction result; updating the original PointNet++ model in reverse according to the second total loss value; repeating S4-S5 until the second total loss value fluctuates within a second preset range, and stopping to obtain a target PointNet++ model; S6, obtaining an image to be recognized and a corresponding 3D original point cloud, and inputting them into the target visual model and the target PointNet++ model for prediction to obtain a target 3D point cloud.

2. The method of claim 1, wherein, The S1 comprises: acquiring an image data set of a ship by a drone, and acquiring a 3D original point cloud set of the ship by a laser radar; classifying each pixel in the image data set to obtain a classification of each pixel; inputting the image data set after classification into a backbone network of an original visual model for feature extraction; inputting the extracted feature maps into a depth value decoder, a focal length offset decoder and a segmentation decoder respectively for prediction, and obtaining a first predicted depth, a first predicted camera internal parameter and a first segmentation feature map of each pixel one by one; generating a predicted class probability of each pixel in each layer of the first segmentation feature map based on the first segmentation feature map and a logistic regression.

3. The method of claim 2, wherein, The S2 comprises: calculating a segmentation loss value based on the predicted class probability of each pixel in each layer of the first segmentation feature map and the classification of each pixel; calculating a camera internal parameter loss value based on the first predicted camera internal parameter obtained by prediction and a real camera internal parameter; calculating a depth loss value based on the first predicted depth of each pixel obtained by prediction and a real depth of each pixel; calculating the first total loss value according to the segmentation loss value, the camera internal parameter loss value and the depth loss value; updating the original visual model in reverse according to the first total loss value; repeating S1-S2 until the first total loss value fluctuates within a first preset range, and stopping to obtain a target visual model; The real depth of each pixel is calculated based on the 3D original point cloud set and the real camera internal parameter.

4. The method of claim 2, wherein, The S3 comprises: Corresponding to the labeling category of each pixel in the image data set in the 3D original point cloud set, the labeling category of each point in the 3D original point cloud set is obtained; The target visual model is frozen, and the 3D original point cloud set corresponding image data set is input into the frozen target visual model for prediction to obtain the second predicted depth, the second predicted camera internal parameter and the second segmentation feature map of each pixel; According to the second predicted depth and the second predicted camera internal parameter of each pixel, the image data set is converted to obtain the predicted 3D point cloud set.

5. The method of claim 4, wherein, The S4 comprises: fuse the predicted 3D point cloud set with the 3D original point cloud set; The fused 3D point cloud set is subjected to multi-stage feature extraction by using the original PointNet++ model, and a two-dimensional segmentation vector is introduced in the multi-stage feature extraction process to obtain a two-dimensional segmentation feature 3D point cloud feature; wherein the two-dimensional segmentation vector is generated by the second segmentation feature map through a self-attention mechanism; The two-dimensional segmentation feature 3D point cloud feature is up-sampled to obtain the predicted category probability of each point in the fused 3D point cloud set.

6. The method of claim 5, wherein, The S5 comprises: Map the predicted category probability of each point in the fused 3D point cloud set to the image data set to obtain the mapped predicted category probability of each pixel; According to the predicted category probability of each point in the fused 3D point cloud set and the labeling category of each point in the 3D original point cloud set, a point cloud loss value is calculated; According to the mapped predicted category probability of each pixel and the labeling category of each pixel, an image loss value is calculated; According to the point cloud loss value and the image loss value, the second total loss value is calculated; According to the second total loss value, the original PointNet++ model is updated reversely; S4-S5 are repeated until the second total loss value fluctuates within a second preset range, and a target PointNet++ model is obtained.

7. The method of claim 5, wherein, The S4 comprises: The second segmentation feature map is subjected to a self-attention mechanism to generate a two-dimensional segmentation vector; The fused 3D point cloud set is subjected to first feature extraction by using the original PointNet++ model to obtain a fused 3D point cloud feature; The two-dimensional segmentation vector is queried in the fused 3D point cloud feature to obtain a 3D point cloud feature matched with the two-dimensional feature information; the 3D point cloud feature matched with the two-dimensional feature information is added to the fused 3D point cloud feature before the query to obtain a one-stage 3D point cloud feature; The one-stage 3D point cloud feature is taken as the fused 3D point cloud feature, and the query and addition are repeated to obtain a two-stage 3D point cloud feature; The two-stage 3D point cloud feature is taken as the fused 3D point cloud feature, and the query and addition are repeated to obtain a three-stage 3D point cloud feature; The three-stage 3D point cloud feature is taken as the fused 3D point cloud feature, and the query and addition are repeated to obtain the two-dimensional segmentation feature 3D point cloud feature.

8. A 3D point cloud enhancement system, characterized in that, The S5 comprises: Map the predicted category probability of each point in the fused 3D point cloud set to the image data set to obtain the mapped predicted category probability of each pixel; According to the predicted category probability of each point in the fused 3D point cloud set and the labeling category of each point in the 3D original point cloud set, a point cloud loss value is calculated; According to the mapped predicted category probability of each pixel and the labeling category of each pixel, an image loss value is calculated; According to the point cloud loss value and the image loss value, the second total loss value is calculated; According to the second total loss value, the original PointNet++ model is updated reversely; S4-S5 are repeated until the second total loss value fluctuates within a second preset range, and a target PointNet++ model is obtained. The S4 comprises: The second segmentation feature map is subjected to a self-attention mechanism to generate a two-dimensional segmentation vector; The fused 3D point cloud set is subjected to first feature extraction by using the original PointNet++ model to obtain a fused 3D point cloud feature; The two-dimensional segmentation vector is queried in the fused 3D point cloud feature to obtain a 3D point cloud feature matched with the two-dimensional feature information; the 3D point cloud feature matched with the two-dimensional feature information is added to the fused 3D point cloud feature before the query to obtain a one-stage 3D point cloud feature; The one-stage 3D point cloud feature is taken as the fused 3D point cloud feature, and the query and addition are repeated to obtain a two-stage 3D point cloud feature; The two-stage 3D point cloud feature is taken as the fused 3D point cloud feature, and the query and addition are repeated to obtain a three-stage 3D point cloud feature; The three-stage 3D point cloud feature is taken as the fused 3D point cloud feature, and the query and addition are repeated to obtain the two-dimensional segmentation feature 3D point cloud feature. The S5 comprises: The first prediction unit is configured to obtain an image data set and a corresponding 3D original point cloud set, and input the image data set into a backbone network of an original visual model for feature extraction; The extracted feature maps are input into a depth value decoder, a focal length offset decoder and a segmentation decoder respectively for prediction; The first reverse updating unit is configured to calculate a first total loss value based on the predicted first result; The original visual model is updated reversely according to the first total loss value; and the steps S1-S2 are repeated until the first total loss value fluctuates within a first preset range, so as to obtain a target visual model; The second prediction and conversion unit is configured to freeze the target visual model, input the image data set into the frozen target visual model for prediction, and convert the image data set based on a predicted second result to obtain a predicted 3D point cloud set; The third prediction unit is configured to fuse the predicted 3D point cloud set with the 3D original point cloud set, extract features of the fused 3D point cloud set by using an original PointNet++ model, introduce a two-dimensional segmentation vector in the feature extraction process, up-sample the extracted feature maps, and obtain a predicted third result; The second reverse updating unit is configured to calculate a second total loss value based on the predicted third result; The original PointNet++ model is updated reversely according to the second total loss value; and the steps S4-S5 are repeated until the second total loss value fluctuates within a second preset range, so as to obtain a target PointNet++ model; The generation unit is configured to obtain an image to be recognized and a corresponding 3D original point cloud, input the image and the 3D original point cloud into the target visual model and the target PointNet++ model for prediction, and obtain a target 3D point cloud.

9. The system of claim 8, wherein, The first prediction unit comprises: An acquisition subunit is configured to acquire, by a UAV, an image data set of a ship and acquire, by a laser radar, a 3D original point cloud set of the ship; A calibration subunit is configured to calibrate each pixel in the image data set to obtain a calibration category of each pixel; A feature extraction subunit is configured to input the image data set after the calibration into a backbone network of an original visual model for feature extraction; A first prediction subunit is configured to input the extracted feature maps into a depth value decoder, a focal length offset decoder and a segmentation decoder respectively for prediction, and obtain a first predicted depth, a first predicted camera internal parameter and a first segmentation feature map of each pixel one by one; A generation subunit is configured to generate, based on a logistic regression, a predicted category probability of each pixel in each layer of the first segmentation feature map.

10. The system of claim 9, wherein, The first reverse updating unit comprises: A first calculation subunit is configured to calculate a segmentation loss value based on the predicted category probability of each pixel in each layer of the first segmentation feature map and the calibration category of each pixel; A second calculation subunit is configured to calculate a camera internal parameter loss value based on the predicted first predicted camera internal parameter and a real camera internal parameter; A third calculation subunit is configured to calculate a depth loss value based on the predicted first predicted depth of each pixel and a real depth of each pixel; The fourth calculation subunit is configured to calculate the first total loss value according to the segmentation loss value, the camera intrinsic loss value and the depth loss value; The first reverse updating subunit is configured to reversely update the original visual model according to the first total loss value; the first prediction subunit and the first reverse updating subunit are repeated until the first total loss value fluctuates within a first preset range, and a target visual model is obtained. The real depth of each pixel is calculated based on a 3D original point cloud set and real camera intrinsic parameters.

Images