An underwater point cloud three-dimensional target detection method and system based on pseudo label optimization

Abstract

The application discloses a kind of underwater point cloud three-dimensional target detection method and system based on pseudo label optimization, wherein the detection method includes the following steps: synchronous acquisition sonar point cloud, optical image and underwater environmental parameters such as turbidity, illumination, construct multimodal sample;Adopt visual basic model to complete two-stage cross-modal pre-labeling, automatically generate initial three-dimensional pseudo label, without artificial three-dimensional labeling;Five-dimensional quality evaluation system including model confidence, visual consistency, cross-modal consistency, geometric consistency, time sequence stability is constructed, and evaluation weight is dynamically distributed relying on underwater environmental parameter, and the quality score of pseudo label is quantified;Based on high and low quality threshold layering processing pseudo label, to intermediate quality sample self-adapting executes point cloud completion, cross-modal label migration, multimodal fusion enhancement.The application solves the problems of existing underwater three-dimensional labeling, large pseudo label noise, poor robustness in degenerative scene, and uneven model training.

Description

Technical Field

[0001] This invention belongs to the field of underwater robot autonomous perception and 3D point cloud target detection technology, and specifically relates to an underwater sonar point cloud 3D target detection method and system that integrates optical-sonar multimodal information and is based on adaptive pseudo-label optimization for turbid, low-light, and long-distance degraded underwater scenarios. Background Technology

[0002] Underwater 3D target detection is the core perception foundation for autonomous underwater robot operations. It requires accurately identifying targets such as pipes, cables, bridge piers, shipwrecks, and marine life from sonar 3D point clouds, and outputting the target's precise 3D coordinates, dimensions, and attitude to provide decision support for obstacle avoidance, inspection, and salvage operations. Currently, mainstream industry solutions suffer from three inherent flaws:

[0003] Manual 3D annotation is extremely costly, samples are scarce, underwater sonar point clouds are naturally sparse, noise interference is strong, and target contours are incomplete. Manual annotation of 3D bounding boxes requires matching spatial geometry, temporal motion, and cross-modal correspondences at the same time. The annotation time for a single frame is more than 10 times that of 2D image annotation. Large-scale underwater 3D annotation datasets are difficult to construct, which directly leads to insufficient training of 3D detection networks and poor model generalization ability.

[0004] Traditional pseudo-labeling schemes lack underwater environment adaptability. When the semi-supervised pseudo-labeling method for terrestrial scenes is directly transferred to underwater scenes, it does not consider the dynamic impact of water turbidity, illumination, and detection distance on image quality: optical images fail in turbid water, the density of long-distance sonar point clouds drops sharply, and self-generated pseudo-labels contain a large number of mislabeled and missing samples; low-quality pseudo-labels participating in training will continuously introduce noise and continuously reduce the detection accuracy of the model.

[0005] Single-modal information is fragmented, and robustness to degraded scenes is insufficient. Pure optical two-dimensional detection cannot output three-dimensional spatial coordinates. In low light and high turbidity water, the signal-to-noise ratio of the image drops sharply, and the detection fails completely. Relying solely on sonar point clouds lacks texture and semantic information, and the ability to distinguish small and overlapping targets is poor. Existing technologies have not established a cross-modal label mutual correction and fusion enhancement mechanism between optics and sonar, resulting in extremely low utilization of single-modal information.

[0006] In summary, existing technologies suffer from four major technical bottlenecks: scarcity of 3D labeled samples, uncontrollable quality of pseudo-labels, lack of multimodal information fusion, and a sharp drop in detection accuracy in degraded underwater scenes. These limitations prevent them from meeting the all-weather, all-water autonomous detection needs of underwater robots. Summary of the Invention

[0007] To address the problems of high cost of manual 3D annotation, difficulty in controlling pseudo-label noise, fragmented multimodal information, weak robustness in degraded waters, and insufficient model generalization ability in existing underwater sonar 3D target detection, this invention relies on a pre-trained visual model to automatically generate 3D pseudo-labels in batches, eliminating the need for manual 3D annotation and alleviating the problem of scarce training samples. A five-dimensional adaptive pseudo-label quality assessment system is constructed to dynamically adapt to the underwater environment and distinguish different quality labels. Three types of repair strategies are designed for medium-quality pseudo-labels to fully utilize unlabeled samples. An environment-driven modal fusion weight allocation logic is adopted, emphasizing sonar information in high-turbidity and long-distance scenes, and enhancing optical semantics in clear close-up scenes. A closed-loop iterative training mechanism is established to dynamically adjust sample sampling weights and aggregate multiple rounds of model replicas, balancing the training effect of rare samples and improving overall detection accuracy.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a method for detecting three-dimensional underwater point clouds based on pseudo-label optimization, comprising the following steps:

[0009] S1. Simultaneously collect two types of multimodal data—sonar 3D point cloud and optical image—using an underwater robot, and collect underwater environmental parameters such as water turbidity, light intensity, target distance, and robot pose; generate an unlabeled training sample set and real-time samples to be detected by matching timestamps.

[0010] S2. Input the optical image into the pre-trained visual base model DINOv3 with parameters frozen throughout the process to extract pixel-level dense visual features and generate two-dimensional target candidate boxes; project the two-dimensional candidate boxes onto the sonar three-dimensional point cloud space and extract the corresponding point cloud subsets; after visual semantic consistency screening and density clustering to remove outliers, generate an initial three-dimensional pseudo-label set containing category, three-dimensional bounding box, confidence score, and frame identifier.

[0011] S3. For each initial 3D pseudo-label, independently calculate the normalized quality scores for five categories: model confidence, visual feature consistency, sonar-optical cross-modal consistency, underwater environment geometric consistency, and temporal motion stability; dynamically allocate the fusion weights of the five categories of scores using only underwater environment parameters, and obtain the pseudo-label comprehensive quality score Q by weighted summation;

[0012] S4. Pre-set high-quality threshold and low-quality threshold; based on the difference in scores between the two modalities, selectively perform three types of repair operations: feature-guided point cloud completion, cross-modal label transfer, and multimodal fusion enhancement, and output an optimized 3D pseudo-label set;

[0013] S5. A Sparse4D sparse convolutional 3D detection network is used to optimize the pseudo-label set for training. After each training cycle, the model performance is evaluated using 3DAP@0.25 on an independent manually labeled validation set. The sample sampling weights are adjusted according to the degree of undertraining of the category / scene, and the sampling weights of low-density samples are increased based on the spatial distribution of visual features in DINOv3. Model copies are saved in the preset training rounds, and the final 3D detection model is obtained by weighted aggregation based on the consistency score of feature representation between copies.

[0014] S6. After synchronizing the time and unifying the coordinates of the samples to be detected to the sonar point cloud coordinate system, input them into the final model; perform confidence threshold filtering and three-dimensional non-maximum suppression on the candidate results output by the model, and output the underwater target category, three-dimensional bounding box, spatial coordinates and confidence.

[0015] Furthermore, the visual semantic consistency screening method described in step S2 is as follows: calculate the cosine similarity between the image patch features corresponding to the sonar point projection position and the average features of the two-dimensional candidate box; if the similarity is <0.55, then remove the point cloud and retain only the subset of point clouds with similarity ≥0.55; the density clustering radius is 0.35m and the minimum number of cluster points is 5.

[0016] Furthermore, the calculation rules for the five quality categories in step S3 are as follows: the scores for the five categories are uniformly normalized to [0,1]: Model confidence q1: In the first round, the average of the YOLO26 2D confidence and the point cloud cluster compactness is used. In iterative training, the 3D model classification confidence is used. Visual consistency q2: Calculate the mean cosine similarity of features between adjacent image blocks in a two-dimensional projection region; Cross-modal consistency q3: The IoU of the projected frame is fused with the center distance attenuation term. The IoU weight is 0.7 when the target distance is <5m and 0.5 when it is ≥5m. Geometric Consistency q4: Based on point density, normal vector, and curvature geometric priors, differentiated constraints are applied to the configuration of pipelines, piers, reefs, and marine organisms; Temporal stability q5: The target temporal state is predicted by Kalman filtering, and the score is calculated by combining the observation bias with Gaussian decay.

[0017] Furthermore, the judgment and implementation logic of the three types of intermediate quality pseudo-label repair operations in step S4 are as follows: Feature-guided point cloud completion: visual consistency score > 0.70 and point cloud density within the bounding box < 50 points / m³; supplementary point clouds are generated through image edge and visual feature interpolation, and the corrected 3D bounding boxes are re-clustered; Cross-modal label transfer: high quality in one modality and degradation in the other; if the image is clear, the 2D semantic labels are mapped to correct the 3D bounding boxes; if the sonar geometry is complete, the candidate regions of the image are corrected using 3D geometry; Multimodal fusion enhancement: both modalities have effective information; the visual comprehensive score is calculated based on turbidity, illumination, and distance, and the optical and sonar modal fusion weights are allocated, and the 3D bounding box parameters of the two modalities are weighted and fused.

[0018] This invention offers the following advantages: Based on a 5000-set independently annotated underwater validation set, using 3DAP@0.25 as the core evaluation metric, compared to three traditional methods, the annotation cost is significantly reduced, and massive training samples can be used to batch construct traditional manual 3D annotations. This invention features a two-stage pre-annotation process: no manual 3D annotation is required; 50,000 sets of samples are used to generate pseudo-labels offline in batches, requiring only 3000 2D images for YOLO26 pre-training, significantly reducing the overall annotation cost; and it solves the bottleneck of scarce underwater 3D annotation samples. Pseudo-label noise is effectively suppressed, and the quality of training data is significantly improved compared to directly using the initial, unoptimized pseudo-label training scheme.

[0019] Strong adaptability across all water areas, significantly improved detection accuracy in degraded scenes; scene-specific quantitative comparison: ①Clear close-up image (turbidity < 20 NTU, distance < 5m): 3DAP@0.25 achieves 89.2%, an improvement of 8.3% compared to traditional pure sonar solutions; ② High turbidity at long distances (turbidity ≥ 40 NTU, distance ≥ 8 m): Traditional pure optical solutions fail (AP < 40%), and traditional fixed-weight pseudo-tag solutions have an AP of only 57.1%; after adaptively adjusting the sonar weights, the AP of this invention reaches 76.8%, an improvement of 19.7 percentage points; through an environment-driven dual-weight adjustment mechanism, stable detection is achieved in all scenarios, including clear water, turbid water, near-field, and far-field, with robustness far exceeding that of existing single-modal / fixed-weight solutions.

[0020] The model training balance and generalization ability are significantly optimized, and the rare target recognition ability is greatly enhanced. Traditional fixed sampling training: the AP of rare categories such as small marine organisms and long-distance cables is only 48~55%; the closed-loop iterative sampling weight adjustment and low-density feature enhancement of this invention: the AP of rare categories is increased to 72~79%; the consistency aggregation of multi-copy features further smooths the training fluctuations, and the generalization accuracy of the model in unfamiliar waters that have not participated in the training is improved by 8~12 percentage points, which can be directly applied to the inspection operation of new and unexplored waters. Attached Figure Description

[0021] Figure 1This is a flowchart illustrating the overall execution of the complete detection method of the present invention.

[0022] Figure 2 This is a flowchart of the two-stage pre-annotation subdivision process of the present invention;

[0023] Figure 3 This is a flowchart of the five-dimensional quality assessment and hierarchical adaptive pseudo-label optimization subdivision process of the present invention;

[0024] Figure 4 This is a flowchart illustrating the closed-loop iterative training process of the present invention.

[0025] Figure 5 Flowchart for deploying real-time detection in this invention;

[0026] Figure 6 This is a schematic diagram of the underwater sonar three-dimensional target detection system of the present invention. Detailed Implementation

[0027] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0028] Reference Figures 1-6 As shown, an underwater point cloud 3D target detection method based on pseudo-label optimization includes the following steps:

[0029] S1. Multimodal Data Acquisition and Underwater Environment Perception: An underwater robot simultaneously acquires two types of multimodal data: sonar 3D point clouds and optical images. It also collects underwater environmental parameters such as water turbidity, light intensity, target distance, and robot pose. Unlabeled training sample sets and real-time samples to be detected are generated by matching timestamps. Simultaneously, three types of underlying data are acquired: sonar geometric information, optical semantic texture, and quantitative indicators of water degradation. This achieves unified spatiotemporal benchmarks across multiple sensors, fully records the degree of underwater imaging degradation, eliminates pseudo-label matching errors caused by spatiotemporal misalignment between images and point clouds and missing environmental parameters, and distinguishes between offline training datasets and online inference datasets. This decouples the training and deployment data links, adapting to the dual-scenario requirements of offline model training and real-time underwater robot detection.

[0030] S2. Two-stage cross-modal pre-labeling: Optical images are input into a pre-trained visual base model DINOv3 with parameters frozen throughout the process to extract pixel-level dense visual features. Two-dimensional target candidate boxes are generated by the YOLO26 detector head. Based on the joint calibration of intrinsic and extrinsic parameters of the camera and sonar, the two-dimensional candidate boxes are projected and mapped to the sonar three-dimensional point cloud space to extract the corresponding point cloud subsets. After visual semantic consistency screening and density clustering to remove outliers, an initial three-dimensional pseudo-label set containing category, three-dimensional bounding box, confidence score, and frame identifier is generated.

[0031] In this invention, DINOv3 is frozen throughout the process, performing only feature extraction. It reuses general semantic representations obtained from pre-training on massive amounts of publicly available images, eliminating the need to retrain the visual backbone and avoiding the overfitting problem of visual features caused by small underwater samples. It stably outputs pixel-level fine-grained semantics. Relying on pre-trained YOLO26, it achieves rapid localization of 2D targets, requiring only a small amount of 2D annotation to complete the pre-training, avoiding the pain point of high cost of underwater 3D annotation. Camera-sonar joint calibration establishes a precise mapping relationship between 2D images and 3D point clouds, realizing cross-modal spatial association and automatically generating 3D bounding boxes, completely eliminating the need for manual 3D annotation. Semantic filtering and density clustering dual noise reduction eliminate invalid point clouds caused by projection misalignment and sonar noise, significantly reducing the basic noise of the initial pseudo-labels, and can generate tens of thousands of sets of 3D pseudo-labels for unlabeled samples in batches.

[0032] In this invention, step S2, visual semantic consistency screening, is implemented as follows: The cosine similarity between the image patch features corresponding to the sonar point projection position and the average features of the two-dimensional candidate box is calculated; if the similarity is <0.55, the point cloud is removed, and only a subset of point clouds with a similarity ≥0.55 is retained; the density clustering radius is 0.35m, and the minimum number of cluster points is 5. The cosine similarity of the high-dimensional semantic features of DINOv3 is used to quantify the semantic matching degree between the image and the point cloud, accurately filtering out semantic mismatch noise points caused by cross-modal projection misalignment and water scattering; the 0.35m clustering radius and the minimum 5-point constraint adapt to the sparse characteristics of underwater sonar point clouds, filtering out isolated outlier noise points, ensuring that the retained point cloud is a continuous and complete target region, and reducing the probability of initial pseudo-label geometric distortion and category mismatch from the source.

[0033] S3. Underwater Scene Adaptive Five-Dimensional Pseudo-Label Quality Assessment: For each initial three-dimensional pseudo-label, five normalized quality scores are independently calculated: model confidence, visual feature consistency, sonar-optical cross-modal consistency, underwater environment geometric consistency, and temporal motion stability. The fusion weights of these five scores are dynamically allocated using only underwater environmental parameters, and the weighted sum is used to obtain the pseudo-label's comprehensive quality score Q. A five-dimensional quantitative assessment system covering prediction credibility, optical integrity, cross-modal matching, geometric compliance, and temporal smoothness is constructed. This differs from traditional coarse-grained discrimination methods that rely solely on single confidence assessment, comprehensively quantifying the sources and severity of pseudo-label defects. Environmental parameters are only used for weight allocation and do not participate in score calculation, enabling the assessment standards to adaptively switch with water turbidity, illumination, and target distance. In turbid, low-light scenes, the optical dimension weights are automatically weakened while the sonar geometric and temporal dimension weights are strengthened. In clear water close-up scenes, the visual semantic weights are increased, significantly improving the accuracy of pseudo-label quality differentiation in degraded underwater scenes and providing precise quantitative judgment criteria for hierarchical optimization.

[0034] In this invention, the calculation rules for the five categories of quality scores in step S3 are as follows: Model confidence score q1: In the first round of training, the average of the YOLO26 2D bounding box confidence score and the point cloud clustering compactness score is used; in subsequent training rounds, the classification confidence score output by the current 3D model is used; in the first round when there is no mature 3D model, the confidence of the label is comprehensively evaluated by the dual indicators of 2D detection and point cloud clustering. During the training iteration process, the confidence score output by the 3D detection model itself is reused to realize that the evaluation indicators are dynamically updated with the training process, which truly reflects the reliability of the current model in recognizing the target category and avoids the evaluation bias caused by a single static confidence score.

[0035] Visual feature consistency score q2: Statistically calculate the average cosine similarity of features of adjacent image blocks within a two-dimensional projection area; quantify the integrity and continuity of the target's optical texture by measuring the semantic similarity of adjacent image blocks, accurately identify semantic breaks in image areas caused by water turbidity and local occlusion, distinguish between complete and clear targets and incomplete and blurred targets, and quantify the quality of the optical modality itself separately.

[0036] Sonar-optical cross-modal consistency score q3: It integrates the IoU between the 2D projection frame and the YOLO26 candidate frame, as well as the Euclidean distance attenuation term of the frame center. When the target distance is <5m, the IoU weight is 0.7, and when it is ≥5m, the IoU weight is 0.5. It also takes into account two matching indicators: the overlap area of ​​the frame and the center offset distance, and dynamically adjusts the cross-union ratio weight with the target distance. When the close-range image is clear, it focuses on the frame overlap matching, and when the far-range image is degraded, it weakens the IoU and strengthens the center position matching. It is suitable for underwater scenes with blurred optical imaging and lost contours at long distances, and accurately quantifies the spatial alignment error between the image and the point cloud target.

[0037] The underwater environment geometric consistency score q4 is based on three types of geometric priors: matching point density, normal vector consistency, and curvature variation. Differentiated geometric constraints are set for pipes, planar structures, bridge piers, reefs, and marine life. Prior constraints are established based on the inherent geometric and physical laws of various underwater targets to distinguish real targets from sonar clutter and floating noise. Artificial rigid structures such as pipes and bridge piers are matched with regular geometric features, while reefs and marine life are matched with rough curved surface features. Erroneous pseudo-labels that violate the morphological laws of underwater objects are screened out from the three-dimensional geometric dimension, making up for the shortcomings of quality assessment in optical failure scenarios.

[0038] Temporal motion stability score q5: Kalman filter predicts the temporal state of the target, and the stability score is calculated using the observation bias Gaussian decay function; continuous frame temporal correlation is used to suppress abrupt pseudo-labels generated by instantaneous noise in a single frame, which greatly reduces the score of pseudo-labels with abrupt motion changes and position jumps, ensuring the temporal smoothness of labels for the same target in multiple frames, and eliminating single-frame mislabeling caused by sonar pulse noise and water disturbance.

[0039] The score ranges for all five categories are normalized to [0,1]. This unified score range ensures that the contribution of each indicator is controllable during the weighted fusion of the five dimensions, preventing any single dimension from dominating the overall score and ensuring the objectivity and balance of the weighted fusion result.

[0040] In this invention, the dynamic weight allocation rule in step S3 is as follows: ①Clear close-up scene: Turbidity < 20 NTU, Illumination > 80 lx, Target distance < 5m; Weight allocation (w1, w2, w3, w4, w5) = (0.20, 0.30, 0.20, 0.15, 0.15); ② Turbid / Low-light / Distant Degradation Scenarios: Turbidity ≥ 20 NTU or Illumination ≤ 80 lx or Target distance ≥ 5m, weight allocation (w1, w2, w3, w4, w5) = (0.25, 0.15, 0.25, 0.20, 0.15); weights satisfy (w1+w2+w3+w4+w5=1). Two sets of weight combinations enable adaptive switching of the evaluation system environment. In clear water close-up scenarios, the visual feature consistency weight is amplified, making full use of optical texture semantics to improve evaluation accuracy; in turbid and distant scenarios, the visual weight is reduced and the cross-modal and geometric dimension weights are increased, relying on sonar geometric information to compensate for optical failure defects; the weight normalization constraint ensures that the comprehensive score value range is stable in [0,1], the evaluation standard is unified and quantifiable, and the defects of fixed weight evaluation cannot be adapted to the variable underwater environment.

[0041] S4. Layered adaptive pseudo-label optimization with pre-set high-quality threshold. =0.75, low quality threshold =0.45; if Q≥ : Keep the pseudo-tag directly; if Q < Remove low-quality, erroneous, and pseudo-labels; if ≤Q< Based on the score differences between the two modalities, three types of repair operations are selectively performed: feature-guided point cloud completion, cross-modal label transfer, and multimodal fusion enhancement, outputting an optimized 3D pseudo-label set;

[0042] A three-layer hierarchical processing strategy is adopted to control pseudo-label classification. High-quality labels are directly reused, completely erroneous labels are filtered out, and incomplete and usable labels are repaired in a targeted manner. This approach not only prevents low-quality noisy labels from entering the training set and causing error accumulation, but also avoids directly discarding intermediate-quality samples with some valid information, thus maximizing the value of unlabeled sample data. Simultaneously, three types of differentiated repair strategies are provided, which automatically match the repair scheme according to the quality of sonar and optical modes, realize complementary correction of modal information, and significantly improve the overall purity and sample utilization of the final training dataset.

[0043] In this invention, the judgment and implementation logic of the three types of intermediate quality pseudo-label repair operations in step S4 are as follows: Feature-guided point cloud completion: visual consistency score > 0.70 and point cloud density within the box < 50 points / m³; supplementary point clouds are generated through image edge and visual feature interpolation, and the corrected 3D bounding boxes are re-clustered; for targets with complete optical semantics but sparse and incomplete sonar point clouds, the missing 3D point clouds are supplemented by interpolation based on clear image semantics and edge contours, filling the point cloud holes caused by long-distance sonar detection, refitting accurate 3D bounding boxes, repairing geometrically incomplete pseudo-labels, and improving the usability of sparse target samples. Cross-modal label migration: high quality in one modality and degradation in another; if the image is clear, the 2D semantic labels are mapped to correct the 3D bounding boxes; if the sonar geometry is complete, the 3D geometry is used to correct the image candidate region; realizing unidirectional correction of effective information from a single modality to the degraded modality; when the water turbidity image fails, the target spatial range is corrected by complete sonar geometry; when the sonar point cloud is incomplete but the image is clear, the 3D bounding box category and contour are corrected by 2D semantics, solving the label distortion problem in the single-modal failure scenario. Multimodal fusion enhancement: Both modalities possess valid information; based on turbidity, illumination, and distance, a comprehensive visual score is calculated to allocate fusion weights for optical and sonar modalities, and the 3D bounding box parameters of the two modalities are weighted and fused. When both modalities are available, the fusion weights are dynamically allocated according to the degree of water degradation. In clean water environments, the optical semantics are enhanced to correct the 3D bounding box, while in turbid water long-distance scenes, the sonar geometric boundaries are emphasized. The weighted fusion yields a high-precision 3D bounding box that takes into account both texture semantics and 3D geometry, fully integrating complementary multimodal information.

[0044] S5. Closed-loop iterative training and model aggregation: The Sparse4D sparse convolutional 3D detection network is used to optimize the pseudo-label set for training. After each training cycle, the model performance is evaluated using 3DAP@0.25 on an independent manually labeled validation set. The sample sampling weights are adjusted according to the degree of undertraining of the category / scene, and the sampling weights of low-density samples are increased based on the spatial distribution of visual features in DINOv3. Model copies are saved in the preset training rounds, and the models are weighted and aggregated according to the consistency scores of feature representations between copies to obtain the final 3D detection model.

[0045] The Sparse4D sparse convolutional network is adapted to sparse underwater sonar point clouds, significantly reducing the computational overhead of empty voxels and improving training and inference efficiency. An independently labeled validation set isolates training data, using 3DAP@0.25 as an objective quantitative indicator to truly reflect the model's generalization ability and avoid inflated performance due to overfitting of the training set. A training feedback loop is built to dynamically increase the sampling weights of under-trained categories, harsh scenarios, and rare feature samples, balancing the sample distribution and addressing the pain points of naturally imbalanced underwater samples and low accuracy in identifying rare targets. Multi-round weighted aggregation of model replicas smooths out single-round training fluctuations, offsetting model shifts caused by minor noise from residual pseudo-labels, and significantly improving the model's generalization ability and full-scene detection accuracy.

[0046] In this invention, step S5, closed-loop iterative training, includes weight adjustment rules: category / scene sampling weights: , where w c AP represents the sampling weight of samples related to category c in the next training cycle. c Represents the AP of category c, This represents the average AP across all categories; AP below 80% of the mean is considered undertrained; sampling weights are negatively correlated with category detection accuracy, with higher weights for lower accuracy, up to a maximum of 2 times to avoid excessively high proportions of rare samples disrupting training stability; it accurately identifies weak target categories and degraded scenarios, automatically strengthening training on weaker samples during training to narrow the accuracy gap between categories and scenarios. Low-density feature samples: DINOv3 feature density clustering is used, doubling the sampling weight for feature clusters with a sample percentage <3%; relying on visual semantic features to identify low-density rare clusters in the sample space, automatically increasing the sampling probability of rare underwater targets and special environment samples, compensating for the inherent deficiency of scarce rare samples during collection, and improving the model's ability to identify unfamiliar and rare underwater targets. Model aggregation: Model copies are saved at rounds 40, 60, and 80, the feature consistency score of each copy's validation set is calculated, aggregation weights are assigned using the softmax function, and a weighted average is used to obtain the final model parameters. Multi-stage model replicas are selected during the mid-to-late training phases. The reliability of the replicas is measured by the consistency of feature representation. Softmax normalization is used to allocate fusion weights, and higher weights are assigned to highly stable replicas. Weighted fusion offsets the local optima defects of single-round training, resulting in an average improvement of 7-12 percentage points in 3DAP compared to single-round model saving.

[0047] S6. Real-time Detection Deployment: After synchronizing the time and unifying the coordinates of the samples to be detected to the sonar point cloud coordinate system, the samples are input into the final model. Confidence thresholding and 3D non-maximum suppression are applied to the candidate output results of the model, outputting the underwater target category, 3D bounding box, spatial coordinates, and confidence score. During the deployment phase, a unified multimodal spatiotemporal reference is used to ensure spatial alignment between the image and the point cloud during inference. Confidence thresholding filters low-confidence false detection targets, and 3D non-maximum suppression eliminates duplicate detection boxes for the same target, reducing redundant output. The final output is structured 3D target parameters, which can be directly supplied to the underwater robot's obstacle avoidance and inspection decision-making unit. The entire post-processing system is lightweight, low-latency, and adaptable to underwater embedded low-computing-power platforms, meeting the engineering requirements for 10Hz real-time detection.

[0048] In a preferred embodiment of the present invention, step S6 deploys the following post-processing parameters: a confidence screening threshold of 0.35 and a three-dimensional non-maximum suppression cross-union threshold of 0.10 to eliminate duplicate and low-confidence detection boxes; the 0.35 confidence threshold is adapted to the underwater noise environment, balancing missed detections and false detections; the three-dimensional IoU=0.10 NMS threshold is adapted to the characteristics of sparse underwater targets and low overlap of distant target boxes, accurately eliminating duplicate detection boxes from the same source, while retaining independent targets with slight overlap at close range, thus balancing detection completeness and output simplicity.

[0049] This invention also provides an underwater point cloud 3D target detection system based on pseudo-label optimization, including a data acquisition module: equipped with an underwater robot, it synchronously outputs optical images and sonar 3D point clouds; the hardware acquisition unit is standardized, the image and sonar acquisition frame rate matches the underwater robot detection rate, the timestamp synchronization unit ensures the time sequence alignment of multimodal data, and it is compatible with mainstream underwater multibeam imaging sonar and industrial underwater cameras, realizing automated batch data acquisition.

[0050] Underwater Environment Perception Module: Collects data from turbidity, illumination, and pose sensors, outputting environmental parameters to the quality assessment and pseudo-label optimization modules; it quantifies water degradation and robot pose in real time, providing real-time environmental input for quality assessment weights and multimodal fusion weights, enabling the hardware implementation of end-to-end environmental adaptive logic. Dual-Stage Pre-Labeling Module: Includes built-in frozen DINOv3, pre-trained YOLO26, and camera-sonar joint calibration units, outputting initial 3D pseudo-labels; it integrates all cross-modal pre-labeling algorithm units, modularly encapsulating them to eliminate the need for repeated development of the visual backbone, 2D detection, and spatial mapping logic, enabling offline batch generation of pseudo-labels, reducing the difficulty of algorithm engineering implementation.

[0051] Quality Assessment Module: Includes a five-dimensional score calculation subunit and an environment-adaptive weight allocation subunit; independently encapsulates the five-dimensional quantitative assessment and dynamic weight calculation logic, and can output the quality index of each pseudo-label separately, facilitating pseudo-label defect tracing and visual analysis. Pseudo-Label Optimization Module: Integrates pseudo-label retention, removal, point cloud completion, label migration, and multi-modal fusion enhancement subunits; modularly decomposes hierarchical optimization strategies, supports the individual activation of any type of repair logic, and flexibly switches optimization schemes to adapt to different water areas and task scenarios. Model Training Module: Built-in Sparse4D network, performance evaluation unit, sampling weight adjuster, and model aggregation unit; integrated encapsulates the entire closed-loop iterative training chain, automatically completing performance evaluation, sampling weight adjustment, and multi-copy aggregation, reducing the workload of 3D detection model training parameter tuning, and automating the training process. Deployment Detection Module: Completes multi-modal time synchronization, coordinate alignment, model inference, detection result post-processing, and output. Lightweight Inference Unit Adapts to Underwater Embedded Boards, Integrates Spatiotemporal Alignment, Model Inference, and Post-processing, Directly Outputs Standardized Target Structured Data, Seamlessly Connecting to Underwater Robot Autonomous Decision Control System.

[0052] This invention also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements all the steps of the underwater sonar point cloud 3D target detection method based on pseudo-label optimization. The entire algorithm logic is solidified in the form of a storage medium, making it portable to any hardware carrier with computing capabilities, such as training servers, edge embedded devices, and shore-based computing platforms.

[0053] The present invention also provides an embedded computing device for an underwater robot, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the real-time deployment and detection steps of the underwater sonar point cloud three-dimensional target detection method based on pseudo-label optimization.

[0054] In this invention, an underwater robot is equipped with a 1920×1080 optical camera, multi-beam imaging sonar, and turbidity, illumination, and pose sensors. The optical image acquisition frame rate is 30 frames / second, the sonar point cloud acquisition frame rate is 10 frames / second, and the environmental sensor sampling frequency is 100Hz. Using a 100ms time difference as the matching threshold, images, point clouds, and environmental parameters are aligned to construct pairs of multimodal samples, which are divided into offline training unlabeled sample sets and online deployment real-time detection samples. Simultaneously, three types of underlying data—sonar geometry, optical semantics, and water degradation—are acquired to unify the spatiotemporal benchmark of the multi-sensor system. Quantitative environmental factors such as turbidity, illumination, and target distance are recorded to provide input for subsequent pseudo-label generation, quality assessment, and modality fusion, eliminating label matching errors caused by spatiotemporal asynchrony.

[0055] In this invention, cross-modal 3D pseudo-labels are automatically generated through a layered approach: Visual feature extraction layer: A DINOv3 model with parameters frozen throughout the process is used to output 1024-dimensional pixel-level dense semantic features, which do not participate in 3D network training; 2D target detection layer: The YOLO26 detection head completes 2D detection based on DINOv3 features. The model is pre-trained using a publicly available underwater dataset and 3000 labeled underwater images, outputting 2D candidate boxes, categories, and confidence scores; Cross-modal spatial mapping layer: Camera-sonar joint calibration intrinsic and extrinsic parameters are loaded, and perspective projection is used to complete bidirectional mapping between 3D points and 2D images, extracting a subset of point clouds corresponding to the 2D boxes; Semantic filtering and clustering layer: Point clouds with a cosine similarity ≥ 0.55 between projected points and candidate box features are retained; DBSCAN with a radius of 0.35m and a minimum number of points of 5 is used to remove outliers, and PCA is used to fit 3D bounding boxes to generate initial 3D pseudo-labels. Three-dimensional pseudo-labels can be generated in batches with only a small number of two-dimensional annotations, without the need for manual three-dimensional annotation. It can generate 50,000 sets of pseudo-labels for unannotated samples at one time; the DINOv3 freezing strategy ensures stable cross-modal semantic alignment.

[0056] In this invention, five independent evaluation dimensions q1~q5 are set. The scores of each dimension are calculated only from the multimodal data of the samples. Turbidity, illumination, and target distance are only used for dynamically allocating fusion weights and do not participate in the score calculation. q1 model confidence: In the first round, the average of the two-dimensional confidence and cluster compactness is used. In iterative training, the three-dimensional model classification confidence is used. q2 visual consistency: the mean similarity of features between adjacent image patches in the projection region, representing the integrity of optical texture; q3 Cross-modal consistency: Fuse the projection frame IoU and center distance attenuation term, and switch the IoU weights according to the target distance; Q4 Geometric Consistency: Configure differentiated geometric constraints for pipelines, bridge piers, reefs, and marine life, and match point density, normal vector, and curvature features; Q5 temporal stability: Kalman filter is used to track targets in consecutive frames, and the score is calculated based on Gaussian decay of observation bias to suppress single-frame abrupt noise. The comprehensive score Q∈[0,1] is obtained by weighted fusion of two sets of fixed weights for clear near-field and turbid / low-light / far-field images, which comprehensively quantifies the defects of pseudo-labels and adapts to different underwater imaging environments.

[0057] In this invention, a high-quality threshold is set. =0.75, low quality threshold =0.45, three processing strategies are adopted: Q≥0.75: pseudo-labels are directly retained; Q<0.45: erroneous and noisy labels are removed; 0.45≤Q<0.75: repair schemes are selected based on modal quality: feature-guided point cloud completion: when visual integrity is complete but point cloud is sparse, point cloud is supplemented by interpolation based on image features and 3D bounding boxes are refitted; cross-modal label migration: when one modality is clear but the other is degraded, the 3D bounding box of the degraded modality is corrected by high-quality modality semantics; multimodal fusion enhancement: when both modalities are effective, visual scores are calculated based on turbidity, illumination, and distance to allocate sonar / optical fusion weights, and 3D bounding boxes are weighted and fused. This hierarchical strategy takes into account noise filtering and reuse of intermediate quality samples. The sonar geometric weight is automatically increased in turbid water bodies to solve the label distortion problem in pure optical failure scenarios.

[0058] This invention employs a Sparse4D sparse convolutional network with 0.1 voxels, an AdamW optimizer, a batch size of 128, and a total of 80 training epochs. The closed-loop training logic is as follows: In each epoch, based on an independent human validation set, the accuracy of each scene and category is evaluated using 3DAP@0.25; if the AP of a single category is lower than 80% of the global mean, the sampling weight is increased, with a maximum weight of 2; for rare feature clusters identified by DINOv3 feature clustering with a proportion <3%, the corresponding sample sampling weight is doubled; model copies are saved at epochs 40, 60, and 80, and weights are allocated using softmax based on the consistency of validation set features, and weighted aggregation yields the final model. Dynamic sampling balances the training intensity of rare targets and high-turbidity scenes, and multi-copy aggregation smooths training fluctuations, thereby improving the model's generalization ability.

[0059] In this invention, real-time samples are standardized to a unified time reference and mapped to the sonar coordinate system. The trained Sparse4D network outputs 3D candidate boxes. Post-processing parameters are fixed: a confidence threshold of 0.35 and a 3D NMS (Non-Maximum Segmentation of Intersection over Union) threshold of 0.10. The final output includes the target category, 3D coordinates, size, attitude, and confidence score. The entire post-processing system is lightweight, adaptable to low-computing-power underwater embedded hardware, and its inference speed meets the 10Hz real-time detection requirement, filtering out false positives and duplicate detection boxes.

[0060] In this invention, pre-annotated pseudo-labels are generated in batches but suffer from projection noise. Five-dimensional evaluation quantifies label defects and locates the source of errors. The two work together to solve the two major problems of label scarcity and pseudo-label noise. The five-dimensional score distinguishes the label defect type, and the layered repair matches the corresponding correction strategy, taking into account both noise filtering and full utilization of samples. Low-noise pseudo-labels serve as the training basis, and the training accuracy feedback dynamically adjusts the sample sampling weight. Multiple copies are aggregated to cancel residual noise, forming a complete positive training closed loop. The quality assessment weight and modality fusion weight introduce environmental parameters such as turbidity, illumination, and distance. Clear water focuses on optical semantics, while turbid water / distant view enhances sonar geometry, adapting to complex degradation scenarios in all water areas and improving detection robustness.

[0061] Example 1: As Figure 1 As shown, an underwater point cloud 3D target detection method based on pseudo-label optimization specifically includes the following steps:

[0062] S1. Collect underwater multimodal scene data, obtain unlabeled training samples for model training and samples to be tested when deploying applications, and obtain underwater environmental parameters corresponding to the underwater multimodal scene data; underwater multimodal scene data refers to multiple modal data collected in the underwater environment, including at least sonar 3D point cloud data and image data.

[0063] In this embodiment, a front-facing camera and an image sonar are installed on the underwater robot. The front-facing camera is used to acquire underwater images, and the image sonar is used to acquire underwater sonar 3D point clouds. The images acquired by the front-facing camera have a resolution of 1920×1080 and an acquisition frequency of 30 frames / second; the image sonar acquires the sonar 3D point cloud at a frequency of 10 frames / second. For each frame of data, using the timestamp as an index, image frames, sonar point cloud frames, and pose data with a time difference of less than 100ms are combined to form an underwater multimodal scene sample. Unlabeled training samples refer to underwater multimodal scene data that has not been manually labeled with 3D bounding boxes or categories.

[0064] In this embodiment, 50,000 sets of unlabeled underwater multimodal scene samples are collected as unlabeled training samples for subsequent pseudo-label generation and model training. An additional 5,000 sets of samples are manually labeled with target categories and 3D bounding boxes to form a validation sample set. The validation sample set does not overlap with the unlabeled training samples and is only used for model performance evaluation, not for training. The samples to be tested refer to the underwater multimodal scene data collected in real time during system deployment and application.

[0065] In this embodiment, underwater multimodal scene samples are collected in real time during application deployment as samples to be tested. Underwater environmental parameters include, but are not limited to, water turbidity, light intensity, target distance, sonar acquisition parameters, and underwater robot pose information. A turbidity sensor, a light sensor, and a pose sensor are installed on the underwater robot. The turbidity sensor is used to collect water turbidity, the light sensor is used to collect light intensity, and the pose sensor is used to collect the position and attitude of the underwater robot. The pose sensor's acquisition frequency is 100Hz.

[0066] S2. Based on the pre-trained visual basic model, perform two-stage pre-labeling on unlabeled training samples to generate an initial three-dimensional pseudo-label set; the pre-trained visual basic model refers to a visual feature extraction model obtained by pre-training with large-scale image data or graphic data, which can extract image-level features, region-level features or pixel-level dense visual features in images.

[0067] In this embodiment, the pre-trained visual base model adopts the open-source self-supervised visual base model DINOv3. DINOv3 is pre-trained on large-scale unlabeled image data through self-supervised learning, and can output high-quality pixel-level dense visual features without manual annotation, making it suitable for scenarios where underwater image annotation data is scarce. Two-stage pre-annotation refers to the process of first generating two-dimensional target candidate regions in the image, and then mapping these two-dimensional target candidate regions to the sonar three-dimensional point cloud space to generate three-dimensional pseudo-labels. The two-stage pre-annotation process specifically includes the following steps: inputting the image into the pre-trained visual base model to extract visual semantic features from the image; generating two-dimensional target candidate regions based on the visual semantic features; associating the two-dimensional target candidate regions with the sonar three-dimensional point cloud space according to the joint calibration parameters between the camera and sonar; obtaining the three-dimensional point cloud subset corresponding to the two-dimensional target candidate regions; and generating initial three-dimensional pseudo-labels by combining the visual semantic features and the spatial distribution of the three-dimensional point cloud subset.

[0068] A 3D pseudo-label set is a dataset consisting of at least one 3D pseudo-label. Each 3D pseudo-label includes target category information, 3D bounding box parameters, confidence information, sample identification information, and time frame information.

[0069] In this embodiment, the three-dimensional pseudo-label set is represented as: ,in, Let represent the i-th 3D pseudo-label, and N represent the number of 3D pseudo-labels. Each 3D pseudo-label is represented as: , where c i Indicates the target category, b i s represents the parameters of the 3D bounding box. i Indicates the confidence level of the pseudo-label, id i Indicates the sample identifier, t i This represents time frame information. The sample identifier uniquely identifies the training sample to which the pseudo-label belongs, such as the sample sequence number or acquisition frame number, facilitating data indexing and result backtracking during training. The 3D bounding box parameter is represented as: b i =(x i ,y i ,z i ,l i ,w i ,h i ,θ i ), (x i ,y i ,z i ) represents the coordinates of the target center point in the sonar 3D point cloud coordinate system, l i ,w i ,h i θ represents the length, width, and height of the target, respectively. i Indicates the target attitude angle.

[0070] In this embodiment, the two-stage pre-annotation process is as follows: Figure 2 As shown, the specific steps are as follows: The image is input into a pre-trained visual base model to extract visual semantic features from the image; the input image is scaled to 518×518, and divided into 14×14 pixel image patches, resulting in 37×37 image patches. The DINOv3 backbone network outputs a dense visual feature map with a feature dimension of 1024. The parameters of the DINOv3 backbone network remain frozen throughout the training and inference processes, used only for feature extraction, and do not participate in training.

[0071] Two-dimensional target candidate regions are generated based on visual semantic features. These regions are generated by the YOLO26 target detection model's head based on dense visual features extracted from DINOv3: the dense visual feature map output from the DINOv3 backbone network is input into the YOLO26 head, which outputs the target category probability and the coordinates of the two-dimensional candidate boxes. The YOLO26 head is pre-trained on publicly available underwater optical image datasets (e.g., DUO dataset, URPC dataset) and a small number of manually annotated underwater images (3000 images in total), covering all detectable target categories with at least 200 images per category. The supervision labels are the manually annotated two-dimensional target boxes and target categories from the aforementioned datasets. During training, the images are first processed by DINOv3 with parameter freeze to extract dense visual features, and then input into the YOLO26 head for training. Only the parameters of the YOLO26 head are updated. Both training and inference follow the same process: "image → DINOv3 → dense visual features → YOLO26 head," ensuring consistent input for both. After training is completed, in step S2, only the DINOv3 and YOLO26 detection heads are used for inference, and no model training is performed. The training of the 3D object detection model is carried out in step S5.

[0072] In this embodiment, the detectable underwater target categories include, but are not limited to: underwater pipelines, underwater cables, underwater structures (dams, piers, jackets), valves and flange components, shipwrecks and debris, reefs, marine life (sea cucumbers, sea urchins, scallops, starfish), and suspended obstacles. Based on the joint calibration parameters between the camera and sonar, the two-dimensional target candidate region is associated with the sonar's three-dimensional point cloud space.

[0073] In this embodiment, the two-dimensional candidate box is spatially associated with the sonar three-dimensional point cloud in the following manner: for any point p in the sonar point cloud... s =(x s ,y s ,z s Based on the camera-sonar joint calibration extrinsic parameter transformation matrix, The camera intrinsic parameter matrix K projects the sonar points onto the image plane: , where u is the projection position of the sonar point in the image, and π represents the perspective projection function.

[0074] Obtain a subset of three-dimensional point clouds corresponding to the two-dimensional target candidate region. If the projected position u falls within a two-dimensional candidate box, add the sonar point to the subset of three-dimensional point clouds corresponding to that two-dimensional candidate box.

[0075] Initial 3D pseudo-labels are generated by combining visual semantic features and the spatial distribution of a subset of 3D point clouds. To reduce the impact of 2D candidate box projection errors and sonar noise, semantic consistency screening is further performed based on visual semantic features. Specifically, the visual semantic feature vectors of the 2D candidate box regions are obtained, and the cosine similarity between the dense visual feature vectors of the image positions corresponding to the projection points and the feature vectors of the candidate box regions is calculated. The cosine similarity calculation formula is: , This represents the dense visual feature vector corresponding to the image location of the projection point. The visual semantic feature vector of the two-dimensional candidate box region (takes the average of the feature vectors of each image patch within the candidate box). The dense visual feature vector refers to the per-image patch feature vector output by the DINOv3 backbone network, with each image patch corresponding to a 1024-dimensional feature vector; the dense visual feature vector at the projection position u is taken from the feature vector of the image patch where u is located. If the similarity is less than 0.55, the point is determined to be a semantically inconsistent point and deleted; if the similarity is not less than 0.55, the point is retained.

[0076] In this embodiment, outliers are removed from the retained subset of target point clouds using a density-based clustering method, with a cluster radius of 0.35m and a minimum number of points of 5. Based on the spatial distribution of the remaining point cloud, principal component analysis is used to estimate the target pose, and an initial 3D bounding box is generated according to the bounding range of the point cloud, thus obtaining an initial 3D pseudo-label set.

[0077] S3. Perform a quality assessment of underwater scene perception on the initial set of 3D pseudo-tags to obtain a comprehensive quality score for each 3D pseudo-tag. The quality assessment of underwater scene perception refers to evaluating the reliability of the 3D pseudo-tags by combining underwater imaging quality, underwater sonar point cloud quality, underwater environmental parameters, and target temporal stability.

[0078] In this embodiment, the quality assessment process is as follows: Figure 3 As shown. For each 3D pseudo-label, the model confidence score, visual feature consistency score, sonar-optical cross-modal consistency score, underwater environment geometric consistency score, and temporal motion stability score are calculated separately. These scores are then weighted and fused based on underwater environment parameters to obtain a comprehensive quality score. The underwater environment parameters are only used to determine the weights in the weighted fusion of the five dimensions and do not participate in the calculation of the scores for each dimension itself.

[0079] In this embodiment, the quality scores of the five dimensions are denoted as q1, q2, q3, q4, and q5, respectively, where q1 is the model confidence score, q2 is the visual feature consistency score, q3 is the sonar-optical cross-modal consistency score, q4 is the underwater environment geometric consistency score, and q5 is the temporal motion stability score.

[0080] The model confidence dimension is used to evaluate the confidence level of the current 3D object detection model or pre-labeled model in detecting the target corresponding to the 3D pseudo-label.

[0081] In this embodiment, before the first round of training, the average of the candidate box confidence scores output by the 2D detector and the 3D point cloud clustering scores in the two-stage pre-annotation is used as the model confidence score, i.e., q1=0.5s. box +0.5s cls , where s box s represents the confidence level of the candidate bounding boxes output by the YOLO26 detector head. cls The 3D point cloud clustering score is obtained by taking the normalized value of the number of points within the cluster and the cluster compactness. In subsequent training rounds, the 3D point cloud region corresponding to the pseudo-label is input into the currently trained 3D object detection model to obtain the object classification confidence score, which is then used as the model confidence score.

[0082] In this embodiment, the 3D point cloud clustering score s cls The calculation method is as follows: Let c be the cluster corresponding to the 3D pseudo-label, and N be the number of points within the cluster. c The cluster center is , then: s cls =0.5s n +0.5s d Among them, the point score s n =min(N c / N0,1), where N0 is the reference point, taken as 200; compactness score. , where is the average distance from a point within the cluster to the cluster center. Take 0.15m; the more points within a cluster and the more compact the distribution, the more s cls The higher the value, the range is [0,1].

[0083] The visual feature consistency dimension is used to evaluate whether the visual features within the image region corresponding to the 3D pseudo-label are continuous, complete, and consistent.

[0084] In this embodiment, the three-dimensional pseudo-tag is first projected onto the image plane: for the sonar point p within the bounding box of the three-dimensional pseudo-tag... s Based on the camera-sonar joint calibration extrinsic parameter transformation matrix And the camera intrinsic parameter matrix K, by Calculate the projection position of each element in the image; the minimum bounding rectangle of all projection positions is the two-dimensional projection region. Then, extract the dense visual features output by DINOv3 within this two-dimensional projection region, calculate the cosine similarity of feature vectors between adjacent image patches, and average the similarity of all adjacent image patch pairs to obtain the visual feature consistency score. N represents the set of all adjacent image patch pairs within the two-dimensional projection region, M represents the number of adjacent image patch pairs, and f m and f m’ Representing the m-th and m-th respectively ’ The feature vectors of adjacent image patches. An adjacent image patch pair refers to a combination of two image patches (i.e., adjacent in the top, bottom, left, and right neighborhoods) that share an edge in the image patch grid within a two-dimensional projection region.

[0085] The sonar-optics cross-modal consistency dimension is used to evaluate whether the correspondence between 3D pseudo-tags in the sonar 3D point cloud space and the image space is consistent.

[0086] In this embodiment, the 3D bounding box of the 3D pseudo-label is projected onto the image plane to obtain a 2D projection box. The formula for calculating the sonar-optics cross-modal consistency score is:

[0087] B proj B represents the two-dimensional projection of the 3D bounding box onto the image plane. 2D This indicates that the first stage of the two-stage pre-annotation is a two-dimensional candidate bounding box generated by the YOLO26 detector head. C represents the intersection-union ratio of two 2D boxes; proj and C 2D These represent the center point coordinates of the two-dimensional projection box and the two-dimensional candidate box, respectively; Take 10% of the diagonal length of the two-dimensional candidate box; γ is a weighting coefficient used to adjust the reliability weights of sonar and optical modes according to the target distance: when the target distance is less than 5m, the optical image is clearer, so take γ=0.7, with the intersection-union term as the main factor; when the target distance is not less than 5m, the optical image degrades, so take γ=0.5, and increase the weight of the center distance consistency term.

[0088] The underwater environment geometric consistency dimension is used to evaluate whether a 3D pseudo-label conforms to the geometric characteristics of an underwater target or structure.

[0089] In this embodiment, the point density, normal vector consistency, and curvature change of the sonar point cloud within the 3D pseudo-label bounding box are statistically analyzed. The specific calculation process is as follows: Point density: , where N box This represents the number of sonar points within the 3D bounding box. The volume of the three-dimensional bounding box;

[0090] Normal vector consistency: For each point within the bounding box, the local plane is fitted using its k nearest neighbors (k=20) to obtain the normal vector n. i Calculate the consistency of normal vectors Where n is the unit mean vector of the normal vectors of all points within the bounding box, and the closer η is to 1, the smoother and more regular the point cloud surface; Curvature variation: eigenvalues ​​are obtained from the covariance matrix fitted to the local plane. The curvature of point i is estimated as follows: The standard deviation of the curvature of all points within the bounding box is taken as the curvature change index k.

[0091] The aforementioned indicators are then matched against pre-established geometric priors for underwater targets. For each geometric indicator, a pre-defined expected range is set for each type of target. The matching score for a single indicator is as follows: if the indicator falls within the expected range, the score is 1; otherwise, it is reduced by a Gaussian factor based on the deviation from the center of the expected range. ,in The center of the expected interval, The expected interval width is [missing information]. The geometric consistency score is the average of the matching scores of the three indicators: In this embodiment, different geometric feature constraints (geometric priors) are used for different target categories, specifically:

[0092] Underwater pipelines and cables: A slender cylindrical prior is adopted, and the variance contribution rate of the first principal component in the principal component analysis is constrained to be no less than 0.8, with the cross-sectional radius between 0.05m and 0.5m; Planar structures such as walls and dams: A plane prior is adopted, and the consistency of the normal vector η is constrained to be no less than 0.9, with the plane fitting residual no greater than 0.05m;

[0093] For artificial structures such as bridge piers and jacket structures: a regular geometric prior is adopted, and the number of corner points and the orientation of edges are matched with a preset template. Specifically:

[0094] Number of corner points: For each point within the 3D bounding box, take its k nearest neighbors (k=20) and fit a local covariance matrix to obtain eigenvalues ​​λ1≥λ2≥λ3. Calculate the curvature response of that point.

[0095] k i =λ3 / (λ1+λ2+λ3), points with curvature response greater than 0.03 are identified as corner points, and the total number of corner points N is counted after nonmaximum suppression. c The preset template provides the expected range of the number of corner points for this type of target. For example, if the number of corner points of a bridge pier is 8, the expected range is [6, 10]. When the number of corner points falls within the expected range, the matching score of this indicator is 1; otherwise, it is Gaussian decayed according to the deviation from the center of the expected range.

[0096] Edge direction: Iteratively perform RANSAC line fitting on the extracted corner point set, extracting one edge line and removing its interior points in each fitting, until the number of remaining corner points is insufficient, thus obtaining the edge direction set {d}. mThe preset template provides a set of main directions for the edges of this type of target. For example, a bridge pier has four vertical edges and two sets of orthogonal horizontal edges. For each extracted edge, the angle between it and the nearest template direction is calculated. If the angle is less than 15°, the edge is considered a match. The ratio of the number of matched edges to the total number of edges in the template is used as the edge orientation score. The geometric consistency score for this type of target is the average of the corner point count score and the edge orientation score.

[0097] Reefs and sediments: A rough surface prior is adopted, and matching is performed using two indices: curvature variation and randomness of normal vector distribution. Specifically: Curvature variation: In this embodiment, the curvature k of each point within the three-dimensional bounding box is taken. i The standard deviation k is used as an indicator of curvature change, and the expected range of this type of target [g] is... min ,g max Set to a higher range, for example, [0.02, 0.1]; when the curvature change falls into the desired range, the matching score of this indicator is 1, otherwise it is Gaussian decayed according to the deviation from the center of the desired range.

[0098] Randomness of normal vector distribution: Calculate all unit normal vectors n within the 3D bounding box. i The mean vector magnitude: B box R represents the number of sonar points within the 3D bounding box, ∈ [0,1]. The closer R is to 1, the more concentrated the normal vector directions are, and the smoother the target surface; the closer R is to 0, the more dispersed the normal vector directions are, and the rougher and more random the target surface is. The expected range of R for this type of target is set to a lower range, such as [0,0.4]. When R falls within the expected range, the matching score for this indicator is 1; otherwise, it is Gaussian decayed based on the deviation from the center of the expected range.

[0099] In this embodiment, the geometric consistency score of this type of target is the average of the curvature change score and the randomness score of the normal vector distribution.

[0100] Marine organisms such as sea cucumbers, sea urchins, scallops, and starfish: Small-size priors are used, the target size is constrained to be between 0.05m and 0.5m, and the point density is within the expected range of the corresponding category.

[0101] The temporal motion stability dimension is used to evaluate the smoothness of changes in the center position, size, and attitude of the same target across consecutive frames. A Kalman filter is used for temporal target tracking. The state vector of each target includes its three-dimensional center position, target size, attitude angles, and velocity information. Specifically, the observation vector in frame t is denoted as z. t (Composed of the center position, size, and attitude angle of the current frame's 3D pseudo-label), the prior prediction given by the Kalman filter is... H is the observation matrix, and the deviation between the current frame's observation result and the prediction result is defined as: The temporal motion stability score is calculated as follows: Where σ is taken as 0.25m. The preset tolerance range is d. t ≤0.5m (i.e., 2σ): when d t When ≤0.5m, The time-series motion stability score is relatively high; when d t When the deviation is greater than 0.5m, q5 < 0.87 and continues to decrease as the deviation increases, indicating a low score for the stability of the time-series motion.

[0102] In this embodiment, the overall quality score Q is expressed as: Q=w1q1+ w2q2+ w3q3+ w4q4+ w5q5, where w1, w2, w3, w4, and w5 represent the weights of the five quality assessment dimensions, and satisfy: w1+ w2+ w3+ w4+ w5=1.

[0103] In this embodiment, when the water turbidity is below 20 NTU, the light intensity is above 80 lx, and the target distance is less than 5 m, the image quality is considered good. In this case, the weights are set as follows: (w1, w2, w3, w4, w5) = (0.20, 0.30, 0.20, 0.15, 0.15). When the water turbidity is not below 20 NTU, or the light intensity is not above 80 lx, or the target distance is not less than 5 m, the image quality is considered to have decreased. In this case, the weights are set as follows: (w1, w2, w3, w4, w5) = (0.25, 0.15, 0.25, 0.20, 0.15). The comprehensive quality score of each 3D pseudo-label is obtained through the above method.

[0104] S4. Based on the comprehensive quality score and the underwater environment parameters, the initial three-dimensional pseudo-label set is adaptively optimized to obtain an optimized three-dimensional pseudo-label set.

[0105] Adaptive optimization processing refers to performing operations such as retention, removal, point cloud completion, cross-modal label migration, or multimodal fusion enhancement on pseudo-labels based on their comprehensive quality scores and underwater environmental parameters, thereby improving the quality of pseudo-labels.

[0106] In this embodiment, the adaptive optimization process is as follows: Figure 3 As shown. Set a high-quality threshold. Low quality threshold For any 3D pseudo-label, perform the following processing based on its overall quality score Q:

[0107] Reserved operation, when If the 3D pseudo-label is determined to be a high-quality pseudo-label, it is directly retained and added to the optimized 3D pseudo-label set. In this embodiment, if the overall quality score of a certain 3D pseudo-label is 0.82, the pseudo-label is directly retained for subsequent 3D object detection model training.

[0108] Remove operation, when < If the 3D pseudo-label is determined to be a low-quality pseudo-label, it will be removed from the training set. In this embodiment, if the overall quality score of a 3D pseudo-label is 0.31, and its cross-modal consistency score and geometric consistency score are both low, then the pseudo-label will be deleted and will not participate in subsequent training.

[0109] Feature-guided point cloud completion operation, when ≤ < Furthermore, if the visual feature consistency score is high but the sonar point cloud is sparse, a feature-guided point cloud completion operation is performed on the 3D pseudo-label. In this embodiment, if the visual feature consistency score is greater than 0.70 and the point cloud density within the 3D bounding box is less than 50 points per cubic meter, the target is determined to have a point cloud sparsity problem.

[0110] The specific implementation process of point cloud completion is as follows: based on the projection formula The process involves determining the 2D projection region of the 3D pseudo-label in the image; obtaining the corresponding DINOv3 dense visual feature map of the 2D projection region and extracting image edges using the Canny operator; projecting the sonar points within the 3D bounding box back onto the image plane along the projection ray and comparing them with the target region in the dense visual feature map; identifying target sub-regions with insufficient projection point coverage as sparse or missing regions; the determination of insufficient projection point coverage is as follows: dividing the 2D projection region into several grid cells of 14×14 pixel image block grid, counting the number of projection points falling into each grid cell; if a grid cell has fewer than 3 projection points, it is considered an insufficiently covered cell; if... If the proportion of grid cells into which the projection point falls is less than 60% of the total number of grid cells in the projection area, the corresponding target sub-region is determined to be a sparse region. If no projection point falls into a sub-region, it is determined to be a missing region. For sparse or missing regions, the RANSAC method is used to fit a local plane using existing sonar points in its neighborhood. Supplementary point clouds are generated by uniform interpolation along the projection ray direction on the local plane according to the image edge direction. The spacing between the supplementary points is taken as the voxel size of 0.1m. The original point cloud and the supplementary point cloud are merged, and density-based clustering and principal component analysis attitude estimation are re-executed. The center position, size, and attitude parameters of the 3D bounding box are regenerated according to the bounding range of the point cloud to obtain the completed 3D pseudo-label.

[0111] Cross-modal label migration operation, when ≤ < When one of the sonar and optical modes has high quality while the other has low quality, a cross-modal tag migration operation is performed on the 3D pseudo-tag.

[0112] In this embodiment, if the target boundary in the image is clear and the visual feature consistency score is higher than 0.75, but the sonar point cloud is sparse, the target category and target region information in the image are transferred to the sonar point cloud space to correct the three-dimensional bounding box. If the sonar point cloud has a clear geometric shape and a geometric consistency score higher than 0.75, but the image is greatly affected by turbid water, the target position and geometric information in the sonar three-dimensional point cloud are used to correct the corresponding region of the image.

[0113] The criteria for determining whether an image is significantly affected by turbid water are: water turbidity not less than 20 NTU, light intensity not more than 80 lx, or image RMS contrast ratio less than 0.15. Meeting any one of these three criteria indicates that the image is significantly affected by turbid water. The formula for calculating image RMS contrast ratio is:

[0114] Where I(i,j) represents the gray value of the (i,j)th pixel in the image. This represents the average gray value of the image, where M and N represent the number of rows and columns of the image, respectively.

[0115] The specific implementation of correcting the corresponding region in the image using the target location and geometric information in the sonar 3D point cloud is as follows: the estimated 3D bounding box of the target in the sonar 3D point cloud is transformed by the camera-sonar joint calibration extrinsic parameter transformation matrix. The camera intrinsic parameter matrix K is projected onto the image plane, and the resulting projection area is the corrected two-dimensional target area. This projection area replaces the original blurred candidate area, thereby realizing the correction of the corresponding area of ​​the image using sonar information.

[0116] Multimodal fusion enhancement operation, when When both the sonar and optical modes have usable information, a multimodal fusion enhancement operation is performed on the 3D pseudo-label.

[0117] In this embodiment, the optical mode fusion weights are determined based on water turbidity, light intensity, and target distance. Sonar mode fusion weights Specifically, this is achieved by calculating three normalized environmental factors—turbidity factor, etc. ,in Water turbidity (unit: NTU); light factor ,in Illumination intensity (in lx); distance factor Where d is the target distance (in meters). Multiplying the three factors yields the overall visual quality score. And make the optical mode fusion weights Sonar mode fusion weights When the water is clear, the sunlight is strong, and the target is close, As the value approaches 1, the optical mode weight increases; however, when the water is turbid, the light is weak, or the target is far away, As the value approaches zero, the sonar mode weights increase.

[0118] In this embodiment, the parameters of the fused 3D bounding box are expressed as follows: ,in, This represents the parameters of the 3D bounding box estimated from the sonar point cloud. The parameters of the 3D bounding box, obtained from image projection and spatial association, are as described in step S2. This represents the parameters of the enhanced 3D bounding box. Through the above adaptive optimization process, an optimized 3D pseudo-label set is obtained.

[0119] S5. Train a 3D target detection model using an optimized 3D pseudo-label set, and iteratively adjust the pseudo-labels and / or training samples based on model training feedback to obtain the trained 3D target detection model. The 3D target detection model is used to detect underwater targets from underwater sonar 3D point cloud data or underwater multimodal scene data, and outputs the target category and 3D location information.

[0120] In this embodiment, the 3D object detection model employs the Sparse4D sparse convolutional 3D object detection network. Iterative adjustments to pseudo-labels and / or training samples based on model training feedback refer to the closed-loop adjustment of pseudo-label quality, training sample sampling weights, and model parameters during model training, based on model detection performance, the degree of inadequate category learning, sample feature distribution, and the model's stage-specific performance. The closed-loop iterative training process specifically includes the following steps:

[0121] A 3D object detection model is trained using the current optimized 3D pseudo-label set; after each training cycle, the performance of the 3D object detection model is evaluated on the validation sample set.

[0122] The degree of undertraining for different categories and scenarios is determined based on the model performance evaluation results; the sample sampling weights for the next training cycle are adjusted according to the degree of undertraining; visual semantic features of unlabeled training samples are extracted using the pre-trained visual base model, and the sampling weights of low-density region samples are increased according to the feature space distribution.

[0123] Save copies of the model parameters in a preset training round, and aggregate the models based on the consistency of feature representations among the model parameter copies; repeat the above training and feedback adjustment process until the training stopping condition is met to obtain the final 3D object detection model.

[0124] In this embodiment, the closed-loop iterative training process is as follows: Figure 4 As shown, specifically: a 3D object detection model is trained using the current optimized 3D pseudo-label set;

[0125] In this embodiment, the three-dimensional point cloud of the sonar is voxelized, and the voxel size is set to 0.1m×0.1m×0.1m; the model training batch size is set to 128; the initial learning rate is set to 0.001; the optimizer is AdamW; and the training epochs are set to 80.

[0126] After each training cycle, the performance of the 3D target detection model is evaluated on the validation sample set. In this embodiment, the validation sample set is constructed as follows: 5000 samples are extracted from the collected underwater multimodal scene samples, and the target categories and 3D bounding boxes are manually labeled to form the validation sample set. The validation sample set does not overlap with the unlabeled training samples and is only used for model performance evaluation, not for model training.

[0127] The degree of undertraining for different categories and scenarios is determined based on the model performance evaluation results. In this embodiment, after every 10 training rounds, the model performance is evaluated using a validation sample set to obtain the average precision (AP) for different target categories. Specifically, the AP metric (3DAP@0.25) is used when the 3D intersection-over-union threshold is 0.25. The evaluation method for the degree of undertraining is as follows: if the AP of a certain target category is lower than 80% of the average AP of all categories, then that category is considered undertrained. At the same time, the validation samples are divided into different scenarios according to underwater environmental parameters (e.g., by water turbidity, by target distance), and the AP in each scenario is statistically analyzed. If the AP of a certain scenario is lower than 80% of the average AP of all scenarios, then that scenario is considered undertrained.

[0128] Adjust the sample sampling weights for the next training cycle based on the degree of undertraining;

[0129] In this embodiment, the sample sampling weights for the next training cycle are given by the following formula: ,in, This represents the sampling weight of samples related to category c in the next training cycle. Represents the AP of category c, This represents the average AP across all categories. For example, when... hour, ; The lower the value, the higher the sampling weight, and the upper limit of the sampling weight is no more than 2; if a category is not identified as undertrained, the sampling weight remains at 1. The sampling weight of samples corresponding to undertrained scenarios is calculated in the same way, and the final sampling weight of the sample is obtained by multiplying the category weight by the scenario weight.

[0130] Visual semantic features of unlabeled training samples are extracted using a pre-trained visual base model, and the sampling weight of samples in low-density regions is increased based on the feature space distribution.

[0131] In this embodiment, a pre-trained visual base model is used to extract the visual semantic feature vectors of all unlabeled training samples, and a density clustering method is used to divide the samples into multiple feature clusters. If the number of samples in a certain feature cluster accounts for less than 3% of the total number of samples, the feature cluster is marked as a low-density feature region, and the sampling weight of the corresponding samples in the next training cycle is increased to twice the original weight.

[0132] Save copies of model parameters during preset training rounds, and aggregate models based on the consistency of feature representations among the model parameter copies.

[0133] In this embodiment, copies of the 3D object detection model parameters are saved in rounds 40, 60, and 80, respectively. For each copy of the model parameters, the feature representation of the validation sample is extracted, and the consistency score of the feature representation between different copies of the model parameters is calculated. The formula for calculating the consistency score of the feature representation is: ,in, The feature representation of the Kth model parameter copy is the consistency score. This represents the Kth model parameter copy against the validation sample. Extracted feature vectors This indicates that each model parameter copy corresponds to a validation sample. The mean of the extracted feature vectors, This indicates the number of validation samples.

[0134] Based on the consistency score represented by the features, the aggregate weights of each model parameter replica are determined using a softmax method: ,in, This represents the aggregate weight of the Kth model parameter copy, where K represents the number of model parameter copies (3 in this example), and is a temperature coefficient, set to 0.1. A weighted average is then applied to multiple model parameter copies to obtain the final trained 3D object detection model.

[0135] ,in, This represents the model parameters of the k-th model parameter copy. This represents the final model parameters after aggregation. Repeat the above training and feedback adjustment process until the training stopping condition is met, resulting in the final 3D object detection model.

[0136] In this embodiment, the training rounds are set to 80 rounds, and the training stops when 80 rounds are reached.

[0137] S6. When deploying the application, the sample to be detected is input into the trained 3D target detection model, and the underwater target category information and 3D position information are output. Deploying the application refers to deploying the trained 3D target detection model to underwater robots, shore-based computing platforms, or other underwater operation equipment to perform target detection on real-time collected underwater multimodal scene data.

[0138] In this embodiment, the application deployment process is as follows: Figure 5 As shown, the specific steps include:

[0139] The underwater robot collects samples to be tested in real time, including sonar three-dimensional point cloud data, image data and underwater environmental parameters.

[0140] Time synchronization and spatial coordinate alignment are performed on sonar 3D point cloud data and image data; among them, spatial coordinate alignment refers to unifying image data and sonar 3D point cloud data into the sonar 3D point cloud coordinate system by jointly calibrating extrinsic parameters of camera-sonar.

[0141] The aligned sonar 3D point cloud data is input into the trained 3D target detection model;

[0142] The 3D target detection model outputs at least one target detection result; in this embodiment, the target detection result is represented as: ,in, This represents the category information of the j-th detected target. This represents the 3D bounding box parameters of the j-th detected target. This represents the confidence level of the j-th detected target.

[0143] The target detection results are subjected to confidence screening and non-maximum suppression processing. In this embodiment, the confidence threshold for deployment detection is set to 0.35, and the 3D intersection-union ratio threshold for non-maximum suppression is set to 0.10. If the confidence of a detection result is lower than 0.35, the detection result is deleted; if the intersection-union ratio of the 3D bounding boxes corresponding to two detection results is greater than 0.10, the detection result with higher confidence is retained.

[0144] The system outputs underwater target category information, 3D bounding box information, target confidence level, and target spatial location. Through the above-described deployment and detection process, the system can output underwater target category, 3D location, and 3D bounding box information, which can be used for underwater robot obstacle avoidance, target recognition, structure inspection, or underwater operation decision-making.

[0145] Example 2: An underwater point cloud 3D target detection system based on pseudo-label optimization, such as... Figure 6 As shown, it includes: a data acquisition module, used to acquire underwater multimodal scene data and obtain unlabeled training samples and samples to be tested. In this embodiment, it includes a camera acquisition unit, a sonar acquisition unit, and a data synchronization unit. Specifically, the camera acquires 1920×1080 images at 30 frames / second, and the image sonar acquires point clouds at 10 frames / second. Frames with a time difference of less than 100ms are grouped into samples according to their timestamps, and a total of 50,000 unlabeled samples are collected as the training set.

[0146] The underwater environment sensing module is used to acquire underwater environmental parameters. In this embodiment, it includes turbidity, illumination, target distance, and pose acquisition units. Specifically, turbidity, illumination, and pose (100Hz) sensors acquire corresponding parameters, and the target distance is calculated by sonar point cloud computing. The output parameters are used by the quality assessment module and the pseudo-tag optimization module.

[0147] The two-stage pre-annotation module is used to perform two-stage pre-annotation based on a pre-trained visual base model to generate an initial set of 3D pseudo-labels. In this embodiment, it includes sub-modules for visual feature extraction, candidate region generation, spatial association, and pseudo-label generation. Specifically, the base model uses DINOv3 (fully frozen), and the YOLO26 detector head is used as input with its dense visual features (pre-trained on a public dataset and 3000 labeled images). 3D pseudo-labels are generated through spatial association and filtering clustering.

[0148] The quality assessment module is used to evaluate the quality of the initial 3D pseudo-label set and obtain a comprehensive quality score. In this embodiment, it includes five assessment sub-modules (confidence, visual consistency, cross-modal consistency, geometric consistency, and temporal stability), which are specifically implemented as follows: the scores of the five dimensions are calculated respectively, and the weights are determined by underwater environmental parameters and then weighted and fused.

[0149] The pseudo-label optimization module is used to adaptively optimize the initial 3D pseudo-label set to obtain an optimized 3D pseudo-label set. In this embodiment, it includes sub-modules for retention, removal, point cloud completion, cross-modal label migration, and multimodal fusion enhancement. Specifically, it is implemented as follows: labels with a score of not less than 0.75 are retained, those with a score less than 0.45 are removed, and the rest are completed, migrated, or fused and enhanced according to modal quality.

[0150] The model training module is used to train a 3D object detection model using an optimized 3D pseudo-label set and iteratively adjust it to obtain a trained model. In this embodiment, it includes model training, performance evaluation, sampling weight adjustment, and model aggregation sub-modules. Specifically, it uses Sparse4D, 0.1m voxels, 128 batches, AdamW, and trains for 80 rounds; every 10 rounds, it uses a 5000-set labeled validation set to evaluate 3DAP@0.25 and adjust the weights; in rounds 40, 60, and 80, it saves copies and aggregates them according to feature representation consistency.

[0151] A detection module is deployed to input the sample to be detected into the trained model and output the target category and 3D location. In this embodiment, it includes sample input, detection inference, result filtering, and result output units. Specifically, after filtering with a confidence level of 0.35 and non-maximum suppression with a 3D intersection-union ratio of 0.10, the target category, 3D bounding box, confidence level, and spatial location are output.

[0152] Experimental results and performance evaluation: The performance of the proposed method was evaluated on an independent validation sample set. The validation sample set consisted of 5000 sets of underwater multimodal scene samples with manually labeled target categories and 3D bounding boxes. These samples did not overlap with the unlabeled training samples and were used only for model performance evaluation, not for training. The evaluation metric was the average accuracy (3DAP@0.25) at a 3D intersection-union (IU) threshold of 0.25.

[0153] The experimental comparison scheme and results are as follows: Pseudo-label quality comparison: When training directly using the initial 3D pseudo-label set, the pseudo-labels contain erroneous labels caused by projection errors and sonar noise; after underwater scene perception quality assessment and adaptive optimization, low-quality pseudo-labels (overall quality score below 0.45) are removed, and intermediate-quality pseudo-labels are enhanced and repaired through point cloud completion, cross-modal label transfer or multimodal fusion, resulting in a significant improvement in the overall quality of pseudo-labels and effective suppression of training noise;

[0154] Detection accuracy comparison: The model trained by closed-loop iteration using an optimized 3D pseudo-label set has a significantly higher 3DAP@0.25 on the validation sample set than the model trained directly using the initial pseudo-label. Moreover, the accuracy improvement is more significant in degradation scenarios (high turbidity, long distance) divided by water turbidity and target distance.

[0155] Environmental adaptability: By adjusting turbidity, illumination and target distance conditions for group testing, the method of this invention can still maintain stable detection performance in scenarios with turbid water, insufficient illumination and distant targets, demonstrating the robustness advantage brought by the adaptive adjustment of quality assessment weights and multimodal fusion weights with the environment.

[0156] Training efficiency: Dual-stage pre-labeling eliminates the need for manual 3D annotation; pseudo-label generation and optimization of 50,000 unlabeled samples can be completed offline in batches; closed-loop iterative training strengthens sampling for under-trained categories and low-density feature regions, and combined with model aggregation based on feature representation consistency, it achieves better generalization performance in the same number of training rounds.

[0157] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting 3D underwater point clouds based on pseudo-label optimization, characterized in that, Includes the following steps: S1. Simultaneously collect two types of multimodal data: sonar 3D point cloud and optical image, using an underwater robot; generate an unlabeled training sample set and a real-time sample to be detected by matching the timestamps. S2. Input the optical image into a pre-trained visual base model with parameters frozen throughout the process to extract pixel-level dense visual features and generate two-dimensional target candidate boxes; project and map the two-dimensional candidate boxes onto the sonar three-dimensional point cloud space and extract the corresponding point cloud subsets. After visual semantic consistency screening and density clustering to remove outliers, an initial set of 3D pseudo-labels is generated. S3. For each initial 3D pseudo-label, calculate the normalized quality scores for five categories: model confidence, visual feature consistency, sonar-optical cross-modal consistency, underwater environment geometric consistency, and temporal motion stability. The fusion weights are dynamically allocated using underwater environmental parameters, and the weighted sum is used to obtain the pseudo-label comprehensive quality score Q. S4. Pre-set high-quality threshold Low quality threshold If Q≥ : Keep the pseudo-label directly; if Q < Remove low-quality, erroneous, and pseudo-labels; if ≤Q< Based on the score differences between the two modalities, three types of repair operations are selectively performed: feature-guided point cloud completion, cross-modal label transfer, and multimodal fusion enhancement, outputting an optimized 3D pseudo-label set; S5. A sparse convolutional 3D detection network is used to optimize the pseudo-label set for training. After each training cycle, the model performance is evaluated. Model copies are saved in the preset training rounds, and the final 3D detection model is obtained by weighted aggregation based on the consistency score of feature representation between copies. S6. After synchronizing the time and unifying the coordinates of the samples to be tested to the sonar point cloud coordinate system, input them into the final model.

2. The underwater point cloud 3D target detection method based on pseudo-label optimization as described in claim 1, characterized in that, The visual semantic consistency screening in step S2 is implemented by calculating the cosine similarity between the image patch features corresponding to the sonar point projection position and the average features of the two-dimensional candidate box. If the similarity is less than 0.55, the point cloud is removed, and a subset of point clouds with similarity greater than or equal to 0.55 is retained; the density clustering radius is 0.35m, and the minimum number of cluster points is 5.

3. The underwater point cloud 3D target detection method based on pseudo-label optimization as described in claim 1, characterized in that, The calculation rules for the quality score in step S3 are as follows: Model confidence score q1: In the first round of training, the average of the YOLO26 2D bounding box confidence score and the point cloud clustering compactness score is used; in subsequent training rounds, the classification confidence score of the current 3D model output is used. Visual feature consistency score q2: The average cosine similarity of features between adjacent image blocks within a two-dimensional projection region; Sonar-optical cross-modal consistency score q3: The IoU between the 2D projection frame and the YOLO26 candidate frame, and the Euclidean distance attenuation term of the frame center are fused. When the target distance is <5m, the IoU weight is 0.7, and when it is ≥5m, the IoU weight is 0.

5. Underwater environment geometric consistency score q4: Matching point density, normal vector consistency, and curvature change are three types of geometric priors. Differentiated geometric constraints are set for pipelines, planar structures, bridge piers, reefs, and marine life. Temporal motion stability score q5: Kalman filter predicts the temporal state of the target, and the stability score is calculated using the observation bias Gaussian decay function.

4. The underwater point cloud 3D target detection method based on pseudo-label optimization as described in claim 1, characterized in that, Step S3 incorporates the following dynamic weight allocation rules: For clear close-up scenes: turbidity < 20 NTU, illumination > 80 lx, target distance < 5m, and the pseudo-label's overall quality score. Q = w1q1 + w2q2 + w3q3 + w4q4 + w5q5; weight allocation (w1, w2, w3, w5, w5) = (0.20, 0.30, 0.20, 0.15, 0.15); For murky / low-light / degraded distant scenes: turbidity ≥ 20 NTU or illumination ≤ 80 lx or target distance ≥ 5m, weight allocation (w1, w2, w3, w5, w5) = (0.25, 0.15, 0.25, 0.20, 0.15).

5. The underwater point cloud 3D target detection method based on pseudo-label optimization as described in claim 1, characterized in that, The logic for determining and implementing the pseudo-label repair operation in step S4 is as follows: Feature-guided point cloud completion: Visual consistency score > 0.70 and point cloud density within the bounding box < 50 points / m³; Supplementary point clouds are generated through image edge and visual feature interpolation, and the corrected 3D bounding boxes are re-clustered. Cross-modal label transfer: high quality in one modality, degradation in another; clear images will have their 2D semantic labels mapped to correct the 3D bounding boxes. If the sonar geometry is complete, the candidate region of the image is corrected using 3D geometry; multimodal fusion enhancement: both modalities have effective information; based on turbidity, illumination, and distance, the visual comprehensive score is calculated and the optical and sonar modal fusion weights are assigned, and the 3D bounding box parameters of the two modalities are weighted and fused.

6. The underwater point cloud 3D target detection method based on pseudo-label optimization as described in claim 1, characterized in that, Step S5, closed-loop iterative training, includes weight adjustment rules: Category / Scene Sampling Weights: , where w c AP represents the sampling weight of samples related to category c in the next training cycle. c Represents the AP of category c, This represents the average AP across all categories; Low-density feature samples: DINOv3 feature density clustering is used, and the sampling weight of feature clusters with a sample ratio of <3% is doubled; Model aggregation: Model copies are saved in rounds 40, 60, and 80, the feature consistency score of each copy on the validation set is calculated, the aggregation weights are assigned through the softmax function, and the final model parameters are obtained by weighted averaging.

7. The underwater point cloud 3D target detection method based on pseudo-label optimization as described in claim 1, characterized in that, Step S6: Deploy post-processing parameters: confidence screening threshold 0.35, 3D non-maximum suppression cross-union threshold 0.10, and remove duplicate and low-confidence detection boxes.

8. A three-dimensional target detection system for underwater point clouds based on pseudo-label optimization, characterized in that, The system is used to execute the detection method according to any one of claims 1 to 7, comprising a data acquisition module, an underwater environment perception module, a two-stage pre-labeling module, a quality assessment module, a pseudo-label optimization module, a model training module, and a deployment detection module; the data acquisition module is equipped with an underwater robot and simultaneously outputs optical images with a resolution of 1920×1080 and a acquisition frequency of 30 frames / second, and sonar three-dimensional point clouds with a acquisition frequency of 10 frames / second; the underwater environment perception module acquires turbidity, illumination, and pose sensor data, and outputs environmental parameters to the quality assessment and pseudo-label optimization modules; the two-stage pre-labeling module includes: The system includes a frozen DINOv3, pre-trained YOLO26, and camera-sonar joint calibration unit, which outputs initial 3D pseudo-labels. The quality assessment module includes a five-dimensional score calculation subunit and an environment-adaptive weight allocation subunit. The pseudo-label optimization module integrates pseudo-label retention, removal, point cloud completion, label transfer, and multimodal fusion enhancement subunits. The model training module incorporates a Sparse4D network, a performance evaluation unit, a sampling weight adjuster, and a model aggregation unit. The deployment detection module performs multimodal time synchronization, coordinate alignment, model inference, and post-processing and output of detection results.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements all the steps of the underwater sonar point cloud three-dimensional target detection method based on pseudo-label optimization as described in any one of claims 1 to 7.

10. An embedded computing device for an underwater robot, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the real-time deployment and detection steps of the underwater sonar point cloud three-dimensional target detection method based on pseudo-label optimization as described in any one of claims 1 to 7.

Images