An underwater close-range target fusion detection and recognition method, system and device

Abstract

The application belongs to the technical field of underwater robots, and discloses a kind of underwater close-range target fusion detection identification method, system and device, the method comprises: obtaining multi-source heterogeneous data and pre-processing, extracting heterogeneous representative features;Cross-modal feature alignment processing is carried out to the heterogeneous representative features, and consistent cross-modal features are obtained;Through the cross-modal attention fusion mechanism built-in in the preset feature fusion network, the attention weight of each modal feature channel in the consistent cross-modal feature is adaptively solved, and the consistent cross-modal feature is weighted and fused according to the attention weight, and the joint feature is output;A multi-task output layer is constructed, the joint feature is input into the multi-task output layer, and the detection and identification result of the underwater target and the target contour accurate positioning information are output.The application integrates multi-source information with high complementarity through an advanced fusion mechanism, realizes high-precision output, and has high automation and good system flexibility.

Description

Technical Field

[0001] This invention relates to the field of underwater robot technology, and in particular to a method, system and device for underwater close-range target fusion detection and recognition. Background Technology

[0002] Underwater robots, also known as unmanned remotely operated vehicles, are robots that perform extreme underwater operations. The underwater environment is harsh and dangerous, and the diving depth of humans is limited. Therefore, underwater robots have become an important tool for ocean development. In close-range operations, such as underwater pipeline inspection, underwater structure repair, and salvage of sunken objects, underwater robots need to accurately detect, identify, and locate the contours of targets.

[0003] However, the underwater close-range environment is extremely complex: the absorption and scattering of light by the water body leads to blurred optical images, color distortion, and low contrast; turbid water body will greatly shorten the effective range of visible light; bubbles and suspended matter generated during operations will cause strong interference. Although sonar is not affected by light and turbidity, its resolution is limited at close range, and its ability to identify the edges and texture details of targets is far lower than that of optical sensors. It also suffers from acoustic shadowing and specular reflection problems.

[0004] Existing technologies often rely on single sensors or simple information fusion (such as data-level stitching or decision-level voting), which have obvious limitations and cause inconvenience to users.

[0005] Therefore, how to provide a method, system, and device for underwater close-range target fusion detection and identification is an urgent problem to be solved. Summary of the Invention

[0006] This invention provides a method, system, and apparatus for underwater close-range target fusion detection and identification to solve the problems in the prior art.

[0007] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or to describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0008] According to a first aspect of the present invention, a method for fusion detection and identification of underwater close-range targets is provided.

[0009] In one embodiment, an underwater near-range target fusion detection and identification method includes:

[0010] Acquire multi-source heterogeneous underwater near-field data within the target area;

[0011] Preprocess multi-source heterogeneous data to extract representative heterogeneous features;

[0012] Cross-modal feature alignment is performed on heterogeneous representative features to obtain consistent cross-modal features. The attention weights of each modal feature channel in the consistent cross-modal features are adaptively solved by the cross-modal attention fusion mechanism built into the preset feature fusion network. The consistent cross-modal features are then weighted and fused according to the attention weights to output joint features.

[0013] A multi-task output layer is constructed to perform target category recognition and pixel-level and / or point cloud-level accurate segmentation. Joint features are input into the multi-task output layer to output the detection and recognition results of underwater targets and the accurate positioning information of target contours.

[0014] In one embodiment, the multi-source heterogeneous data includes color images, 3D point cloud data, and acoustic images.

[0015] In one embodiment, preprocessing multi-source heterogeneous data to extract heterogeneous representative features includes:

[0016] Underwater scattering enhancement processing is performed on the color image to obtain an enhanced color image;

[0017] Statistical outlier removal and voxel downsampling are performed on the 3D point cloud data to remove invalid noise points and reduce the amount of data, resulting in enhanced 3D point cloud data.

[0018] Acoustic images are subjected to speckle noise suppression and contrast enhancement processing to eliminate inherent noise in acoustic imaging and obtain enhanced acoustic images;

[0019] Texture color features, geometric shape features, and material reflection intensity features are extracted from enhanced color images, enhanced 3D point cloud data, and enhanced acoustic images, respectively, to form heterogeneous representative features.

[0020] In one embodiment, performing underwater scattering enhancement processing on a color image to obtain an enhanced color image includes:

[0021] Based on the color image, global background light estimation is performed to obtain the global background light vector;

[0022] Based on the global background light vector, the transmittance of the color image is estimated pixel by pixel to generate a transmittance map;

[0023] Based on underwater optical propagation characteristics, and combining global background light vector and transmittance map, the target radiance is retrieved from the color image to obtain a clear image;

[0024] The clear image is sequentially subjected to white balance correction and adaptive local contrast enhancement processing to output an enhanced color image.

[0025] In one embodiment, texture color features, geometric shape features, and material reflection intensity features are extracted from enhanced color images, enhanced 3D point cloud data, and enhanced acoustic images, respectively, to form heterogeneous representative features, including:

[0026] Multi-scale convolution processing is performed on the enhanced color image to obtain feature maps of different resolutions in the shallow, middle and deep layers. After channel alignment and upsampling fusion, the texture color features are output.

[0027] Extract 3D spatial features from enhanced 3D point cloud data and output geometric shape features;

[0028] The enhanced acoustic image is convolved to output the material reflection intensity characteristics.

[0029] By integrating multi-scale texture color features, geometric shape features, and material reflection intensity features, heterogeneous representative features are formed.

[0030] In one embodiment, cross-modal feature alignment processing is performed on heterogeneous representative features to obtain consistent cross-modal features, including:

[0031] Based on the pose and depth data corresponding to the target region, the time stamp alignment process is performed on each feature in the heterogeneous representative features to obtain time-aligned heterogeneous representative features.

[0032] Based on the rotation matrix and translation vector corresponding to the pose data, each feature in the time-aligned heterogeneous representative feature is uniformly mapped to the same spatial reference coordinate system to obtain the spatially aligned heterogeneous representative feature.

[0033] Adaptive scale normalization is performed on spatially aligned heterogeneous representative features to map features of different dimensions and magnitudes to a unified feature scale range, resulting in consistent cross-modal features.

[0034] In one embodiment, the attention weights of each modal feature channel in the consistent cross-modal features are adaptively solved by using the cross-modal attention fusion mechanism built into the preset feature fusion network, and the consistent cross-modal features are weighted and fused according to the attention weights to output joint features including:

[0035] By using a pre-defined feature fusion network, the modal features in the consistent cross-modal features are mapped and adjusted to a unified feature dimension, resulting in the adjusted consistent cross-modal features.

[0036] Based on the cross-modal attention fusion mechanism built into the preset feature fusion network, global pooling sampling is performed on each modality feature in the adjusted consistent cross-modal features, the channel description information of all modalities is fused, and attention weight vectors corresponding to each modality feature channel are adaptively generated.

[0037] By using a cross-modal attention fusion mechanism, each modal feature is multiplied by its corresponding attention weight vector channel by channel to obtain weighted modal features;

[0038] By using a pre-defined feature fusion network, the weighted modal features are spliced ​​and fused along the channel dimension. The spliced ​​features are then subjected to dimensionality reduction, filtering, and integration processing to output joint features.

[0039] In one embodiment, based on the cross-modal attention fusion mechanism built into the preset feature fusion network, global pooling sampling is performed on each modality feature in the adjusted consistent cross-modal features, fusing the channel description information of all modalities, and adaptively generating the attention weight vector corresponding to each modality feature channel, including:

[0040] Global average pooling is performed on each modal feature in the adjusted consistency cross-modal features to obtain the channel description vector of each modal feature;

[0041] The channel description vector of the current modality feature is concatenated with the channel description vectors of all remaining modality features to form a cross-modal context description vector;

[0042] The cross-modal context description vector is sequentially processed through fully connected dimensionality reduction, nonlinear activation, fully connected dimensionality increase, and normalization to generate the attention weight vector corresponding to the current modal feature channel.

[0043] Iterate through all modalities to obtain the attention weight vector corresponding to the feature channels of each modality.

[0044] In one embodiment, the multi-task output layer includes a classification branch and a splitting branch;

[0045] The classification branch is used to perform global pooling and fully connected mapping on the joint features in sequence, and output the target category recognition result.

[0046] The segmentation branch is used to perform layer-by-layer upsampling and convolution to restore spatial resolution of the joint features and output pixel-level segmentation results; or, it is used to map the joint features back to each sampling point and then perform point-by-point category determination processing to output point cloud-level segmentation results.

[0047] The classification branch and the segmentation branch are supervised by a joint loss function during the training phase. The joint loss function includes a classification loss function, a cross-entropy loss function, and a Dice loss function.

[0048] According to a second aspect of the present invention, an underwater near-range target fusion detection and identification system is provided.

[0049] In one embodiment, an underwater near-range target fusion detection and identification system includes:

[0050] Heterogeneous data acquisition unit, used to acquire multi-source heterogeneous underwater near-field data within the target area;

[0051] The heterogeneous data processing unit is used to preprocess multi-source heterogeneous data and extract heterogeneous representative features;

[0052] The joint feature construction unit is used to perform cross-modal feature alignment processing on heterogeneous representative features to obtain consistent cross-modal features. Through the cross-modal attention fusion mechanism built into the preset feature fusion network, the attention weights of each modal feature channel in the consistent cross-modal features are adaptively solved, and the consistent cross-modal features are weighted and fused according to the attention weights to output the joint features.

[0053] The detection and recognition unit is used to construct a multi-task output layer for performing target category recognition and pixel-level / point cloud-level accurate segmentation. It inputs joint features into the multi-task output layer and outputs the detection and recognition results of underwater targets and the accurate positioning information of the target contour.

[0054] According to a third aspect of the present invention, an underwater near-range target fusion detection and identification device is provided.

[0055] In one embodiment, an underwater near-range target fusion detection and identification device includes:

[0056] Drive mechanism, heterogeneous data acquisition module, feature processing module, drive module and adjustment module;

[0057] The drive mechanism includes a floating plate, with a battery installed at one end of the bottom of the inner cavity of the floating plate and a feature processing module installed at the other end of the bottom of the inner cavity of the floating plate.

[0058] In one embodiment, the feature processing module includes a feature extractor and a feature processor, with the feature extractor located at the bottom of the other end of the inner cavity of the floating plate and the feature processor located at the top of the other end of the inner cavity of the floating plate.

[0059] In one embodiment, the drive module includes a second motor disposed at one end of the top of the floating plate, the output shaft of the second motor is provided with a third bracket, a third motor is disposed on one side of the third bracket, and the output shaft of the third motor is provided with fan blades;

[0060] In one embodiment, a second auxiliary rod is provided at the other end of the bottom of the third support, a second locking plate is provided at the bottom of the second auxiliary rod, and the second locking plate is provided at the top of the floating plate.

[0061] In one embodiment, the adjustment module includes a first motor disposed at the middle of the top of the floating board, the output shaft of the first motor is provided with an electric telescopic rod, the telescopic end of the electric telescopic rod is provided with a first bracket, and the front of the first bracket is provided with a fourth motor.

[0062] The bottom of the fixed end of the electric telescopic pole is provided with a first card plate, and several first auxiliary rods are slidably arranged at equal intervals on the bottom of the first card plate, and the first auxiliary rods are set on the top of the floating plate.

[0063] In one embodiment, the heterogeneous data acquisition module includes a second bracket mounted on the output shaft of the fourth motor, and an inertial measurement instrument is mounted on one end of the top of the second bracket.

[0064] An optical camera is installed in the middle of one side of the second bracket, and high-frequency forward-looking sonars are installed at both ends of one side of the second bracket. A laser scanner is installed at one end of the bottom of the second bracket, and a depth pressure sensor is installed at the bottom of the second bracket and on the other side of the laser scanner.

[0065] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0066] 1. The invention has strong information complementarity: the advantages of optics, laser and sonar complement each other, overcoming the limitations of a single sensor in a specific environment. Laser provides accurate 3D shape, optics provides texture and color, and sonar provides information on penetrating turbid water and material.

[0067] 2. The fusion mechanism of this invention is advanced: It adopts feature-level fusion based on attention mechanism, which can adaptively weigh the importance of different sensor information. Especially when the data quality of some sensors deteriorates, it can automatically rely on more reliable sensor information, which is extremely robust. Moreover, due to the addition of feature alignment module, the spatiotemporal deviation between different modalities can be reduced before fusion, which improves the stability and recognition accuracy of subsequent attention fusion.

[0068] 3. The output results of this invention are accurate: it simultaneously completes classification and pixel-level segmentation, not only knowing what it is, but also knowing its precise location and outline, providing key information for subsequent grasping, operation and other tasks;

[0069] 4. This invention has a high degree of automation: it is based on end-to-end deep learning, which reduces the reliance on manual feature design and complex rule formulation in traditional methods;

[0070] 5. The invention has high overall flexibility: With the cooperation of the drive module and the adjustment module, the overall flexibility of the heterogeneous data acquisition module can be effectively improved, thereby achieving the purpose of identifying targets at different locations.

[0071] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0072] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0073] Figure 1 This is a flowchart illustrating an underwater near-field target fusion detection and recognition method according to an exemplary embodiment;

[0074] Figure 2 This is a schematic diagram illustrating the principle of an underwater near-range target fusion detection and recognition system according to an exemplary embodiment;

[0075] Figure 3 This is one of the three-dimensional structural schematic diagrams of an underwater near-range target fusion detection and identification device according to an exemplary embodiment;

[0076] Figure 4 This is a second three-dimensional structural schematic diagram of an underwater near-range target fusion detection and identification device according to an exemplary embodiment;

[0077] Figure 5 This is the third three-dimensional structural schematic diagram of an underwater near-range target fusion detection and identification device according to an exemplary embodiment;

[0078] Figure 6 This is a schematic diagram of the internal structure of a floating plate in an underwater near-range target fusion detection and identification device according to an exemplary embodiment.

[0079] In the picture:

[0080] 1. Drive mechanism; 2. Heterogeneous data acquisition module; 3. First support; 4. Second auxiliary rod; 5. Fan blade; 6. Second clamping plate; 7. First motor; 8. Third support; 9. First auxiliary rod; 10. First clamping plate; 11. High-frequency forward-looking sonar; 12. Inertial measurement unit; 13. Second support; 14. Fourth motor; 15. Depth pressure sensor; 16. Optical camera; 17. Laser scanner; 18. Floating board; 19. Third motor; 20. Electric telescopic rod; 21. Battery; 22. Feature processing module; 23. Feature extractor; 24. Feature processor; 25. Drive module; 26. Adjustment module; 27. Second motor. Detailed Implementation

[0081] The following description and accompanying drawings fully illustrate specific embodiments described herein to enable those skilled in the art to practice them. Some portions and features of certain embodiments may be included in or replace portions and features of other embodiments. The scope of the embodiments herein includes the entire scope of the claims and all available equivalents thereof. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0082] The modules in the apparatus or system of this application can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0083] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0084] Figure 1 An embodiment of an underwater close-range target fusion detection and identification method of the present invention is shown.

[0085] In this optional embodiment, the underwater near-range target fusion detection and identification method includes:

[0086] S101. Acquire multi-source heterogeneous underwater near-field data within the target area;

[0087] In this optional embodiment, the multi-source heterogeneous data includes color images, 3D point cloud data, and acoustic images.

[0088] It should be explained that the optical camera 16 acquires RGB images, the high-frequency forward-looking sonar 11 emits and receives sound echoes to form a high-resolution acoustic image, the laser scanner 17 scans the target surface to obtain a three-dimensional point cloud, and the inertial measurement unit 12 and the depth and pressure sensor 15 provide attitude and depth information, respectively.

[0089] S102. Preprocess the multi-source heterogeneous data and extract representative features of the heterogeneous data;

[0090] In this optional embodiment, preprocessing the multi-source heterogeneous data to extract heterogeneous representative features includes:

[0091] Underwater scattering enhancement processing is performed on the color image to obtain an enhanced color image;

[0092] Statistical outlier removal and voxel downsampling are performed on the 3D point cloud data to remove invalid noise points and reduce the amount of data, resulting in enhanced 3D point cloud data.

[0093] Acoustic images are subjected to speckle noise suppression and contrast enhancement processing to eliminate inherent noise in acoustic imaging and obtain enhanced acoustic images;

[0094] Texture color features, geometric shape features, and material reflection intensity features are extracted from enhanced color images, enhanced 3D point cloud data, and enhanced acoustic images, respectively, to form heterogeneous representative features.

[0095] In this optional embodiment, performing underwater scattering enhancement processing on the color image to obtain an enhanced color image includes:

[0096] Based on the color image, global background light estimation is performed to obtain the global background light vector;

[0097] Based on the global background light vector, the transmittance of the color image is estimated pixel by pixel to generate a transmittance map;

[0098] Based on underwater optical propagation characteristics, and combining global background light vector and transmittance map, the target radiance is retrieved from the color image to obtain a clear image;

[0099] The clear image is sequentially subjected to white balance correction and adaptive local contrast enhancement processing to output an enhanced color image.

[0100] In this optional embodiment, texture color features, geometric shape features, and material reflection intensity features are extracted from the enhanced color image, enhanced 3D point cloud data, and enhanced acoustic image, respectively, to form heterogeneous representative features, including:

[0101] Multi-scale convolution processing is performed on the enhanced color image to obtain feature maps of different resolutions in the shallow, middle and deep layers. After channel alignment and upsampling fusion, the texture color features are output.

[0102] Extract 3D spatial features from enhanced 3D point cloud data and output geometric shape features;

[0103] The enhanced acoustic image is convolved to output the material reflection intensity characteristics.

[0104] By integrating multi-scale texture color features, geometric shape features, and material reflection intensity features, heterogeneous representative features are formed.

[0105] It should be explained that the preprocessing of the RGB image includes underwater image enhancement, employing a physics-based underwater scattering algorithm. The preprocessing of the 3D point cloud data includes outlier removal and voxel-based downsampling. The preprocessing of the acoustic image includes speckle noise suppression and contrast enhancement. Specifically, the underwater scattering algorithm first estimates the background light value and transmittance based on the acquired image, then uses an underwater imaging model to recover the scene's radiance, and combines white balance correction and local contrast enhancement to obtain the enhanced RGB image.

[0106] In specific implementation, the underwater scattering algorithm first estimates the background light of the original R32GB image, then calculates the transmittance based on pixel attenuation, and subsequently uses an underwater imaging model to recover the target radiance. The steps are as follows: An improved Jaffe-McGlamory underwater imaging physical model is used, and its imaging process can be represented as follows:

[0107] ;

[0108] In the formula, I(x) represents the observed value of the original underwater RGB image at pixel x; J(x) represents the true radiance of the target scene under scattering-free conditions (i.e., the clear image to be restored); t(x) represents the water transmittance at pixel x; and A represents the global background light vector, which is estimated from the brightest region in the image.

[0109] Based on the above model, the target radiance J(x) can be obtained by inverse calculation:

[0110] ;

[0111] In the formula, t0 is the lower limit threshold of transmittance (set to 0.1 in this embodiment) to avoid numerical instability when the transmittance is too low; the meanings of the other parameters are the same as above.

[0112] A lightweight convolutional neural network based on the above physical model is used to implement the underwater imaging physical model, which consists of three cascaded modules: a background light estimation module, which estimates the global background light vector A from the input image through multi-scale pooling and attention mechanisms; a transmittance estimation module, which takes the original image and the background light vector as input and outputs a pixel-wise transmittance map t(x) using an encoding and decoding structure; and a radiance reconstruction module, which calculates the target radiance J(x) from the observed image I(x) using the estimated A and t(x) according to the above formula and outputs an enhanced and clear image.

[0113] The underwater imaging physical model training process is as follows:

[0114] Training data: The training was conducted using a mixture of publicly available underwater image datasets and self-made underwater real-shot images, including underwater scene images under different turbidity and lighting conditions;

[0115] Loss function: The loss function is a weighted sum of reconstruction loss and perception loss.

[0116] ;

[0117] In the formula, L1 is Norm loss, The perceptual loss is based on the VGG network, with weights α=1 and β=0.1.

[0118] Optimization and training strategy: The Adam optimizer is used, and the initial learning rate is set to... The batch size is 16, and the training rounds are 100. The learning rate decreases to 0.5 times every 20 rounds, and an early stopping mechanism is used to prevent overfitting.

[0119] Key parameter settings: lower limit threshold for transmittance t0 = 0.1; local window size for background light estimation is 15×15.

[0120] The underwater imaging physical model output image is then subjected to white balance correction and adaptive local contrast enhancement to obtain the final clear RGB image after preprocessing. Point cloud preprocessing employs statistical outlier filtering to remove outliers, followed by voxel downsampling. Acoustic images undergo speckle noise suppression before contrast enhancement to improve the stability of subsequent feature extraction. Outlier removal uses statistical filtering to eliminate isolated points, and voxel downsampling aggregates the point cloud according to a preset voxel grid. Speckle noise suppression combines median filtering with adaptive smoothing, and contrast enhancement uses histogram stretching or adaptive equalization.

[0121] The system combines white balance and local contrast enhancement to output clear images; point cloud preprocessing uses statistical outlier filtering to remove abnormal scattered points, and then uses voxel grids for downsampling; acoustic images are first subjected to speckle noise suppression, and then contrast enhancement is performed to improve the stability of subsequent feature extraction.

[0122] Feature extraction is performed using a deep convolutional neural network. RGB images are input into a 2DCNN to extract multi-scale visual features, 3D point clouds are input into PointNet++ or VoxelNet to extract geometric features, and acoustic images are input into another 2DCNN to extract acoustic features.

[0123] The preprocessed enhanced 3D point cloud contains spatial coordinates (x, y, z), with a uniform number of points N=1024, and no additional normal or curvature information. The PointNet++ network architecture adopts a hierarchical multi-scale structure with 4 layers of ensemble abstraction (SA), 2 layers of feature propagation (FP), and global pooling. Specifically, the processing of each layer is as follows:

[0124] SA1 layer: Farthest point sampling (FPS) reduces 1024 points to 512 points; spherical neighborhood radius is 0.1m, with 32 points per neighborhood; built-in MLP (multilayer perceptron) is [64,64,128], outputting 512×128 local features.

[0125] SA2 layer: FPS reduced to 256 points; neighborhood radius 0.2m, 32 points; MLP is [128, 128, 256], outputting 256×256 features.

[0126] SA3 layer: FPS reduced to 128 points; neighborhood radius 0.4m, 32 points; MLP is [256,256,512], outputting 128×512 features.

[0127] SA4 layer: FPS reduced to 64 points; neighborhood radius 0.8m, 32 points; MLP is [512, 512, 1024], outputting 64×1024 high-level features.

[0128] FP1 layer: Upsample SA4 features to the SA3 scale, interpolate and concatenate with SA3 features. MLP is [1536,512,512], outputting 128×512 features.

[0129] FP2 layer: Upsampled to SA2 scale, the stitched MLP is [768,256,256], and the output is 256×256 fused features.

[0130] Global pooling: Perform global max pooling on the FP2 output to output a 256-dimensional geometric feature vector.

[0131] The training process and parameter settings for the PointNet++ network architecture are as follows: The training data uses an underwater near-field scene point cloud dataset, including categories such as pipes, reefs, sunken objects, and underwater structures, with category labels and a sample size of ≥10,000 frames. The joint loss function is defined as... Among them, classification loss Cross-entropy, segmentation loss For Dice loss, weights =1.0. The optimizer used is Adam, with an initial learning rate of 1×10. -3The batch size is 8, and the training rounds are 80. The learning rate decays to 0.5 times every 20 rounds. The weights are initialized using Xavier uniform initialization, and the activation function is ReLU. In addition, BatchNorm and Dropout (dropout=0.5) regularization are added to the PointNet++ network.

[0132] The VoxelNet network architecture takes the same input data as PointNet++, which is a preprocessed augmented 3D point cloud (x, y, z). Specifically, the processing at each level is as follows:

[0133] Voxelization: The spatial range is limited to [-4m, 4m] × [-4m, 4m] × [-2m, 2m], the voxel size is 0.1m × 0.1m × 0.1m, the output is a 48 × 48 × 40 voxel grid, and the mean feature of each voxel is stored.

[0134] VFE layer: VFE1 encodes each voxel in-point using MLP

[64] and outputs voxel features by max pooling; VFE2 encodes twice using MLP

[128] and outputs a 48×48×40×128 feature tensor.

[0135] 3D convolutional layers: Conv3D1 (3×3×3 convolution, 64 channels, stride 2) → 24×24×20×64; Conv3D2 (3×3×3, 128 channels, stride 2) → 12×12×10×128; Conv3D3 (3×3×3, 256 channels, stride 2) → 6×6×5×256.

[0136] Global pooling: Global average pooling outputs a 256-dimensional geometric feature vector.

[0137] The training process and parameter settings for the VoxelNet network architecture are consistent with those for PointNet++. The optimizer used is Adam, with an initial learning rate of 1×10⁻⁶. -3 The batch size is 4, and the training rounds are 80. The learning rate decays by 0.5 times every 20 rounds. In addition, the VoxelNet network incorporates the ReLU activation function and BatchNorm regularization.

[0138] Acoustic Image Material Reflection Intensity Feature Extraction (2DCNN): The preprocessed enhanced acoustic image is a single-channel acoustic intensity grayscale image with a uniform size of 256×256. A lightweight structure of 4 convolutional layers, 3 pooling layers, and global pooling is adopted. Specifically, the processing of each layer is as follows:

[0139] Input: 256×256×1 acoustic image.

[0140] Conv1: 3×3 convolution, 32 channels, ReLU activation, Same padding → 256×256×32; MaxPool1 (2×2, stride 2) → 128×128×32.

[0141] Conv2: 3×3 convolution, 64 channels, ReLU→128×128×64; MaxPool2→64×64×64.

[0142] Conv3: 3×3 convolution, 128 channels, ReLU→64×64×128; MaxPool3→32×32×128.

[0143] Conv4: 3×3 convolution, 256 channels, ReLU → 32×32×256.

[0144] Global average pooling: Outputs a 256-dimensional feature vector of material reflection intensity.

[0145] The training process and parameter settings for the 2DCNN network are as follows: The training data uses an underwater acoustic image dataset containing four material types: metal, concrete, rock, and mud. Material labels are provided, and the sample size is ≥8000. The loss function used is classification cross-entropy loss. The Adam optimizer is set with an initial learning rate of 1×10⁻⁶. -4 The batch size is 16, and the training run is 60 epochs. The learning rate decays by 0.5 times every 15 epochs. In addition, the ReLU activation function and BatchNorm+Dropout (dropout=0.3) regularization are added to the 2DCNN network. The color features of the color image texture (256-dimensional), the geometric features of the point cloud (256-dimensional), and the material features of the acoustic image (256-dimensional) are concatenated to obtain 768-dimensional heterogeneous representative features, which are then input into the subsequent cross-modal feature alignment module.

[0146] The specific process of extracting multi-scale visual features is as follows: RGB images are input into the backbone convolutional network, and feature maps of different resolutions are obtained in the shallow, middle and deep layers respectively. Then, the number of channels is unified by 1×1 convolution, and fusion is performed by combining top-down upsampling and lateral connections to form multi-scale visual features that take into account both texture details and semantic information.

[0147] S103. Perform cross-modal feature alignment processing on heterogeneous representative features to obtain consistent cross-modal features; adaptively solve the attention weights of each modal feature channel in the consistent cross-modal features through the cross-modal attention fusion mechanism built into the preset feature fusion network, and perform weighted fusion of the consistent cross-modal features according to the attention weights to output joint features;

[0148] In this optional embodiment, cross-modal feature alignment processing is performed on heterogeneous representative features to obtain consistent cross-modal features, including:

[0149] Based on the pose and depth data corresponding to the target region, the time stamp alignment process is performed on each feature in the heterogeneous representative features to obtain time-aligned heterogeneous representative features.

[0150] Based on the rotation matrix and translation vector corresponding to the pose data, each feature in the time-aligned heterogeneous representative feature is uniformly mapped to the same spatial reference coordinate system to obtain the spatially aligned heterogeneous representative feature.

[0151] Adaptive scale normalization is performed on spatially aligned heterogeneous representative features to map features of different dimensions and magnitudes to a unified feature scale range, resulting in consistent cross-modal features.

[0152] Before being input into the feature fusion network, heterogeneous features undergo a feature alignment module. Based on the attitude and depth information provided by the inertial measurement unit 12 and the depth pressure sensor 15, the following alignment processing is performed on the RGB image (i.e., color) features, 3D point cloud features, and acoustic image features:

[0153] 1. Time synchronization: Using the high-frequency timestamp of the inertial measurement instrument 12 as a reference, the asynchronous sampling data of the three types of sensors are time-series interpolated and aligned to eliminate the timing deviation caused by the sampling time difference;

[0154] The time synchronization interpolation formula is as follows:

[0155] ;

[0156] In the formula, t' is the aligned target timestamp; t_IMU represents the timestamps of two adjacent frames of raw data; t_IMU represents the reference timestamp output by the inertial measurement unit.

[0157] 2. Coordinate unification: Based on the pose information output by the inertial measurement unit 12 and the depth data of the depth pressure sensor 15, the three-dimensional point cloud features and acoustic image features are uniformly converted to the world coordinate system of the RGB camera to complete spatial registration.

[0158] The coordinate transformation formula is as follows:

[0159] P_cam = R × P_sensor + T;

[0160] In the formula, P_cam represents the coordinates of the feature point transformed to the camera coordinate system; P_sensor represents the coordinates of the feature point in the original sensor coordinate system; R is the rotation matrix provided by the inertial measurement instrument; and T is the translation vector provided by the inertial measurement instrument.

[0161] 3. Scale normalization: Based on the spatial resolution and physical scale differences of the three types of features, an adaptive normalization method is used to map each modal feature to the same feature scale range, reducing the impact of scale differences on subsequent fusion. Through the above processing, the temporal, spatial and scale bias of cross-modal features is effectively reduced, improving the accuracy and robustness of subsequent cross-modal attention fusion.

[0162] The scale normalization formula is as follows:

[0163] x'=(x-x_min) / (x_max-x_min);

[0164] In the formula, x' is the normalized eigenvalue; x is the original eigenvalue; and x_max and x_min are the maximum and minimum values ​​of the feature channel of this modality.

[0165] In this optional embodiment, the attention weights of each modal feature channel in the consistent cross-modal features are adaptively solved through the cross-modal attention fusion mechanism built into the preset feature fusion network, and the consistent cross-modal features are weighted and fused according to the attention weights to output joint features including:

[0166] By using a pre-defined feature fusion network, the modal features in the consistent cross-modal features are mapped and adjusted to a unified feature dimension, resulting in the adjusted consistent cross-modal features.

[0167] Based on the cross-modal attention fusion mechanism built into the preset feature fusion network, global pooling sampling is performed on each modality feature in the adjusted consistent cross-modal features, the channel description information of all modalities is fused, and attention weight vectors corresponding to each modality feature channel are adaptively generated.

[0168] By using a cross-modal attention fusion mechanism, each modal feature is multiplied by its corresponding attention weight vector channel by channel to obtain weighted modal features;

[0169] By using a pre-defined feature fusion network, the weighted modal features are spliced ​​and fused along the channel dimension. The spliced ​​features are then subjected to dimensionality reduction, filtering, and integration to output joint features.

[0170] In this optional embodiment, based on the cross-modal attention fusion mechanism built into the preset feature fusion network, global pooling sampling is performed on each modal feature in the adjusted consistent cross-modal features, fusing the channel description information of all modalities, and adaptively generating the attention weight vector corresponding to each modal feature channel, including:

[0171] Global average pooling is performed on each modal feature in the adjusted consistency cross-modal features to obtain the channel description vector of each modal feature;

[0172] The channel description vector of the current modality feature is concatenated with the channel description vectors of all remaining modality features to form a cross-modal context description vector;

[0173] The cross-modal context description vector is sequentially processed through fully connected dimensionality reduction, nonlinear activation, fully connected dimensionality increase, and normalization to generate the attention weight vector corresponding to the current modal feature channel.

[0174] Iterate through all modalities to obtain the attention weight vector corresponding to the feature channels of each modality.

[0175] It should be explained that the features of each modality are first processed by global average pooling to obtain channel description vectors, then concatenated with the context vectors of other modalities and input into the first fully connected layer for dimensionality reduction. After being processed by the activation function, they are input into the second fully connected layer to restore the channel dimension. Finally, the Sigmoid function outputs weight coefficients between 0 and 1 to complete the adaptive allocation of the importance of each channel.

[0176] The architecture, training process, and parameter settings of the small neural network are as follows:

[0177] Model architecture: A lightweight cross-modal guided channel attention network is adopted, which consists of a global average pooling layer, a first fully connected layer, a ReLU activation layer, a second fully connected layer, and a Sigmoid normalization layer connected in series. When calculating the weight of a certain modality, the channel description vectors of the other two modalities are used as context information, concatenated with the pooling output of the current modality, and then input into the network to achieve cross-modal interactive guidance.

[0178] Training process: This small neural network is jointly trained end-to-end with a pre-defined feature fusion network.

[0179] The preset feature fusion network adopts a cross-modal feature fusion structure, which consists of a dimension-aligned sub-network, a cross-modal attention sub-network, and a feature fusion sub-network. It is jointly trained end-to-end with a small attention neural network and a multi-task output layer.

[0180] The specific architecture of the preset feature fusion network is as follows:

[0181] 1. Dimension Alignment Sub-Network: For the preprocessed and feature-aligned optical texture features, 3D point cloud geometric features, and acoustic image material features, independent 1×1 convolutional or fully connected layers are used to map each modality feature to a unified dimension. The output size is B×C×H×W or B×D, where B is the batch size, C is the number of channels, and D is the feature dimension. In this embodiment, the unified mapping is 512-dimensional features.

[0182] 2. Cross-modal attention sub-network: This is a small attention neural network that performs global average pooling on the 512-dimensional features of each modality to obtain the description vector of each modality channel. The description vector of the current modality is concatenated with the description vectors of the other two modalities to form a cross-modal context vector. The vector is then processed sequentially through fully connected dimensionality reduction, non-linear activation, fully connected dimensionality increase, and Sigmoid normalization to generate the 512-dimensional attention weight vector corresponding to each modality feature channel.

[0183] 3. Feature Fusion Sub-network: Multiply each modal feature with its corresponding attention weight channel by channel to obtain a weighted feature; concatenate the three weighted features along the channel dimension to obtain a 1536-dimensional fused feature; then perform feature integration and dimensionality reduction through two fully connected layers, successively 1536→1024 and 1024→512; batch normalization, ReLU activation function and Dropout regularization are added in between, finally outputting a 512-dimensional joint feature.

[0184] The pre-defined feature fusion network, small attention neural network, and multi-task output layer are jointly trained end-to-end. The training input is a dimension-aligned multimodal feature map, and the training data consists of paired samples of color images, 3D point clouds, and acoustic images from underwater close-up scenes. Sample annotations include target category labels and pixel-level / point cloud-level segmentation masks. The number of samples is no less than 12,000, covering different turbidity, lighting conditions, and material types. A joint loss function is used during training, with the total loss consisting of a weighted average of classification loss, segmentation cross-entropy loss, and Dice loss. All values ​​are set to 1.0. During training, forward propagation sequentially completes dimension alignment, attention weight generation, feature weighted fusion, joint feature output, and classification and segmentation prediction; backpropagation updates all network parameters of the preset feature fusion network, the small attention neural network, and the multi-task output layer simultaneously based on the total loss, achieving end-to-end joint optimization.

[0185] The optimizer uses Adam, with an initial learning rate set to 1×10⁻⁶. -4 The weight decay coefficient is 1×10 -5 The batch size was set to 16, the total training epochs were 120, and the learning rate decayed to 0.5 times its original value every 30 epochs. ReLU was used as the network activation function, and Sigmoid activation was used for the attention output layer. Regularization employed batch normalization and Dropout, with the Dropout dropout rate set to 0.3. Xavier uniform initialization was used for weight initialization. During the inference phase, all network weights were frozen. The input was a dimensionally aligned multimodal feature map, and the forward propagation output joint features were used by the subsequent multi-task output layer to complete object detection and recognition, and precise contour localization.

[0186] The input consists of dimension-aligned feature maps from each modality, supervised by both the final classification and segmentation losses; the weights are calculated using the cross-modal fusion loss function L. fusion =L cls +λ1L ce +λ2L dice Backpropagation updates automatically learn the importance of each channel in cross-modal scenarios.

[0187] Among them, L fusion L is a cross-modal fusion loss function used for the joint optimization of the supervised feature fusion network and attention weights; cls Classification Loss is used for supervision of target category identification; L ce Cross-entropy loss is used for pixel-level / point cloud-level segmentation supervision; L dice λ1 and λ2 are Dice loss, used to alleviate the imbalance between positive and negative samples in the segmentation task; λ1 and λ2 are loss weight coefficients used to balance the losses of different tasks, with a value of 1.0.

[0188] Parameter settings: The output dimension of the first fully connected layer is 1 / 16 of the number of input channels, and it uses Xavier uniform initialization; the activation function is ReLU, and the output layer uses the Sigmoid function; the optimizer is the same as the main network, using the Adam optimizer, with an initial learning rate of... The batch size is 16, the training rounds are 100, and the learning rate decays to 0.5 times its original value every 20 rounds.

[0189] S104. Construct a multi-task output layer for performing target category recognition and pixel-level and / or point cloud-level accurate segmentation, and input joint features into the multi-task output layer to output the detection and recognition results of underwater targets and the accurate positioning information of target contours.

[0190] In this optional embodiment, the multi-task output layer includes a classification branch and a segmentation branch;

[0191] The classification branch is used to perform global pooling and fully connected mapping on the joint features in sequence, and output the target category recognition result.

[0192] The segmentation branch is used to perform layer-by-layer upsampling and convolution to restore spatial resolution of the joint features and output pixel-level segmentation results; or, it is used to map the joint features back to each sampling point and then perform point-by-point category determination processing to output point cloud-level segmentation results.

[0193] The classification branch and the segmentation branch are supervised by a joint loss function during the training phase. The joint loss function includes a classification loss function, a cross-entropy loss function, and a Dice loss function.

[0194] The multi-task output layer includes a fully connected layer for object classification and a U-Net-style decoder network for upsampling and pixel-level segmentation; the segmentation task is supervised by a joint loss function, which is a weighted sum of cross-entropy loss and Dice loss.

[0195] The U-Net-style decoder network employs layer-by-layer upsampling and convolution to restore resolution, and uses skip connections with corresponding layer features from the encoder to preserve edge and detail information. For point cloud-level segmentation, the fused features are back-projected onto the corresponding point set before point-by-point classification is performed. The joint loss function can be written as:

[0196] L total =L cls +λ1L ce +λ2L dice ;

[0197] In the formula, L total L represents the total loss function. cls L represents the classification loss. ce L represents the cross-entropy loss. dice Let λ represent the Dice loss, and λ1 and λ2 represent the weighting coefficients of each loss term.

[0198] The joint loss function is a weighted sum of the classification loss, cross-entropy loss, and Dice loss. The definitions and calculation processes of each component are as follows:

[0199] (1) Classification loss L cls It is used to supervise target category recognition tasks, and cross-entropy loss is used for calculation:

[0200] ;

[0201] In the formula, C represents the total number of categories; yc represents the category label (one-hot encoding); y c This represents the class probabilities predicted by the model.

[0202] (2) Cross-entropy loss Lce is used to supervise pixel-level / point cloud-level segmentation tasks and measures the difference between the predicted probability distribution and the true label distribution:

[0203] ;

[0204] In the formula, N is the total number of pixels / points; yi is the true label (0 or 1) of the i-th pixel / point; and y^i is the predicted probability of the i-th pixel / point.

[0205] (3) Dice loss L dice .

[0206] Used to address the imbalance between positive and negative samples in segmentation tasks, and to measure the overlap between the predicted segmentation region and the real region:

[0207] ;

[0208] In the formula, This is a smoothing term to avoid a denominator of 0; the meanings of the other parameters are the same as above.

[0209] (4) Loss weights λ1 and λ2 are hyperparameters that are adjusted according to the sample distribution of the underwater scene. The range of values ​​is λ1∈[0.5,2] and λ2∈[0.5,2]. In this embodiment, λ1=1.0 and λ2=1.0 are set to balance the training weights of the classification task and the segmentation task.

[0210] Figure 2 An embodiment of an underwater close-range target fusion detection and identification system according to the present invention is shown.

[0211] In this optional embodiment, the underwater near-range target fusion detection and recognition system includes:

[0212] The heterogeneous data acquisition unit 201 is used to acquire multi-source heterogeneous underwater near-field data within the target area;

[0213] The heterogeneous data processing unit 202 is used to preprocess multi-source heterogeneous data and extract heterogeneous representative features;

[0214] The joint feature construction unit 203 is used to perform cross-modal feature alignment processing on heterogeneous representative features to obtain consistent cross-modal features. Through the cross-modal attention fusion mechanism built into the preset feature fusion network, the attention weights of each modal feature channel in the consistent cross-modal features are adaptively solved, and the consistent cross-modal features are weighted and fused according to the attention weights to output the joint features.

[0215] The detection and recognition unit 204 is used to construct a multi-task output layer for performing target category recognition and pixel-level / point cloud-level accurate segmentation, and inputs joint features into the multi-task output layer to output the detection and recognition results of underwater targets and the accurate positioning information of target contours.

[0216] Figure 3-6 An embodiment of an underwater close-range target fusion detection and identification device according to the present invention is shown.

[0217] In this optional embodiment, the underwater near-range target fusion detection and identification device includes:

[0218] Drive mechanism 1, heterogeneous data acquisition module 2, feature processing module 22, drive module 25 and adjustment module 26;

[0219] The drive mechanism 1 includes a floating plate 18, with a battery 21 installed at one end of the bottom of the inner cavity of the floating plate 18 and a feature processing module 22 installed at the other end of the bottom of the inner cavity of the floating plate 18.

[0220] Specifically, when the drive mechanism 1 moves, the battery 21 supplies power to the second motor 27 and the third motor 19 respectively. The second motor 27 drives the third bracket 8 to rotate around the mounting shaft to adjust the propulsion direction. The third motor 19 drives the fan blade 5 to rotate at high speed to generate propulsion force, thereby pushing the floating plate 18 to move underwater to the designated position.

[0221] In this optional embodiment, the feature processing module 22 includes a feature extractor 23 and a feature processor 24. The feature extractor 23 is disposed at the bottom of the other end of the inner cavity of the floating plate 18, and the feature processor 24 is disposed at the top of the other end of the inner cavity of the floating plate 18.

[0222] In this optional embodiment, the drive module 25 includes a second motor 27 disposed at one end of the top of the floating plate 18, the output shaft of the second motor 27 is provided with a third bracket 8, a third motor 19 is disposed on one side of the third bracket 8, and the output shaft of the third motor 19 is provided with a fan blade 5.

[0223] In this optional embodiment, a second auxiliary rod 4 is provided at the other end of the bottom of the third support 8, a second locking plate 6 is provided at the bottom of the second auxiliary rod 4, and the second locking plate 6 is provided at the top of the floating plate 18.

[0224] Specifically, when the drive module 25 drives the heterogeneous data acquisition module 2, the output shaft of the second motor 27 drives the third bracket 8 to deflect, and the third bracket 8 then drives the third motor 19 and the fan blade 5 to change direction synchronously. The second auxiliary rod 4 forms a stable guide in the second card plate 6 to maintain stability when the third bracket 8 rotates, so that the floating plate 18 can turn smoothly, thereby driving the heterogeneous data acquisition module 2 to move in the target direction as a whole.

[0225] In this optional embodiment, the adjustment module 26 includes a first motor 7 disposed at the top center of the floating plate 18, the output shaft of the first motor 7 is provided with an electric telescopic rod 20, the telescopic end of the electric telescopic rod 20 is provided with a first bracket 3, and the front of the first bracket 3 is provided with a fourth motor 14.

[0226] The bottom of the fixed end of the electric telescopic rod 20 is provided with a first card plate 10, and a number of first auxiliary rods 9 are slidably arranged at equal intervals on the bottom of the first card plate 10, and the first auxiliary rods 9 are located on the top of the floating plate 18.

[0227] Specifically, when the adjustment module 26 is working, the first motor 7 drives the electric telescopic rod 20 to extend and retract to drive the first bracket 3 to rise and fall. The first clamping plate 10 and multiple first auxiliary rods 9 form a guide pair to suppress swaying. The fourth motor 14 then drives the second bracket 13 to rotate, thereby realizing the fine adjustment of the height, pitch angle and orientation of the heterogeneous data acquisition module 2. Multiple equidistantly distributed first auxiliary rods 9 can improve the stability and anti-shaking ability during the adjustment process.

[0228] In this optional embodiment, the heterogeneous data acquisition module 2 includes a second bracket 13 disposed on the output shaft of the fourth motor 14, and an inertial measurement instrument 12 is disposed at one end of the top of the second bracket 13;

[0229] An optical camera 16 is installed at the middle of one side of the second bracket 13. High-frequency forward-looking sonars 11 are installed at both ends of one side of the second bracket 13. A laser scanner 17 is installed at one end of the bottom of the second bracket 13. A depth pressure sensor 15 is installed at the bottom of the second bracket 13 and on the other side of the laser scanner 17.

[0230] Specifically, when the heterogeneous data acquisition module 2 acquires data, the fourth motor 14 first drives the second support 13 to rotate towards the target orientation. The optical camera 16 acquires RGB images of the target surface, the high-frequency forward-looking sonar 11 emits and receives acoustic echoes to form an acoustic image, the laser scanner 17 scans the target surface to obtain a three-dimensional point cloud, and the inertial measurement unit 12 and the depth pressure sensor 15 simultaneously output attitude and depth parameters, providing a benchmark for subsequent multi-source data registration and fusion. In a specific application scenario, this method can be used for subsea pipeline inspection, sediment identification, or close-range detection of underwater structural components. After approaching the target, the system simultaneously completes multi-source acquisition, feature alignment, fusion recognition, and contour segmentation to improve the reliability of detection in turbid waters.

[0231] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "setting," "connection," "fixing," "screw connection," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal connection of two components or the interaction between two components. Unless otherwise explicitly limited, those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0232] This invention is not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this invention is limited only by the appended claims.

Claims

1. A method for fusion detection and recognition of underwater near-range targets, characterized in that, The method includes: Acquire multi-source heterogeneous underwater near-field data within the target area; Preprocess multi-source heterogeneous data to extract representative heterogeneous features; Cross-modal feature alignment is performed on heterogeneous representative features to obtain consistent cross-modal features. The attention weights of each modal feature channel in the consistent cross-modal features are adaptively solved by the cross-modal attention fusion mechanism built into the preset feature fusion network. The consistent cross-modal features are then weighted and fused according to the attention weights to output joint features. A multi-task output layer is constructed to perform target category recognition and pixel-level and / or point cloud-level accurate segmentation. Joint features are input into the multi-task output layer to output the detection and recognition results of underwater targets and the accurate positioning information of target contours.

2. The underwater near-range target fusion detection and identification method according to claim 1, characterized in that, The multi-source heterogeneous data includes color images, 3D point cloud data, and acoustic images.

3. The underwater near-range target fusion detection and identification method according to claim 2, characterized in that, The preprocessing of multi-source heterogeneous data to extract representative heterogeneous features includes: Underwater scattering enhancement processing is performed on the color image to obtain an enhanced color image; Statistical outlier removal and voxel downsampling are performed on the 3D point cloud data to remove invalid noise points and reduce the amount of data, resulting in enhanced 3D point cloud data. Acoustic images are subjected to speckle noise suppression and contrast enhancement processing to eliminate inherent noise in acoustic imaging and obtain enhanced acoustic images; Texture color features, geometric shape features, and material reflection intensity features are extracted from enhanced color images, enhanced 3D point cloud data, and enhanced acoustic images, respectively, to form heterogeneous representative features.

4. The underwater close-range target fusion detection and identification method according to claim 3, characterized in that, The underwater scattering enhancement process for the color image to obtain the enhanced color image includes: Based on the color image, global background light estimation is performed to obtain the global background light vector; Based on the global background light vector, the transmittance of the color image is estimated pixel by pixel to generate a transmittance map; Based on underwater optical propagation characteristics, and combining global background light vector and transmittance map, the target radiance is retrieved from the color image to obtain a clear image; The clear image is sequentially subjected to white balance correction and adaptive local contrast enhancement processing to output an enhanced color image.

5. The underwater close-range target fusion detection and identification method according to claim 3, characterized in that, The process of extracting texture color features, geometric shape features, and material reflection intensity features from enhanced color images, enhanced 3D point cloud data, and enhanced acoustic images to form heterogeneous representative features includes: Multi-scale convolution processing is performed on the enhanced color image to obtain feature maps of different resolutions in the shallow, middle and deep layers. After channel alignment and upsampling fusion, the texture color features are output. Extract 3D spatial features from enhanced 3D point cloud data and output geometric shape features; The enhanced acoustic image is convolved to output the material reflection intensity characteristics. By integrating multi-scale texture color features, geometric shape features, and material reflection intensity features, heterogeneous representative features are formed.

6. The underwater near-range target fusion detection and recognition method according to claim 1, characterized in that, The cross-modal feature alignment process performed on heterogeneous representative features to obtain consistent cross-modal features includes: Based on the pose and depth data corresponding to the target region, the time stamp alignment process is performed on each feature in the heterogeneous representative features to obtain time-aligned heterogeneous representative features. Based on the rotation matrix and translation vector corresponding to the pose data, each feature in the time-aligned heterogeneous representative feature is uniformly mapped to the same spatial reference coordinate system to obtain the spatially aligned heterogeneous representative feature. Adaptive scale normalization is performed on spatially aligned heterogeneous representative features to map features of different dimensions and magnitudes to a unified feature scale range, resulting in consistent cross-modal features.

7. The underwater near-range target fusion detection and recognition method according to claim 1, characterized in that, The method utilizes a pre-defined feature fusion network with a built-in cross-modal attention fusion mechanism to adaptively solve for the attention weights of each modal feature channel in the consistent cross-modal features. Based on these attention weights, the consistent cross-modal features are weighted and fused to output joint features, including: By using a pre-defined feature fusion network, the modal features in the consistent cross-modal features are mapped and adjusted to a unified feature dimension, resulting in the adjusted consistent cross-modal features. Based on the cross-modal attention fusion mechanism built into the preset feature fusion network, global pooling sampling is performed on each modality feature in the adjusted consistent cross-modal features, the channel description information of all modalities is fused, and attention weight vectors corresponding to each modality feature channel are adaptively generated. By using a cross-modal attention fusion mechanism, each modal feature is multiplied by its corresponding attention weight vector channel by channel to obtain weighted modal features; By using a pre-defined feature fusion network, the weighted modal features are spliced ​​and fused along the channel dimension. The spliced ​​features are then subjected to dimensionality reduction, filtering, and integration processing to output joint features.

8. The underwater close-range target fusion detection and identification method according to claim 7, characterized in that, The step of using the cross-modal attention fusion mechanism built into the preset feature fusion network to perform global pooling sampling on each modality feature in the adjusted consistent cross-modal features, fusing the channel description information of all modalities, and adaptively generating the attention weight vector corresponding to each modality feature channel includes: Global average pooling is performed on each modal feature in the adjusted consistency cross-modal features to obtain the channel description vector of each modal feature; The channel description vector of the current modality feature is concatenated with the channel description vectors of all remaining modality features to form a cross-modal context description vector; The cross-modal context description vector is sequentially processed through fully connected dimensionality reduction, nonlinear activation, fully connected dimensionality increase, and normalization to generate the attention weight vector corresponding to the current modal feature channel. Iterate through all modalities to obtain the attention weight vector corresponding to the feature channels of each modality.

9. The underwater near-range target fusion detection and identification method according to claim 1, characterized in that, The multi-task output layer includes classification branches and segmentation branches; The classification branch is used to perform global pooling and fully connected mapping on the joint features in sequence, and output the target category recognition result. The segmentation branch is used to perform layer-by-layer upsampling and convolution to restore spatial resolution of the joint features and output pixel-level segmentation results; or, it is used to map the joint features back to each sampling point and then perform point-by-point category determination processing to output point cloud-level segmentation results. The classification branch and the segmentation branch are supervised by a joint loss function during the training phase. The joint loss function includes a classification loss function, a cross-entropy loss function, and a Dice loss function.

10. An underwater near-range target fusion detection and recognition system, characterized in that, The system includes: Heterogeneous data acquisition unit, used to acquire multi-source heterogeneous underwater near-field data within the target area; The heterogeneous data processing unit is used to preprocess multi-source heterogeneous data and extract heterogeneous representative features; The joint feature construction unit is used to perform cross-modal feature alignment processing on heterogeneous representative features to obtain consistent cross-modal features. Through the cross-modal attention fusion mechanism built into the preset feature fusion network, the attention weights of each modal feature channel in the consistent cross-modal features are adaptively solved, and the consistent cross-modal features are weighted and fused according to the attention weights to output the joint features. The detection and recognition unit is used to construct a multi-task output layer for performing target category recognition and pixel-level / point cloud-level accurate segmentation. It inputs joint features into the multi-task output layer and outputs the detection and recognition results of underwater targets and the accurate positioning information of the target contour.

11. An underwater close-range target fusion detection and identification device, used to implement the underwater close-range target fusion detection and identification method according to any one of claims 1-9, characterized in that, The device includes: Drive mechanism (1), heterogeneous data acquisition module (2), feature processing module (22), drive module (25) and adjustment module (26); The driving mechanism (1) includes a floating plate (18), a battery (21) is provided at one end of the bottom of the inner cavity of the floating plate (18), and a feature processing module (22) is provided at the other end of the bottom of the inner cavity of the floating plate (18).

12. The underwater near-range target fusion detection and identification device according to claim 11, characterized in that, The feature processing module (22) includes a feature extractor (23) and a feature processor (24). The feature extractor (23) is located at the bottom of the other end of the inner cavity of the floating plate (18), and the feature processor (24) is located at the top of the other end of the inner cavity of the floating plate (18).

13. The underwater near-range target fusion detection and identification device according to claim 11, characterized in that, The drive module (25) includes a second motor (27) disposed at one end of the top of the floating plate (18). The output shaft of the second motor (27) is provided with a third bracket (8). A third motor (19) is disposed on one side of the third bracket (8). The output shaft of the third motor (19) is provided with a fan blade (5).

14. The underwater near-range target fusion detection and identification device according to claim 13, characterized in that, The other end of the bottom of the third support (8) is provided with a second auxiliary rod (4), the bottom of the second auxiliary rod (4) is provided with a second card plate (6), and the second card plate (6) is provided on the top of the floating plate (18).

15. The underwater near-range target fusion detection and identification device according to claim 11, characterized in that, The adjustment module (26) includes a first motor (7) located at the top center of the floating plate (18). The output shaft of the first motor (7) is provided with an electric telescopic rod (20). The telescopic end of the electric telescopic rod (20) is provided with a first bracket (3). The front of the first bracket (3) is provided with a fourth motor (14). The bottom of the fixed end of the electric telescopic rod (20) is provided with a first card plate (10), and a number of first auxiliary rods (9) are slidably arranged at equal intervals on the bottom of the first card plate (10), and the first auxiliary rods (9) are located on the top of the floating plate (18).

16. The underwater near-range target fusion detection and identification device according to claim 15, characterized in that, The heterogeneous data acquisition module (2) includes a second bracket (13) mounted on the output shaft of the fourth motor (14), and an inertial measurement instrument (12) is mounted on one end of the top of the second bracket (13). An optical camera (16) is provided at the middle of one side of the second bracket (13), and high-frequency forward-looking sonar (11) is provided at both ends of one side of the second bracket (13). A laser scanner (17) is provided at one end of the bottom of the second bracket (13), and a depth pressure sensor (15) is provided at the bottom of the second bracket (13) and on the other side of the laser scanner (17).

Images