Marine ranch shellfish high-precision fishing method and equipment based on multi-source information and visual enabling

Abstract

The invention discloses a marine ranch shellfish high-precision fishing method and equipment based on multi-source information and visual enabling, and belongs to the technical field of crossing of computer vision, an underwater robot technology and intelligent marine equipment. According to the method, multi-modal input is constructed by fusing optical, sonar and vibration information, and sand surface distortion features are extracted by using deformable convolution and a weighted bidirectional feature pyramid. Further, the relevant local texture of the shellfish is actively enhanced through a deformable local texture enhancement module, and the regional perception robustness is enhanced in combination with graph structure reasoning. And finally, time sequence fusion and probability generation modeling are used, and a high-precision probabilistic positioning result with uncertainty measurement is output. According to the method, the problems of poor underwater visual quality, weak target features and unreliable positioning results are effectively solved, the accuracy, robustness and decision intelligent level of identifying and positioning the buried shellfish by the underwater robot are remarkably improved, and the method is suitable for automatic and accurate catching operation of a marine ranch.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision, underwater robot technology and intelligent marine equipment. Based on deep fusion of multimodal sensor information, deformable local texture enhancement and graph structure reasoning, as well as spatiotemporal context fusion and probabilistic generation modeling, it performs visual recognition and high-precision positioning and harvesting of shellfish. Specifically, it involves a method and equipment for high-precision harvesting of shellfish in marine ranches based on multi-source information and vision empowerment. Background Technology

[0002] With the development of underwater robot technology (such as ROV and AUV), using robots to achieve targeted, precise, and automated harvesting has become a research hotspot in the industry. However, underwater vision systems remain a key bottleneck in achieving automated harvesting. The underwater environment has its own unique characteristics: First, light undergoes severe attenuation and scattering when propagating in water, resulting in low image contrast, color distortion, and blurred details; second, the sandy seabed is easily affected by water flow, biological activity, and robot disturbance, which can generate suspended particles and create visual noise; third, target shellfish (such as razor clams) are often buried under the sand layer and can only be characterized by indirect features such as micro-indentations, bulges, or local texture distortions on the sand surface after lying on the sand. These features have extremely low distinguishability from the surrounding sandy background, making visual detection very difficult.

[0003] Most existing underwater identification and localization methods rely on a single visual modality, lacking effective fusion of multi-source information such as sonar and vibration. This leads to large localization errors and insufficient robustness in complex seabed topography. Furthermore, traditional convolutional neural networks and attention mechanisms lack the proactive and targeted perception of localized structured texture distortions caused by shellfish lying on the sand (such as concave edges in specific directions or radial cracks). In terms of localization output, mainstream methods typically employ deterministic coordinate regression, which cannot quantitatively assess the localization uncertainty caused by occlusion, overlap, and feature ambiguity, making it difficult to meet the reliability requirements of high-risk, high-precision underwater operations.

[0004] Therefore, there is an urgent need for a visual recognition and positioning method that can effectively overcome the bottlenecks of underwater vision technology, achieve deep fusion of multi-source information, possess active local feature perception capabilities, and output probabilistic, interpretable positioning results and their uncertainties, so as to support the accurate, autonomous, and reliable operation of next-generation underwater robot intelligent harvesting equipment. Summary of the Invention

[0005] The purpose of this invention is to provide a high-precision harvesting method for shellfish in marine ranches based on multi-source information and visual empowerment, in order to achieve rapid and accurate identification and positioning of shellfish targets in marine ranches through multimodal feature fusion and visual empowerment.

[0006] Another object of the present invention is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the above-described method.

[0007] To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0008] A high-precision harvesting method for shellfish in marine ranching based on multi-source information and vision-enabled methods includes the following steps:

[0009] S1. Information Collection and Deep Fusion

[0010] Optical images, sonar density maps, and vibration response maps of the target area are acquired simultaneously. After spatial registration and scale normalization, they are stitched together along the channel dimension to form an initial fusion tensor. An adaptive weighting of each channel of the initial fusion tensor is performed through a channel attention module to generate weighted multimodal input features.

[0011] S2. Adaptive Multi-Scale Feature Extraction

[0012] Using the weighted multimodal input features as input, multi-scale features are extracted using a deformable convolution-based backbone network; the multi-scale features are then weighted and bidirectionally fused through a weighted bidirectional feature pyramid network to obtain enhanced multi-scale fused features.

[0013] S3. Area Awareness

[0014] Feature maps are selected from the enhanced multi-scale fusion features and input into the deformable local texture enhancement module. This module dynamically generates deformable convolution kernels related to the potential shellfish texture features, performs adaptive convolution on the input features to enhance the local texture response, and outputs texture enhancement features. Subsequently, the texture enhancement features are superpixel segmented to construct a graph structure, and spatial context reasoning is performed using a graph neural network to obtain region-aware features with stronger spatial coherence.

[0015] S4. Positioning and Uncertainty Estimation

[0016] The region-aware features of the current frame are temporally fused with the motion-compensated features of historical frames to obtain spatiotemporal fusion features. Based on the spatiotemporal fusion features, the posterior probability distribution of the target position is modeled using a conditional variational autoencoder and a hybrid density network framework, and a probabilistic localization result containing the target coordinates and its uncertainty measure is output.

[0017] As a limitation, in step S1, the specific implementation of the channel attention module includes: performing global average pooling on the initial fusion tensor to obtain the global statistical description vector of each channel; inputting the description vector into a two-layer fully connected network with a bottleneck structure to learn the importance weights of each channel; normalizing the importance weights to the (0,1) interval using the Sigmoid function; and weighting the corresponding channels of the initial fusion tensor with the normalized weights to obtain the weighted multimodal input features.

[0018] As a second limitation, in step S2, the weighted bidirectional feature pyramid network introduces learnable weight parameters for each input feature map during feature fusion, and performs feature fusion through a normalized weighted sum, expressed by the formula:

[0019] ,

[0020] in, The input feature maps come from paths with different resolutions. = 0.0001 is used to prevent numerical instability. Here, Conv represents the convolution operation, where Conv is the corresponding learnable weight.

[0021] As a third limitation, in step S3, the operation of the deformable local texture enhancement module includes:

[0022] S31. Construct a learnable prototype convolutional kernel library, where each kernel represents a potential local structure detector;

[0023] S32. Using a light quantum network, predict a set of kernel offsets and kernel weights for each spatial location and each prototype kernel based on the local context of the input feature map;

[0024] S33. The prototype kernel is deformed using the predicted kernel offset, and the deformed kernel is weighted and combined using kernel weights. An adaptive convolution operation is then performed on the input feature map to generate the texture enhancement feature.

[0025] As a fourth limitation, in step S3, the inference process of the graph neural network is as follows: superpixel regions are used as graph nodes, the node features are the mean values ​​of features within the region, and edge connections are established based on spatial proximity and feature similarity; message passing is performed through graph convolution or graph attention mechanism to update node features; finally, the updated node features are mapped back to the original spatial dimension to generate the region-aware features.

[0026] As a fifth limitation, in step S4, the conditional variational autoencoder and hybrid density network framework includes a prior network, a posterior network, and a decoder.

[0027] The prior network takes the spatiotemporal fusion features as input and outputs the prior distribution parameters of the latent variables;

[0028] The posterior network takes the spatiotemporal fusion features and the real location labels as input during training and outputs the posterior distribution parameters of the latent variables.

[0029] The decoder samples from the distribution of the latent variables and, in conjunction with the spatiotemporal fusion features, regresses the parameters of the Gaussian mixture model, including the mixing weights, mean vector, and covariance matrix of each component.

[0030] As a further limitation, in step S4, the uncertainty measure is characterized by the covariance matrix of the component with the largest weight in the Gaussian mixture model; the final predicted target position is the mean vector of that component.

[0031] As another limitation on step S4, the loss function used for model training in step S4 is:

[0032]

[0033] in, As a feature of spatiotemporal fusion, For the actual location, As latent variables, For posterior networks, For a priori network, For decoder, Let KL divergence be the KL divergence. To balance the reconstruction loss with the hyperparameters of the regularization term.

[0034] As a sixth limitation, in step S4, the specific process of spatiotemporal fusion is as follows: using pose transformation information from motion sensors or visual odometry, the optimized feature map from the previous moment is warped and aligned to the coordinate system of the current frame; the aligned historical features are channel-separated with the region perception features of the current frame, and then fused through a convolutional layer to generate the spatiotemporal fusion features.

[0035] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the above-described high-precision shellfish harvesting method for marine ranching based on multi-source information and vision empowerment.

[0036] The present invention, by adopting the above-described technical solution, achieves the following technical advancements compared to existing technologies:

[0037] (1) This invention significantly improves the robustness and environmental adaptability of the sensing system. This invention constructs complementary and complete feature representations by deeply fusing multimodal information from optics, sonar, and vibration sensing, and by using a channel attention mechanism for adaptive weighting. This method effectively overcomes the inherent defects of single visual modalities being susceptible to light attenuation, water turbidity, and interference from suspended particles underwater, ensuring the stable and reliable operation of the sensing system even under harsh visual conditions.

[0038] (2) This invention achieves proactive and accurate perception of weak and irregular local features. The deformable local texture enhancement module proposed in this invention can dynamically generate adaptive convolution kernels that match the target features, actively "exploring" and enhancing key texture patterns such as micro-indentations, edges, and cracks on the sand surface formed by shellfish lying on the sand. By combining graph structure reasoning to model and optimize spatial context relationships, the system's ability to perceive regions of low-contrast and low-discrimination targets is greatly improved, and the false alarm rate is significantly reduced.

[0039] (3) This invention provides high-precision and interpretable probabilistic positioning output, significantly enhancing the system's decision-making intelligence and operational reliability. This invention utilizes historical information through a temporal fusion mechanism to ensure the smoothness of the positioning trajectory and its robustness against instantaneous disturbances. The core innovation lies in employing a conditional variational autoencoder and a hybrid density network framework to transform the positioning problem from deterministic regression to probabilistic generative modeling. This system not only outputs sub-pixel-level precise coordinates, but more importantly, it can simultaneously output quantified positioning uncertainty estimates (such as covariance ellipses), thus providing a crucial reliability criterion for high-risk underwater precision fishing operations. This enables the robot to make intelligent decisions such as requesting re-detection or human intervention based on the confidence level.

[0040] (4) The present invention has been fully experimentally verified, and the method provided is significantly superior to the existing mainstream methods in key indicators such as detection accuracy (mAP), positioning error and uncertainty calibration. The entire technical solution is well-designed, with rigorous logic from multi-source information preprocessing to probabilistic positioning output, demonstrating excellent practicality and generalization ability, and providing core and reliable technical support for the research and development of intelligent and refined harvesting equipment for marine ranches.

[0041] This invention belongs to the interdisciplinary field of computer vision, underwater robot technology and intelligent marine equipment, and can achieve rapid and accurate identification and positioning of shellfish targets in marine ranches. Attached Figure Description

[0042] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0043] In the attached diagram:

[0044] Figure 1 This is a sample diagram of the dataset in an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram illustrating the working principle of the multimodal input data and DLTEM module in an embodiment of the present invention.

[0046] Figure 3 This is a schematic diagram of the graph structure reasoning (GSR) process according to an embodiment of the present invention;

[0047] Figure 4 This is a visualization of the output of the probabilistic localization network in a complex scene according to an embodiment of the present invention. Detailed Implementation

[0048] The preferred embodiments of the present invention will now be described with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit the scope of the invention.

[0049] Example

[0050] This embodiment provides a complete implementation process for a high-precision shellfish harvesting method for marine ranching based on multi-source information and vision-enabled methods. This embodiment aims to verify the effectiveness of the technical solution of this invention on realistically simulated and collected underwater datasets, demonstrating its entire process from multi-source information input to probabilistic positioning result output.

[0051] I. Overall System Implementation Framework

[0052] The implementation system of this embodiment consists of two parts: a hardware data acquisition platform and a software processing algorithm. The hardware platform comprises an underwater robot (ROV) equipped with a high-definition optical camera, a high-frequency side-scan sonar, a micro vibration sensor, an underwater thruster, a platform motion sensor (DVL), an inertial measurement unit (IMU), and a main control computer. The software algorithm is developed based on the PyTorch deep learning framework and deployed on the main control computer to process sensor data in real time.

[0053] II. Data Preparation and Preprocessing

[0054] To train and test the algorithm model of this embodiment, a dedicated underwater shellfish visual dataset was constructed.

[0055] Synthetic Data: Using Blender 3D modeling software, we meticulously simulated different lighting angles and intensities, varying water turbidity levels, diverse sandbed textures, and the varying depths, sizes, and shapes of depressions and ridges formed by shellfish lying on the sand. A total of 150 high-resolution synthetic images (1280×720) were generated, each providing precise target bounding boxes, center point coordinates, and pixel-level segmentation mask labels. The synthetic data was primarily used for model pre-training and data augmentation. The diverse sandbed textures included smooth and wavy surfaces. The shellfish razor clam was used as the model.

[0056] Real-world data: In collaboration with a marine ranch, the aforementioned hardware platform was used to collect data in a sandy aquaculture area at a water depth of approximately 4 meters. By controlling the ROV to cruise at different heights, angles, and speeds, approximately 150 real underwater scene images were collected, for example... Figure 1 As shown, 120 images were finely annotated, and the remaining 30 were used for testing. Annotation information included target category, bounding box, and precise center point coordinates. All real images were simultaneously recorded with corresponding sonar echo intensity data and vibration sensor response data. The sonar echo intensity data was post-processed to generate a sand layer density intensity map S, and the vibration sensor response data was processed to generate a vibration response map V.

[0057] III. Specific Implementation of the Algorithm Module

[0058] The method provided in this embodiment mainly includes steps S1 to S4. Each step is described in detail below.

[0059] S1. Information Acquisition and Deep Fusion. This step involves acquiring information from multiple sources to address the issues of incompleteness and susceptibility to interference from single visual information sources.

[0060] The system simultaneously acquires optical images of the target area from a high-definition optical camera. Sonar density map from high-frequency side-scan sonar. Vibration response graph from miniature vibration sensor First, spatial registration and scale normalization are performed on the three to achieve a uniform size. The initial fusion tensor is formed by splicing along the channel dimension as follows:

[0061] (1)

[0062] in, The operation copies the single-channel S and V three times along the channel dimension to align with the number of channels in the RGB three-channel image I, facilitating subsequent convolution processing.

[0063] Sonar density maps, also known as sand density intensity maps, reflect the density differences within the sand layer's internal structure. Areas where shellfish are present typically exhibit density anomalies. Following vibration, shellfish activity may cause micro-deformation of the sand surface, and vibration response maps quantify this deformation response.

[0064] To further improve the fusion effect and suppress the influence of irrelevant or noisy channels, this invention introduces a lightweight channel attention module. The channel attention module adaptively weights each channel of the initial fusion tensor to generate weighted multimodal input features.

[0065] In this step, the specific implementation of the channel attention module includes: processing the initial fusion tensor. Perform global average pooling (GAP) to obtain global statistical information for each channel, i.e., the global statistical description vector for each channel.

[0066] (2)

[0067] Next, a two-layer fully connected network with a bottleneck structure is used to learn the nonlinear interactions between channels, and the weights of each channel are output:

[0068] (3)

[0069] In the formula, It is a global description vector. and The weight matrix is ​​a learnable matrix. This is the reduction ratio, used to reduce model complexity. Represents the ReLU activation function. This represents the Sigmoid function, which normalizes the weights to the range (0,1).

[0070] The corresponding channels of the initial fusion tensor are weighted using normalized weights to obtain the weighted multimodal input features:

[0071] (4).

[0072] S2. Adaptive Multi-Scale Feature Extraction. This step involves adaptive multi-scale feature extraction based on deformable convolution and a weighted bidirectional feature pyramid.

[0073] To effectively capture geometric distortion features such as minute depressions and bulges on the sand surface, this step designs an enhanced feature extraction backbone network. This network uses weighted multimodal input features as input and extracts multi-scale features using a deformable convolution-based backbone network. Then, a weighted bidirectional feature pyramid network is used to perform weighted bidirectional fusion of the multi-scale features, resulting in enhanced multi-scale fused features.

[0074] Specifically, in this step, the pre-processed... For input, the backbone uses the CSPDarknet53 architecture, but replaces its standard convolutional part with deformable convolutions. Deformable convolutions, by introducing learned spatial offsets, enable the convolutional kernel to adaptively focus on irregular regions, making them more suitable for extracting local deformation features caused by shellfish.

[0075] Let the sampling grid of the standard convolution kernel at position p be... (For 3x3 convolution). Deformable convolution, on the other hand, is for each sampling point. Learn an offset The actual sampling location is Feature map of layer l The deformable convolution operation on can be represented as:

[0076] (5)

[0077] in, These are convolution weights. From an additional convolutional layer The offset field learned and related to position 𝑝.

[0078] To further integrate feature information at different scales, this embodiment constructs a weighted bidirectional feature pyramid network (Weighted BiFPN). Let the multi-scale features extracted by the backbone be... (Resolution decreasing). BiFPN performs multiple fusions via top-down and bottom-up paths. Unlike classic FPN, this embodiment introduces learnable weights for each input feature during fusion. This adaptively measures its contribution. Taking a specific fusion node as an example, its output... It can be represented as:

[0079] , 0 (6)

[0080] in, The input feature maps come from paths with different resolutions. = 0.0001 is used to prevent numerical instability. For its corresponding learnable weights, This represents a convolution operation. Through this weighted fusion, the network can more effectively integrate multi-level information, from high-resolution details to low-resolution semantics.

[0081] S3. Region Perception: This step involves region perception of mollusks based on deformable local texture enhancement and graph structure inference. Traditional attention mechanisms, such as SE and CBAM, can improve feature response, but they are essentially content-independent global or local weightings, lacking the ability to specifically model the local structured texture distortions formed by mollusks lying in sand, such as concave edges in specific directions and radial cracks. To address this issue, this embodiment proposes a perception method combining a Deformable Local Texture Enhancement Module (DLTEM) with a graph structure inference mechanism. Feature maps are selected from the enhanced multi-scale fusion features and input into the Deformable Local Texture Enhancement Module. This module dynamically generates deformable convolution kernels related to potential mollusk texture features, adaptively convolves the input features to enhance the local texture response, and outputs texture-enhanced features. Subsequently, the texture-enhanced features are superpixel segmented to construct a graph structure, and spatial context inference is performed using a graph neural network to obtain region-perception features with stronger spatial coherence.

[0082] Specifically, the core idea of ​​the deformable local texture enhancement module is to actively generate a series of deformable convolutional kernels related to potential shellfish features, such as edges, ridges, and depressions, and apply them to the input feature map to enhance the response associated with these structures.

[0083] First, define a learnable prototype convolutional kernel library. Each of them This represents a potential local structure detector, such as an edge detector in different orientations.

[0084] For the input feature map For example, a lightweight subnetwork derived from the output of a layer in a BiFPN. according to Local context information for each spatial location and each prototype core Predict a set of kernel offsets and kernel weight .

[0085] (7)

[0086] in, Defined for the kernel Spatial offset at each sampling point This refers to the importance weight of the adaptive kernel at its current position. Lightweight subnetwork. It consists of two 1*1 convolutional layers.

[0087] Then, in position At this point, the offset and weighted prototype kernel is applied for convolution:

[0088] (8)

[0089] in, yes The sampling grid of the convolution. This operation allows the network to dynamically adjust its receptive field, actively "exploring" and enhancing local texture patterns associated with mollusks, rather than passively relying on feedforward features.

[0090] To further utilize the spatial relationships between different regions in the enhanced features, this embodiment introduces graph-based reasoning. Simple algorithms such as SLIC are used to transform the feature map... Divided into N superpixel regions Construct a graph using each superpixel region as a node. Each node Features It is the mean of the features within its corresponding region.

[0091] Edges E between nodes are established based on spatial proximity and feature similarity. Then, a graph convolutional network (GCN) or a graph attention network (GAT) is applied for message passing to update node features.

[0092] (9)

[0093] in, It is a node Neighbors It is a normalization constant. It is a learnable weight matrix. Through graph reasoning, the system can smooth noise and solidify the true target region based on context (e.g., "a certain region is a concave edge, and its neighboring regions are likely also part of a mollusk"). Finally, the updated node features are mapped back to the original spatial dimension, resulting in an enhanced feature map with stronger spatial coherence. .

[0094] S4. Localization and Uncertainty Estimation. This step aims to overcome the limitations of traditional deterministic localization and achieve high-precision probabilistic localization with temporal smoothness and uncertainty quantification capabilities. In this step, the region-aware features of the current frame are temporally fused with the motion-compensated features of historical frames to obtain spatiotemporal fusion features. Using these spatiotemporal fusion features as conditions, a posterior probability distribution of the target location is modeled based on a conditional variational autoencoder and a hybrid density network framework, outputting a probabilistic localization result containing the target coordinates and their uncertainty measure. The conditional variational autoencoder and hybrid density network framework includes a prior network, a posterior network, and a decoder. The prior network takes the spatiotemporal fusion features as input and outputs the prior distribution parameters of the latent variables. The posterior network, during training, takes the spatiotemporal fusion features and the real location labels as input and outputs the posterior distribution parameters of the latent variables. The decoder samples from the latent variable distribution and, combined with the spatiotemporal fusion features, regresses the parameters of the Gaussian mixture model, including the mixing weights, mean vector, and covariance matrix of each component. The detailed process of this step is further described below.

[0095] S41. Spatiotemporal context fusion:

[0096] Considering that the acquisition platform is in continuous motion, this embodiment uses timing information to assist in the positioning of the current frame, thereby improving the system's stability and robustness to momentary occlusion.

[0097] Feature Alignment: Let the feature map after graph inference at time t be... Pose transformation estimated using platform motion sensors such as DVL, IMU, or visual odometry. The optimized features from the previous time step Spatial warping is performed using bilinear interpolation, aligned with the current frame:

[0098] (10)

[0099] Temporal fusion: fusing aligned historical features with current frame features.

[0100] (11)

[0101] This positioning system thus possesses short-term memory, enabling it to use historical information to suppress transient noise and make more robust predictions of targets that are temporarily occluded or have blurred features.

[0102] S42. Probabilistic Positioning and Uncertainty Estimation:

[0103] This network uses a Conditional Variational Autoencoder (CVAE) framework to model the posterior distribution of the target location and regresses a Hybrid Density Network (MDN) to output probabilistic localization results.

[0104] Model framework:

[0105] Prior network Based on spatiotemporal fusion characteristics Given conditions, learn the prior distribution of the latent variable z. z encodes implicit factors such as the target's appearance, shape, and its contextual relationship with the environment.

[0106] Posterior network During training, it receives features z and the true location z, approximating the posterior distribution of the latent variable z. .

[0107] decoder Sample 𝑧 from the prior or posterior distribution, and together with the feature 𝐹, decode the probability location distribution of the target. Here, the parameters of a Gaussian Mixture Model (GMM) are regressed:

[0108]

[0109] Among them, 𝜋 𝑘 , 𝑘 Σ and Σ are both predicted by the decoder network and are functions of Σ and Σ. 𝑘 It is a mixed weight, 𝜇 𝑘 ∈𝑅 2 The coordinates of the center of the Σk Gaussian component are Σk∈k. 2×2 It is its covariance matrix, which naturally characterizes the uncertainty of the location (the shape, size, and orientation of the ellipse).

[0110] Loss function: Loss function in the form of β-VAE:

[0111]

[0112] The first term is the reconstruction loss, which encourages the decoder to reconstruct the true location from the latent code . The second term is the KL divergence, which constrains the posterior distribution . 𝜙 The regularization approximates the prior distribution ẋ, providing a reasonable prior for testing. ẋ is a hyperparameter that controls the strength of the regularization.

[0113] Reasoning process:

[0114] 1. Input the current frame feature 𝐹 into the prior network to obtain... ;

[0115] 2. From Extract L latent codes from ;

[0116] 3. Place each By inputting the decoder with the input, we obtain L sets of GMM parameters. .

[0117] 4. Aggregate these GMMs. For example, calculate the average GMM parameters:

[0118] 5. The final target position is the mean of the Gaussian component with the largest mixing weight, i.e. Simultaneously, the system outputs the covariance matrix at that location. As a measure of uncertainty.

[0119] To comprehensively evaluate the performance of the method in this embodiment, an underwater shellfish visual detection dataset containing both synthetic and real-world collected data was constructed. The following is combined with... Figures 1 to 4 Detailed explanation.

[0120] Network training details:

[0121] Hardware and software: Training was performed on a single NVIDIA RTX 3080 GPU using the PyTorch deep learning framework.

[0122] Input preprocessing: Images were uniformly scaled to 640x640 pixels and normalized using standard methods. Data augmentation techniques such as random horizontal flipping, color dithering, Gaussian noise injection, and motion blur were applied to improve model robustness.

[0123] Optimizer and Hyperparameters: The AdamW optimizer was used, with an initial learning rate of 1e-3 and cosine annealing scheduling applied. Weight decay was 1e-4. Batch size was set to 16. Total training epochs were 300.

[0124] Loss function: The total loss is the weighted sum of the losses from multiple tasks.

[0125] For the detection part (up to step S3), classification loss with Focal Loss and GIoU loss are used.

[0126] For the probabilistic localization network (step S4), use , where 𝛽 is set to 0.1.

[0127] Total loss: ,in =0.5.

[0128] Experimental Results and Analysis:

[0129] The method proposed in this embodiment (denoted as Ours (Full)) was compared with current mainstream object detection methods, including YOLOv5s, YOLOv5m, Faster R-CNN (ResNet-50-FPN), and several ablation experiment versions of this invention (Ours w / o DLTEM&GSR: step S3 removed, using ordinary convolution and attention; Ours w / o Prob: step S4 removed, using traditional heatmap regression; Ours w / o Temp: temporal fusion removed) on the same test set. Evaluation metrics included mean accuracy (mAP@0.5), mean mean accuracy (mAP@0.5:0.95), location error (LE), and uncertainty calibration (evaluated using negative log-likelihood NLL, with smaller values ​​indicating more accurate probability prediction). All comparison methods were retrained and evaluated on the same training and test sets using the same image preprocessing and data augmentation strategies to ensure fairness in the comparison.

[0130] Table 1. Evaluation Indicators for Other Advanced Methods and Ablation Tests

[0131]

[0132] Figure 2 This is a schematic diagram illustrating the working principle of the multimodal input data and the DLTEM module in this embodiment. Figure 3 This is a schematic diagram of the graph structure reasoning (GSR) process in this embodiment, including superpixel segmentation, graph construction, graph convolution, and feature reconstruction.

[0133] The results show that:

[0134] The complete model in this embodiment (Ours (Full)) achieves optimality across all quantitative metrics.

[0135] Ablation experiments demonstrated the effectiveness of each innovative module: DLTEM and GSR (step 3) significantly improved detection accuracy (mAP) and reduced localization error; the probabilistic model (step 4) further reduced localization error while providing good uncertainty estimation (lowest NLL); and temporal fusion (step 4) also contributed to stability and accuracy.

[0136] Practicality of Uncertainty Estimation: In the test set, the top 10% of samples with the highest positioning uncertainty (trace of the covariance matrix) were selected. These samples had an average positioning error of 3.8px, significantly higher than the overall average error of 2.0px. This demonstrates that the positive correlation between uncertainty measurement and positioning error has been experimentally verified, proving its effectiveness as a reliability indicator. This proves that the uncertainty measurement provided by this invention can effectively identify unreliable positioning samples, providing a crucial basis for system decision-making.

[0137] Visual analysis is attached. Figure 4 As shown, in complex scenarios (such as target proximity, partial occlusion, and sand ripple interference), baseline methods are prone to missed detections, false detections, or positioning drift. The method of this invention, however, can more accurately and stably identify and locate targets, and can visually display the range of positioning uncertainty in elliptical form. In time-series sequences, the position trajectory output by the method of this embodiment is smooth and robust to transient occlusion.

[0138] This embodiment possesses excellent detection performance, robustness, positioning capability, and uncertainty quantification capability, providing a solid, reliable, and intelligent technical foundation for the engineering application of intelligent harvesting of shellfish in marine ranches.

Claims

1. A high-precision harvesting method for shellfish in marine ranching based on multi-source information and vision-enabled methods, characterized in that, Includes the following steps: S1. Information Collection and Deep Fusion Optical images, sonar density maps, and vibration response maps of the target area are acquired simultaneously. After spatial registration and scale normalization, they are stitched together along the channel dimension to form an initial fusion tensor. An adaptive weighting of each channel of the initial fusion tensor is performed through a channel attention module to generate weighted multimodal input features. S2. Adaptive Multi-Scale Feature Extraction Using the weighted multimodal input features as input, multi-scale features are extracted using a deformable convolution-based backbone network; The multi-scale features are weighted and bidirectionally fused using a weighted bidirectional feature pyramid network to obtain enhanced multi-scale fused features. S3. Area Awareness Feature maps are selected from the enhanced multi-scale fusion features and input into the deformable local texture enhancement module. This module dynamically generates deformable convolution kernels related to the potential shellfish texture features, performs adaptive convolution on the input features to enhance the local texture response, and outputs texture enhancement features. Subsequently, the texture enhancement features are segmented into superpixels to construct a graph structure, and a graph neural network is used for spatial context reasoning to obtain region-aware features with stronger spatial coherence. S4. Positioning and Uncertainty Estimation The region-aware features of the current frame are temporally fused with the motion-compensated features of historical frames to obtain spatiotemporal fusion features. Based on the spatiotemporal fusion features, the posterior probability distribution of the target position is modeled using a conditional variational autoencoder and a hybrid density network framework, and a probabilistic localization result containing the target coordinates and its uncertainty measure is output.

2. The high-precision harvesting method for marine ranching shellfish based on multi-source information and vision empowerment according to claim 1, characterized in that, In step S1, the specific implementation of the channel attention module includes: performing global average pooling on the initial fusion tensor to obtain the global statistical description vector of each channel; inputting the description vector into a two-layer fully connected network with a bottleneck structure to learn the importance weights of each channel; normalizing the importance weights to the (0,1) interval using the Sigmoid function; and weighting the corresponding channels of the initial fusion tensor with the normalized weights to obtain the weighted multimodal input features.

3. The high-precision harvesting method for marine ranching shellfish based on multi-source information and vision empowerment according to claim 1 or 2, characterized in that, In step S2, the weighted bidirectional feature pyramid network introduces learnable weight parameters for each input feature map during feature fusion, and performs feature fusion through a normalized weighted sum, expressed by the formula: , in, The input feature maps come from paths with different resolutions. = 0.0001 is used to prevent numerical instability. For its corresponding learnable weights, This indicates a convolution operation.

4. The high-precision harvesting method for marine ranching shellfish based on multi-source information and vision empowerment according to claim 1, characterized in that, In step S3, the operation of the deformable local texture enhancement module includes: S31. Construct a learnable prototype convolutional kernel library, where each kernel represents a potential local structure detector; S32. Using a light quantum network, predict a set of kernel offsets and kernel weights for each spatial location and each prototype kernel based on the local context of the input feature map; S33. The prototype kernel is deformed using the predicted kernel offset, and the deformed kernel is weighted and combined using kernel weights. An adaptive convolution operation is then performed on the input feature map to generate the texture enhancement feature.

5. The high-precision harvesting method for marine ranching shellfish based on multi-source information and vision empowerment according to claim 1, characterized in that, In step S3, the inference process of the graph neural network is as follows: superpixel regions are used as graph nodes, and the node features are the mean values ​​of features within the region. Edge connections are established based on spatial proximity and feature similarity. Message passing is performed through graph convolution or graph attention mechanisms to update node features. Finally, the updated node features are mapped back to the original spatial dimension to generate the region-aware features.

6. The high-precision harvesting method for marine ranching shellfish based on multi-source information and vision empowerment according to claim 1, characterized in that, In step S4, the conditional variational autoencoder and hybrid density network framework includes a prior network, a posterior network, and a decoder. The prior network takes the spatiotemporal fusion features as input and outputs the prior distribution parameters of the latent variables; The posterior network takes the spatiotemporal fusion features and the real location labels as input during training and outputs the posterior distribution parameters of the latent variables. The decoder samples from the distribution of the latent variables and, in conjunction with the spatiotemporal fusion features, regresses the parameters of the Gaussian mixture model, including the mixing weights, mean vector, and covariance matrix of each component.

7. The method for high-precision harvesting of shellfish in marine ranching based on multi-source information and vision empowerment according to claim 6, characterized in that, In step S4, the uncertainty measure is represented by the covariance matrix of the component with the largest weight in the Gaussian mixture model; the final predicted target position is the mean vector of this component.

8. The method for high-precision harvesting of shellfish in marine ranching based on multi-source information and vision empowerment according to claim 6, characterized in that, In step S4, the loss function used for model training is: in, As a feature of spatiotemporal fusion, For the actual location, As latent variables, For posterior networks, For a priori network, For decoder, Let KL divergence be the KL divergence. To balance the reconstruction loss with the hyperparameters of the regularization term.

9. The high-precision harvesting method for marine ranching shellfish based on multi-source information and vision empowerment according to claim 1, characterized in that, In step S4, the specific process of spatiotemporal fusion is as follows: using pose transformation information from motion sensors or visual odometry, the optimized feature map from the previous moment is warped and aligned to the coordinate system of the current frame; the aligned historical features are concatenated with the region perception features of the current frame, and then fused through a convolutional layer to generate the spatiotemporal fusion features.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the high-precision harvesting method for marine ranching shellfish based on multi-source information and vision empowerment as described in any one of claims 1 to 9.

Images