Deep sea polymetallic nodule coverage rate distribution diagram compilation method based on camera data

Abstract

The invention relates to the field of image processing, and discloses a shooting data-based deep sea polymetallic nodule coverage rate distribution diagram compiling method, which comprises the following steps of S1, obtaining seabed shooting data of deep sea polymetallic nodules, and performing preprocessing; s2, constructing a semantic segmentation data set of the polymetallic nodule; s3, training the semantic segmentation data set by using the improved semantic segmentation neural network model; s4, analyzing the new camera data based on the trained model to obtain a segmentation result of the polymetallic nodule; and S5, according to a segmentation result, compiling a polymetallic nodule resource coverage rate distribution map. According to the invention, the YOLOv7-PMN (YOLOv7for Polymetal Nodules) model has an excellent image processing capability, and high-precision tuberculosis identification and segmentation can be realized in a complex seabed environment.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for compiling a deep-sea polymetallic nodule coverage distribution map based on camera data. Background Art

[0002] Existing deep-sea camera data processing models, such as traditional neural networks, often require a lot of computing resources, which may not be practical when processing large-scale data or real-time applications. In addition, the performance of these models is often limited in complex seabed environments, especially under conditions of insufficient lighting and severe underwater visual interference. The generalization ability of the model is also often challenged because in practical applications, each new detection scene may be very different, making it difficult for pre-trained models to adapt to new environments.

[0003] In addition, existing methods also have shortcomings in segmentation accuracy. Many models often miss or misdetect when distinguishing seafloor nodules from complex backgrounds (such as sediments, biological cover, etc.). These errors may lead to inaccurate assessments of nodule resources and affect subsequent resource development decisions. In addition, many high-precision models are often accompanied by high parameter counts and large memory requirements, which is particularly disadvantageous in resource-constrained application scenarios.

[0004] Therefore, there is a need for an improved technology that can effectively overcome the above-mentioned defects in order to improve the accuracy, efficiency and adaptability of deep-sea polymetallic nodule identification and segmentation, especially in application environments with limited resources and requiring real-time processing. Summary of the invention

[0005] In view of the shortcomings of the existing technology, the present invention provides a method for compiling a deep-sea polymetallic nodule coverage distribution map based on camera data, which solves the problems of low accuracy, weak generalization ability and insufficient processing efficiency in traditional technologies in deep-sea polymetallic nodule identification and segmentation.

[0006] To achieve the above objectives, the present invention is implemented by the following technical scheme: A method for compiling a deep-sea polymetallic nodule coverage distribution map based on camera data comprises the following steps:

[0007] Acquire seafloor camera data of deep-sea polymetallic nodules and perform preprocessing;

[0008] Construct a semantic segmentation dataset of polymetallic nodules;

[0009] Use the improved semantic segmentation neural network model to train the semantic segmentation dataset;

[0010] Analyze new camera data based on the trained model to obtain the segmentation results of polymetallic nodules;

[0011] Prepare a distribution map of polymetallic nodule resource coverage based on the segmentation results;

[0012] Wherein, the improved semantic segmentation neural network model includes:

[0013] The feature extraction network is composed of multiple bneck modules, each of which includes a depthwise separable convolution for efficient feature extraction.

[0014] The feature fusion network fuses features of different scales through upsampling and downsampling operations and the expansion layer aggregation network module to generate a fused multi-scale feature map;

[0015] The segmentation head network receives the fused multi-scale feature maps and performs semantic segmentation to identify and segment structures of different sizes in the image.

[0016] Preferably, the bneck module includes:

[0017] Depth-wise separable convolutional layer: It realizes independent extraction of spatial features and mixes features between channels through 1x1 point-by-point convolutional layers to improve parameter and computational efficiency.

[0018] Squeeze-and-Excitation module: adjusts the importance of each channel through global average pooling and fully connected layers to enhance the feature selection ability of the model;

[0019] Non-linear activation function: Hard-Swish activation function is used to provide non-linear processing and enhance the functional expression ability of the model.

[0020] Preferably, the depth-wise separable convolutional layer comprises:

[0021] A depthwise convolutional layer that applies a convolution kernel to each input channel independently, performing convolution across spatial dimensions;

[0022] Point-wise convolutional layer, applies 1x1 convolution to combine the output of spatial convolution, mix channel information and generate the final output feature map.

[0023] Preferably, the feature fusion network includes:

[0024] Upsampling operation, which uses bilinear interpolation or other appropriate upsampling techniques to increase the low-resolution feature map to a high resolution to restore image details and maintain the spatial continuity of features;

[0025] Downsampling operation, which reduces the high-resolution feature map to a low resolution through maximum pooling or other effective downsampling methods to reduce the amount of calculation and extract feature information in a wider area;

[0026] The splicing operation horizontally stacks feature maps from different scales to combine the advantages of features at each scale and enhance the model's ability to detect objects of different scales.

[0027] Preferably, the segmentation head network comprises:

[0028] Extension layer aggregation network module: By adaptively aggregating features from different network depths, the features are processed using a multi-layer perceptron and a nonlinear activation function to generate feature maps that are highly responsive to objects of various sizes;

[0029] Segmentation operation: Through the depth of the multi-scale feature fusion map, the channel dimension is finely adjusted through 1x1 convolution, and finally the softmax or sigmoid activation function is applied for pixel-by-pixel classification to output the category probability of each pixel to achieve accurate semantic segmentation.

[0030] Preferably, the step of analyzing new camera data based on the trained model includes:

[0031] Load the pre-trained semantic segmentation neural network model;

[0032] Input the preprocessed new camera data into the loaded model;

[0033] Applying the model to perform forward propagation to generate a probability map of each pixel belonging to a polymetallic nodule;

[0034] Applying thresholding techniques, the probability map is converted into a binary segmentation map, where pixels are labeled as polymetallic nodules or background;

[0035] Post-processing steps were performed to improve the quality of the segmentation maps and enhance nodule boundary clarity and continuity.

[0036] Preferably, the step of compiling a polymetallic nodule resource coverage distribution map according to the segmentation results comprises:

[0037] Collect segmentation results, including location, area and morphology information of each identified polymetallic nodule;

[0038] The segmentation results are matched with the geographic coordinate data to determine the specific location of each nodule on the seafloor;

[0039] Using geographic information system software, create resource distribution maps based on the spatial distribution information of nodules;

[0040] Different visual symbols and color coding are applied on the resource distribution map to represent different types or sizes of nodules.

[0041] The present invention also provides a system for compiling a deep-sea polymetallic nodule resource coverage distribution map based on camera data, comprising:

[0042] A camera data acquisition module, used to acquire seabed camera data of deep-sea polymetallic nodules;

[0043] A data processing module, used for preprocessing the camera data;

[0044] The semantic segmentation neural network model includes a feature extraction module, a multi-scale feature fusion module, and a segmentation head module, which is used to train and analyze camera data;

[0045] A distribution map preparation module is used to prepare a polymetallic nodule resource coverage distribution map based on the segmentation results;

[0046] Display module, used to display and analyze resource distribution maps.

[0047] The present invention provides a method for compiling a deep-sea polymetallic nodule coverage distribution map based on camera data.

[0048] It has the following beneficial effects:

[0049] 1. The YOLOv7-PMN model used in this paper has excellent image processing capabilities and can achieve high-precision nodule recognition and segmentation in complex seabed environments. Through fine-tuned feature extraction and optimized deep learning algorithms, the model can accurately identify nodules of various sizes and shapes, and maintain good performance even in the presence of high background noise. This high-precision segmentation capability is crucial for subsequent resource assessment and environmental impact studies.

[0050] 2. The model of the present invention provides high-precision recognition while also having efficient image processing speed, which is achieved by using lightweight MobileNetV3 and deep separable convolution technology. This high-efficiency processing not only reduces the demand for computing resources, but also enables the model to run on devices with lower power consumption, which is very suitable for application scenarios that require fast response, such as real-time seabed monitoring and operation.

[0051] 3. The design of the model of the present invention takes into account the application requirements on resource-constrained devices. By reducing the number of model parameters and memory usage, the demand for storage and computing resources is reduced. This optimization not only improves the deployment flexibility of the model, but also reduces the operating cost, making the technology easier to be widely used on various ocean detection equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0053] Figure 2 It is a schematic diagram of the YOLOv7 semantic segmentation network of the present invention;

[0054] Figure 3 It is a schematic diagram of the MobileNetV3-Small network structure of the present invention;

[0055] Figure 4 It is a schematic diagram of the YOLOv7-MobileNetV3 structure and modules of the present invention;

[0056] Figure 5 It is a schematic diagram of the YOLOv7-DSC network structure of the present invention;

[0057] Figure 6 It is a schematic diagram of the YOLOv7-PMN network structure of the present invention;

[0058] Figure 7 It is a schematic diagram of the structure of the device of the present invention;

[0059] Figure 8 A schematic diagram of a seabed image according to an embodiment of the present invention;

[0060] Fig. 9 Schematic diagram of data enhancement according to an embodiment of the present invention, wherein (A) is a manually annotated image, and (1)(2)(3) are enhanced images;

[0061] Fig.10 The detection results of (1) using (2) YOLOv7, (2) YOLOv7-MobileNetV3, (3) YOLOv7-DSC, and (4) YOLOv7-PMN for a subset of three different scenarios of the embodiment of the present invention. YOLOv7-PMN has the least number of missed nodules and false border detections;

[0062] Fig.11 Schematic diagram of the detection results of an embodiment of the present invention, where (1)(2) are prediction subsets of 6 different scenarios, and (3)(4) are the detection results of YOLOv7-PMN on (1)(2);

[0063] Fig.12 This is a schematic diagram of the number of pixels in the deposition background (white) area of ​​an embodiment of the present invention;

[0064] Fig.13 A schematic diagram of a three-dimensional visualization of ArcGIS software according to an embodiment of the present invention;

[0065] Fig.14 This is a tuberculosis coverage diagram of an embodiment of the present invention.

[0066] Among them, 100, camera data acquisition module; 200, data processing module; 300, semantic segmentation neural network model; 400, distribution map compilation module; 500, display module. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] Please refer to the attached Figure 1 The embodiment of the present invention provides a method for compiling a deep-sea polymetallic nodule coverage distribution map based on camera data, comprising the following steps:

[0069] S1. Obtain seafloor camera data of deep-sea polymetallic nodules and perform preprocessing

[0070] First, use underwater video equipment designed for deep-sea environments to obtain video data of the deep sea bottom. These data are usually captured in the form of high-resolution videos or images, which can show the seafloor topography and surface features of polymetallic nodules in detail. During the data acquisition process, the camera device is equipped with stabilization technology to reduce perturbations caused by currents or operations and ensure image quality.

[0071] The acquired camera data undergoes preprocessing steps, including denoising, color correction, and contrast enhancement. Denoising removes random noise in the image through a digital filter; color correction adjusts the color deviation caused by changes in underwater lighting conditions; and contrast enhancement improves the light and dark details of the image through adaptive histogram equalization technology, making the contrast between the nodules and the bottom more obvious.

[0072] S2. Constructing a semantic segmentation dataset of polymetallic nodules

[0073] Building the dataset involves manually or semi-automatically annotating polymetallic nodules in seafloor camera data. Using image processing software, technicians mark the boundaries of nodules in each image and create masks, which are then used to train semantic segmentation models. The construction of the dataset also includes enhancements such as random cropping, rotation, and scaling of images to improve the model's adaptability to different viewing angles and nodule sizes.

[0074] S3. Use the improved semantic segmentation neural network model to train the semantic segmentation dataset

[0075] The improved semantic segmentation neural network model is based on the YOLOv7-PMN architecture, which greatly reduces the amount of calculation and speeds up processing by integrating the lightweight MobileNetV3 feature extraction network and deep separable convolution. Model training includes two stages: feedforward propagation and feedback optimization. The cross entropy loss function is used to optimize the model parameters to ensure the accuracy and reliability of the segmentation results. The training process uses transfer learning technology, using weights pre-trained on similar image data as initialization to accelerate the learning efficiency of the model on a specific data set.

[0076] S4. Analyze the new camera data based on the trained model to obtain the segmentation results of polymetallic nodules

[0077] The trained model is used to analyze newly acquired seafloor camera data. The model will automatically identify and segment polymetallic nodules in the image, and output a probability map of whether each pixel belongs to a nodule. Thresholding is applied to convert the probability map into a binary image, where the nodule area is clearly marked.

[0078] S5. Prepare a distribution map of polymetallic nodule resource coverage based on the segmentation results

[0079] Based on the results of the model segmentation, the coverage data of each image is matched with the seafloor map using geographic information system (GIS) software to compile a resource distribution map. The distribution map not only shows the spatial distribution of nodules, but also can classify and code them according to their density for resource assessment and development planning.

[0080] The embodiments of the present invention greatly improve the efficiency and accuracy of deep-sea resource exploration through automated polymetallic nodule identification and accurate resource distribution map compilation. In addition, the improved neural network model reduces the demand for computing resources, making the method suitable for real-time analysis on board or at a remote terminal to support rapid decision-making. The implementation of this technology helps optimize resource development plans and reduce interference with the seabed environment, and has important economic value and environmental significance.

[0081] Please refer to the attached Figure 2 -Attached Figure 6 In the existing polymetallic nodule analysis technology, although the conventional deep-sea camera data processing model can handle basic image segmentation tasks, facing the extreme deep-sea environment, such as insufficient light, seawater disturbance and poor equipment stability, these models are often difficult to adapt due to excessive computational load or insufficient segmentation accuracy. In order to solve these problems, the present invention proposes an improved semantic segmentation neural network model based on the YOLOv7-PMN structure, which is as follows:

[0082] like Figure 2 and Figure 3As shown in the figure, the traditional YOLOv7 semantic segmentation network and the MobileNetV3-Small network structure are respectively shown. The present invention makes the following improvements to the traditional YOLOv7 basic model:

[0083] 1) Use the MobileNetV3-Small feature extraction network to replace the backbone network of the YOLOv7 instance segmentation model to form a new network structure of YOLOv7-MobileNetV3;

[0084] 2) Use the depth-separable convolutional layer to replace the ordinary convolution Conv module of the YOLOv7 instance segmentation model to form a new network structure of YOLOv7-DSC;

[0085] 3) The backbone network of the YOLOV7 instance segmentation model is replaced by the MobileNetV3-Small feature extraction network, and the ordinary convolution Conv module of the head network of the YOLOV7 instance segmentation model is replaced by the depthwise separable convolution. The new network is named YOLOv7-PMN.

[0086] The new network model structure is as follows:

[0087] YOLOv7-MobileNetV3

[0088] As Figure 4 As shown, the present invention uses the MobileNetV3-Small feature extraction network to replace the backbone network of the traditional YOLOv7 instance segmentation model. MobileNetV3-Small utilizes a lightweight backneck module structure, and each block is equipped with a depth-separable convolution and Squeeze-and-Excitation (SE) module. These innovative designs significantly improve the efficiency of feature extraction while reducing the complexity and computational requirements of the model. Through this replacement, the new YOLOv7-MobileNetV3 network model not only runs more efficiently on resource-constrained devices, but also improves segmentation accuracy due to more sophisticated feature processing capabilities.

[0089] YOLOv7-DSC

[0090] like Figure 5As shown in the figure, in the traditional YOLOv7 model, the standard convolution operation is powerful, but its computational cost increases with the increase of convolution layers. The improved YOLOv7-DSC model uses a depth-separable convolution layer to replace the standard convolution Conv module. This improvement significantly reduces the number of parameters and computational complexity by decomposing the standard convolution operation into two steps: spatial convolution and point-by-point convolution. Spatial convolution processes each input channel independently, while point-by-point convolution is responsible for merging the results of these independent processes, improving processing efficiency and reducing memory requirements.

[0091] YOLOv7-PMN

[0092] like Figure 6 As shown in the figure, the above two improved YOLOv7-PMN models are combined. As shown in the figure, the multi-scale feature fusion structure and ISegment segmentation head of YOLOv7 are retained. These designs optimize the model's ability to recognize and segment nodules of different sizes in images. By using the feature extraction network and deep separable convolution of MobileNetV3-Small, the model not only reduces the number of parameters and computational load, but also improves the accuracy of nodule image segmentation, especially in complex seabed environments.

[0093] By introducing a lightweight network architecture and an efficient convolution mechanism, the improved model of the present invention significantly improves processing speed and energy efficiency while maintaining or improving the accuracy of image segmentation. This makes the model not only suitable for running on high-performance computing devices, but also can be deployed on offshore platforms or shipboard systems with limited resources, providing strong technical support for the rapid and accurate identification and resource assessment of deep-sea polymetallic nodules. In addition, the improvement of the model also brings lower energy consumption and operation and maintenance costs, which is conducive to the continued deep-sea exploration activities and promotes the sustainable development of deep-sea resources.

[0094] In a preferred embodiment of the present invention, the improved neural network model particularly adopts the bneck module structure, which is designed to improve the efficiency and accuracy of feature extraction in image segmentation tasks. The specific structure is as follows:

[0095] Depth-wise separable convolutional layer, the depth-wise separable convolutional layer in each bneck module first processes each input channel in the image independently through spatial convolution. This step is performed using convolution kernels assigned channel by channel, effectively capturing spatial features without adding additional computational burden. Subsequently, a 1x1 point-by-point convolutional layer is used to merge these independently processed channel features. In this way, the depth-wise separable convolutional layer not only significantly reduces the number of parameters of the model, but also improves computational efficiency, making the model more efficient when processing large-scale image data.

[0096] Squeeze-and-Excitation module, in order to further enhance the model's ability to capture key features, the bneck module integrates the Squeeze-and-Excitation (SE) module. This module first spatially compresses the features of each channel through a global average pooling layer to generate channel-level descriptors. These descriptors are then transformed through two fully connected layers (a compression layer and an expansion layer), and the sigmoid activation function is applied to generate the weights of each channel. In this way, the SE module effectively recalibrates the feature response of each channel, enhances the model's sensitivity to information-rich features, and thus improves the overall segmentation performance.

[0097] Nonlinear activation function, in order to provide better functional expression capabilities, the bneck module uses the Hard-Swish activation function. The Hard-Swish activation function is a variant of the Swish activation function. It combines the characteristics of the ReLU and sigmoid functions, provides nonlinear processing capabilities, and maintains low computational complexity. This activation function is particularly suitable for use in mobile devices or resource-constrained environments because it can maintain high computational efficiency without sacrificing model performance.

[0098] By adopting the above-mentioned bneck module, the model of the present invention is not only more efficient in resource utilization, but also can provide more accurate image segmentation results. These optimization measures ensure that the model can accurately identify and segment polymetallic nodules in extreme deep-sea environments, providing strong technical support for resource exploration and evaluation. In addition, the model of this structure is particularly suitable for deployment in real-time or near-real-time ocean exploration tasks due to its excellent performance and efficiency, which helps to conduct data analysis and decision-making in real time, thereby greatly improving the efficiency of deep-sea exploration activities and the reliability of results.

[0099] In a preferred embodiment of the present invention, the improved neural network model uses a depth-separable convolutional layer to improve processing efficiency and reduce model complexity, which is particularly suitable for processing a large amount of deep-sea camera data. The structure of the depth-separable convolutional layer is described in detail as follows:

[0100] Spatial convolution layer: The main function of the spatial convolution layer is to perform convolution operations independently on each channel of the image. This layer uses convolution kernels specially designed for each input channel, which only perform convolution processing in the spatial dimension on their corresponding channels. This approach significantly reduces the number of parameters because it does not require the convolution kernel to be shared between all channels. In addition, since the convolution kernel is applied to each channel independently, this also reduces the amount of computation, making the entire processing more efficient. The spatial convolution layer can effectively capture the spatial features within each channel and provide refined local feature information for subsequent feature fusion.

[0101] Point-wise convolutional layer: After the spatial convolutional layer is processed, the point-wise convolutional layer takes over the processing result and applies a 1x1 convolution kernel to combine the features of each independent channel. This step is to mix the information from different channels to generate the final output feature map. The 1x1 convolution not only helps to mix channel features while keeping the number of parameters low, but also improves the nonlinear ability of the network, enabling the model to better learn the complex relationship between the features extracted from each independent channel. This design of the point-by-point convolutional layer is a key step in optimizing information flow and improving feature utilization efficiency. It ensures that the network can fully understand and express cross-channel feature dependencies.

[0102] By reducing the number of parameters and computational operations required, the model is more suitable for running on resource-constrained devices, such as remotely operated vehicles (ROVs) or autonomous underwater vehicles (AUVs) used in ocean exploration. Due to the reduced computing requirements, the model processes faster and can provide near real-time data analysis, which is critical for rapid response and decision-making.

[0103] In a preferred embodiment of the present invention, a feature fusion network is designed to effectively process and integrate multi-scale image features to optimize the recognition and segmentation of polymetallic nodules. This network includes upsampling, downsampling, and splicing operations, which are described as follows:

[0104] Upsampling operation: The upsampling operation of the feature fusion network uses bilinear interpolation or other advanced upsampling techniques (such as transposed convolution) to process low-resolution feature maps. The purpose of this operation is to convert the coarser feature map into a higher-resolution version to restore the lost image details and maintain the spatial continuity of the features. Through upsampling, the model is able to refine the feature representation so that small polymetallic nodules are more clearly represented, which is particularly important for subsequent accurate segmentation.

[0105] Downsampling operation: In the feature fusion network, the downsampling operation is performed by the maximum pooling or average pooling method to reduce the high-resolution feature map to a lower resolution. This step helps the model reduce the computational burden and capture image features from a wider field of view, thereby enhancing the understanding of large-scale structures. The downsampled feature map provides a compressed but information-rich feature representation, which is essential for understanding the contextual relationships in the entire image scene.

[0106] Splicing operation: The splicing operation is a key component of the feature fusion network, which horizontally stacks feature maps of different scales. This operation does not simply stack features, but merges feature maps from different processing stages in a strategic way to ensure that the advantages of features at each scale are fully utilized. This strategy enables the model to take into account both local details and global structures, greatly enhancing the detection ability of targets of different scales, from tiny nodule particles to larger areas of nodule aggregation, which can be effectively identified and accurately segmented.

[0107] Through the above-mentioned feature fusion strategy, the present invention significantly improves the flexibility and efficiency of the model in processing multi-scale images. The combination of upsampling and downsampling ensures that the model can capture wider-area feature information while maintaining high-resolution details, and the splicing operation optimizes the integration of this information. This multi-scale feature fusion capability enables the model to maintain high accuracy and high robustness when identifying and segmenting deep-sea polymetallic nodules, regardless of the size of the nodules, greatly improving the efficiency and accuracy of deep-sea resource exploration. In addition, the design of this feature fusion network takes into account the optimized use of computing resources, making the model particularly suitable for resource-constrained remote operation environments.

[0108] In a preferred embodiment of the present invention, an improved segmentation head network is designed to process and optimize the multi-scale feature maps from the feature fusion network to achieve efficient and accurate semantic segmentation of polymetallic nodules. The composition and operation of the segmentation head network are detailed as follows:

[0109] Extension layer aggregation network module: The extension layer aggregation network module is the core component of the segmentation head network. Its main function is to adaptively aggregate features from network layers of different depths. This module uses a multi-layer perceptron (MLP) and nonlinear activation functions such as ReLU or Leaky ReLU to process and transform features. In this way, the extension layer aggregation network is able to generate feature maps that are highly responsive to objects of various sizes, thereby improving the model's ability to recognize nodules of different scales. This adaptive aggregation method ensures that key features from different depths of the network are fully utilized, optimizing information flow and feature utilization efficiency.

[0110] Segmentation operation: The segmentation operation is the last stage of the segmentation head network, which is responsible for converting the aggregated multi-scale feature map into a specific semantic segmentation result. This operation first processes the multi-scale feature fusion map through a depth-wise separable convolution, which maintains the efficiency of the processing while retaining rich spatial and channel information. Subsequently, a 1x1 convolution is used to fine-tune the channel dimension of the feature map to ensure that the features of each channel are properly represented and weighted.

[0111] Finally, the segmentation operation applies a pixel-by-pixel classification activation function, usually softmax or sigmoid, to calculate the probability of each pixel belonging to a nodule or background. The Softmax function is suitable for multi-class classification problems, while the Sigmoid function is used for binary classification problems, such as nodules and non-nodules in the present invention. Through this pixel-by-pixel classification method, the segmentation head network can output a high-precision pixel-level segmentation map that accurately marks the location and range of polymetallic nodules.

[0112] By adaptively aggregating multi-scale features, the model can effectively identify nodules of various sizes from small to large, improving the coverage and accuracy of segmentation. The use of depthwise separable convolution and 1x1 convolution to optimize the calculation process significantly reduces the computational complexity and improves the processing speed, making the model suitable for real-time or near-real-time processing needs. Pixel-by-pixel classification ensures segmentation accuracy, allowing the model to carefully distinguish nodules from the background, providing reliable data support for subsequent resource assessment and development decisions.

[0113] In a preferred embodiment of the present invention, the steps of analyzing new camera data based on the trained model are described in detail, aiming to provide a high-precision and high-efficiency method for identifying and segmenting deep-sea polymetallic nodules.

[0114] 1. Load the pre-trained semantic segmentation neural network model

[0115] First, the system loads a pre-trained semantic segmentation neural network model. This model is based on a modified YOLOv7-PMN architecture and has been trained on a large amount of deep-sea camera data to learn to identify polymetallic nodules of various types and sizes. Model training includes the recognition of nodules in various complex environments, ensuring its wide applicability and high accuracy.

[0116] 2. Input the preprocessed new camera data into the loaded model

[0117] Input the preprocessed new camera data into the loaded model. The preprocessing step includes image denoising, contrast enhancement, and color correction to optimize image quality and provide the best input for model analysis. This step ensures that the model can receive clear and accurate image data in the best condition, thereby improving the accuracy of the analysis.

[0118] 3. Apply the model to perform forward propagation to generate a probability map of each pixel belonging to a polymetallic nodule

[0119] The model performs a forward pass on the input camera data to calculate the probability of each pixel belonging to a polymetallic nodule. This calculation process involves deep separable convolution, feature fusion, and multi-scale processing, so that the generated probability map not only accurately reflects the location of the nodule, but also reveals its possible shape and boundary.

[0120] 4. Apply threshold processing technology to convert the probability map into a binary segmentation map

[0121] The probability map is converted into a binary segmentation map using a thresholding technique, where pixels above a certain threshold are marked as nodules and pixels below the threshold are considered background. The threshold is set according to the specific application requirements and environment to ensure the highest segmentation accuracy and the lowest false positive rate.

[0122] 5. Perform post-processing steps to improve the quality of the segmentation map

[0123] After generating the binary segmentation map, a series of post-processing steps are performed, including morphological operations such as dilation and erosion, and smoothing filtering. These steps help improve the overall quality of the segmentation map, enhance the boundary clarity of the nodules, eliminate small noise points, and improve the continuity of the nodule region. This step is critical for providing reliable downstream analysis and decision support.

[0124] The analysis method based on the trained model provided by the embodiment of the present invention provides an efficient and accurate solution for the identification and segmentation of deep-sea polymetallic nodules. Through sophisticated model training and effective post-processing, the method can achieve fast and accurate nodule detection in complex seabed environments, significantly improving the capabilities of resource exploration and environmental monitoring. In addition, the implementation of the method reduces the demand for high-performance computing resources, making it more suitable for use in remote or resource-constrained environments, thereby providing greater flexibility and feasibility for deep-sea exploration.

[0125] In a preferred embodiment of the present invention, the steps of compiling a polymetallic nodule resource coverage distribution map based on the segmentation results are described in detail to effectively utilize the information obtained from deep-sea camera data.

[0126] 1. Collect segmentation results

[0127] First, the system collects the results generated by the segmentation network, which includes the coverage of polymetallic nodules in each image identified from the seafloor camera data. These data are obtained through advanced image analysis technology, and each nodule is accurately annotated to determine its boundaries in the image. The collected data is not limited to two-dimensional coordinates, but also includes a detailed description of the shape of each nodule, which is extremely important for subsequent resource assessment.

[0128] 2. Match the segmentation results with the geographic coordinate data

[0129] Next, the segmentation results are matched to geographic coordinate data. This step involves assigning each nodule image its actual seafloor geographic location. This is usually done by synchronizing with bottom scanning equipment or other geographic positioning systems to ensure that the location information of each nodule image is accurate. This process is crucial for the subsequent production of resource distribution maps because it ensures the accuracy and practicality of the geographic information.

[0130] 3. Use GIS software to create resource distribution maps

[0131] Subsequently, geographic information system (GIS) software was used to create a resource distribution map based on the spatial distribution information of nodules. GIS software is capable of processing large amounts of geographic data and generating detailed maps that not only show the distribution of nodules, but can also perform a variety of complex spatial analyses as needed. In this step, GIS software integrates information such as the location and coverage value of nodule images into a detailed distribution map, providing valuable visual information and data support for researchers and decision makers.

[0132] 4. Apply visual symbols and color coding

[0133] Finally, different visual symbols and color coding will be applied to the resource distribution map to represent different levels of nodule coverage. This visual strategy not only enhances the readability of the map, but also helps users quickly identify and assess the type and potential value of the resource. Color coding can be set based on factors such as the metal content of the nodules, their economic value or their environmental impact, while visual symbols can be used to distinguish morphological categories or clusters of nodules.

[0134] Through the above steps, the present invention can effectively convert the segmentation results of deep-sea polymetallic nodules into detailed resource distribution maps. This not only improves the accuracy and efficiency of resource assessment, but also enhances the scientific nature of resource management and development plans. In addition, this method greatly improves the technical level of deep-sea resource exploration by combining advanced image analysis technology with geographic information systems, providing strong support for the sustainable development of deep-sea resources.

[0135] The system for compiling a deep-sea polymetallic nodule resource coverage distribution map based on camera data described below and the method for compiling a deep-sea polymetallic nodule coverage distribution map based on camera data described above can be referenced to each other.

[0136] Please refer to the attached Figure 7 The present invention also provides a system for compiling a deep-sea polymetallic nodule resource coverage distribution map based on camera data, comprising:

[0137] The camera data acquisition module 100 is used to acquire the seabed camera data of deep-sea polymetallic nodules;

[0138] The data processing module 200 is used to pre-process the camera data;

[0139] A semantic segmentation neural network model 300, including a feature extraction module, a multi-scale feature fusion module and a segmentation head module, is used to train and analyze camera data;

[0140] A distribution map preparation module 400 is used to prepare a polymetallic nodule resource coverage distribution map according to the segmentation results;

[0141] The display module 500 is used to display and analyze the resource distribution map.

[0142] The system of this embodiment can be used to execute the above method embodiments, and its principles and technical effects are similar, which will not be repeated here.

[0143] Embodiment 1:

[0144] 1. Dataset construction:

[0145] like Figure 8 As shown, the image data is obtained from the seabed photography taken by the submersible and is divided into three categories:

[0146] Long-range image (1): taken when the submersible is at a higher altitude from the seabed, covering a larger area, and the image is blue-green (bluish). In this type of image, the nodules are smaller in size and more numerous, with approximately 550 to 1,000 nodules.

[0147] Medium-range image (2): taken at a moderate distance from the seafloor, with a moderate coverage area and a blue-green (greenish) image. The nodules are of moderate size, with a number of 200 to 550 nodules.

[0148] Close-up image (3): taken very close to the seafloor, with a smaller coverage area and a yellow image. The nodules are larger in size and fewer in number, ranging from 30 to 200.

[0149] In these images, white microorganisms are often seen on the surface of polymetallic nodules, with large variations in shape, and a small amount of sediment covering the middle of the top surface of the nodules. In addition, the dataset also includes some images with large environmental impacts and unclear images to increase the diversity and complexity of the data.

[0150] 2. Image processing and annotation:

[0151] The clip function of OpenCV was used to crop a 512×512 pixel area from the center of each original image to reduce the overlap between adjacent images. Images were selected for annotation based on the characteristics of nodules, such as obvious features, rich categories, diverse environments, and different sizes. The annotation work was semi-automatically processed using the SAM model in the open source software X-AnyLabeling to separate the polymetallic nodules from the sedimentary background in each image.

[0152] like Fig. 9 As shown in the figure, in order to improve the generalization ability of the model and enhance the robustness, the image and label are enhanced simultaneously. The following operations are performed using the Python-based image enhancement package Augmentor:

[0153] Random flip or rotation: simulate the effect of underwater camera at different shooting angles.

[0154] Brightness change: simulates the change of underwater light source and randomly adjusts the brightness of the image.

[0155] Contrast transformation: simulate the change of image color when the submersible is at different heights from the bottom, and randomly adjust the contrast.

[0156] Random equal reduction: Simulation of small nodule imaging.

[0157] After these enhancements, the dataset finally contains 310 original images and corresponding 310 label files, with a total of 39,540 polymetallic nodule labels.

[0158] 3. Comparison of semantic segmentation models for polymetallic nodules:

[0159] First, configure the training environment. The system software is Windows 10 Professional Edition, the programming language is Python-3.11.0, and the deep learning framework is pytorch-2.1.0+cu118; the GPU processor is 2 NVIDIA GeForce RTX 4090, each with 24564MiB of memory, and the CPU processor is 13th Gen Intel(R) Core(TM) i9-13900K, 3.00GHz. Secondly, set the model training parameters. The optimizer is Adam, the initial learning rate is 0.01, and a total of 100 rounds of training are performed, with a batch size of 16 images per round. The data set is randomly divided into training set and test set at a ratio of 9:1 and input into the YOLOv7 model, YOLOv7-MobileNetV3 model, YOLOv7-DSC model, and YOLOv7-PMN model for training.

[0160] 4. Model Evaluation Metrics

[0161] Recall rate: calculated through the confusion matrix, reflecting the proportion of true tuberculosis identified by the model among all true tuberculosis, and is a key indicator for evaluating the recognition ability of the model.

[0162] Model parameter count: Indicates the complexity and learning ability of the model. The more parameters, the more complex features the model can learn, but increases the risk of overfitting and the computational burden.

[0163] Weight memory usage: reflects the storage requirements of the model, which is particularly important for memory-constrained application scenarios.

[0164] Inference speed: directly affects the deployment efficiency and practicality of the model. Fast inference speed is a key requirement for real-time application scenarios.

[0165] 5. Test results:

[0166] Model performance: All improved models have improved recall by about 3% compared to the baseline YOLOv7 model, and achieved at least 30% reduction in model parameters and weight memory usage, and increased inference speed by at least 15.21FPS.

[0167] Recall rate: The recall rates of the three improved models were all 97%, indicating that the improved network architecture effectively improved the detection accuracy of tuberculosis.

[0168] Model comparison:

[0169] YOLOv7: has the lowest recall rate, the largest number of parameters, the largest weight memory usage, and the lowest inference speed.

[0170] YOLOv7-MobileNetV3: High recall rate.

[0171] YOLOv7-DSC: High recall.

[0172] YOLOv7-PMN: High recall rate, minimum model parameters and weight memory usage, and fastest inference speed.

[0173] Table 1 Comparison of model training evaluation indicators

[0174]

[0175]

[0176] like Fig.10 As shown in Table 1, the nodule recognition and segmentation effects of the baseline model and the three improved models on near, medium and long distance images are shown. The YOLOv7-PMN model performs best in reducing the number of missed nodules and the number of false boundary detections, while the baseline YOLOv7 model performs worst in these aspects.

[0177] like Fig.11As shown, images (1)(2) are from the prediction set. These images contain organisms of different types, colors, shapes, and sizes from those in the training set (1c)(2a), suspended solids of different shapes, areas, and concentrations (1a), sediment blanks of different shapes and areas (1b), and bright bands in the image caused by camera failure (2b). In addition, the number of medium-range and long-range images is much larger than that in the training set. The YOLOv7-PMN model detects and segments nodules in images (1)(2) to obtain images (3)(4). Almost all nodules are detected, and the predicted contours of nodules are almost consistent with the true contours. In Figure (2a), some small nodules that are pressed around the blue organisms are also detected. Figures (1c)(2c) are even more blurred than the training set images, but the nodules in the images can still be detected and segmented. In addition, in Figure (1c), the creature (white fish) suspended in the water forms a shadow on the surface of the nodule. YOLOv7-PMN can clearly detect and segment the nodules in the shadow, indicating that the model can distinguish the categories to which pixels with similar colors belong. All these show that the model has good generalization ability.

[0178] Based on comprehensive quantitative evaluation indicators and qualitative comparison of images, YOLOv7-PMN is the preferred application model. Its recognition accuracy, inference speed, weight memory usage, and actual image application effect can meet the requirements.

[0179] 6. Estimation of polymetallic nodule coverage

[0180] The semantic segmentation results of tuberculosis images are as follows Fig.12 As shown, white pixels (pixel value = 255) are deposition background, and gray pixels are polymetallic nodules (pixel value ≠ 255). In combination with the characteristics of the new model, the coverage calculation formula of this embodiment is as follows:

[0181] Let the number of pixels in the background area of ​​a mask image be W i ,like Fig.12 shown.

[0182] The coverage of polymetallic nodules in this image is:

[0183]

[0184] Where N width and N height are the number of pixels of image width and image height respectively. Fig.12 The polymetallic nodule coverage rate is 55.52%.

[0185] The YOLOv7-PMN model was used to predict the polymetallic nodule images of the survey line. Due to the blurring of some images and the small size of the nodules, no nodules were detected in 82 images, and the nodule detection efficiency was 99.08%. The site sampling records showed that the majority of nodules in this area were epigenetic nodules, and the coverage of nodules in the video images could represent the estimated coverage of deck photography to a certain extent.

[0186] After the previous processing, the nodule coverage, longitude and latitude of each sampled image are known. In ArcGIS software 3D visualization, such as Fig.13 The red dots in the figure represent tuberculosis coverage values ​​between 0.01% and 21.62%, the yellow dots represent tuberculosis coverage values ​​between 21.63% and 43.15%, and the green dots represent coverage values ​​between 43.15% and 61.84%.

[0187] In each coverage interval, randomly sampled and compared tuberculosis images, it can be seen that the images in the 0.01%-21.62% interval are mostly long-distance images, the tuberculosis images are deformed and blurred, and the tuberculosis boundaries cannot be accurately identified, so they are eliminated as low-quality data. The data after elimination is used as statistical data. Fig.14 As shown in the figure, the distribution of tuberculosis coverage shows obvious changes and can be roughly divided into four stages. In stage I, the tuberculosis coverage rate is about 53%; in stage II, the tuberculosis coverage rate decreases to about 50%; in stage III, the tuberculosis coverage rate rises to 55%; in stage IV, the tuberculosis coverage rate drops rapidly to 45%; in stage V, the tuberculosis coverage rate fluctuates between 50% and 57%.

[0188] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. Method for compiling a deep-sea polymetallic nodule coverage distribution map based on camera data, It is characterized in that The following steps are involved: Acquire seafloor camera data of deep-sea polymetallic nodules and perform preprocessing; Construct a semantic segmentation dataset of polymetallic nodules; Use the improved semantic segmentation neural network model to train the semantic segmentation dataset; Analyze new camera data based on the trained model to obtain the segmentation results of polymetallic nodules; Prepare a distribution map of polymetallic nodule resource coverage based on the segmentation results; Wherein, the improved semantic segmentation neural network model includes: The feature extraction network is composed of multiple bneck modules, each of which includes a depthwise separable convolution for efficient feature extraction. The feature fusion network fuses features of different scales through upsampling and downsampling operations and the expansion layer aggregation network module to generate a fused multi-scale feature map; The segmentation head network receives the fused multi-scale feature maps and performs semantic segmentation to identify and segment structures of different sizes in the image.

2. The method for compiling a deep-sea polymetallic nodule coverage distribution map based on camera data according to claim 1, It is characterized in that The bneck module includes: Depth-wise separable convolutional layer: It realizes independent extraction of spatial features and mixes features between channels through 1x1 point-by-point convolutional layers to improve parameter and computational efficiency. Squeeze-and-Excitation module: adjusts the importance of each channel through global average pooling and fully connected layers to enhance the feature selection ability of the model; Non-linear activation function: Hard-Swish activation function is used to provide non-linear processing and enhance the functional expression ability of the model.

3. The method for compiling a deep-sea polymetallic nodule coverage distribution map based on camera data according to claim 1, It is characterized in that The depth-wise separable convolutional layer comprises: A depthwise convolutional layer that applies a convolution kernel to each input channel independently, performing convolution across spatial dimensions; Point-wise convolutional layer, applies 1x1 convolution to combine the output of spatial convolution, mix channel information and generate the final output feature map.

4. The method for compiling a deep-sea polymetallic nodule coverage distribution map based on camera data according to claim 1, It is characterized in that The feature fusion network includes: Upsampling operation, which uses bilinear interpolation or other appropriate upsampling techniques to increase the low-resolution feature map to a high resolution to restore image details and maintain the spatial continuity of features; Downsampling operation, which reduces the high-resolution feature map to a low resolution through maximum pooling or other effective downsampling methods to reduce the amount of calculation and extract feature information in a wider area; The splicing operation horizontally stacks feature maps from different scales to combine the advantages of features at each scale and enhance the model's ability to detect objects of different scales.

5. The method for compiling a deep-sea polymetallic nodule coverage distribution map based on camera data according to claim 1, It is characterized in that The segmentation head network comprises: Extension layer aggregation network module: By adaptively aggregating features from different network depths, the features are processed using a multi-layer perceptron and a nonlinear activation function to generate feature maps that are highly responsive to objects of various sizes; Segmentation operation: Through the depth of the multi-scale feature fusion map, the channel dimension is finely adjusted through 1x1 convolution, and finally the softmax or sigmoid activation function is applied for pixel-by-pixel classification to output the category probability of each pixel to achieve accurate semantic segmentation.

6. The method for compiling a deep-sea polymetallic nodule coverage distribution map based on camera data according to claim 1, It is characterized in that The step of analyzing new camera data based on the trained model includes: Load the pre-trained semantic segmentation neural network model; Input the preprocessed new camera data into the loaded model; Applying the model to perform forward propagation to generate a probability map of each pixel belonging to a polymetallic nodule; Applying thresholding techniques, the probability map is converted into a binary segmentation map, where pixels are labeled as polymetallic nodules or background; Post-processing steps were performed to improve the quality of the segmentation maps and enhance nodule boundary clarity and continuity.

7. The method for compiling a deep-sea polymetallic nodule coverage distribution map based on camera data according to claim 1, It is characterized in that The step of compiling a polymetallic nodule resource coverage distribution map according to the segmentation results comprises: Collect segmentation results, including coverage value and shooting time of each image; Match the segmentation results with the geographic coordinate data to determine the specific geographic location of each image; Using geographic information system software, create resource distribution maps based on the spatial distribution information of nodules; Different visual symbols and color coding are applied on the resource distribution map to represent nodules of different individual types or sizes.

8. A system for compiling a deep-sea polymetallic nodule resource coverage distribution map based on camera data, used to implement the method according to any one of claims 1 to 7, It is characterized in that include: A camera data acquisition module, used to acquire seabed camera data of deep-sea polymetallic nodules; A data processing module, used for preprocessing the camera data; The semantic segmentation neural network model includes a feature extraction module, a multi-scale feature fusion module, and a segmentation head module, which is used to train and analyze camera data; A distribution map preparation module is used to prepare a polymetallic nodule resource coverage distribution map based on the segmentation results; Display module, used to display and analyze resource distribution maps.

Images