Deep-sea polymetallic nodule classification and segmentation method and system based on multi-task model

Abstract

The application belongs to the technical field of image processing, and specifically discloses a deep-sea polymetallic nodule classification and segmentation method and system based on a multi-task model. In the method, an improved multi-task model for polymetallic nodule classification and segmentation is proposed. In the model input aspect, the application constructs an accurate polymetallic nodule classification and segmentation data set by using a LabelMe tool. In addition, in the model structure aspect, the application innovatively proposes a network structure containing a kernel pixel branch, a horizontal and vertical distance branch, and a kernel classification branch three-branch structure, which can simultaneously realize three sub-tasks of semantic segmentation, instance segmentation and pixel classification of polymetallic nodules. In addition, in the model structure aspect, the application introduces a feature enhancement module, a cross-attention module and designs different loss functions for each network branch, which greatly improves the classification and segmentation accuracy of polymetallic nodules.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing and deep-sea polymetallic nodule classification and segmentation, and particularly relates to a deep-sea polymetallic nodule classification and segmentation method and system based on a multi-task model. BACKGROUND

[0002] With the continuous development of deep-sea mineral deposits, deep-sea images as an important remote sensing technology have been proven to play an important role in evaluating seabed features, achieving rapid mapping and visualization of underwater landscapes, especially in capturing seabed mineral images and exploring polymetallic nodules, providing valuable data support. In order to better evaluate the distribution, characteristics and economic value of polymetallic nodules, accurate image segmentation technology becomes the key. Especially in determining the coverage, granularity and abundance of nodules, image segmentation technology plays a crucial role. However, due to the tedious process of nodule image annotation collected by underwater photography, the amount of labeled data is limited and there is an imbalance in nodule types, resulting in a serious lack of annotated samples in the dataset. Under this background, how to use limited labeled data and effectively improve the performance in the segmentation model has become a problem to be solved.

[0003] Currently, for the segmentation of seabed minerals, two methods of semantic segmentation and instance segmentation are mainly used. Among them, the semantic segmentation model can quickly perform image segmentation and is suitable for calculating the coverage of nodules, while the instance segmentation model can more accurately determine the specific position, size and shape of the nodule, but its computational load is large, consuming time and memory. Therefore, in the process of polymetallic nodule coverage evaluation, how to balance the segmentation accuracy and computational efficiency has become a key technical problem. With the further development of deep-sea mineral deposits, the accurate evaluation of the distribution, coverage and other characteristics of polymetallic nodules becomes increasingly important.

[0004] Existing patent document 1 discloses a deep-sea polymetallic nodule coverage distribution map compilation method based on camera data, which uses the YOLOV7 model to achieve high-precision semantic segmentation, but its model can only segment the nodules of good quality images, cannot effectively segment the nodules and the model can only be used for segmentation but cannot be classified. Existing patent document 2 discloses a polymetallic nodule image recognition method based on machine learning, which uses a single-branch ResNet as a feature extractor and obtains the nodule coordinates and categories through a fully connected layer, which is slow in running speed, poor in effect, and easy to miss nodules.

[0005] It can be seen that the prior art has the problems of complex and high cost of underwater image acquisition, tedious and time-consuming tuberculosis image labeling, limited amount of labeled data, unbalanced tuberculosis type data (it is difficult to obtain labeled samples of some tuberculosis types), and the need for professional personnel to accurately calibrate tuberculosis and handle differences between different tuberculosis types. The special shooting environment makes the morphology and structure of tuberculosis in the image differ greatly, further increasing the difficulty of labeling. In addition, existing segmentation models can only segment tuberculosis and cannot classify it, and few models can handle adjacent contact nuclei to separate them. In addition, due to the significant differences in the number and size of various types of polymetallic nodules (disc-shaped, spheroid-shaped, synangium-shaped, and detritus-shaped), there is a data imbalance problem during model training. That is, the number and pixel ratio of synangium-shaped nodules are significantly less than those of other types of nodules, resulting in a serious lack of training for the classification and segmentation of synangium-shaped nodules.

[0006] REFERENCES

[0007] Patent Document 1: Chinese Invention Patent Application, Publication No. CN119107544A, Publication Date: December 10, 2024;

[0008] Patent Document 2: Chinese Invention Patent Application, Publication No. CN118982745A, Publication Date: November 19, 2024. SUMMARY

[0009] The present application aims to provide a deep-sea polymetallic nodule classification and segmentation method based on a multi-task model, which overcomes the problems of insufficient labeled data, inability to simultaneously classify and segment, inability to separate contact nuclei, and data imbalance in the prior art, facilitating accurate classification and segmentation of deep-sea polymetallic nodules.

[0010] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0011] The deep-sea polymetallic nodule classification and segmentation method based on a multi-task model comprises the following steps:

[0012] Step 1. Obtain deep-sea polymetallic nodule images, pre-process the images, and construct a polymetallic nodule classification and segmentation dataset;

[0013] Step 2. Construct an improved multi-task model for polymetallic nodule classification and segmentation; the improved multi-task model includes three encoders, a feature fusion module composed of a double-cross attention module, and three decoders;

[0014] Each encoder has the same structure and is defined as the first, second, and third encoders;

[0015] The respective decoders are identical in structure and are defined as a first decoder, a second decoder and a third decoder respectively;

[0016] The first encoder, the feature fusion module and the first decoder constitute a core pixel branch, which is used for realizing semantic segmentation;

[0017] The second encoder, the feature fusion module and the second decoder constitute a horizontal and vertical distance branch, which is used for realizing distance prediction output and fusing the semantic segmentation result of the core pixel branch to separate contact cores and realize core instance segmentation;

[0018] The third encoder, the feature fusion module and the third decoder constitute a core classification branch, which is used for type prediction of each pixel, and for each segmented core instance, the most common class in the pixel prediction in the instance is taken as the final type of the core by using majority voting;

[0019] The encoder comprises a plurality of improved residual units, and the decoder comprises a plurality of transpose convolution modules;

[0020] Step 3. Based on the multi-metal core classification and segmentation data set constructed in step 1, the improved multi-task model is trained, and the trained improved multi-task model is used to realize the classification and segmentation operation of the deep-sea multi-metal core.

[0021] In addition, on the basis of the deep-sea multi-metal core classification and segmentation method based on the multi-task model, the application also proposes a deep-sea multi-metal core classification and segmentation system based on the multi-task model, which adopts the following technical scheme:

[0022] The deep-sea multi-metal core classification and segmentation system based on the multi-task model comprises a camera, a light source and a computer device carried on a deep submergence vehicle; the camera is used for shooting deep-sea multi-metal core images and uploading the images to the computer device; the computer device comprises a memory and one or more processors; the executable code is stored in the memory; when the processor executes the executable code, the steps of the deep-sea multi-metal core classification and segmentation method based on the multi-task model are realized.

[0023] The application has the following advantages:

[0024] As described above, the present application relates to a deep-sea polymetallic nodule classification and segmentation method and system based on a multi-task model. The present application builds an improved multi-task model for polymetallic nodule classification and segmentation. Among them, in terms of model input, the present application uses LabelMe tool to build an accurate data set for polymetallic nodule classification and segmentation. In terms of model structure, the present application innovatively proposes a three-branch structure multi-task model to realize the classification and segmentation of nodule at the same time. Specifically, the three-branch structure is the nodule pixel branch, the horizontal and vertical distance branch, and the nodule classification branch. Among them, the nodule pixel branch and the horizontal and vertical distance branch first separate the nodule pixels from the background to separate the contact nodule and realize the nodule instance segmentation; the nodule classification branch adopts the majority voting method to take the most category in the pixel prediction of the instance as the final type of the nodule; the present application realizes the semantic segmentation, instance segmentation and pixel-level classification three sub-tasks through the improved multi-task model of the three-branch structure. In addition, the feature enhancement module, the cross attention module are introduced in the model, and different loss functions are designed for each branch to suppress the decline of classification and segmentation accuracy caused by data imbalance of nodule. The present application realizes accurate classification and segmentation of nodule through the multi-branch deep learning network designed carefully, improves the accuracy and robustness of the classification and segmentation result, at the same time, explores new image processing and deep learning methods to cope with complex underwater environment and image labeling problems, and the present application can further estimate the coverage of nodule based on the segmentation result, so as to realize efficient and accurate coverage estimation. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The network structure diagram of the improved multi-task model for polymetallic nodule classification and segmentation built in the deep-sea polymetallic nodule classification and segmentation method based on a multi-task model in the embodiment of the present application;

[0026] Figure 2 The data set annotation visualization schematic diagram in the embodiment of the present application;

[0027] Figure 3 The schematic diagram of a single branch structure in the improved multi-task model in the embodiment of the present application;

[0028] Figure 4 The structure schematic diagram of the feature enhancement module in the embodiment of the present application;

[0029] Figure 5 The structure schematic diagram of the cross attention module in the embodiment of the present application;

[0030] Figure 6 The subjective experiment effect diagram in the specific example of the present application. DETAILED DESCRIPTION

[0031] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0032] Example 1

[0033] This embodiment describes a method for classifying and segmenting deep-sea polymetallic nodules based on a multi-task model. The method proposes an improved multi-task model for classifying and segmenting polymetallic nodules, used to achieve nodule instance classification and segmentation in deep-sea polymetallic nodules. The improved multi-task model's network architecture includes, for example... Figure 1 As shown. This invention uses ResNet34 as the backbone of the encoder structure to reduce the risk of overfitting. The encoder mainly consists of the first to fourth residual blocks in ResNet34. The decoder uses four consecutive transposed convolutional layers to perform upsampling. Skip connections are embedded between the encoder and decoder modules to effectively integrate multi-scale semantic features. Notably, the encoder and decoder architectures of the three branches are consistent. However, each branch has its own network parameters, allowing each task to better learn its own feature information without excessive interference from other tasks. Specifically, the kernel pixel branch (NP) and the horizontal / vertical distance branch (HV) first separate kernel pixels from the background to isolate contact kernels, thereby achieving kernel instance segmentation. The kernel classification branch (NC) determines the type of each kernel by aggregating pixel-level kernel type predictions in each instance. To improve the perception range of image context and promote feature interaction between different network branches, such as... Figure 3 As shown, a Feature Enhancement Module (FEM) and a Cross-Attention Module (CCA) are introduced into each branch of the network model. Furthermore, in the deep-sea polymetallic nodule image dataset, the imbalance in the number of different nodule types is particularly prominent. For example, spheroidal and discoid nodules account for a large proportion, while conjoined nodules and fragmentary nodules often constitute only a small portion. This imbalance in nodule type leads to a decrease in segmentation and classification performance, especially for a minority of nodules. To address this problem, this invention designs a novel loss function scheme, with different loss functions designed for each task branch.

[0034] A classification and segmentation method for deep-sea polymetallic nodules based on a multi-task model includes the following steps:

[0035] Step 1. Obtain images of deep-sea polymetallic nodules, preprocess the images, and construct a polymetallic nodule classification and segmentation dataset.

[0036] Step 1.1. Data collection.

[0037] The deep-sea polymetallic nodule image data used in this embodiment was taken by a deep-sea submersible. There is no sunlight in the deep sea, and the lighting in the images was provided by the deep-sea submersible. Therefore, the images include pictures with different light intensities, distributions, and nodule abundances.

[0038] All images have the same pixel resolution, for example, all images have a resolution of 3648×5472 pixels.

[0039] Step 1.2. Data filtering and cropping.

[0040] The original pixel (3648×5472 pixels) deep-sea polymetallic nodule image is cropped, for example, to 1000×1000 pixels, and multiple images (e.g., 60 images) that are evenly distributed and easy to label are selected to form a dataset.

[0041] The dataset is divided into a training set and a test set according to a preset ratio (e.g., 5:1); the training set is used for training the improved multi-task model described below, and the test set is used for testing the improved multi-task model described below.

[0042] Step 1.3. Use the LabelMe tool to label the dataset.

[0043] The boundaries of each polymetallic nodule in the deep-sea polymetallic nodule image are delineated using a polygon tool, which involves marking points on the nodule boundary to form a polygon that encircles the nodule.

[0044] The center point of the nodule in the image is marked using the point tool, indicating the (approximate) center of the nodule.

[0045] Tuberculosis was categorized into four types based on morphological differences: spherical, discoid, concretionary, and fragmentary, with corresponding label names of Spherical, Disc, Concretionary, and Fragmental.

[0046] The annotation of the dataset is as follows Figure 2 As shown, for ease of viewing, the present invention assigns different colors to different categories.

[0047] Step 1.4. After completing the annotation, save a JSON file with the same name as the dataset, which includes the location, mask, and category information of each nodule. Then parse the JSON file and convert it into a MAT file for subsequent reading of training data.

[0048] Step 2. Construct an improved multi-task model for classifying and segmenting polymetallic nodules, such as... Figure 1 As shown, the improved multi-task model includes an encoder, a feature fusion module consisting of a dual cross-attention module, and a decoder.

[0049] This invention innovatively employs three branches to achieve three subtasks: semantic segmentation, instance segmentation, and pixel-level classification.

[0050] There are three encoders and three decoders, each corresponding to one of the three branches. The encoders have the same structure and are defined as the first, second, and third encoders, respectively; the decoders have the same structure and are defined as the first, second, and third decoders, respectively.

[0051] The first encoder, the feature fusion module, and the first decoder together form the kernel pixel branch NP, which is used to achieve semantic segmentation.

[0052] The second encoder, feature fusion module, and second decoder together form the horizontal and vertical distance branches hv, which are used to realize distance prediction output and are fused with the semantic segmentation results of the kernel pixel branch to separate the contact kernel and realize kernel instance segmentation.

[0053] This invention calculates the horizontal and vertical distances of a single nodule from its center using the horizontal and vertical distance calculation branch HV, thereby achieving separation between adjacent contacting nodules. The specific method is as follows:

[0054] First, utilize kernel pixel probability (i.e., the output of the kernel pixel branch NP) and horizontal and vertical gradient maps (Obtained by calculating the gradient of the horizontal and vertical distances of the HV branch output), the labeled map M is calculated using the following formula:

[0055] .

[0056] in, This is a binary thresholding function that thresholds the input image based on empirical parameters h and k. To remove negative values ​​and ensure the label values ​​are non-negative, the resulting M is a binary image (0, 1), with pixels having a value of 1 serving as the center seed for the kernel candidate region. In this embodiment, h and k are, for example, 0.5 and 0.5.

[0057] Then, the energy map E is generated, and the calculation formula is as follows: .

[0058] in, gradient map High-threshold suppression is applied, retaining only low-gradient regions to ensure that low-value regions in the energy map serve as segmentation boundaries. Ensure that segmentation is performed only in high-confidence kernel pixel regions.

[0059] The final energy map E is a floating-point image. The lower the value, the more likely the pixel is to be a nuclear boundary, thus separating the contact nucleus.

[0060] The third encoder, feature fusion module, and third decoder form the kernel classification branch, which is used to predict the type of each pixel. For each segmented kernel instance, the majority voting method is used to determine the category with the most predicted pixel type in the instance as the final type of the kernel.

[0061] The encoder consists of three identical and independent networks, all of which have the same structure.

[0062] like Figure 3 As shown, taking any one of the branches as an example, the encoder of each branch uses ResNet34 as the backbone (improved residual units are obtained by improving the first to fourth residual blocks), and each branch includes 4 improved residual units.

[0063] Each improved residual unit consists of several standard residual units and a feature enhancement module (FEM); the number of standard residual units in the first to fourth improved residual units are 3, 4, 6, and 3, respectively. Figure 3 As shown.

[0064] The structure of the Feature Enhancement Module (FEM) is as follows: Figure 4 As shown, Figure 4 As shown, FEM adopts a 4-branch architecture, in which:

[0065] Branch 1 is the standard convolution path, with 1×1 convolution used for inter-channel feature fusion and 3×3 convolution for extracting local spatial features.

[0066] In branch two, a 1×1 convolution is first performed, followed by a 1×3 convolution and a 3×1 convolution to decompose the horizontal and vertical features, enhancing the ability to extract features in multiple directions. Then, a sparse convolution (Dilated Convolution, or DConv) is performed to expand the receptive field by introducing dilation, capturing long-distance dependencies.

[0067] In branch three, a 1×1 convolution is first performed, followed by a 3×1 convolution and a 1×3 convolution to decompose the vertical and horizontal features, enhancing the ability to extract features in multiple directions. Then, a sparse convolution (Dilated Convolution, or DConv) is performed to expand the receptive field by introducing dilation, capturing long-distance dependencies.

[0068] Branch four is a direct channel mapping, which preserves the high-frequency information of the original feature map of the input FEM and avoids information loss.

[0069] The output features from branch one, branch two, branch three, and branch four are fused to produce an enhanced feature map.

[0070] The two cross-attention modules are cascaded to enable each pixel in the feature map to effectively collect information along its cross path, thereby achieving feature fusion of three branches: kernel pixel branch, horizontal and vertical distance branch, and kernel classification branch.

[0071] like Figure 5 As shown, each cross-attention module divides the input feature map into three paths, used to generate query Q, key K, and value V, respectively. Q and K are then multiplied to generate an affinity matrix, representing the relationship between different positions in the feature map. The affinity matrix is ​​normalized using the Softmax function to generate the attention map.

[0072] The attention map is multiplied by the value V, and residual connections are introduced to generate an enhanced feature map.

[0073] Two cascaded cross-attention modules are defined as the first and second cross-attention modules. The first cross-attention module takes three feature maps from the three branches after feature extraction by the encoder as input, concatenates them by channel, and then performs three parallel 1×1 convolutions to obtain Q, K, and V, respectively. The second cross-attention module takes the feature map processed by the first cross-attention module as input and performs the same operations as the first cross-attention module.

[0074] The cross-attention module not only reduces computational requirements but also captures global context. In this embodiment, the parameters of the cross-attention module are shared by all tasks, thus achieving effective integration between tasks; that is, the outputs of the three branches are processed through the same cross-attention module. Furthermore, compared to traditional multi-task learning methods, this invention, through a continuous cross-attention mechanism, ensures that tasks only interact with information at the cross-attention layer, avoiding meaningless generalization sharing. This reduces performance degradation caused by task conflicts and significantly reduces interference between tasks. Simultaneously, it optimizes feature representation for similar target tasks, enhances the propagation of useful information using the cross-attention module, and improves the model's performance and efficiency.

[0075] The decoder is also a three-branch network, and the three branch networks are identical.

[0076] Each branch's decoder includes four transposed convolutional modules and one 1×1 convolution. Each transposed convolution contains a deconvolutional layer to improve the resolution of the feature map and two convolutional layers for further feature extraction.

[0077] The feature map after each transposed convolutional module is fused with the skip connection feature map from the corresponding layer of the corresponding branch encoder to retain more detailed information; finally, the number of channels and aggregated features are adjusted by 1×1 convolution.

[0078] Step 3. Based on the polymetallic nodule classification and segmentation dataset constructed in Step 1, train the improved multi-task model and use the trained improved multi-task model to perform classification and segmentation operations on deep-sea polymetallic nodules.

[0079] This invention presents a novel method for predicting the horizontal and vertical distances of nodules to achieve the separation of contact nuclei and thus complete the instance segmentation task. In addition, this invention proposes a combined loss function design that integrates cross-entropy, Dice, focus loss, and dynamic cost-sensitive loss, aiming to improve the classification penalty for a minority of nodule classes, alleviate class bias during model training, and ultimately improve the overall classification and segmentation accuracy and the robustness of the model.

[0080] Step 3.1. Use the deep-sea polymetallic nodule image as the sole input to the improved multi-task model, and input it into the encoders of the three branches of the model respectively. The encoder of each branch outputs the feature map of the corresponding branch.

[0081] Step 3.2. Input the feature maps of the three branches of the encoder output in Step 3.1 into the feature fusion module composed of dual cross-attention modules for feature fusion, and output the fused enhanced feature map.

[0082] Step 3.3. Use the fused enhanced feature map output from Step 3.2 as the input to the decoder of the three branches, and input it into the decoder of the corresponding branch to obtain the prediction results of the three branches respectively.

[0083] In step 3.3, the prediction results for each branch are as follows:

[0084] I. The output of the kernel pixel branch (NP) is the semantic segmentation result;

[0085] Background pixels are represented by 0, and nodule pixels are represented by 1. The predicted value is between [0,1], representing the probability that a pixel belongs to a nodule.

[0086] II. The horizontal and vertical distance prediction branch HV outputs the horizontal and vertical distance prediction results;

[0087] The background pixel distance is set to 0. In the horizontal distance prediction, the pixels on the same vertical plane as the center of the nodule are set to 0, and the horizontal distance of the pixels on both sides from the center is normalized to [-1,1]. In the vertical distance prediction, the pixels on the same horizontal plane as the center of the nodule are set to 0, and the vertical distance of the pixels above and below from the center is normalized to [-1,1].

[0088] III. The output of the kernel classification branch (NC) is the classification result, and the categories are encoded using one-hot encoding.

[0089] [1 0 0 0 0] is the background, [0 1 0 0 0], [0 0 1 0 0], [0 0 0 1 0], [0 0 0 1 0], and [0 0 0 0 1] represent four categories: spherical, discoid, conjoined, and fragmented, respectively. The prediction output is the probability of each pixel belonging to one of the five categories, for example, [0.03 0.01 0.03 0.02 0.91].

[0090] Step 3.4. Design novel loss function schemes, and design different loss functions for each task branch.

[0091] The designed loss function is as follows:

[0092] .

[0093] in This represents the loss of the entire model; , and These represent the loss functions for the kernel pixel branch, the horizontal and vertical distance branches, and the kernel classification branch, respectively. , These are cross-entropy loss and Dice loss, respectively. For mean square error loss, This represents the mean squared error loss used to calculate the horizontal and vertical distance gradient maps.

[0094] , and Dynamic cost-sensitive cross-entropy loss, Alpha-Dice loss, and focus loss are the core solutions to the data imbalance problem. , , , , , , These represent the weighting coefficients for the corresponding losses.

[0095] Dynamic cost-sensitive cross-entropy loss Defined as:

[0096] .

[0097] Where N is the number of pixels and C is the number of categories. If pixel i belongs to class c, set the value to 1; otherwise, set the value to 0. The probability that pixel i is incorrectly predicted as class k. For period t, category c is the cost weight for being misclassified as analogy k.

[0098] The formula for updating the value weight is:

[0099] ;

[0100] in, , Representing the period , Weight of time; This represents the relative frequency (error rate) of samples with the true label of class c being misclassified as class k by the model during training epoch t. This represents the average error rate for all classes during training epoch t. This represents the adjustment factor, used to control the speed and magnitude of weight adjustments; the default value is 1.

[0101] Alpha-Dice loss, by dynamically adjusting class weights To improve the recognition of minority categories, the function is:

[0102] .

[0103] Where N is the number of pixels. If pixel i belongs to class c, set the value to 1; otherwise, set the value to 0. The probability that pixel i is predicted to be class c. The weight of category c is defined as:

[0104] .

[0105] in, It is a constant slightly greater than 1 to ensure that all category weights are positive. Indicates whether pixel i belongs to class c; if it does, set to 1, otherwise set to 0. N represents the total number of pixels.

[0106] Focus loss introduces two new parameters into the traditional cross-entropy loss: weighting factors. and modulation index ;

[0107] The formula for focus loss is defined as follows:

[0108] .

[0109] Where N is the number of pixels and C is the number of categories. This indicates whether pixel i belongs to class c; a value of 1 indicates it belongs, and a value of 0 indicates it belongs to class c. The probability that pixel i is predicted to be class c, weight factor and modulation index Hyperparameters are set by humans.

[0110] Step 3.5 Based on the loss function, backpropagate to optimize and improve the parameters of the multi-task model.

[0111] Step 3.6 Repeat steps 3.1 to 3.5 for iterative optimization until the improved multi-task model converges.

[0112] After training the improved multi-task model, it is used to classify and segment deep-sea polymetallic nodules. After instance segmentation, the coverage rate of nodules can be calculated based on the classification and segmentation results.

[0113] Example 2

[0114] This embodiment 2 describes a method for calculating the coverage of deep-sea polymetallic nodules based on a multi-task model. This method is based on the results of the deep-sea polymetallic nodule classification and segmentation method based on a multi-task model mentioned in embodiment 1 above.

[0115] The method for calculating the coverage of deep-sea polymetallic nodules based on a multi-task model includes the following steps:

[0116] Step 4.1. Place the test image Input into the trained improved multi-task model.

[0117] Step 4.2. Utilize kernel pixel probability (i.e., the output of the kernel pixel branch NP) and horizontal and vertical gradient maps (Obtained by calculating the gradient of the horizontal and vertical distances of the HV branch output), the labeled map M is calculated using the following formula:

[0118] .

[0119] in, This is a binary thresholding function that thresholds the input image based on empirical parameters h and k. To remove negative values ​​and ensure the label values ​​are non-negative, the resulting M is a binary image (0, 1), with pixels having a value of 1 serving as the center seed for the kernel candidate region. In this embodiment, h and k are, for example, 0.5 and 0.5.

[0120] Step 4.3. Generate the energy map E, calculated using the following formula:

[0121] .

[0122] in, gradient map High-threshold suppression is applied, retaining only low-gradient regions to ensure that low-value regions in the energy map serve as segmentation boundaries. Ensure that segmentation is performed only in high-confidence kernel pixel regions.

[0123] The final energy map E is a floating-point image. The lower the value, the more likely the pixel is to be a nuclear boundary, thus separating the contact nucleus.

[0124] Step 4.4. Use the watershed algorithm to generate the final instance segmentation result. The calculation formula is as follows:

[0125] .

[0126] in The watershed algorithm is represented by E and M, which are the marker map and energy map of steps 4.3 and 4.2, respectively. S=j represents the pixel being assigned to the j-th kernel instance, and S=0 represents the pixel being judged as the background.

[0127] Step 4.5. For the output of the kernel classification branch, i.e. the probability of a pixel belonging to each category. The maximum value among them represents the category of the pixel.

[0128] Step 4.6. Obtain the nodule coverage rate by calculating the proportion of pixels belonging to nodules (pixels with S=j) to the total number of pixels N in Step 4.4 (i.e., the number of pixels belonging to nodules divided by the total number of pixels N).

[0129] Step 4.7. By querying the pixel category of the kernel instance in Step 4.4 in Step 4.5, take the pixel with the most occurrences in the instance as the kernel instance category, and calculate the number of kernels in each category.

[0130] In addition, the following experiments were conducted to verify the effectiveness of the method of the present invention.

[0131] Experiments were conducted on a server equipped with an Nvidia Tesla V100S-PCIE-32GB GPU, running Ubuntu 18.04.6 LTS. This invention uses the Adam optimizer with a learning rate of 1.0e-4, a StepLR learning rate scheduler, 200 training epochs, and batch sizes of 16 for both training and validation. To demonstrate the effectiveness of the proposed method, it is compared with state-of-the-art methods such as Unet, Hover-Net, and PointNu-Net.

[0132] UNet, a classic network for segmentation tasks, uses an encoder that progressively reduces the spatial dimension of an image through multiple convolutional layers to extract higher-level features. The decoder, on the other hand, progressively restores the image resolution through multiple convolutional and upsampling layers to generate the segmentation result. Skip connections directly connect feature maps from the encoder to the corresponding layers in the decoder, reducing information loss during upsampling. Hover-Net uses a pre-activated ResNet50 as a multi-task shared encoder, and the decoder implements a nearest-neighbor upsampling strategy through three independent identical branches to simultaneously achieve accurate kernel instance segmentation and classification. PointNu-Net combines dynamic image segmentation techniques to detect and classify kernels and introduces a novel Joint Pyramid Fusion Module (JPFM) to enhance the cross-scale dependency of features, thereby improving the accuracy of detection and classification.

[0133] In the comparative experiment, Unet is a single-branch network and cannot simultaneously satisfy classification, semantic segmentation and instance segmentation. Therefore, we repeated its network three times in the experiment to achieve multi-task operation. The other settings are the same as the method of this invention. Hover-Net and PointNu-Net are multi-task models and were run according to their default settings in the experiment.

[0134] Subjective experiment. (By...) Figure 6 As can be seen, UNet's advantage of precise segmentation does not positively impact classification accuracy due to the independent and non-interactive features of its three networks, as shown in Experiments I and III. While Hover-Net's shared encoder reduces model parameters, excessive feature cross-interaction severely impacts the segmentation task, leading to oversegmentation. For example, it misclassifies a single nodule into multiple smaller parts, such as the disc-shaped class in Experiment II, and misclassifies multiple nodules into one, as shown in Experiment I. Furthermore, it suffers from misclassification for classes with fewer samples, such as misclassifying conjoined organisms into spherical classes, as shown in Experiment III. PointNu-Net treats the detection and classification of each kernel as a new semantic keypoint estimation problem to determine the center point of each kernel, thus significantly improving the missegmentation problem. However, it does not effectively address the issue of inaccurate classification of classes with few samples, as seen in the conjoined organism class in Experiment III. The method of this invention employs independent encoders and decoders and introduces a cross-attention mechanism. Each branch avoids unnecessary influence, achieving benign feature interaction and solving the missegmentation problem. It also introduces cost-sensitive loss and other loss functions to address the problem of misclassification of a few samples.

[0135] Objective Experiments. To objectively evaluate the performance of different classification and segmentation algorithms, four statistical metrics—Accuracy (Acc), Precision (Prec), Recall, and IoU—were used in an objective comparative experiment. Accuracy reflects the proportion of correctly predicted pixels out of the total pixels; Precision refers to the proportion of pixels predicted as positive that are actually positive; Recall measures the proportion of true positive pixels that are accurately predicted; and Intersection over Union (IoU) assesses the degree of overlap between the predicted and true regions by calculating the ratio of the overlap between the predicted and true regions to their union.

[0136] As shown in Table 1, the performance metrics mentioned above were used to compare and evaluate the results of classification and segmentation algorithms on the multimetallic nodule dataset. The table shows that algorithms such as Unet and Hover-Net did not adequately consider the data imbalance problem, which affected the overall accuracy of classification and segmentation tasks to some extent. Furthermore, due to the lack of effective solutions to the challenges of multi-scale representation of multimetallic nodules, smaller nodules often resulted in misclassification, thus reducing recall. These factors collectively led to the severity of missegmentation, resulting in a low intersection-over-union ratio (IoU). PointNu-Net combines dynamic image segmentation techniques and a Joint Pyramid Fusion Module (JPFM) to improve the accuracy of nodule detection and classification. However, although this algorithm theoretically improves feature fusion, it is prone to misclassification in samples with dense nodule populations. The method of this invention effectively solves the data imbalance problem and improves the accuracy of classification and segmentation by employing an independent encoder, introducing a multi-scale module, and using a cost-sensitive loss function. These improvements not only enhance the algorithm's ability to identify nodule edges and small nodules but also improve overall performance, especially showing significant advantages in accuracy and IoU.

[0137] Table 1 Objective Evaluation of Classification and Segmentation Results

[0138]

[0139] Ablation experiments. To evaluate the effectiveness of the proposed and improved modules of this invention, ablation experiments were conducted on a deep-sea polymetallic nodule dataset. The specific design is as follows:

[0140] (a): Basic three-branch network,

[0141] (b): Introduce a Feature Enhancement Module (FEM) into the base network;

[0142] (c): Add a cross-attention module (CCA) to the base network;

[0143] (d): Add a cross-attention module (CCA) and a feature enhancement module (FEM) to the base network;

[0144] (e): Add dynamic cost-sensitive loss to (d). and focus loss .

[0145] Table 2 Ablation Experiment

[0146]

[0147] As shown in Table 2, the feature enhancement module improves the classification and segmentation of small targets by increasing feature richness and enlarging the receptive field; the cross-attention module selectively aggregates context based on the spatial attention map, making information between multiple branches interconnected and improving the classification and segmentation results of nodules; the introduction of dynamic cost-sensitive loss and focus loss solves the problem of data imbalance and improves the accuracy of classification and segmentation.

[0148] Resource Assessment. The experiments described above show that the method of this invention achieved the best results among the comparative algorithms. Based on this conclusion, we used the experimental results of the method presented in this paper to assess the nodule coverage and the number of nodules of each category (spheroidal: NS, discoid: ND, intercalary: NC, fragmented: NF, total: NT) on the polymetallic nodule test set. The results are shown in Table 3.

[0149] Table 3 Resource Assessment Experiment

[0150]

[0151] Experiments show that the present invention has an error of only 0.84% ​​in nodule coverage assessment, controlling the overall coverage error to within 1%; and the total statistical error is less than 0.6%, meeting the accuracy requirements for coverage and morphological classification in deep-sea polymetallic nodule resource exploration.

[0152] Example 3

[0153] This embodiment 3 describes a deep-sea polymetallic nodule classification and segmentation system based on a multi-task model, which includes a camera, a light source, and a computer device mounted on a deep-sea submersible. Since there is no sunlight in the deep sea, the illumination in the image is provided by the submersible's light source. The camera is used to capture images of deep-sea polymetallic nodules and upload them to the computer device; the images captured by the camera include pictures with different light intensities, distribution patterns, and nodule abundance. The computer device includes a memory and one or more processors.

[0154] The memory stores executable code; when the processor executes the executable code, it implements the steps of the deep-sea polymetallic nodule classification and segmentation method based on a multi-task model as described in Embodiment 1 above.

[0155] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A classification and segmentation method for deep-sea polymetallic nodules based on a multi-task model, characterized in that, Includes the following steps: Step 1. Obtain images of deep-sea polymetallic nodules, preprocess the images, and construct a polymetallic nodule classification and segmentation dataset; Step 2. Construct an improved multi-task model for the classification and segmentation of polymetallic nodules; The improved multi-task model includes three encoders, a feature fusion module consisting of a dual cross-attention module, and three decoders; Each encoder has the same structure and is defined as the first, second, and third encoders, respectively. Each decoder has the same structure and is defined as the first, second, and third decoders, respectively. The first encoder, the feature fusion module, and the first decoder form the kernel pixel branch, which is used to achieve semantic segmentation. The second encoder, feature fusion module, and second decoder form a horizontal and vertical distance branch, which is used to realize the distance prediction output and fuse it with the semantic segmentation result of the kernel pixel branch to separate the contact kernel and realize kernel instance segmentation. The third encoder, feature fusion module and third decoder form the kernel classification branch, which is used to predict the type of each pixel. For each segmented kernel instance, the majority voting method is used to determine the most frequently predicted class in the pixels within the instance as the final type of the kernel. The encoder includes several improved residual units, and the decoder includes several transposed convolutional modules; Step 3. Based on the polymetallic nodule classification and segmentation dataset constructed in Step 1, train the improved multi-task model and use the trained improved multi-task model to perform classification and segmentation operations on deep-sea polymetallic nodules. The two cross-attention modules are cascaded to enable each pixel in the feature map to effectively collect information along its cross path, thereby achieving feature fusion of three branches: kernel pixel branch, horizontal and vertical distance branch, and kernel classification branch. Each cross-attention module divides the input feature map into three paths, which are used to generate query Q, key K, and value V, respectively. Q and K are multiplied by matrix to generate an affinity matrix, which represents the relationship between different positions in the feature map. The affinity matrix is ​​normalized using the Softmax function to generate an attention map. The attention map is then multiplied with the value V, and residual connections are introduced to generate an enhanced feature map. Define two cascaded cross-attention modules as the first and second cross-attention modules, respectively; The first cross-attention module takes three feature maps from three branches after feature extraction by the encoder as input, concatenates them by channel, and then performs three parallel 1×1 convolutions to obtain Q, K and V respectively. The second cross-attention module takes the feature map processed by the first cross-attention module as input and performs the same operations as the first cross-attention module.

2. The method for classifying and segmenting deep-sea polymetallic nodules based on a multi-task model according to claim 1, characterized in that, Step 1 specifically involves: Step 1.

1. The deep-sea polymetallic nodule image data was taken by the submersible. The lighting in the images was provided by the submersible. The images include pictures with different lighting intensities, distributions, and nodule abundances. All pictures have the same pixel resolution. Step 1.

2. Extract the original pixel images of deep-sea polymetallic nodules, and select multiple evenly distributed and easily labeled images to form a dataset. Divide the dataset into training set and test set according to a preset ratio. Step 1.

3. Label the dataset using the LabelMe tool; The boundaries of each polymetallic nodule in the deep-sea polymetallic nodule image are delineated using a polygon tool, which involves marking points on the nodule boundary to form a polygon to delineate the nodule. The center point of the nodules in the image is marked using the point tool; Tuberculosis is categorized into four types based on morphological differences: spherical, discoid, concretionary, and fragmentary, with corresponding label names of Spherical, Disc, Concretionary, and Fragmental. Step 1.

4. After completing the annotation, save a JSON file with the same name as the dataset, which includes the location, mask, and category information of each nodule. Then parse the JSON file and convert it into a MAT file for subsequent reading of training data.

3. The method for classifying and segmenting deep-sea polymetallic nodules based on a multi-task model according to claim 1, characterized in that, Each branch encoder uses ResNet34 as the backbone and includes 4 improved residual units, each of which consists of several standard residual units and a feature enhancement module (FEM). The number of standard residual units in each improved residual unit is 3, 4, 6, and 3, respectively.

4. The method for classifying and segmenting deep-sea polymetallic nodules based on a multi-task model according to claim 3, characterized in that, The Feature Enhancement Module (FEM) adopts a 4-branch architecture, wherein: Branch 1 is the standard convolution path, with 1×1 convolution used for inter-channel feature fusion and 3×3 convolution for extracting local spatial features; In branch two, the horizontal and vertical features are first decomposed by a 1×1 convolution, then by a 1×3 convolution and a 3×1 convolution to enhance the ability to extract features in multiple directions. Finally, the receptive field is expanded by introducing holes through sparse convolution. In branch three, a 1×1 convolution is first performed, followed by a 3×1 convolution and a 1×3 convolution to decompose the vertical and horizontal features, thereby enhancing the ability to extract features in multiple directions. Then, sparse convolution is performed to expand the receptive field by introducing holes. Branch four is a direct channel mapping, which preserves the high-frequency information of the original feature map of the input FEM and avoids information loss; The output features from branch one, branch two, branch three, and branch four are fused to produce an enhanced feature map.

5. The method for classifying and segmenting deep-sea polymetallic nodules based on a multi-task model according to claim 1, characterized in that, Each branch's decoder includes four transposed convolutional modules and one 1×1 convolution. Each transposed convolution contains a deconvolutional layer to improve the resolution of the feature map and two convolutional layers to further extract features. The feature map after each transposed convolutional module is fused with the skip connection feature map from the corresponding layer of the corresponding branch encoder to retain more detailed information; finally, the number of channels and aggregated features are adjusted by 1×1 convolution.

6. The method for classifying and segmenting deep-sea polymetallic nodules based on a multi-task model according to claim 1, characterized in that, Step 3 specifically involves: Step 3.

1. Use the deep-sea polymetallic nodule image as the sole input to the improved multi-task model, and input it into the encoders of the three branches of the model respectively. The encoder of each branch outputs the feature map of the corresponding branch. Step 3.

2. The feature maps of the three branches of the encoder output in Step 3.1 are concatenated by channel and then input into the feature fusion module composed of dual cross-attention modules for feature fusion, and the fused enhanced feature map is output. Step 3.

3. Use the fused enhanced feature map output from Step 3.2 as the input to the decoder of the three branches, and input it into the decoder of the corresponding branch to obtain the prediction results of the three branches respectively; Step 3.

4. Design loss function schemes, designing different loss functions for each task branch; Step 3.5 Based on the loss function, backpropagate to optimize and improve the parameters of the multi-task model; Step 3.6 Repeat steps 3.1 to 3.5 for iterative optimization until the improved multi-task model converges.

7. The method for classifying and segmenting deep-sea polymetallic nodules based on a multi-task model according to claim 6, characterized in that, In step 3.3, the prediction results for each branch are as follows: I. The output of the kernel pixel branch is the semantic segmentation result; Background pixels are represented by 0, and nodule pixels are represented by 1. The predicted value is between [0,1], representing the probability that a pixel belongs to a nodule. II. The horizontal and vertical distance prediction branch outputs the horizontal and vertical distance prediction results; The background pixel distance is set to 0. In the horizontal distance prediction, the pixels on the same vertical plane as the center of the nodule are set to 0, and the horizontal distance of the pixels on both sides from the center is normalized to [-1,1]. In the vertical distance prediction, the pixels on the same horizontal plane as the center of the nodule are set to 0, and the vertical distance of the pixels above and below from the center is normalized to [-1,1]. III. The output of the kernel classification branch is the classification result, and the categories are encoded using one-hot encoding; [1 0 0 0 0] is the background, and [0 1 0 0 0], [0 0 1 0 0], [0 0 0 1 0], and [0 0 0 0 1] represent four categories: spherical, discoid, conjoined, and fragmented, respectively. The prediction output is the probability of each pixel belonging to one of the five categories.

8. The method for classifying and segmenting deep-sea polymetallic nodules based on a multi-task model according to claim 6, characterized in that, In step 3.4, the designed loss function is as follows: ； in This represents the loss of the entire model; , and These represent the loss functions for the kernel pixel branch, the horizontal and vertical distance branch, and the kernel classification branch, respectively. , These are cross-entropy loss and Dice loss, respectively. For mean square error loss, This represents the mean squared error loss used to calculate the horizontal and vertical distance gradient maps; , and For dynamic cost-sensitive cross-entropy loss, Alpha-Dice loss, and focus loss; , , , , , , These represent the weighting coefficients for the corresponding losses.

9. A deep-sea polymetallic nodule classification and segmentation system based on a multi-task model, including a camera, a light source, and a computer device mounted on a deep-sea submersible; wherein the camera is used to capture images of deep-sea polymetallic nodules and upload them to the computer device; Computer devices include memory and one or more processors; The memory stores executable code; its characteristic is that... When the processor executes the executable code, it implements the steps of the deep-sea polymetallic nodule classification and segmentation method based on a multi-task model as described in any one of claims 1 to 8.

Images