Deep-sea mineral identification method and system based on deep learning

Abstract

The application discloses a deep-sea mineral identification method and system based on deep learning, which comprises collecting data, shooting a video of underwater mineral distribution and recording geographical position coordinates of the mineral distribution; processing data, reading the collected video data of underwater mineral distribution according to the time length of one frame, screening out pictures related to the mineral distribution from the video data as a data set; dividing the data set into a training set, a test set and a verification set; deep learning training, putting the pictures of the training set into a deep learning network model for training to obtain a trained deep learning network model; deep learning verification, putting the pictures of the verification set into the trained deep learning network model for verification until the verification is qualified; outputting the pictures of the verification set in the form of a heat map; mining vehicle operation, according to the geographical position coordinates of the mineral, the mining vehicle operates, and the heat map is output to guide the mining vehicle to reach the mineral distribution area. The application can realize high-precision real-time identification of seabed mineral distribution.

Description

Technical Field

[0001] This invention relates to the field of deep-sea mineral identification, and in particular to a deep learning-based method and system for deep-sea mineral identification. Background Technology

[0002] Polymetallic nodules are mineral resources located at depths of over 4,000 meters below the seabed, and commercial exploitation of deep-sea mineral resources is still in its early stages worldwide. Deep-sea mining is very costly, and most current development processes simply exchange information between the exploration and mining stages, failing to deeply integrate exploration and mining, which significantly reduces mining efficiency and increases mining costs.

[0003] Existing technology (CN107967452A) discloses a video-based method for identifying the distribution of deep-sea minerals, including the following steps: acquiring seabed geographic information; dividing the sea area according to the seabed geographic information to obtain multiple sea areas to be identified, and establishing corresponding mineral characterization libraries for each sea area; capturing seabed video of the sea area to be identified and recording the corresponding coordinate and orientation data; extracting mineral characterization images from the seabed video and comparing the mineral characterization images with the mineral characterization information; when the similarity between the mineral characterization images and the mineral characterization information reaches a similarity threshold, identifying the target characterization object and obtaining the corresponding coordinate and orientation data; determining the distribution of the target characterization object in the sea area to be identified based on the obtained coordinate and orientation data, thereby preliminarily determining the distribution of seabed minerals. The aforementioned existing technology requires a large number of seabed mineral images to establish the mineral characterization library, typically tens of thousands of images for at least one mineral, which is time-consuming and inefficient.

[0004] Therefore, the above problems urgently need to be solved. Summary of the Invention

[0005] Purpose of the invention: The first objective of this invention is to provide a deep learning-based method for identifying deep-sea minerals. During the exploration phase, a small dataset can be collected and trained using deep learning. The system outputs a heat map showing the location of minerals and the probability of mineral distribution at that location. This facilitates high-precision real-time identification of seabed mineral distribution during the mining phase, enabling intelligent and visual deep-sea mineral development.

[0006] The second objective of this invention is to provide a deep-sea mineral identification system based on deep learning.

[0007] Technical Solution: To achieve the above objectives, this invention discloses a deep-sea mineral identification system and method based on deep learning, comprising the following steps:

[0008] (1) Collect data and prepare an underwater exploration AUV vehicle. Install an underwater camera at each of the four corners of the vehicle to capture videos of the underwater mineral distribution. At the same time, install a Beidou satellite positioning system and a long baseline positioning sensor inside the AUV vehicle to record the geographical coordinates of the mineral distribution. Install lighting equipment above each underwater camera.

[0009] (2) Data processing: The video data of underwater mineral distribution collected is read in one-frame time length, generating fewer than 3000 and more than 500 images. Images related to mineral distribution are selected from the images and used as the training dataset. The outer bounding boxes of seabed minerals are labeled using the labelme tool as detection boxes for identifying mineral distribution. Category labels are added to each labeled detection box. The dataset is divided into training set, test set and validation set. The training set, test set and validation set are independent of each other and cannot have duplicate images.

[0010] (3) Deep learning training: The images in the training set are put into the deep learning network model for training. The deep learning network model is trained using the YOLOv8 network model. The number of training iterations is set. After training, the loss function change graph and the training accuracy change graph will be output. When the curves of the output loss function and training accuracy converge, it indicates that the training result is stable. Training ends and the trained deep learning network model is obtained.

[0011] (4) Deep learning validation: The images in the validation set are put into the trained deep learning network model for validation. The mean precision mAP, mean precision AP, frames per second (FPS), accuracy P and recall R are calculated. When the mean precision mAP is greater than 0.9, the mean precision AP is greater than 0.9, the frames per second (FPS) is greater than 120, the accuracy P is greater than 0.8, and the recall R is greater than 0.8, it indicates that the deep learning network model is validated.

[0012] When the mean precision mAP is less than 0.9, the mean precision AP is less than 0.9, the number of frames transmitted per second (FPS) is less than 120, the accuracy P is less than 0.8, and the recall R is less than 0.8, return to step (3), increase the number of training iterations, and perform deep learning training again to obtain the iterated deep learning training model until it is verified as qualified.

[0013] (5) Output the images of the validation set as heatmaps using the GradCam method to generate activation class heatmaps. First, in the deep learning network model, place the images of the validation set and the program that generates the heatmaps in the same path. Then, in the deep learning network model, set the path of the weight file and the parameters of the GradCam method. Finally, output the heatmap of mineral distribution.

[0014] (6) Mining vehicle operation: Based on the mineral geographical coordinates stored in the Beidou satellite positioning system in step (1), determine the mineral distribution range; through the Beidou satellite positioning system, deploy the mining vehicle at the center point of the given range; control the mining vehicle operation through wireless transmission, and output a heat map to guide the mining vehicle to the mineral distribution area.

[0015] In step (2), the image selection criteria are as follows: images with more than 3 minerals are retained, images with fewer than 3 minerals are removed, and images with a resolution higher than 640*640 are retained.

[0016] Preferably, the loss function used in step (3) is the cross-entropy loss function, and the formula for the cross-entropy loss function is as follows:

[0017]

[0018] Where x represents a sample, ∑ represents the summation over sample x, y represents the label, a represents the predicted output, Ina represents the logarithmic value with base e and argument a, n represents the total number of samples, and C represents the cross-entropy loss function value.

[0019] Furthermore, in step (4), the mean accuracy mAP represents the average accuracy of target recognition for all categories, and the calculation formula is:

[0020]

[0021] Average precision (AP) represents the accuracy of target recognition for a certain category, and is calculated using the following formula:

[0022]

[0023] P represents accuracy, and dr represents the derivative of P with respect to r. This indicates that the value of r ranges from 0 to 1;

[0024] Accuracy P: represents the proportion of samples predicted as positive that are actually positive. The formula is:

[0025] P = TP / (TP + FP)

[0026] Recall R: represents the proportion of true positive samples that are identified, and is calculated using the following formula:

[0027] R = TP / (TP + FN)

[0028] Where TP represents the number of true positive samples, FP represents the number of negative samples that were misclassified as positive samples, and FN represents the number of unidentified positive samples;

[0029] Frames per second (FPS) is used to evaluate the speed of object recognition, that is, the number of images that can be processed per second or the time required to process one image. The formula is:

[0030]

[0031] The time (pre-process + inference + NMS) represents the recognition time of the YOLOv8 network model. The recognition time of the YOLOv8 network model is divided into three parts: image pre-process, inference, and non-maximum suppression (NMS).

[0032] Furthermore, in step (5), the GradCam method is a gradient-based activation class heatmap. The calculation process is as follows: First, the gradient of the model score for class c with respect to a certain convolutional layer is calculated. At the same time, based on the gradient information obtained in the above process, the pixel values ​​are averaged in each channel dimension to obtain the neuron importance weights. The calculation formula is as follows:

[0033]

[0034] Where Z represents the number of pixels in the feature map, A ij y represents the pixel value at position i,j in the k-th feature map. c This represents the model score for category c. This represents the neuron importance weight of the k-th feature map in the c-class model. Indicates y c right Partial derivatives;

[0035] Then, the features of the selected convolutional layer are weighted using the neuron importance weights obtained above; finally, the coarse heatmap output after weighting is processed by ReLU (activation function) to obtain the heatmap, calculated as follows:

[0036]

[0037] Where A k Represents the pixel value of the k-th feature map, and ReLU represents an activation function.

[0038] This invention discloses a deep-sea mineral identification system based on deep learning, comprising:

[0039] The data acquisition module is used to capture videos of underwater mineral distribution and record the geographical coordinates of the mineral distribution.

[0040] The data processing module reads the collected underwater mineral distribution video data frame by frame, generating fewer than 3000 and more than 500 images. It then selects images relevant to mineral distribution from these images to use as the training dataset. The module uses the labelme tool to annotate the outer bounding boxes of seabed minerals, which serve as detection boxes for identifying mineral distribution. Each annotated detection box is labeled with a category label. The dataset is then divided into training, testing, and validation sets, which are independent of each other and cannot contain duplicate images.

[0041] The deep learning training module is used to feed the images in the training set into the deep learning network model for training. The deep learning network model is trained using the YOLOv8 network model. The number of training iterations is set. After training, the loss function change graph and the training accuracy change graph will be output. When the curves of the output loss function and training accuracy converge, it indicates that the training result is stable, the training ends, and the trained deep learning network model is obtained.

[0042] The deep learning validation module is used to feed the images from the validation set into the trained deep learning network model for validation. It calculates the mean precision (mAP), mean accuracy (AP), frames per second (FPS), precision (P), and recall (R). When the mean precision (mAP) is greater than 0.9, the mean accuracy (AP) is greater than 0.9, the frames per second (FPS) is greater than 120, the precision (P) is greater than 0.8, and the recall (R) is greater than 0.8, it indicates that the deep learning network model has passed the validation.

[0043] When the mean precision mAP is less than 0.9, the mean precision AP is less than 0.9, the number of frames transmitted per second (FPS) is less than 120, the accuracy P is less than 0.8, and the recall R is less than 0.8, return to step (3), increase the number of training iterations, and perform deep learning training again to obtain the iterated deep learning training model until it is verified as qualified.

[0044] The output heatmap module is used to output the images of the validation set as heatmaps, using the GradCam method to generate activation class heatmaps. First, in the deep learning network model, the images of the validation set and the program that generates the heatmaps are placed in the same path. Then, in the deep learning network model, the path of the weight file and the parameters of the GradCam method are set. Finally, the heatmap of mineral distribution is output.

[0045] The mining operation module is used to determine the mineral distribution range based on the stored mineral geographical coordinates; to deploy mining vehicles at the center point of the given range via the Beidou satellite positioning system; and to control the operation of the mining vehicles via wireless transmission, outputting heat maps to guide the mining vehicles to the mineral distribution area.

[0046] The image selection criteria in the data processing module are as follows: images with more than 3 minerals are retained, images with fewer than 3 minerals are removed, and images with a resolution higher than 640*640 are retained.

[0047] Preferably, the loss function used in the deep learning training module is the cross-entropy loss function, the formula of which is as follows:

[0048]

[0049] Where x represents a sample, ∑ represents the summation over sample x, y represents the label, a represents the predicted output, Ina represents the logarithmic value with base e and argument a, n represents the total number of samples, and C represents the cross-entropy loss function value.

[0050] Furthermore, in the deep learning validation module, the mean accuracy (mAP) represents the average accuracy of target recognition across all categories, and its calculation formula is:

[0051]

[0052] Average precision (AP) represents the accuracy of target recognition for a certain category, and is calculated using the following formula:

[0053]

[0054] P represents accuracy, and dr represents the derivative of P with respect to r. This indicates that the value of r ranges from 0 to 1;

[0055] Accuracy P: represents the proportion of samples predicted as positive that are actually positive. The formula is:

[0056] P = TP / (TP + FP)

[0057] Recall R: represents the proportion of true positive samples that are identified, and is calculated using the following formula:

[0058] R = TP / (TP + FN)

[0059] Where TP represents the number of true positive samples, FP represents the number of negative samples that were misclassified as positive samples, and FN represents the number of unidentified positive samples;

[0060] Frames per second (FPS) is used to evaluate the speed of object recognition, that is, the number of images that can be processed per second or the time required to process one image. The formula is:

[0061]

[0062] The time (pre-process + inference + NMS) represents the recognition time of the YOLOv8 network model. The recognition time of the YOLOv8 network model is divided into three parts: image pre-process, inference, and non-maximum suppression (NMS).

[0063] Furthermore, the GradCam method in the output heatmap module is a gradient-based activation class heatmap. The calculation process is as follows: First, the gradient of the model score for class c with respect to a certain convolutional layer is calculated. Simultaneously, the gradient information obtained in the above process is averaged across the pixel values ​​in each channel dimension to obtain the neuron importance weights. The calculation formula is:

[0064]

[0065] Where Z represents the number of pixels in the feature map, A ij y represents the pixel value at position i,j in the k-th feature map. c This represents the model score for category c. This represents the neuron importance weight of the k-th feature map in the c-class model. Indicates y c right Partial derivatives;

[0066] Then, the features of the selected convolutional layer are weighted using the neuron importance weights obtained above; finally, the coarse heatmap output after weighting is processed by ReLU (activation function) to obtain the heatmap, calculated as follows:

[0067]

[0068] Where A k Represents the pixel value of the k-th feature map, and ReLU represents an activation function.

[0069] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: This invention installs lighting equipment above each underwater camera, improving image acquisition quality. Simultaneously, it filters the captured images, further improving the quality of the dataset and laying the foundation for subsequent deep learning network training and validation. The use of the labelme annotation tool improves annotation efficiency. This invention utilizes the YOLOv8 algorithm to increase the number of training iterations, ensuring training convergence and guaranteeing mAP greater than 0.9 and FPS greater than 120, achieving high-precision real-time identification of deep-sea mineral distribution. The training of this invention employs a cross-entropy loss function, converging to a value less than 1, ensuring the accuracy and robustness of the algorithm. Attached Figure Description

[0070] Figure 1This is a flowchart illustrating the overall scheme of the present invention;

[0071] Figure 2 This is a schematic diagram of the underwater exploration AUV vehicle in this invention;

[0072] Figure 3 This is a structural diagram of the deep learning algorithm used in this invention;

[0073] Figure 4 This is a graph showing the changes in the loss function and training accuracy during deep learning training in this invention.

[0074] Figure 5 This is a grayscale image of the heatmap output from the test set in this invention. Detailed Implementation

[0075] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0076] Example 1

[0077] like Figure 1 As shown, this invention discloses a deep learning-based deep-sea mineral identification system and method. 1. A deep learning-based deep-sea mineral identification method, characterized by comprising the following steps:

[0078] (1) Data collection: Prepare an underwater exploration AUV vehicle. Install an underwater camera at each of the four corners of the vehicle. Simultaneously, equip the AUV with a BeiDou satellite positioning system and a long-baseline positioning sensor to record the geographical coordinates of mineral distribution. Install lighting equipment above each underwater camera, such as... Figure 2 As shown; the underwater camera is responsible for capturing videos of the underwater mineral distribution, which are used to create the dataset in step S2; the purpose of installing the Beidou positioning system is to record the geographical coordinates of the minerals, so that the mining vehicle can quickly reach the mineral area during mining operations in step S6;

[0079] Deep-sea minerals (polymetallic nodules) are mainly distributed at depths of 4000 to 6000 meters. Example 1 uses the "Qianlong-1" 6000-meter Autonomous Underwater Robot (AUV), a pressure-resistant hull capable of withstanding 62 MPa of pressure, suitable for operations at depths of 6000 meters. A recess is created on the exterior of the AUV to house the underwater camera; this recess reduces the impact of water currents and improves shooting stability. The underwater camera used is a Kongsberg OE14-408, characterized by its anti-disturbance properties, long endurance, high resolution, and large memory capacity. It prevents motion blur during AUV movement, is well-suited for large-scale, long-duration, and complex shooting tasks during a single exploration, and its high-resolution images improve the quality of dataset production. Lighting is installed above the underwater camera because the deep-sea exploration environment is almost completely dark, requiring adequate lighting for the camera. The AUV is equipped with a BeiDou satellite positioning system and long-baseline positioning sensors to record the geographical coordinates of mineral distribution. The mother ship's terminal system controls the AUV's operation via wireless communication. The AUV can be deployed and retrieved at fixed points in a fixed sea area by the mother ship. The sea state for deployment and retrieval can reach level 4 (level 4 sea state is moderate waves with a wave height of 1.25m-2.5m; theoretically, deployment and retrieval are prohibited in sea states above level 4).

[0080] (2) Data processing: The collected underwater mineral distribution video data is uploaded to the terminal system of the mother ship. The video is read in one frame time length, and the number of generated images is less than 3000 and more than 500. Images related to mineral distribution are selected from the images. The image selection criteria are: images with more than 3 minerals are retained, images with less than 3 minerals are removed, and images with a resolution higher than 640*640 are retained as training datasets. The outer bounding boxes of seabed minerals are labeled with the labeling tool labelme as detection boxes for identifying mineral distribution. A category label is added to each labeled detection box. The dataset is divided into training set, test set and validation set. The three datasets are independent of each other and there should be no duplicate images.

[0081] (3) Deep learning training: The images in the training set are fed into the deep learning network model for training; the YOLOv8 network model is used for training, and the structure diagram of the algorithm is shown below. Figure 3As shown, the YOLOv8 network model mainly consists of three parts: Backbone, Neck, and Head. The Backbone is the feature extraction network, responsible for extracting information from the image for use by subsequent networks. The Neck, placed between the Backbone and Head, is used to more effectively utilize the features extracted by the Backbone, acting as a feature fusion layer. The Head extracts features from previous layers and performs recognition. The Backbone consists of three modules: CBS, C2f, and SPPF. CBS consists of a two-dimensional convolution (Conv2d), a two-dimensional batch processing layer (BatchNorm2d, BN), and a SiLU activation function. C2f consists of two CBS convolutional layers, a split layer, several Bottleneck layers, and a concat layer. The split layer halves the channels of the feature map, and the feature maps before and after the bottleneck are concatenated using residual connections. The n bottlenecks are sequentially connected, and each bottleneck is concatenated with the features from the last bottleneck. The SPPF module first introduces a CBS convolutional layer, followed by three consecutive max-pooling layers. The feature maps obtained before and after each max-pooling operation are concatenated to achieve feature fusion. The Neck network has four concatenation operations, concatenating the original feature maps with the results obtained after the convolution operation. This method achieves network lightweighting. The Neck network also borrows the idea from the Feature Pyramid Network (FPN), performing upsampling followed by downsampling, with two cross-layer fusion connections between the upsampling and downsampling branches. The Head network first branches into two CBS convolutional modules, then passes them through a 2D convolutional layer (Conv2d), and finally calculates the classification loss and detection loss (bbox loss) separately. The Head network uses the currently mainstream decoupled-head structure, separating the classification and detection heads.

[0082] The training iterations are set to 100. After training, the loss function and training accuracy changes will be output, as shown below. Figure 4 As shown, lower loss function values ​​and higher training accuracy indicate better training results. When the curves representing the output loss function and training accuracy converge, it indicates that the training results have stabilized, and training can be terminated. Figure 4As can be seen, when the number of iterations is set to 100, the training loss function curve converges to a value less than 1, and the accuracy of the training process eventually converges to a value close to 100%, indicating that the algorithm has good robustness. The loss function used is the cross-entropy loss function, which measures the closeness between the model's predicted values ​​and the true values, directly reflecting the model's performance. The formula for the cross-entropy loss function is as follows:

[0083]

[0084] Where x represents a sample, Σ represents the summation over sample x, y represents the label, a represents the predicted output, represents the logarithm of base e and argument a, n represents the total number of samples, and C represents the cross-entropy loss function value. The smaller the value of C, the better the model performance.

[0085] (4) Deep learning validation: Images from the validation set are fed into a deep learning network for validation. The validation results can be used as a measure of the algorithm's performance in practical applications. In practical applications, deep-sea mineral identification must consider both accuracy and speed. High accuracy and high speed can effectively improve work efficiency and reduce mining costs. mAP represents recognition accuracy; the higher the mAP value, the higher the accuracy. FPS represents recognition speed; the higher the FPS value, the faster the speed. When the FPS value is greater than 60, real-time identification can be achieved. The validation set has an mAP of 0.988 and an FPS of 188, indicating that the YOLOv8 algorithm can complete the high-precision real-time identification task of seabed minerals. When the mean accuracy mAP is greater than 0.9, the mean accuracy AP is greater than 0.9, the FPS is greater than 120, the accuracy P is greater than 0.8, and the recall R is greater than 0.8, it indicates that the YOLOv8 algorithm can complete the high-precision real-time identification task of seabed minerals. In deep learning target identification tasks, mAP and FPS are the two most important evaluation metrics.

[0086] The mean accuracy (mAP) represents the average accuracy of target recognition across all categories, and is calculated using the following formula:

[0087]

[0088] Average precision (AP) represents the accuracy of target recognition for a certain category, and is calculated using the following formula:

[0089]

[0090] P represents accuracy, and dr represents the derivative of P with respect to r. This indicates that the value of r ranges from 0 to 1;

[0091] Accuracy P: represents the proportion of samples predicted as positive that are actually positive. The formula is:

[0092] P = TP / (TP + FP) (4)

[0093] Recall R: represents the proportion of true positive samples that are identified, and is calculated using the following formula:

[0094] R = TP / (TP + FN) (5)

[0095] Where TP represents the number of true positive samples, FP represents the number of negative samples that were misclassified as positive samples, and FN represents the number of unidentified positive samples;

[0096] Frames per second (FPS) is used to evaluate the speed of object recognition, that is, the number of images that can be processed per second or the time required to process one image. The formula is:

[0097]

[0098] time(pre-process+inference+NMS) represents the recognition time of the YOLOv8 algorithm. The recognition time of the YOLOv8 algorithm is divided into three parts: image pre-process, inference, and non-maximum suppression (NMS).

[0099] When the mean precision mAP is less than 0.9, the mean precision AP is less than 0.9, the FPS is less than 120, the accuracy P is less than 0.8, and the recall R is less than 0.8, return to step (3), set the number of iterations to 150, and then perform deep learning training to obtain the iterated deep learning training model.

[0100] (5) Output the images of the validation set as heatmaps. In deep learning, heatmaps help to understand which part of an image led the neural network to make the final classification decision, and can intuitively reflect the location of the target. The activation class heatmap is generated using the GradCam method. First, in the YOLOv8 algorithm, the images of the validation set and the program for generating the heatmap are placed in the same path. Then, in the YOLOv8 algorithm program, the path of the weight file and the parameters of the GradCam method are set. The weight file represents the best weights obtained during the training process of the deep learning network. Methods for generating activation class heatmaps include GradCamPlus, XGradCam, and GradCam. In actual testing, GradCam performed the best. Finally, running the program outputs a heatmap of mineral distribution. In the heatmap, warm colors represent large values, and cool colors represent small values. In the output heatmap, warm-toned areas indicate a high probability of mineral distribution, while cool-toned areas indicate a low probability of mineral distribution. In practical applications, mining vehicles can directly select warm-toned areas in the heatmap for mining based on the warmth or coolness of the heatmap, effectively improving operational efficiency. In this embodiment, to facilitate demonstration of its effect, the heatmap is converted into its corresponding grayscale image. In the grayscale image corresponding to the heatmap, the brighter the color of a region, the higher the probability that minerals are distributed in that region. The heatmap displays the borders and probabilities of the identified minerals; the borders help determine the location of the minerals, and the probability values ​​represent the likelihood that minerals are distributed within the bordered area.

[0101] GradCam is a gradient-based activation class heatmap. The calculation process is as follows: First, the gradient of the model score for class c with respect to a certain convolutional layer is calculated. Simultaneously, the gradient information obtained in the above process is averaged across the pixel values ​​in each channel dimension to obtain the neuron importance weights, as shown in Equation 7. Here, Z represents the number of pixels in the feature map, and Aij represents the pixel value at position i,j in the k-th feature map. Then, the neuron importance weights obtained above are used to weight the features of the selected convolutional layer. Finally, the weighted coarse heatmap output is processed through the ReLU activation function to obtain the final heatmap, as shown in Equation 8.

[0102]

[0103] Where Z represents the number of pixels in the feature map, A ij y represents the pixel value at position i,j in the k-th feature map. c This represents the model score for category c. This represents the neuron importance weight of the k-th feature map in the c-class model. Indicates y c right Partial derivatives;

[0104]

[0105] Where A k Represents the pixel value of the k-th feature map, and ReLU represents an activation function.

[0106] (6) In actual operation of the mining vehicle, the mineral distribution range is roughly determined based on the mineral geographical coordinates stored in the positioning system of step (1). The algorithm is deployed to the mother ship terminal control system, and the mother ship deploys the mining vehicle at the center point of the given range via the Beidou satellite positioning system. The mother ship controls the operation of the mining vehicle through wireless transmission, and the terminal system outputs a heat map to guide the mining vehicle directly to the mineral distribution area. Multiple cameras are installed around the mining vehicle to take videos of each mining process, which are then wirelessly transmitted to the mother ship terminal system and fed into the deep learning algorithm for further training and optimization, thereby further improving speed and accuracy.

[0107] Example 2

[0108] This invention discloses a deep learning-based deep-sea mineral identification system, comprising:

[0109] The data acquisition module is used to capture videos of underwater mineral distribution and record the geographical coordinates of the mineral distribution.

[0110] The data processing module reads the collected underwater mineral distribution video data frame by frame, generating fewer than 3000 and more than 500 images. It then selects images relevant to mineral distribution from these images to use as the training dataset. The module uses the labelme tool to annotate the outer bounding boxes of seabed minerals, which serve as detection boxes for identifying mineral distribution. Each annotated detection box is labeled with a category label. The dataset is then divided into training, testing, and validation sets, which are independent of each other and cannot contain duplicate images.

[0111] The image selection criteria in the data processing module are as follows: images with more than 3 minerals are retained, images with fewer than 3 minerals are removed, and images with a resolution higher than 640*640 are retained.

[0112] The deep learning training module is used to feed the images in the training set into the deep learning network model for training. The deep learning network model is trained using the YOLOv8 network model. The number of training iterations is set. After training, the loss function change graph and the training accuracy change graph will be output. When the curves of the output loss function and training accuracy converge, it indicates that the training result is stable, the training ends, and the trained deep learning network model is obtained.

[0113] The loss function used in the deep learning training module is the cross-entropy loss function, and the formula for the cross-entropy loss function is as follows:

[0114]

[0115] Where x represents a sample, ∑ represents the summation over sample x, y represents the label, a represents the predicted output, Ina represents the logarithmic value with base e and argument a, n represents the total number of samples, and C represents the cross-entropy loss function value.

[0116] The deep learning validation module is used to feed the images from the validation set into the trained deep learning network model for validation. It calculates the mean precision (mAP), mean accuracy (AP), frames per second (FPS), precision (P), and recall (R). When the mean precision (mAP) is greater than 0.9, the mean accuracy (AP) is greater than 0.9, the frames per second (FPS) is greater than 120, the precision (P) is greater than 0.8, and the recall (R) is greater than 0.8, it indicates that the deep learning network model has passed the validation.

[0117] When the mean precision mAP is less than 0.9, the mean precision AP is less than 0.9, the number of frames transmitted per second (FPS) is less than 120, the accuracy P is less than 0.8, and the recall R is less than 0.8, return to step (3), increase the number of training iterations, and perform deep learning training again to obtain the iterated deep learning training model until it is verified as qualified.

[0118] In the deep learning validation module, the mean accuracy (mAP) represents the average accuracy of target recognition across all categories. The calculation formula is:

[0119]

[0120] Average precision (AP) represents the accuracy of target recognition for a certain category, and is calculated using the following formula:

[0121]

[0122] P represents accuracy, and dr represents the derivative of P with respect to r. This indicates that the value of r ranges from 0 to 1;

[0123] Accuracy P: represents the proportion of samples predicted as positive that are actually positive. The formula is:

[0124] P = TP / (TP + FP)

[0125] Recall R: represents the proportion of true positive samples that are identified, and is calculated using the following formula:

[0126] R = TP / (TP + FN)

[0127] Where TP represents the number of true positive samples, FP represents the number of negative samples that were misclassified as positive samples, and FN represents the number of unidentified positive samples;

[0128] Frames per second (FPS) is used to evaluate the speed of object recognition, that is, the number of images that can be processed per second or the time required to process one image. The formula is:

[0129]

[0130] The time (pre-process + inference + NMS) represents the recognition time of the YOLOv8 network model. The recognition time of the YOLOv8 network model is divided into three parts: image pre-process, inference, and non-maximum suppression (NMS).

[0131] The output heatmap module is used to output the images of the validation set as heatmaps, using the GradCam method to generate activation class heatmaps. First, in the deep learning network model, the images of the validation set and the program that generates the heatmaps are placed in the same path. Then, in the deep learning network model, the path of the weight file and the parameters of the GradCam method are set. Finally, the heatmap of mineral distribution is output.

[0132] The GradCam method in the output heatmap module is a gradient-based activation class heatmap. The calculation process is as follows: First, the gradient of the model score for class c with respect to a certain convolutional layer is calculated. Then, the gradient information obtained in the above process is averaged across the pixel values ​​in each channel dimension to obtain the neuron importance weights. The calculation formula is:

[0133]

[0134] Where Z represents the number of pixels in the feature map, A ij y represents the pixel value at position i,j in the k-th feature map. c This represents the model score for category c. This represents the neuron importance weight of the k-th feature map in the c-class model. Indicates y c right Partial derivatives;

[0135] Then, the features of the selected convolutional layer are weighted using the neuron importance weights obtained above; finally, the coarse heatmap output after weighting is processed by ReLU (activation function) to obtain the heatmap, calculated as follows:

[0136]

[0137] Where A k Represents the pixel value of the k-th feature map, and ReLU represents an activation function.

[0138] The mining operation module is used to determine the mineral distribution range based on the stored mineral geographical coordinates; to deploy mining vehicles at the center point of the given range via the Beidou satellite positioning system; and to control the operation of the mining vehicles via wireless transmission, outputting heat maps to guide the mining vehicles to the mineral distribution area.

[0139] The preferred embodiments of the present invention have been described in detail above, but the design concept of the present invention is not limited thereto. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solution of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A deep learning-based method for identifying deep-sea minerals, characterized in that, Includes the following steps: (1) Collect data and prepare an underwater exploration AUV vehicle. Install an underwater camera at each of the four corners of the vehicle to capture videos of the underwater mineral distribution. At the same time, install a Beidou satellite positioning system and a long baseline positioning sensor inside the AUV vehicle to record the geographical coordinates of the mineral distribution. Install lighting equipment above each underwater camera. (2) Data processing: The video data of underwater mineral distribution collected is read in one-frame time length, generating fewer than 3000 and more than 500 images. Images related to mineral distribution are selected from the images and used as the training dataset. The outer bounding boxes of seabed minerals are labeled using the labelme tool as detection boxes for identifying mineral distribution. Category labels are added to each labeled detection box. The dataset is divided into training set, test set and validation set. The training set, test set and validation set are independent of each other and cannot have duplicate images. (3) Deep learning training: The images in the training set are put into the deep learning network model for training. The deep learning network model is trained using the YOLOv8 network model. The number of training iterations is set. After training, the loss function change graph and the training accuracy change graph will be output. When the curves of the output loss function and training accuracy converge, it indicates that the training result is stable. Training ends and the trained deep learning network model is obtained. (4) Deep learning validation: The images in the validation set are put into the trained deep learning network model for validation. The mean precision mAP, mean precision AP, frames per second (FPS), accuracy P and recall R are calculated. When the mean precision mAP is greater than 0.9, the mean precision AP is greater than 0.9, the frames per second (FPS) is greater than 120, the accuracy P is greater than 0.8, and the recall R is greater than 0.8, it indicates that the deep learning network model is validated. When the mean precision mAP is less than 0.9, the mean precision AP is less than 0.9, the number of frames transmitted per second (FPS) is less than 120, the accuracy P is less than 0.8, and the recall R is less than 0.8, return to step (3), increase the number of training iterations, and perform deep learning training again to obtain the iterated deep learning training model until it is verified as qualified. (5) Output the images of the validation set as heatmaps using the GradCam method to generate activation class heatmaps. First, in the deep learning network model, place the images of the validation set and the program that generates the heatmaps in the same path. Then, in the deep learning network model, set the path of the weight file and the parameters of the GradCam method. Finally, output the heatmap of mineral distribution. (6) Mining vehicle operation: Based on the mineral geographical coordinates stored in the Beidou satellite positioning system in step (1), determine the mineral distribution range; through the Beidou satellite positioning system, deploy the mining vehicle at the center point of the given range; control the mining vehicle operation through wireless transmission, and output a heat map to guide the mining vehicle to the mineral distribution area.

2. The deep learning-based deep-sea mineral identification method according to claim 1, characterized in that: The image selection criteria in step (2) are as follows: images with more than 3 minerals are retained, images with fewer than 3 minerals are removed, and images with a resolution higher than 640*640 are retained.

3. The deep learning-based deep-sea mineral identification method according to claim 2, characterized in that: The loss function used in step (3) is the cross-entropy loss function, and the formula for the cross-entropy loss function is as follows: Where x represents a sample, ∑ represents the summation over sample x, y represents the label, a represents the predicted output, Ina represents the logarithmic value with base e and argument a, n represents the total number of samples, and C represents the cross-entropy loss function value.

4. The deep learning-based deep-sea mineral identification method according to claim 3, characterized in that: In step (4), the mean accuracy mAP represents the average accuracy of target recognition for all categories, and the calculation formula is: Average precision (AP) represents the accuracy of target recognition for a certain category, and is calculated using the following formula: P represents accuracy, and dr represents the derivative of P with respect to r. This indicates that the value of r ranges from 0 to 1; Accuracy P: represents the proportion of samples predicted as positive that are actually positive. The formula is: P = TP / (TP + FP) Recall R: represents the proportion of true positive samples that are identified, and is calculated using the following formula: R = TP / (TP + FN) Where TP represents the number of true positive samples, FP represents the number of negative samples that were misclassified as positive samples, and FN represents the number of unidentified positive samples; Frames per second (FPS) is used to evaluate the speed of object recognition, that is, the number of images that can be processed per second or the time required to process one image. The formula is: The time (pre-process + inference + NMS) represents the recognition time of the YOLOv8 network model. The recognition time of the YOLOv8 network model is divided into three parts: image pre-process, inference, and non-maximum suppression (NMS).

5. The deep learning-based deep-sea mineral identification method according to claim 4, characterized in that: In step (5), the GradCam method is a gradient-based activation class heatmap. The calculation process is as follows: First, the gradient of the model score of class c with respect to a certain convolutional layer is calculated. At the same time, based on the gradient information obtained in the above process, the pixel values ​​are averaged in each channel dimension to obtain the neuron importance weights. The calculation formula is as follows: Where Z represents the number of pixels in the feature map, A ij y represents the pixel value at position i,j in the k-th feature map. c This represents the model score for category c. This represents the neuron importance weight of the k-th feature map in the c-class model. Indicates y c right Partial derivatives; Then, the features of the selected convolutional layer are weighted using the neuron importance weights obtained above; finally, the coarse heatmap output after weighting is processed by ReLU (activation function) to obtain the heatmap, calculated as follows: Where A k Represents the pixel value of the k-th feature map, and ReLU represents an activation function.

6. A deep learning-based deep-sea mineral identification system, characterized in that, include: The data acquisition module is used to capture videos of underwater mineral distribution and record the geographical coordinates of the mineral distribution. The data processing module reads the collected underwater mineral distribution video data frame by frame, generating fewer than 3000 and more than 500 images. It then selects images relevant to mineral distribution from these images to use as the training dataset. The module uses the labelme tool to annotate the outer bounding boxes of seabed minerals, which serve as detection boxes for identifying mineral distribution. Each annotated detection box is labeled with a category label. The dataset is then divided into training, testing, and validation sets, which are independent of each other and cannot contain duplicate images. The deep learning training module is used to feed the images in the training set into the deep learning network model for training. The deep learning network model is trained using the YOLOv8 network model. The number of training iterations is set. After training, the loss function change graph and the training accuracy change graph will be output. When the curves of the output loss function and training accuracy converge, it indicates that the training result is stable, the training ends, and the trained deep learning network model is obtained. The deep learning validation module is used to feed the images from the validation set into the trained deep learning network model for validation. It calculates the mean precision (mAP), mean accuracy (AP), frames per second (FPS), precision (P), and recall (R). When the mean precision (mAP) is greater than 0.9, the mean accuracy (AP) is greater than 0.9, the frames per second (FPS) is greater than 120, the precision (P) is greater than 0.8, and the recall (R) is greater than 0.8, it indicates that the deep learning network model has passed the validation. When the mean precision mAP is less than 0.9, the mean precision AP is less than 0.9, the number of frames transmitted per second (FPS) is less than 120, the accuracy P is less than 0.8, and the recall R is less than 0.8, return to step (3), increase the number of training iterations, and perform deep learning training again to obtain the iterated deep learning training model until it is verified as qualified. The output heatmap module is used to output the images of the validation set as heatmaps, using the GradCam method to generate activation class heatmaps. First, in the deep learning network model, the images of the validation set and the program that generates the heatmaps are placed in the same path. Then, in the deep learning network model, the path of the weight file and the parameters of the GradCam method are set. Finally, the heatmap of mineral distribution is output. The mining operation module is used to determine the mineral distribution range based on the stored mineral geographical coordinates; to deploy mining vehicles at the center point of the given range via the Beidou satellite positioning system; and to control the operation of the mining vehicles via wireless transmission, outputting heat maps to guide the mining vehicles to the mineral distribution area.

7. A deep-learning-based deep-sea mineral identification system according to claim 6, characterized in that: The image selection criteria in the data processing module are as follows: images with more than 3 minerals are retained, images with fewer than 3 minerals are removed, and images with a resolution higher than 640*640 are retained.

8. A deep-learning-based deep-sea mineral identification system according to claim 7, characterized in that: The loss function used in the deep learning training module is the cross-entropy loss function, and the formula for the cross-entropy loss function is as follows: Where x represents a sample, ∑ represents the summation over sample x, y represents the label, a represents the predicted output, Ina represents the logarithmic value with base e and argument a, n represents the total number of samples, and C represents the cross-entropy loss function value.

9. A deep-learning-based deep-sea mineral identification system according to claim 8, characterized in that: In the deep learning verification module, the mean accuracy (mAP) represents the average accuracy of target recognition across all categories, and its calculation formula is as follows: Average precision (AP) represents the accuracy of target recognition for a certain category, and is calculated using the following formula: P represents accuracy, and dr represents the derivative of P with respect to r. This indicates that the value of r ranges from 0 to 1; Accuracy P: represents the proportion of samples predicted as positive that are actually positive. The formula is: P = TP / (TP + FP) Recall R: represents the proportion of true positive samples that are identified, and is calculated using the following formula: R = TP / (TP + FN) Where TP represents the number of true positive samples, FP represents the number of negative samples that were misclassified as positive samples, and FN represents the number of unidentified positive samples; Frames per second (FPS) is used to evaluate the speed of object recognition, that is, the number of images that can be processed per second or the time required to process one image. The formula is: The time (pre-process + inference + NMS) represents the recognition time of the YOLOv8 network model. The recognition time of the YOLOv8 network model is divided into three parts: image pre-process, inference, and non-maximum suppression (NMS).

10. A deep-learning-based deep-sea mineral identification system according to claim 9, characterized in that: The GradCam method in the output heatmap module is a gradient-based activation class heatmap. The calculation process is as follows: First, the gradient of the model score for class c with respect to a certain convolutional layer is calculated. Then, the gradient information obtained in the above process is averaged across the pixel values ​​in each channel dimension to obtain the neuron importance weights. The calculation formula is: Where Z represents the number of pixels in the feature map, A ij y represents the pixel value at position i,j in the k-th feature map. c This represents the model score for category c. This represents the neuron importance weight of the k-th feature map in the c-class model. Indicates y c right Partial derivatives; Then, the features of the selected convolutional layer are weighted using the neuron importance weights obtained above; Finally, the weighted coarse heatmap is processed by ReLU (activation function) to obtain the final heatmap, calculated using the following formula: Where A k Represents the pixel value of the k-th feature map, and ReLU represents an activation function.

Images