Zooplankton scanning imaging analysis method, medium and electronic device

Abstract

The present invention provides a zooplankton scanning image analysis method, medium and electronic device, belonging to the technical field of zooplankton scanning images, comprising: performing denoising and contrast enhancement preprocessing on underwater images; detecting zooplankton targets on the preprocessed images using a target detection algorithm to obtain a bounding box of each target; segmenting the detected zooplankton target instances according to the bounding box to separate the background; initializing a CUDA environment and creating multiple CUDA streams, and starting two layers of sub-thread grids in each stream: one layer for a posture calculation model and one layer for a target recognition model; dividing multiple target images into batches, and assigning each batch to different CUDA streams in parallel for processing. The processing results of each CUDA stream are merged to establish a temporary zooplankton data set. The data set is clustered and analyzed to count the number of zooplankton of various types. The technical problem of insufficient efficiency of detection and analysis in the prior art is solved.

Description

Technical Field

[0001] The invention belongs to the technical field of zooplankton scanning images, and in particular, relates to a zooplankton scanning imaging analysis method, a medium and an electronic device. Background Art

[0002] Underwater ecological environment monitoring is an important task in the fields of environmental protection, resource management, and marine scientific research. Among them, zooplankton is an important indicator organism in the aquatic ecosystem. Their species composition and quantity distribution can reflect key information such as the nutritional status and pollution level of the water body. Therefore, automated scanning imaging and quantity statistics of underwater zooplankton are of great significance for aquatic ecological environment monitoring.

[0003] Existing underwater zooplankton scanning and analysis technologies mainly include the following methods:

[0004] 1. Manual sampling and microscopic observation: This is the most traditional method for zooplankton detection. Through the steps of water sample collection, fixation, staining, etc., the researchers made the zooplankton samples into slides, and identified and counted them one by one under a microscope. Although this method can accurately obtain species and quantity information, it is labor-intensive, inefficient, and requires professional personnel to operate, making it difficult to achieve rapid and large-scale underwater ecological monitoring.

[0005] 2. Acoustic imaging method: Use the reflection characteristics of sound waves to image and detect underwater organisms. Echo signals are collected through underwater sonar equipment, and then the echo data is analyzed and interpreted using signal processing algorithms to infer the species and quantity information of underwater organisms. This method can achieve continuous and automatic underwater scanning, but due to the complexity of the reflection characteristics of sound waves, the recognition accuracy of small zooplankton is relatively low, and it is difficult to obtain detailed species composition data.

[0006] 3. Image processing and computer vision method: With the rapid development of digital imaging and computer vision technology in recent years, the use of underwater camera equipment to image and analyze zooplankton has become a new research direction. Through image preprocessing, target detection, feature extraction, classification and recognition, the species and quantity information of underwater zooplankton can be automatically obtained. This method has the advantages of simple sample collection, high analysis efficiency, and strong operability. However, due to the complex underwater environment, uneven lighting, cluttered background, target occlusion and other problems, the image quality of zooplankton is poor, which affects the accuracy of subsequent target detection and species recognition.

[0007] In general, the existing underwater zooplankton scanning and analysis technology has the technical problem of insufficient detection and analysis efficiency. Summary of the invention

[0008] In view of this, the present invention provides a zooplankton scanning and imaging analysis method, medium and electronic device, which can solve the technical problem of insufficient detection and analysis efficiency in the existing underwater zooplankton scanning and analysis technology.

[0009] The present invention is achieved in that:

[0010] A first aspect of the present invention provides a zooplankton scanning imaging analysis method, which comprises the following steps:

[0011] S10, preprocessing the underwater field of view image, including denoising and contrast enhancement, to improve image quality;

[0012] S20, performing zooplankton target detection on the processed image using a deep learning target detection algorithm to obtain a bounding box of each zooplankton target;

[0013] S30, performing instance segmentation on each detected zooplankton target according to the bounding box, separating the background, and obtaining multiple target images;

[0014] S40, initializing the CUDA environment, creating a sufficient number of CUDA streams, and starting a multi-layer sub-thread grid in each CUDA stream, including a first-layer sub-thread grid for running a pre-trained posture calculation model, and a second-layer sub-thread grid for running a pre-trained target recognition model;

[0015] S50, dividing the multiple target images into batches, wherein the number of images in each batch does not exceed the number of CUDA streams, and assigning the images in each batch to different CUDA streams in parallel for processing;

[0016] S60, in each CUDA stream, using the first layer of sub-thread grids to run the posture calculation model for each target image in parallel, calculate the posture of the zooplankton in each target image, and perform a plane rotation adjustment on the posture to obtain a posture-adjusted image;

[0017] S70, in each CUDA stream, using the second layer of sub-thread grids to run the target recognition model on each posture-adjusted image in parallel to identify the zooplankton species in each posture-adjusted image;

[0018] S80, merging the processing results of each CUDA stream, adjusting the zooplankton species corresponding to the image based on all the postures, and establishing a temporary zooplankton dataset, including the species, center coordinates, bounding box, and size of each zooplankton;

[0019] S90, using a clustering algorithm to perform cluster analysis on each record in the temporary zooplankton feature set to obtain statistical information on the number of various types of zooplankton in the underwater shooting field of view.

[0020] Among them, the posture calculation model is used to predict the 2D key point coordinates of zooplankton in the input image, and then calculate the posture information. The model architecture adopts a fully convolutional network structure based on a feature pyramid, which can simultaneously detect and locate instances of different scales.

[0021] The zooplankton target recognition model adopts a deep convolutional network structure based on feature pyramid and attention mechanism to identify the species of the input zooplankton images and give the category probability distribution of each target.

[0022] Furthermore, the zooplankton posture calculation model includes:

[0023] Pre-trained ResNet-50 backbone network;

[0024] Feature pyramid network, outputs feature pyramids at five scale levels P3-P7;

[0025] For each scale level Pi, an independent key point detection head is set to output the zooplankton key point prediction map;

[0026] The key point fusion module performs weighted fusion of key point prediction images at all scale levels.

[0027] Furthermore, the keypoint detection head contains 4 convolutional layers, outputting an N x N x K tensor, where N is the output feature map size and K is the number of zooplankton keypoints.

[0028] Furthermore, the zooplankton target recognition model includes:

[0029] Pre-trained ResNet-50 backbone network;

[0030] Feature pyramid network, outputs feature pyramids at five scale levels P3-P7;

[0031] Attention pyramid pooling module, which fuses feature responses of different scales;

[0032] The classification head includes a global average pooling layer, a category prediction branch, and a bounding box regression branch.

[0033] Furthermore, the attention pyramid pooling module performs spatial pyramid pooling of 1x1, 3x3, 5x5 and 7x7 scales on the P3-P7 feature layers, and cascades and splices feature responses of different scales.

[0034] Furthermore, the category prediction classification branch uses Softmax to output category probability distribution, and the boundary regression branch predicts the fine bounding box coordinates of the target location.

[0035] Among them, the step S10 specifically includes: pre-processing the field of view image taken underwater, including using the median filter algorithm to reduce the noise of the image to remove salt and pepper noise and other impulse noise, and improve the stability of subsequent image processing; secondly, using the histogram equalization algorithm to enhance the contrast of the image, by adjusting the image grayscale distribution to make it flat, stretching the image contrast, thereby highlighting the significant features of zooplankton in the image, and providing better input for subsequent target detection. For the parameter setting of the histogram equalization algorithm, the threshold can be set to 0.4, that is, when the cumulative probability of the image grayscale histogram exceeds 40%, saturation cutoff is performed to effectively prevent the loss of details caused by excessive enhancement.

[0036] Wherein, the step S20 uses the Mask R-CNN algorithm to detect zooplankton targets, and the algorithm can simultaneously perform target detection and instance segmentation. The main steps of the Mask R-CNN algorithm include: using ResNet-50 as the backbone network of the feature extraction network, and pre-training on the ImageNet data set; adding the extraction candidate box network k (RPN) module on the basis of the pre-trained feature extraction network to generate target candidate boxes, and two parallel classification and regression heads for target recognition and bounding box regression; using Focal Loss as the classification loss function, smooth L1 Loss as the bounding box regression loss function, and Binary Cross Entropy Loss as the mask segmentation loss function, and performing end-to-end joint training on the network; during the training process, using the marine life categories in the COCO data set as the training samples of zooplankton, using random scaling, flipping, color jittering and other data enhancement methods; in the inference stage, forward propagation is performed on the input image to obtain the bounding box, category probability and instance segmentation mask of each detected target.

[0037] Among them, the step S30 performs instance segmentation on each detected zooplankton target according to the bounding box, separates the background, and obtains multiple target images. Specifically, it includes: traversing each target bounding box detected by the Mask R-CNN algorithm, for each bounding box, using the corresponding instance segmentation mask, cutting out the target area from the original image to form a new target image; and normalizing the scale of the cut target image to a uniform size of 224x224 pixels. Through this step, a set of segmented and uniformly scaled zooplankton target images can be obtained, preparing for subsequent parallel processing.

[0038] Wherein, the step S40 includes: initializing the CUDA operating environment, including allocating GPU video memory and creating 8-16 CUDA streams; in each CUDA stream, starting two layers of sub-thread grids, the first layer is used to run the pre-trained zooplankton posture calculation model, and the second layer is used to run the pre-trained zooplankton species recognition model. The zooplankton posture calculation model adopts a key point detection-based method, pre-trained on the MPII human posture dataset, and then fine-tuned to the marine biological dataset; the zooplankton species recognition model adopts a convolutional neural network-based method, and is end-to-end trained on the iNaturalist 2017 marine biological dataset.

[0039] Wherein, the step S50 includes: traversing all zooplankton target images extracted in step S30, grouping these images according to batch size (for example, 32), and the number of images in each batch does not exceed the number of CUDA streams (for example, 8-16); for each batch, the images therein are parallelly assigned to different CUDA streams for processing, and each CUDA stream is responsible for processing part of the images in a batch. In this way, the parallel computing capability of the GPU can be fully utilized to improve the overall processing efficiency, while avoiding the problem of GPU memory overflow caused by too many images in a single batch.

[0040] Wherein, the step S60 comprises: in each CUDA stream, starting the first layer of sub-thread grid, each sub-thread is responsible for processing an image; in the sub-thread, inputting the image into a pre-trained zooplankton posture calculation model to obtain the 2D key point coordinates of the zooplankton; according to the key point coordinates, calculating the zooplankton posture information, including translation, rotation, scaling, etc.; performing a plane rotation transformation on the original image, so that the main orientation of the zooplankton is perpendicular to the image plane, eliminating the randomness of the zooplankton posture, and providing a more stable input for subsequent species identification; and saving the image after rotation adjustment as the output of the CUDA stream.

[0041] Wherein, the step S70 includes: in each CUDA stream, starting the second-layer sub-thread grid, each sub-thread is responsible for processing an image; in the sub-thread, inputting the posture-adjusted image into a pre-trained zooplankton species recognition model to obtain the category probability distribution of the zooplankton in each image; selecting the category with the highest probability as the recognition result of the zooplankton in the image; and saving the recognition result of each image as the output of the CUDA stream.

[0042] Wherein, the step S80 includes: traversing all CUDA streams and collecting the zooplankton identification results output by each stream; for each identification result, extracting the zooplankton species label, center coordinates, bounding box, size and other information to form a record; aggregating all records to form a temporary zooplankton data set.

[0043] Among them, step S90 uses the DBSCAN clustering algorithm to perform cluster analysis on the temporary zooplankton data set. According to the scale characteristics of the zooplankton target image, the neighborhood radius Eps=50 pixels and the density threshold MinPts=5 are set, and the DBSCAN algorithm is run to obtain the label information of each cluster, each cluster represents a type of zooplankton, and finally the number of each type of zooplankton is counted to obtain the number distribution information of various zooplankton in the underwater shooting field of view.

[0044] The zooplankton posture calculation model adopts a key point detection-based method, with the zooplankton target image as input and the 2D key point coordinates of the zooplankton as output. The model is pre-trained on the MPII human posture dataset and then fine-tuned on the marine organism dataset.

[0045] The zooplankton species recognition model adopts a convolutional neural network-based method, with the zooplankton target image as input and the zooplankton category probability distribution as output. The model is trained end-to-end on the iNaturalist2017 marine biological dataset.

[0046] The following is a description of the specific structure of the two models:

[0047] 1. Zooplankton posture calculation model

[0048] The main task of the model is to predict the 2D keypoint coordinates of zooplankton in the input image and then calculate its pose information. The model architecture adopts a fully convolutional network structure based on feature pyramid, which can simultaneously detect and locate instances of different scales, including:

[0049] 1. Backbone network

[0050] The zooplankton posture calculation model uses ResNet-50 as the backbone feature extraction network and is pre-trained on the ImageNet dataset. ResNet uses residual connections to solve the gradient vanishing problem of deep networks and can effectively extract multi-scale features.

[0051] 2. Feature Pyramid Network (FPN)

[0052] FPN consists of a bottom-up path and a parallel top-down path to build pyramids with features at different scales. Specifically:

[0053] Bottom-up path: Use the feature layers of three scales, C3, C4, and C5, output by ResNet to perform feature fusion through 3x3 convolution.

[0054] Top-down path: high-level semantic features are cascaded to lower levels through upsampling and 3x3 convolution.

[0055] The final FPN output contains feature pyramids at five scale levels: P3, P4, P5, P6, and P7.

[0056] 3. Key point detection head

[0057] For each scale level Pi (i = 3, 4, 5, 6, 7) feature pyramid, add an independent keypoint detection head. Each keypoint detection head contains 4 convolutional layers and outputs an N x N x K tensor, where N is the output feature map size and K is the number of zooplankton keypoints (e.g. 18). With this design, zooplankton keypoints can be detected and located at different scale levels.

[0058] 4. Key point fusion module

[0059] In order to fuse the multi-scale keypoint prediction results, a keypoint fusion module is designed. This module first upsamples all keypoint prediction maps to make them have the same spatial resolution. Then, for each keypoint, the weighted average of all scale levels is calculated to obtain the fused final keypoint heat map. The weight coefficient is generated by the credibility prediction branch of each scale level.

[0060] 5. Loss Function

[0061] The weighted binary cross entropy loss function is used as the supervisory signal for key point detection. For each key point, only the area near the real coordinates of the key point will produce a valid gradient signal. In addition, a credibility branch is added, and focal loss is used to supervise the credibility of different scale levels.

[0062] Through the above-mentioned network structure design, the attitude calculation model can efficiently detect the key points of zooplankton at various scale levels and fuse the multi-scale prediction results to achieve accurate positioning of zooplankton targets of different scales.

[0063] 2. Zooplankton Target Recognition Model

[0064] The task of this model is to identify the species of the input zooplankton images and give the category probability distribution of each target. A deep convolutional network structure based on feature pyramid and attention mechanism is adopted.

[0065] include:

[0066] 1. Backbone network

[0067] Similar to the posture calculation model, the pre-trained ResNet-50 is used as the backbone feature extraction network.

[0068] 2. Feature Pyramid Network (FPN)

[0069] The structure of FPN is consistent with that in the posture calculation model, and it also outputs a feature pyramid at five scale levels P3-P7.

[0070] 3. Attention Pyramid Pooling Module

[0071] In order to improve the recognition ability of small objects, the attention pyramid pooling (APM) module is introduced. APM consists of a group of convolution modules with different pooling kernel sizes, which are used to capture semantic and detail information from feature pyramids of different scales. Specifically:

[0072] Perform 1x1 convolution on the P3-P7 feature layers respectively to fuse their channel information.

[0073] The fused features are subjected to spatial pyramid pooling operations of 1x1, 3x3, 5x5 and 7x7 scales to generate feature responses with different receptive fields.

[0074] Feature responses of different scales are cascaded and spliced ​​to form a feature representation of multi-scale information fusion.

[0075] 4. Classification Head

[0076] At the output of the APM module, a global average pooling layer and two parallel fully connected layers are added for the final category prediction and category bounding box regression, respectively. The category prediction classification branch uses Softmax to output the category probability distribution, while the boundary regression branch predicts the fine bounding box coordinates of the target location.

[0077] 5. Loss Function

[0078] Multi-task loss function is used for joint training, including:

[0079] Classification loss: Log Softmax loss

[0080] Regression loss: smooth L1 loss

[0081] Because zooplankton in the image may be far away and therefore relatively small, accurate recognition is achieved through multi-level recognition. By introducing the attention pyramid pooling module and multi-scale feature fusion, the target recognition model can focus on both semantic and detail information, improving the recognition ability of zooplankton targets of different scales. At the same time, the additional bounding box regression branch also helps to improve the accuracy of target positioning. In general, the design of the above two deep learning models takes into account the diversity of underwater zooplankton target scales, and achieves high-precision detection, positioning and recognition of targets of different scales through feature pyramid networks, attention mechanisms and other technologies. Compared with the previous single-scale model, this multi-scale architecture can greatly improve the robustness and accuracy in complex underwater environments.

[0082] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned zooplankton scanning imaging analysis method.

[0083] A third aspect of the present invention provides an electronic device, which comprises a processor and a memory, wherein the memory is used to store the step program of the above method, and the processor reads the memory and executes the steps.

[0084] Compared with the prior art, the zooplankton scanning imaging analysis method, medium and electronic device provided by the present invention have the following beneficial effects:

[0085] 1. Greatly improved the accuracy of zooplankton target detection. Using the Mask R-CNN deep learning algorithm for target detection and instance segmentation, it can accurately extract zooplankton targets from complex underwater environments and finely segment the outline of each zooplankton. This lays a good foundation for subsequent posture calculation and species recognition.

[0086] 2. Rapid recognition of zooplankton posture and species is achieved. The present invention makes full use of the parallel computing capability of the GPU, and simultaneously runs the posture calculation model and the species recognition model in each CUDA stream, greatly improving the overall processing efficiency. At the same time, by adjusting the plane rotation of the detection target, the interference caused by the randomness of the posture is eliminated, and the accuracy of species recognition is further improved.

[0087] Therefore, the solution of the present invention solves the technical problem of insufficient detection and analysis efficiency in the existing underwater zooplankton scanning and analysis technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0089] Figure 1 A flow chart of the method provided by the present invention; DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0091] like Figure 1 FIG. 1 is a flow chart of a zooplankton scanning and imaging analysis method provided by the present invention, and the method comprises the following steps:

[0092] S10, preprocessing the underwater field of view image, including denoising and contrast enhancement, to improve image quality;

[0093] S20, performing zooplankton target detection on the processed image using a deep learning target detection algorithm to obtain a bounding box of each zooplankton target;

[0094] S30, for each detected zooplankton target, perform instance segmentation according to the bounding box, separate the background, and obtain multiple target images;

[0095] S40, initializing the CUDA environment, creating a sufficient number of CUDA streams, and starting a multi-layer sub-thread grid in each CUDA stream, including a first-layer sub-thread grid for running a pre-trained posture calculation model, and a second-layer sub-thread grid for running a pre-trained target recognition model;

[0096] S50, dividing the multiple target images into batches, wherein the number of images in each batch does not exceed the number of CUDA streams, and assigning the images in each batch to different CUDA streams in parallel for processing;

[0097] S60, in each CUDA stream, using the first layer of sub-thread grids to run the posture calculation model for each target image in parallel, calculate the posture of the zooplankton in each target image, and perform a plane rotation adjustment on the posture to obtain a posture-adjusted image;

[0098] S70, in each CUDA stream, using the second layer of sub-thread grids to run the target recognition model on each posture-adjusted image in parallel to identify the zooplankton species in each posture-adjusted image;

[0099] S80, merging the processing results of each CUDA stream, adjusting the zooplankton species corresponding to the image based on all the postures, and establishing a temporary zooplankton dataset, including the species, center coordinates, bounding box, and size of each zooplankton;

[0100] S90. Use a clustering algorithm to perform cluster analysis on each record in the temporary zooplankton feature set to obtain statistical information on the number of various types of zooplankton in the underwater shooting field of view.

[0101] The specific implementation methods of the above steps are described in detail below:

[0102] The specific implementation of step S10 is: pre-processing the field of view image taken underwater. First, the image is subjected to noise reduction processing using a median filter algorithm. The median filter algorithm can effectively remove salt and pepper noise and other impulse noise by replacing the value of the central pixel with the median of the neighboring pixels. The purpose of this step is to improve the stability of subsequent image processing.

[0103] Secondly, the histogram equalization algorithm is used to enhance the contrast of the image. Histogram equalization adjusts the grayscale distribution of the image to make it flat, thereby stretching the contrast of the image. The purpose of this step is to highlight the significant features of zooplankton in the image and provide better input for subsequent target detection.

[0104] For the parameter setting of the histogram equalization algorithm, the threshold can be set to 0.4, that is, when the cumulative probability of the image grayscale histogram exceeds 40%, saturation cutoff is performed, which can effectively prevent the loss of details caused by over-enhancement.

[0105] The specific implementation of step S20 is: using a deep learning target detection algorithm to perform zooplankton target detection on the preprocessed image. The Mask R-CNN algorithm is used here, which can perform target detection and instance segmentation at the same time.

[0106] The main steps of the Mask R-CNN algorithm include:

[0107] 1. Use ResNet-50 as the backbone network of the feature extraction network and pre-train it on the ImageNet dataset.

[0108] 2. Based on the pre-trained feature extraction network, a proposed box candidate network (RPN) module is added to generate target candidate boxes, as well as two parallel classification and regression heads for target recognition and bounding box regression.

[0109] 3. Use Focal Loss as the classification loss function, Smooth L1 Loss as the bounding box regression loss function, and Binary Cross Entropy Loss as the mask segmentation loss function to perform end-to-end joint training on the network.

[0110] 4. During the training process, the marine organisms in the COCO dataset are used as training samples for zooplankton. Data enhancement methods include random scaling, flipping, color jittering, etc.

[0111] 5. In the inference stage, the input image is forward propagated to obtain the bounding box, category probability and instance segmentation mask of each detected target.

[0112] Through the Mask R-CNN algorithm, zooplankton targets in the image can be accurately detected and the bounding box information of each target can be extracted.

[0113] The specific implementation of step S30 is: for each detected zooplankton target, instance segmentation is performed according to the boundary box to separate the background and obtain multiple target images. The purpose of this step is to extract a separate zooplankton sample image from the original image to prepare for subsequent posture calculation and species recognition.

[0114] The implementation is as follows:

[0115] 1. Traverse each target bounding box detected by the Mask R-CNN algorithm.

[0116] 2. For each bounding box, use the corresponding instance segmentation mask to crop the target area from the original image to form a new target image.

[0117] 3. Normalize the scale of the cropped target image to a uniform size of 224x224 pixels.

[0118] Through this operation, a set of segmented zooplankton target images with uniform scale can be obtained, preparing for subsequent parallel processing.

[0119] The specific implementation method of step S40 is: initialize the CUDA environment, create a sufficient number of CUDA streams, and start a multi-layer sub-thread grid in each CUDA stream, including a first-layer sub-thread grid for running a pre-trained posture calculation model and a second-layer sub-thread grid for running a pre-trained target recognition model.

[0120] First, initialize the CUDA operating environment, including allocating GPU memory, creating CUDA streams, and other operations. CUDA streams are a set of ordered GPU operations that can achieve parallel computing. Depending on the computing power of the GPU, 8-16 CUDA streams can be created to fully utilize GPU resources.

[0121] Secondly, within each CUDA stream, two layers of sub-thread grids are started. The first layer of sub-thread grids is used to run the pre-trained zooplankton posture calculation model, and the second layer of sub-thread grids is used to run the pre-trained zooplankton species recognition model.

[0122] The zooplankton posture calculation model adopts a key point detection-based method, with the zooplankton target image as input and the 2D key point coordinates of the zooplankton as output. The model is pre-trained on the MPII human posture dataset and then fine-tuned on the marine organisms dataset.

[0123] The zooplankton species recognition model uses a convolutional neural network-based method, with the zooplankton target image as input and the zooplankton category probability distribution as output. The model is trained end-to-end on the iNaturalist 2017 marine life dataset.

[0124] By initializing the CUDA environment and starting a multi-layer sub-thread grid, the parallel computing capabilities of the GPU can be fully utilized to improve the overall processing efficiency.

[0125] The specific implementation method of step S50 is: divide the multiple target images into batches, the number of images in each batch does not exceed the number of CUDA streams, and the images in each batch are allocated to different CUDA streams in parallel for processing.

[0126] The specific operations are as follows:

[0127] 1. Traverse all zooplankton target images extracted in step S30.

[0128] 2. Group these images into batches of size (e.g. 32), where the number of images in each batch does not exceed the number of CUDA streams (e.g. 8-16).

[0129] 3. For each batch, the images in it are assigned to different CUDA streams for processing in parallel. Each CUDA stream is responsible for processing part of the images in a batch.

[0130] The benefits of doing this are:

[0131] - Make full use of the parallel computing capabilities of GPU to improve overall processing efficiency.

[0132] -Avoid the problem of GPU memory overflow caused by too many images in a single batch.

[0133] - The batch size and the number of CUDA streams can be flexibly adjusted according to actual conditions to achieve the best performance.

[0134] The specific implementation method of step S60 is: in each CUDA stream, the first layer of sub-thread grid is used to run the posture calculation model for each target image in parallel, calculate the posture of the zooplankton in each target image, and perform plane rotation adjustment on the posture to obtain a posture adjusted image.

[0135] The specific operations are as follows:

[0136] 1. For each CUDA stream, start the first layer of sub-thread grid. The number of threads in the sub-thread grid is equal to the number of images in the batch, and each sub-thread is responsible for processing one image.

[0137] 2. In the child thread, input the image into the pre-trained zooplankton posture calculation model to obtain the 2D key point coordinates of the zooplankton.

[0138] 3. According to the coordinates of key points, calculate the posture information of zooplankton, including translation, rotation, scaling, etc.

[0139] 4. Perform a plane rotation transformation on the original image so that the main orientation of the zooplankton is perpendicular to the image plane. This step can eliminate the randomness of the zooplankton posture and provide a more stable input for subsequent species identification.

[0140] 5. Save the rotated image as the output of the CUDA stream.

[0141] Through this step, the position information of each zooplankton target can be efficiently calculated and standardized to provide better input for subsequent species identification.

[0142] The specific implementation method of step S70 is: in each CUDA stream, using the second-layer sub-thread grid to run the target recognition model in parallel on each posture-adjusted image to identify the zooplankton species in each posture-adjusted image.

[0143] The specific operations are as follows:

[0144] 1. For each CUDA stream, start the second-level sub-thread grid. The number of threads in the sub-thread grid is equal to the number of images in the batch, and each sub-thread is responsible for processing one image.

[0145] 2. In the child thread, input the pose-adjusted image into the pre-trained zooplankton species recognition model to obtain the category probability distribution of zooplankton in each image.

[0146] 3. Select the category with the highest probability as the recognition result of zooplankton in the image.

[0147] 4. Save the recognition results of each image as the output of the CUDA stream.

[0148] Through this operation, the species of each zooplankton target image can be efficiently identified and the category information of each target can be obtained.

[0149] The specific implementation method of step S80 is: merging the processing results of each CUDA stream, adjusting the zooplankton species corresponding to the image based on all postures, and establishing a temporary zooplankton dataset, including the species, center coordinates, bounding box and size of each zooplankton.

[0150] The specific operations are as follows:

[0151] 1. Traverse all CUDA streams and collect the zooplankton identification results output by each stream.

[0152] 2. For each recognition result, extract the zooplankton species label, center coordinates, bounding box, size and other information to form a record.

[0153] 3. Aggregate all records to form a temporary zooplankton dataset.

[0154] This dataset contains detailed information of all detected zooplankton in the image, ready for subsequent clustering analysis.

[0155] The specific implementation method of step S90 is: using a clustering algorithm to perform cluster analysis on each record in the temporary zooplankton feature set to obtain statistical information on the number of various types of zooplankton in the underwater shooting field of view.

[0156] The specific operations are as follows:

[0157] 1. Select the DBSCAN clustering algorithm to cluster the temporary zooplankton dataset. DBSCAN is a density-based clustering algorithm that can adaptively discover clustering structures of any shape and is highly robust to noise points.

[0158] 2. Set two key parameters of the DBSCAN algorithm:

[0159] -Eps (neighborhood radius): According to the scale characteristics of the zooplankton target image, Eps can be set to 50 pixels.

[0160] -MinPts (density threshold): Set according to actual situation, you can set it to 5.

[0161] 3. Run the DBSCAN algorithm to obtain the label information of each cluster. Each cluster represents a type of zooplankton.

[0162] 4. By counting the number of each type of zooplankton, you can get the distribution information of the number of various zooplankton in the underwater shooting field of view.

[0163] Through the DBSCAN clustering algorithm, each species can be automatically discovered from a large number of zooplankton targets and the quantity distribution of each category can be given, providing valuable information for subsequent ecological surveys.

[0164] The detailed structures of the two models are described below:

[0165] 1. Zooplankton posture calculation model

[0166] The main task of the model is to predict the 2D keypoint coordinates of zooplankton in the input image and then calculate its pose information. The model architecture adopts a fully convolutional network structure based on feature pyramid, which can simultaneously detect and locate instances of different scales, including:

[0167] 1. Backbone network

[0168] The zooplankton posture calculation model uses ResNet-50 as the backbone feature extraction network and is pre-trained on the ImageNet dataset. ResNet uses residual connections to solve the gradient vanishing problem of deep networks and can effectively extract multi-scale features.

[0169] 2. Feature Pyramid Network (FPN)

[0170] FPN consists of a bottom-up path and a parallel top-down path to build pyramids with features at different scales. Specifically:

[0171] Bottom-up path: Use the feature layers of three scales, C3, C4, and C5, output by ResNet to perform feature fusion through 3x3 convolution.

[0172] Top-down path: high-level semantic features are cascaded to lower levels through upsampling and 3x3 convolution.

[0173] The final FPN output contains feature pyramids at five scale levels: P3, P4, P5, P6, and P7.

[0174] 3. Key point detection head

[0175] For each scale level Pi (i = 3, 4, 5, 6, 7) feature pyramid, add an independent keypoint detection head. Each keypoint detection head contains 4 convolutional layers and outputs an N x N x K tensor, where N is the output feature map size and K is the number of zooplankton keypoints (e.g. 18). With this design, zooplankton keypoints can be detected and located at different scale levels.

[0176] 4. Key point fusion module

[0177] In order to fuse the multi-scale keypoint prediction results, a keypoint fusion module is designed. This module first upsamples all keypoint prediction maps to make them have the same spatial resolution. Then, for each keypoint, the weighted average of all scale levels is calculated to obtain the fused final keypoint heat map. The weight coefficient is generated by the credibility prediction branch of each scale level.

[0178] 5. Loss Function

[0179] The weighted binary cross entropy loss function is used as the supervisory signal for key point detection. For each key point, only the area near the real coordinates of the key point will produce a valid gradient signal. In addition, a credibility branch is added, and focal loss is used to supervise the credibility of different scale levels.

[0180] Through the above-mentioned network structure design, the attitude calculation model can efficiently detect the key points of zooplankton at various scale levels and fuse the multi-scale prediction results to achieve accurate positioning of zooplankton targets of different scales.

[0181] 2. Zooplankton Target Recognition Model

[0182] The task of this model is to identify the species of the input zooplankton image and give the category probability distribution of each target. A deep convolutional network structure based on feature pyramid and attention mechanism is adopted.

[0183] include:

[0184] 1. Backbone network

[0185] Similar to the posture calculation model, the pre-trained ResNet-50 is used as the backbone feature extraction network.

[0186] 2. Feature Pyramid Network (FPN)

[0187] The structure of FPN is consistent with that in the posture calculation model, and it also outputs a feature pyramid at five scale levels P3-P7.

[0188] 3. Attention Pyramid Pooling Module

[0189] In order to improve the recognition ability of small objects, the attention pyramid pooling (APM) module is introduced. APM consists of a group of convolution modules with different pooling kernel sizes, which are used to capture semantic and detail information from feature pyramids of different scales. Specifically:

[0190] Perform 1x1 convolution on the P3-P7 feature layers respectively to fuse their channel information.

[0191] The fused features are subjected to spatial pyramid pooling operations of 1x1, 3x3, 5x5 and 7x7 scales to generate feature responses with different receptive fields.

[0192] Feature responses of different scales are cascaded and spliced ​​to form a feature representation of multi-scale information fusion.

[0193] 4. Classification Head

[0194] At the output of the APM module, a global average pooling layer and two parallel fully connected layers are added for the final category prediction and category bounding box regression, respectively. The category prediction classification branch uses Softmax to output the category probability distribution, while the boundary regression branch predicts the fine bounding box coordinates of the target location.

[0195] 5. Loss Function

[0196] Multi-task loss function is used for joint training, including:

[0197] Classification loss: Log Softmax loss

[0198] Regression loss: smooth L1 loss

[0199] Because zooplankton in the image may be relatively small because they are far away, accurate recognition is achieved through multi-level recognition. By introducing the attention pyramid pooling module and multi-scale feature fusion, the target recognition model can focus on semantic and detail information at the same time, improving the recognition ability of zooplankton targets of different scales. At the same time, the additional bounding box regression branch also helps to improve the accuracy of target positioning. In general, the design of the above two deep learning models takes into account the diversity of underwater zooplankton target scales. Through the feature pyramid network, attention mechanism and other technologies, high-precision detection, positioning and recognition of targets of different scales are achieved. Compared with the previous single-scale model, this multi-scale architecture can greatly improve the robustness and accuracy in complex underwater environments.

[0200] The following describes the training of the two models in detail:

[0201] 1. Zooplankton posture calculation model training

[0202] 1. Data preparation

[0203] The MPII human posture dataset is used as the pre-training data source. The dataset contains about 25K images and 40K human instances, and 16 key points are annotated for each instance. These human key points are converted into corresponding key points of zooplankton, such as head, antennae, torso, etc.

[0204] For the fine-tuning stage, a dedicated marine life dataset was used, including about 10K underwater images, with key point coordinates of 20 common zooplankton species manually annotated.

[0205] 2. Pre-training

[0206] First, the whole model is end-to-end pre-trained on the MPII human posture dataset using traditional key point detection loss functions (such as weighted binary cross entropy loss). During the pre-training process, common data augmentation techniques such as random cropping, horizontal flipping, color transformation, etc. are used.

[0207] 3. Fine-tuning

[0208] The network weights are initialized using the parameters of the pre-trained model and then fine-tuned on the marine biological dataset. In addition to the key point detection loss, the focal loss of the credibility branch is introduced to supervise the credibility prediction at different scale levels.

[0209] A phased training strategy is adopted, where the weights of the backbone network and FPN are fixed first, and only the parameters of the key point detection head and fusion module are optimized until convergence. Then the high-level feature layers of the backbone network are unfrozen to further fine-tune the entire network.

[0210] 4. Hyperparameter Settings

[0211] Some key hyperparameter settings during training are as follows:

[0212] Batch size: 8

[0213] Learning rate: 1e-4, using a step-down strategy in the later stages of training

[0214] Optimizer: Adam

[0215] Training epochs: 100

[0216] 5. Model Ensemble

[0217] In order to improve the robustness of the model, multiple models were trained, and the output results were averaged using model integration to obtain the final key point coordinate prediction.

[0218] 2. Zooplankton Target Recognition Model Training

[0219] 1. Data preparation

[0220] The iNaturalist 2017 marine life dataset is used for training, which contains about 1.2 million underwater images, including 5089 marine life categories. 50 categories related to zooplankton are selected from them, with a total of about 200,000 images.

[0221] 2. Data Preprocessing

[0222] The training images are preprocessed by data cleaning and standardization, including removing low-quality images, cropping, and resizing. Common data augmentation strategies are also introduced, such as random cropping, horizontal flipping, color distortion, etc.

[0223] 3. Pre-trained backbone network

[0224] First, the ResNet-50 backbone network is pre-trained on the ImageNet dataset to obtain general image feature representation capabilities.

[0225] 4. End-to-end training

[0226] The pre-trained ResNet-50 network is used to initialize the backbone network weights and the parameters of other modules are randomly initialized. Then, end-to-end joint training is performed on the marine biological dataset, optimizing both the classification and bounding box regression branches.

[0227] Cross entropy loss is used as classification loss, smooth L1 loss is used as bounding box regression loss, and the weighted sum of the two losses is used as the final multi-task loss for back propagation.

[0228] 5. Hyperparameter Settings

[0229] The key hyperparameter settings during training are as follows:

[0230] Batch size: 16

[0231] Initial learning rate: 1e-3, using cosine annealing scheduling strategy during training

[0232] Optimizer: SGD with Momentum

[0233] Weight decay: 1e-4

[0234] Training epochs: 50

[0235] 6. Model fine-tuning

[0236] For a specific zooplankton category, the model can be further fine-tuned on the dataset of that category to improve the detection accuracy. During the fine-tuning process, the parameters of the backbone network and FPN module remain frozen, and only the parameters of the classification head are optimized.

[0237] Through the above training strategies, both deep learning models can learn effective feature representations on large-scale datasets and achieve good generalization performance on the target dataset, thereby completing accurate posture calculation and species recognition of underwater zooplankton.

[0238] In summary, this zooplankton scanning and imaging analysis method makes full use of deep learning and parallel computing technology, and can efficiently complete underwater zooplankton target detection, posture calculation, species identification, and quantity statistics. Among them, the key technologies include:

[0239] 1. Mask R-CNN is used for target detection and instance segmentation to accurately extract zooplankton targets from the original image.

[0240] 2. Use CUDA streams and sub-thread grids to parallel process zooplankton images, give full play to the computing power of the GPU, and improve overall processing efficiency.

[0241] 3. Use the pre-trained zooplankton posture calculation model and species recognition model to identify the posture and species information of each zooplankton target.

[0242] 4. Using the DBSCAN clustering algorithm, each species is automatically discovered from a large number of zooplankton targets and the quantity distribution in the underwater field of view is given.

[0243] In general, this method can provide fast, accurate and comprehensive information support for underwater ecological surveys and has broad application prospects.

[0244] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned zooplankton scanning imaging analysis method.

[0245] A third aspect of the present invention provides an electronic device, which comprises a processor and a memory, wherein the memory is used to store the step program of the above method, and the processor reads the memory and executes the steps.

[0246] In order to better understand and implement the present invention, a specific embodiment 1 of the present invention in a computer-readable storage medium or an electronic device for a computer program is provided below. In embodiment 1, the specific implementation method of step S10 is to pre-process the field of view image taken underwater, including denoising and contrast enhancement.

[0247] For the denoising part, the median filter algorithm is used. Let the original image be I(x,y) and the processed image be I′(x,y). The mathematical formula of median filter is:

[0248] I′(x,y)=median{I(xk,yl)|(k,l)∈W}

[0249] Where W is an m×m filter window centered at (x, y), and median means finding the median of the pixel values ​​in the window. Median filtering can effectively remove salt and pepper noise and other impulse noise, and improve the stability of subsequent image processing.

[0250] For the contrast enhancement part, the histogram equalization algorithm is used. The goal of histogram equalization is to adjust the grayscale distribution of the image to a state as uniform as possible, thereby stretching the contrast of the image. Suppose the grayscale histogram of the original image is h(I) and the cumulative histogram is H(I), then the transformation function of the histogram equalization is:

[0251]

[0252] Where L is the number of gray levels (usually 256), and N is the total number of pixels in the image. Histogram equalization can highlight the significant features of zooplankton in the image and provide better input for subsequent target detection.

[0253] For the parameter setting of histogram equalization, the threshold can be set to T = 0.4, that is, when the cumulative probability of the image grayscale histogram exceeds 40%, saturation cutoff is performed, which can effectively prevent the loss of details caused by over-enhancement.

[0254] Step S20 uses the deep learning target detection algorithm Mask R-CNN to detect zooplankton targets on the processed image. The core idea of ​​Mask R-CNN is to first use the extraction candidate box network k (RPN) to generate target candidate boxes, then classify and regress each candidate box, and finally obtain the target detection result.

[0255] Assume the input image is The candidate box set output by the RPN network is where b i =(x i ,y i ,w i ,h i ) represents the coordinates and size of the i-th candidate box. For each candidate box bi, the classification network outputs the category probability (where C is the number of categories, +1 is the background class), the regression network outputs the bounding box offset The loss function of Mask R-CNN can be expressed as:

[0256]

[0257] in, and is the ground truth, m i and are the predicted instance segmentation mask and groundtruth respectively, and λ1 and λ2 are weight hyperparameters.

[0258] The training of Mask R-CNN consists of the following steps:

[0259] 1. Use ImageNet pre-trained ResNet-50 as the feature extraction backbone network.

[0260] 2. Add RPN module, classification network and regression network based on the backbone network.

[0261] 3. Perform end-to-end training on the marine life category of the COCO dataset to optimize the above loss function.

[0262] 4. Use data enhancement techniques, including random scaling, flipping, color jittering, etc.

[0263] In the inference stage, the input image I is forward propagated through Mask R-CNN to obtain the bounding box coordinates, category probability, and instance segmentation mask of each detected target.

[0264] Step S30 is to perform instance segmentation on each detected zooplankton target according to the bounding box, separate the background, and obtain multiple target images. Let the bounding box of the i-th detected target be b i =(x i ,y i ,w i ,h i ), the corresponding instance segmentation mask is m i ∈{0,1} H×W , the zooplankton target image can be extracted by the following formula:

[0265] I i =I[y i :y i +h i ,x i :x i Ten w i ]☉m i

[0266] Where ⊙ represents element-by-element multiplication. The extracted target image I i They are uniformly scaled to 224×224 pixels.

[0267] Step S40 is to initialize the CUDA environment, create a sufficient number of CUDA streams, and start a multi-layer sub-thread grid in each CUDA stream. First, initialize the CUDA operating environment, including allocating GPU video memory, creating N CUDA streams, and starting a multi-layer sub-thread grid in each CUDA stream. According to the computing power of the GPU, N can be set to 8 to 16.

[0268] In each CUDA stream S j Internally, two layers of sub-thread grids are started. The first layer of sub-thread grids is used to run the pre-trained zooplankton posture calculation model, and the second layer of sub-thread grids is used to run the pre-trained zooplankton species recognition model.

[0269] The zooplankton posture calculation model adopts a key point detection method. Assume that the input image is I i , the output is K 2D key point coordinates The network structure can be designed as: ResNet-50 as the feature extraction backbone network, K fully connected layers regress the coordinates of each key point respectively. The loss function of the network is:

[0270]

[0271] in, The kth keypoint coordinates predicted by the network.

[0272] The zooplankton species recognition model adopts a convolutional neural network-based method. Assume that the input image is I i , the output is the probability distribution p of type C zooplankton i =(p i,1 ,p i,2 ,…,p i,C ). The network structure can be designed as: several convolutional layers, pooling layers and fully connected layers. The loss function of the network is cross entropy loss:

[0273]

[0274] Among them, y i =(y i,1 ,y i,2 ,…,y i,C ) is a one-hot tag.

[0275] Step S50 is to divide the multiple target images into batches, the number of images in each batch does not exceed the number of CUDA streams, and the images in each batch are assigned to different CUDA streams for processing in parallel. Where M is the total number of images. Divide into N batches The number of images in each batch does not exceed N, that is Then, each batch Assigned to different CUDA streams S in parallel j to be processed.

[0276] Step S60 is to use the first layer of sub-thread grids to run the posture calculation model for each target image in parallel in each CUDA stream, calculate the posture of zooplankton in each target image, and perform plane rotation adjustment on the posture to obtain a posture-adjusted image. Suppose the batch processed by the jth CUDA stream is Each image I i The coordinates of the key points output by the posture calculation model are:

[0277] First, calculate the position information of each zooplankton, including translation, rotation, scaling, etc. Let the translation vector be t i =(t i,x ,t i,y ), the rotation angle is θ i , the scaling factor is s i , then the transformation matrix can be expressed as:

[0278]

[0279] Secondly, for the original image I i Perform an affine transformation so that the zooplankton's primary orientation is perpendicular to the image plane:

[0280] I i ′=T i I i

[0281] This operation can eliminate the randomness of zooplankton posture and provide more stable input for subsequent species identification.

[0282] Step S70 is to use the second layer of sub-thread grids in each CUDA stream to run the target recognition model on each posture-adjusted image in parallel to identify the zooplankton species in each posture-adjusted image. Suppose the batch processed by the jth CUDA stream is The probability distribution of each image I′ output by the category recognition model is p i =(p i,1 ,p i,2 ,…,p i,C ), where C is the number of zooplankton categories.

[0283] The loss function of the category recognition model is cross entropy loss:

[0284]

[0285] Among them, y i =(y i,1 ,y i,2 ,…,y i,C) is a one-hot label. In the inference stage, the category with the highest probability is selected as the final recognition result.

[0286] Step S80 is to merge the processing results of each CUDA stream, adjust the zooplankton species corresponding to the images based on all the postures, and establish a temporary zooplankton dataset. Suppose the output of the jth CUDA stream is where b i =(x i ,y i ,w i ,h i ) is the bounding box of the i-th target. Merge to get the complete data set Contains images, species probability distribution, and bounding box information for each zooplankton.

[0287] Step S90 is to use a clustering algorithm to perform cluster analysis on each record in the temporary zooplankton feature set to obtain statistical information on the number of various types of zooplankton in the underwater shooting field of view.

[0288] Set data set Each record in x i =(I i ′,p i ,b i ), including image, category probability and bounding box information. The DBSCAN clustering algorithm is used to cluster {x i} Clustering is performed to obtain K clusters The core idea of ​​the DBSCAN algorithm is:

[0289] 1. For each sample x i , find all samples in its ∈ neighborhood, recorded as

[0290] 2. If If the number of samples is greater than the threshold MinPts, then x i The core samples form a new cluster.

[0291] 3. The non-core samples in x are classified into i in the same cluster.

[0292] 4. Repeat the above steps until all samples are clustered.

[0293] For the parameter setting of DBSCAN algorithm, the neighborhood radius ∈ = 50 pixels and the density threshold MinPts = 5 can be set according to the scale characteristics of zooplankton target image. It represents a type of zooplankton in the underwater field of vision. Finally, by counting the number of each type, we can get the quantity distribution information of various zooplankton.

[0294] Specifically, the principle of the present invention is:

[0295] 1. Image preprocessing. De-noising and contrast enhancement are performed on the original underwater images to improve image quality and provide better input for subsequent target detection and recognition tasks. Among them, median filtering is used to remove salt and pepper noise, and histogram equalization is used to enhance image contrast. These preprocessing operations can significantly improve the visibility and saliency of zooplankton in the image.

[0296] 2. Deep learning target detection. The Mask R-CNN algorithm is used to detect zooplankton targets and segment instances in the preprocessed images. Mask R-CNN, combined with the region proposal network (RPN) and instance segmentation mask, can accurately locate zooplankton targets in the image and extract the bounding box and fine contour information of each target. This lays a good foundation for subsequent posture calculation and species recognition.

[0297] 3. Parallel posture calculation and species identification. The present invention makes full use of the parallel computing capability of the GPU. By creating multiple CUDA streams, the pre-trained zooplankton posture calculation model and species identification model are simultaneously run in each stream. The posture calculation model is based on key point detection technology and can accurately estimate the 2D posture information of each zooplankton target, including translation, rotation, and scaling. The species identification model uses a convolutional neural network method to output the species probability distribution of each target image. Through parallel processing, the overall processing speed is greatly improved.

[0298] 4. Plane rotation adjustment. Based on the posture calculation, a plane rotation transformation is performed on each zooplankton target image so that its main orientation is perpendicular to the image plane. This step can eliminate the interference caused by the randomness of the zooplankton posture and provide a more stable input for subsequent species identification.

[0299] 5. Cluster analysis. Finally, the DBSCAN clustering algorithm is used to classify all detected zooplankton targets, automatically discover various types of zooplankton in the underwater field of view, and give their quantitative statistics. The DBSCAN algorithm can adaptively discover clustering structures of arbitrary shapes and is also highly robust to noise points, making it very suitable for biological species analysis tasks in such complex environments.

[0300] Because the size of zooplankton samples in the underwater shooting field of view is very different, the number is large, the distance is different, the posture is different, and the size is different, there will be occlusion and overlap on the plane picture, which affects the analysis. Through the close cooperation of the above technical links, the underwater zooplankton scanning and analysis method of the present invention can efficiently and accurately complete the detection, posture calculation, species identification and quantity statistics of zooplankton, greatly improving the scientificity and objectivity of underwater ecological monitoring. In particular, by making full use of the parallel computing capabilities of GPU, the overall processing speed can be greatly improved under the premise of ensuring the accuracy of detection, thereby realizing continuous and large-scale underwater zooplankton scanning and analysis, meeting the urgent needs of water ecological environment monitoring.

[0301] The following is an example of a specific application scenario of the present invention: A marine ecological research institute is currently carrying out a comprehensive underwater ecological environment monitoring project, in which a comprehensive investigation of the species composition and quantity changes of underwater zooplankton is one of the key contents. Traditional manual sampling and microscope identification methods are difficult to meet the needs of the project due to the large workload and low efficiency. Therefore, the institute decided to adopt the underwater zooplankton scanning imaging analysis method proposed in the present invention to achieve rapid and efficient underwater biological monitoring.

[0302] The specific implementation process is as follows:

[0303] 1. Installation and debugging of underwater camera equipment

[0304] The researchers first deployed multiple underwater cameras in typical waters of a certain sea area to continuously capture images of underwater zooplankton. These cameras use 4K high-definition camera technology to capture clear images of tiny zooplankton. At the same time, the researchers also conducted detailed debugging of the camera's installation position, viewing angle, exposure parameters, etc. to ensure that the quality of the underwater field of view images obtained was good, laying the foundation for subsequent image analysis.

[0305] 2. Image Preprocessing

[0306] The researchers input the collected original underwater vision images into the computer system for preprocessing. First, the median filter algorithm was used to reduce the noise of the image, effectively removing the salt and pepper noise caused by water turbulence, light reflection and other factors. After the median filter processing, the noise points of the image were significantly suppressed, which is conducive to the subsequent target detection and recognition.

[0307] Secondly, the image contrast is enhanced using the histogram equalization algorithm. The algorithm stretches the image contrast by adjusting the grayscale histogram of the image to make it tend to be evenly distributed. After this step, the salient features of zooplankton in the underwater environment are highlighted, which helps the Mask R-CNN algorithm to more accurately locate and extract zooplankton targets. In the histogram equalization process, the researchers set the threshold to 0.4, that is, when the cumulative probability of the image grayscale histogram exceeds 40%, a saturation cutoff is performed to prevent excessive enhancement from causing loss of details.

[0308] 3. Zooplankton Target Detection and Segmentation

[0309] The preprocessed underwater vision images were input into the pre-trained Mask R-CNN deep learning model for zooplankton target detection and instance segmentation. The backbone network of the model uses ResNet-50, which is pre-trained on the ImageNet dataset. On this basis, the researchers added the Extraction Candidate Box Network k (RPN) module to generate target candidate boxes, as well as two parallel classification and regression heads for target recognition and bounding box regression. During the training process, the research team used the marine life categories in the COCO dataset as training samples for zooplankton, and used data enhancement methods such as random scaling, flipping, and color jittering to improve the robustness of the model.

[0310] During the inference phase, the Mask R-CNN model can accurately detect zooplankton targets from complex underwater environment images and output the bounding box coordinates, category probability, and fine instance segmentation mask of each target. Through this step, the researchers obtained detailed location and contour information of all zooplankton targets in the underwater field of view.

[0311] As an example, taking an image obtained from an underwater video, the Mask R-CNN model detected a total of 36 zooplankton targets, including 14 copepod zooplankton, 12 tropod zooplankton, 6 truncate zooplankton, and 4 other types of zooplankton. The bounding box coordinates and instance segmentation mask information of each target were accurately extracted, laying the foundation for subsequent posture calculation and species recognition.

[0312] 4. Zooplankton posture calculation

[0313] For each zooplankton target image detected by Mask R-CNN, the researchers used a pre-trained zooplankton posture calculation model to estimate their posture information on the 2D plane, including parameters such as translation, rotation, and scaling.

[0314] The posture calculation model uses the key point detection method. The network structure includes ResNet-50 as the feature extraction backbone network, and several fully connected layers are used to regress the 2D key point coordinates of each zooplankton target. During the training process, the research team first pre-trained on the MPII human posture dataset to learn the general key point detection capabilities, and then fine-tuned on the marine biological dataset to enable the model to adapt to the appearance characteristics of zooplankton.

[0315] Taking the 36 zooplankton targets detected above as an example, after processing by the attitude calculation model, the researchers obtained their various attitude parameters on the 2D image plane. For example, 12 targets showed an upward vertical attitude, 18 targets showed a sideways horizontal attitude, and 6 targets showed a certain degree of tilt. These attitude information provides an important basis for subsequent species identification.

[0316] 5. Identification of zooplankton species

[0317] For each zooplankton target image that has been pose-calculated, the researchers used another pre-trained deep learning model to identify the species. This model uses a convolutional neural network structure and is trained end-to-end on the iNaturalist 2017 marine biological dataset, which can output the probability distribution of the zooplankton category in each target image.

[0318] In actual application, the researchers identified species of 36 target images. The results showed that 14 targets were identified as copepod zooplankton, with a probability of more than 90%; 12 targets were identified as tropod zooplankton, with a probability of about 80%; 6 targets were identified as interceptor zooplankton, with a probability of about 75%; and the remaining 4 targets were identified as other types of zooplankton, with a lower probability. Through this operation, the researchers obtained the species composition information of various zooplankton in the underwater field of view.

[0319] 6. Zooplankton plane rotation adjustment

[0320] After completing the posture calculation and species identification, the researchers also performed a plane rotation adjustment on each zooplankton target image. Specifically, based on the rotation angle information obtained from the posture calculation, an affine transformation was performed on the original image so that the main orientation of the zooplankton was perpendicular to the image plane. The purpose of this operation is to eliminate the interference caused by the randomness of the zooplankton posture and provide a more stable input for subsequent species identification.

[0321] After this processing, the postures of the 36 zooplankton target images were standardized, preparing for the next step of parallel computing.

[0322] 7. Parallel processing and result merging

[0323] To improve the overall processing efficiency, the researchers initialized the CUDA operating environment on the GPU server and created 16 CUDA streams. Two layers of sub-thread grids were started inside each CUDA stream: the first layer was used to execute the posture calculation model in parallel, and the second layer was used to execute the type recognition model in parallel.

[0324] Specifically, the 36 standardized zooplankton target images are first grouped into two batches of 32 images each. Then, the two batches of images are input into 16 CUDA streams in parallel for processing.

[0325] Inside each CUDA stream, the first-layer sub-thread grid is used to run the posture calculation model in parallel to obtain the 2D key point coordinates of each target image. Then, the translation, rotation and scaling parameters of each target are calculated based on the key point information, and an affine transformation is performed on the original image to eliminate posture randomness.

[0326] Secondly, within the same CUDA stream, the second-layer sub-thread grid is used to run the species recognition model in parallel to output the category probability distribution of each target image. This parallel processing method greatly improves the efficiency of posture calculation and species recognition of zooplankton targets.

[0327] Finally, the researchers combined the processing results of 16 CUDA streams to obtain a complete underwater zooplankton information dataset, which contains information such as the image, species probability distribution, and bounding box coordinates of each target.

[0328] During the entire parallel processing process, the researchers observed that the peak GPU memory usage was about 12GB and the computing time was about 2.5 minutes, which greatly improved efficiency. In contrast, if a single-threaded CPU method is used for serial processing, it is expected to take about 15 minutes.

[0329] 8. Cluster analysis and quantitative statistics

[0330] With a complete underwater zooplankton dataset, the researchers used the DBSCAN clustering algorithm to analyze it, automatically discovering various types of zooplankton in the underwater field of view and providing their population statistics.

[0331] The key parameters of DBSCAN clustering are set as follows: the neighborhood radius ε is set to 50 pixels, and the density threshold MinPts is set to 5. After cluster analysis, the researchers found that there were 5 obvious clusters in the data set, corresponding to 5 different types of zooplankton. By counting each cluster, specific quantitative information was obtained:

[0332] -Copepod zooplankton: 14

[0333] - Zooplankton: 12

[0334] -Interceptor zooplankton: 6

[0335] -Other types of zooplankton: 4

[0336] These results provide detailed and reliable biological indicator data for subsequent water ecological environment monitoring.

[0337] In summary, the institute successfully completed the underwater ecological monitoring task by using the underwater zooplankton scanning imaging analysis method proposed in this invention. Compared with traditional manual sampling and microscopic identification, this method has the following significant advantages:

[0338] 1. Detection accuracy is significantly improved. The Mask R-CNN algorithm can accurately extract zooplankton targets from complex underwater environments and finely segment the outline of each target, which is much better than traditional image processing methods.

[0339] 2. Processing efficiency is greatly improved. By making full use of the parallel computing power of GPU and combining the joint optimization of posture calculation and type recognition, the overall data processing time is only 2.5 minutes, which is 6 times faster than serial CPU processing.

[0340] 3. Fully automated analysis is achieved. This method does not require human intervention and can continuously and large-scale scan and analyze underwater zooplankton, greatly improving the efficiency and scalability of ecological monitoring.

[0341] 4. Obtained rich biological indicator data. Through cluster analysis, the researchers not only mastered the species composition of various zooplankton, but also obtained their quantity distribution information, providing a scientific basis for subsequent water ecological environment assessment.

[0342] In general, the underwater zooplankton scanning and analysis technology of the present invention has injected new vitality into marine ecological monitoring and will surely play an important role in future applications.

[0343] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A zooplankton scanning and imaging analysis method, characterized in that: The following steps are involved: S10, preprocessing the underwater field of view image, including denoising and contrast enhancement, to improve image quality; S20, using a deep learning target detection algorithm to perform zooplankton target detection on the processed image to obtain a bounding box for each zooplankton target; S30, for each detected zooplankton target, performing instance segmentation according to the bounding box, separating the background, and obtaining multiple target images; S40, initialize the CUDA environment, create a sufficient number of CUDA streams, and start a multi-layer sub-thread grid in each CUDA stream, including a first-layer sub-thread grid for running a pre-trained posture calculation model, and a second-layer sub-thread grid for running a pre-trained target recognition model; S50, dividing the multiple target images into batches, wherein the number of images in each batch does not exceed the number of CUDA streams, and assigning the images in each batch to different CUDA streams for processing in parallel; S60, in each CUDA stream, using the first layer of sub-thread grids to run the posture calculation model for each target image in parallel, calculate the posture of the zooplankton in each target image, and perform a plane rotation adjustment on the posture to obtain a posture-adjusted image; S70, in each CUDA stream, using the second layer of sub-thread grids to run the target recognition model on each pose-adjusted image in parallel to identify the zooplankton species in each pose-adjusted image; S80, merging the processing results of each CUDA stream, adjusting the zooplankton species corresponding to the image based on all the postures, and establishing a temporary zooplankton dataset, including the species, center coordinates, bounding box, and size of each zooplankton; S90, using a clustering algorithm to perform cluster analysis on each record in the temporary zooplankton data set to obtain statistical information on the number of various types of zooplankton in the underwater shooting field of view.

2. The zooplankton scanning and imaging analysis method according to claim 1, characterized in that: The posture calculation model is used to predict the 2D key point coordinates of zooplankton in the input image, and then calculate the posture information. The model architecture adopts a fully convolutional network structure based on feature pyramid, which can simultaneously detect and locate instances of different scales.

3. The zooplankton scanning and imaging analysis method according to claim 1, characterized in that: The target recognition model adopts a deep convolutional network structure based on feature pyramid and attention mechanism to identify the species of input zooplankton images and give the category probability distribution of each target.

4. The zooplankton scanning and imaging analysis method according to claim 2, characterized in that: The posture calculation model includes: Pre-trained ResNet-50 backbone network; Feature pyramid network, outputs feature pyramids at five scale levels P3-P7; For each scale level Pi, an independent key point detection head is set to output the zooplankton key point prediction map; The key point fusion module performs weighted fusion of key point prediction images at all scale levels.

5. The zooplankton scanning and imaging analysis method according to claim 4, characterized in that: The keypoint detection head contains 4 convolutional layers and outputs an N x N x K tensor, where N is the output feature map size and K is the number of zooplankton keypoints.

6. The zooplankton scanning and imaging analysis method according to claim 3, characterized in that: The target recognition model comprises: Pre-trained ResNet-50 backbone network; Feature pyramid network, outputs feature pyramids at five scale levels P3-P7; Attention pyramid pooling module, which fuses feature responses of different scales; The classification head includes a global average pooling layer, a category prediction branch, and a bounding box regression branch.

7. The zooplankton scanning and imaging analysis method according to claim 6, characterized in that: The attention pyramid pooling module performs spatial pyramid pooling of 1x1, 3x3, 5x5 and 7x7 scales on the P3-P7 feature layers, and cascades and splices feature responses of different scales.

8. The zooplankton scanning and imaging analysis method according to claim 6, characterized in that: The category prediction branch uses Softmax to output category probability distribution, and the boundary regression branch predicts the fine bounding box coordinates of the target location.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the zooplankton scanning imaging analysis method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: A processor and a memory are provided, wherein the memory is used to store the step program of the method described in any one of claims 1 to 8, and the processor reads the memory and executes the steps.

Images