Marine fish and coral identification method and equipment based on deep learning

Abstract

The invention relates to a marine fish and coral identification method based on deep learning, and the method comprises the following steps: collecting underwater image samples of marine fish and coral as original images; performing defogging enhancement on the original image, collecting the defogged enhanced image as a detection data set, and training a fish and coral recognition model; and storing underwater video samples monitored in real time as underwater images frame by frame, performing defogging enhancement processing, and inputting the processed underwater images into the trained fish and coral recognition model to obtain coordinates and types of corals and fishes. The invention also provides identification equipment corresponding to the identification method. The method has the advantages that the image quality is greatly improved through the underwater image enhancement technology, so that the adaptability of the recognition model to different marine environment conditions is improved, and automatic and accurate recognition of marine fishes and corals under various complex environment conditions is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and image processing, and particularly to a method and device for identifying marine fish and corals based on deep learning. Background Art

[0002] With the enhancement of marine environmental protection awareness, the identification and monitoring of marine fish and corals have become particularly important. Traditional identification methods mainly rely on manual observations by divers and manual appraisals by experts. This method is not only time-consuming and laborious but also has subjective errors, making it difficult to achieve large-scale monitoring and data collection. With the rapid development of deep learning and computer vision technologies, many researchers have begun to explore applying these technologies to the automatic identification of marine organisms. During the deep learning process, the quality of images plays a crucial role in the recognition effect of deep learning. Fish and corals live in complex underwater environments. Factors such as insufficient light, variable light, interlaced light and shadow, micro-particles, and micro-bubbles in the deep sea affect the quality of the collected images. Especially during the process of identifying green corals, it is easily interfered by the color of seaweed on rocks. Therefore, to improve the accuracy of identifying marine fish and corals, it is necessary to solve the problem of underwater image quality under different environmental conditions first.

[0003] Secondly, since there are small-sized species of corals and fish or they are currently in a small-sized state, and in the underwater environment, due to the absorption and scattering effects of light, the imaging is often affected by blurring, low contrast, and color deviation, and the edge features of the target will be weaker. Especially small targets may be difficult to detect due to various types of noise interference in the background. Therefore, the existing fish recognition models based on deep learning also have the problem of low success rate in capturing small targets. In addition, there are also problems such as slow recognition speed / high processing delay, which are difficult to meet the requirements of real-time monitoring. Summary of the Invention

[0004] To solve the above problems, the purpose of the present invention is to provide a method for identifying marine fish and corals based on deep learning. By using underwater image enhancement technology, the image quality is greatly improved, thereby enhancing the adaptability of the recognition model to different marine environmental conditions and achieving automatic and accurate identification of marine fish and corals under various complex environmental conditions.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions:

[0006] Technical Solution 1

[0007] A method for identifying marine fish and corals based on deep learning, comprising the following steps:

[0008] Step 1, collect underwater image samples of marine corals as the original images;

[0009] Step 2: Dehaze and enhance the original image, including: calculating the dark channel of the image through a minimum filter for the original image; calculating the guided filter parameters using the dark channel, and then performing guided filtering on the original image to remove noise and preserve edge features, obtaining a filtered image; determining the pixel region representing the atmospheric light source from the dark channel, and then taking the maximum value of each RGB channel from the original image according to this pixel region as the atmospheric light value; calculating the original image using the filtered image and the atmospheric light value to obtain a dehazed and enhanced image;

[0010] Step 3: Collect the dehazed and enhanced images as a detection dataset, and annotate the coral species in the images and their positions in the images;

[0011] Step 4: Improve based on the YoloV8 neural network model: Add the CBAM module to the C2f module of the YoloV8 neural network model to obtain the C2f-CMBA module, which enhances the feature expression ability of the C2f module, and obtain the improved YoloV8 neural network model;

[0012] Step 5: Use the detection dataset annotated in Step 3 to train the improved YoloV8 neural network model. After training, obtain a marine coral recognition model.

[0013] More preferably, Step 2 is specifically executed as follows:

[0014] Step 21: Calculate the dark channel V1(x,y) of the image by passing the input original image I(x,y) through a minimum filter: V1(x,y) = min c∈{r,g,b} (min (x',y')∈Ω(x,y) I c (x',y')); where I represents the original image matrix, Ω(x,y) represents a fixed-size sliding window centered on (x,y), and I c (x',y') represents the pixel value of the original image matrix on channel c, where {r,g,b} are the pixel value channels of the red, green, and blue colors;

[0015] Step 22: Calculate the guided filter parameters using the dark channel V1(x,y), including coefficient a and constant b:

[0016] where cov(V1,I) and var(V1) are covariance and variance respectively: where, is to perform mean filtering on the image matrices V1, V1 2 , V1·I, I, and reduce the noise in the image by taking the average of neighboring pixels. ∈ is the dielectric constant;

[0017] Step 23: Perform guided filtering on the original image I(x, y) to obtain the filtered image V1'(x, y). The formula is: V' 1 (x, y) = a * I(x, y) + b;

[0018] Step 24: Determine the pixel region representing the atmospheric light source from the dark channel. Calculate the histogram of the dark channel V1(x, y), and set the number of pixel intervals bins to K: hist(V1) =

[0019] Historgram(V1, bins = K);

[0020] Step 25: Calculate the cumulative distribution function CDF of the pixel intensity value l:

[0021] Determine the pixel intensity value l at which the cumulative distribution function in the histogram reaches 99.9%: max : l max = max{l | CDF(l) ≤ 0.999}; Select the positions in the dark channel V1 where the pixel values are higher than l max as the pixel region of the brightest 0.001% of the image, and use them to represent the source of atmospheric light;

[0022] Step 26: Take the maximum value of each RGB channel in the original image according to this pixel region as the atmospheric light value A;

[0023] Step 27: Use the filtered image V1'(x, y) and the atmospheric light value A to calculate the original image I(x, y) to obtain the defogged and enhanced image I':

[0024] More preferably, Step 4 is specifically executed as follows: Build a CBAM module. The CBAM module includes a channel attention module and a spatial attention module, guiding the network to focus on significant features. The input feature F enters the CBAM module and passes through the channel attention module M c and the spatial attention module M s , to obtain the enhanced feature F out ; where represents element-wise multiplication; Build a CMAM-C2f module: Add the CBAM module after the Bottleneck module of the C2f module and use it as the last layer of the CMAM-C2f module to further enhance the features extracted by multiple Bottleneck modules through the CBAM module; Embed the CMAM-C2f module into the YoloV8 neural network model to replace the original C2f module to obtain the improved YoloV8 neural network model.

[0025] Preferably, underwater image samples of marine fish are also collected in step 1 as the original images;

[0026] Execute step 2; in step 3, collect the dehazed and enhanced images as the detection dataset, and label the fish species and their positions in the images; execute step 4; execute step 6, use the labeled detection dataset in step 3 to train another improved YoloV8 neural network model, and after training is completed, obtain the marine fish recognition model.

[0027] Based on the same inventive concept, the present invention also provides a device for recognizing marine fish and corals based on deep learning.

[0028] Technical solution two

[0029] A device for recognizing marine fish and corals based on deep learning includes a memory storing an executable program and a processor. The processor runs the program and executes the method steps in technical solution one.

[0030] The present invention has the following beneficial effects:

[0031] 1. By performing minimum filtering and guided filtering on the underwater image to obtain a filtered image, and then combining the filtered image with the atmospheric light value to perform dehazing and enhancement processing on the original image, the quality of the underwater image is greatly improved, the adaptability of the recognition model to underwater images under different environmental conditions is enhanced, and accurate recognition can be ensured under various complex conditions. It can be applied to a variety of sea area environments and has a wide range of application scenarios.

[0032] 2. By using a YOLOv8 recognition model with a smaller density, the recognition speed is improved, and the application effect of strong real-time image recognition is enhanced.

[0033] 3. Reduce labor costs: Through automatic recognition, the dependence on manual labor is reduced, the labor and material costs are reduced, and the work efficiency is improved. Description of the drawings

[0034] Figure 1 is the flowchart of the present invention;

[0035] Figure 2 is the structural diagram of the improved YOLOv8 neural network of the present invention;

[0036] Figure 3 is the schematic diagram of the CBAM module of the present invention;

[0037] Figure 4 is the schematic diagram of the C2f-CBAM module of the present invention;

[0038] Figure 5Schematic diagram of the Bottleneck module of the present invention;

[0039] Figure 6 Schematic diagram of the ConvModule module of the present invention;

[0040] Figure 7 Schematic diagram of the SPPF module of the present invention;

[0041] Figure 8 Sample diagram of coral recognition of the present invention;

[0042] Figure 9 Sample diagram of fish recognition of the present invention. Detailed implementation manners

[0043] The following further describes the present invention in detail with reference to the accompanying drawings and specific embodiments:

[0044] Embodiment 1

[0045] See Figure 1 , a method for recognizing marine fish and corals based on deep learning, including the following steps:

[0046] Step 1, collect underwater image samples of marine fish and corals as the original images. Specifically, through the underwater camera equipment installed in advance at the specified underwater point in the ocean, the video stream is transmitted back using a network cable and saved to the host device near the point, or through the video samples taken by a diver carrying a mobile recording device along the underwater path in the specified sea area. The collected video is saved frame by frame as an image sample as the original image.

[0047] Step 2, perform haze removal and enhancement on the original image to obtain a haze-removed and enhanced image. Specifically: the dark channel of the image is calculated from the original image through a minimum filter;

[0048] Step 21, calculate the dark channel V1(x, y) of the input original image I(x, y) through a minimum filter: V1(x, y) = min c∈{r,g,b} (min (x',y')∈Ω(x,y) I c (x', y')); where I represents the original image matrix, I c (x', y') represents the pixel value of the original image matrix on channel c, where {r, g, b} are the pixel value channels of the red, green, and blue colors; Ω(x, y) represents a fixed-size sliding window centered on (x, y), for example, a 3x3 sliding window, and there are 9 pixel points in this sliding window;

[0049] Step 22, calculate the guided filter parameters using the dark channel, and then perform guided filtering on the original image to remove noise and preserve edge features to obtain a filtered image;

[0050] The guided filtering parameters include coefficient a and constant b:

[0051] where cov(V1, I) and var(V1) are covariance and variance respectively:

[0052] where performs mean filtering on the image matrices V1, V1 2 , V1·I, I, and reduces the noise in the image by taking the average of the neighboring pixels. Exemplarily, taking the image matrix G as an example, the mean filtering process is illustrated: the image matrix after mean filtering is The filtering formula is:

[0053] where m*n is the window size of the filter. In the present invention, the filter window is set to m = 3 and n = 3.

[0054] Step 23: Perform guided filtering on the original image I(x, y) to obtain the filtered image V1'(x, y). The formula is: V' 1 (x, y) = a*I(x, y) + b;

[0055] Step 24: Determine the pixel region representing the atmospheric light source from the dark channel: Calculate the histogram of the dark channel V1(x, y), and set the number of pixel intervals bins to K: hist(V1) =

[0056] Historgram(V1, bins = K); where bins is the parameter of the number of pixel intervals. For example, K can take the value of 2000;

[0057] Step 25: Calculate the cumulative distribution function CDF of the pixel intensity value l:

[0058] Determine the pixel intensity value l at which the cumulative distribution function in the histogram reaches 99.9% max : l max = max{l|CDF(l) ≤ 0.999}; Select the pixels in the dark channel V1 with pixel values higher than l maxAt its position, it determines the pixel region of the 0.001% brightest pixels in the image and uses them to represent the source of the atmospheric light. There are very few high-light outliers in the underwater image, such as noise or reflections. These extreme values will affect the accuracy of calculating the atmospheric light value. Therefore, by taking the maximum intensity value of the 99.9% cumulative distribution, extreme noise is ignored, effectively avoiding the selection of noise points or abnormally dark points. Therefore, the present invention uses the intensity value at the 99.9% cumulative distribution position to determine the atmospheric light value, which can well exclude abnormal high-light values, thereby improving the accuracy and robustness of the defogging algorithm.

[0059] Step 26: Take the maximum value in each RGB channel from the original image according to this pixel region as the atmospheric light value A;

[0060] Step 27: Use the filtered image V1'(x, y) and the atmospheric light value A to calculate the original image I(x, y) to obtain the defogged enhanced image I': Among them, It is used to correct the image brightness. Haze will reduce the contrast of the scene and make the image blurred. Through this correction factor, the original contrast of the image can be restored. The closer the denominator is to 0, it indicates that the position is more affected by haze. By dividing by this value, the brightness of this position can be amplified, thus achieving a good defogging effect.

[0061] Step 3: Collect the defogged enhanced images as a detection data set, label the coral species in the images and their positions in the images to generate a coral detection data set, and label the fish species in the images and their positions in the images to generate a fish detection data set. Each detection data set is divided into a training set, a test set and a validation set, where the image samples in the training set account for 80% of the total images, and the validation set and the test set each account for 10%.

[0062] Step 4: In the underwater environment, due to the absorption and scattering effects of light, imaging is often affected by blurring, low contrast and color cast. The edge features of the target will be weaker. Especially small targets may be difficult to detect due to various noises in the background. This embodiment proposes to improve based on the YoloV8 neural network model to enhance the recognition accuracy of the neural network model in this scenario. Please refer to Figures 2 to 7, The Chinese translations of each layer in the figure are as follows: ConvModule: Convolution Module; SPPF: Spatial Pyramid Pooling Module; Concat: Tensor Concatenation Operation; UpSample: Upsampling Operation; Conv2d: 2D Convolution Operation; MaxPool2d: Max Pooling Layer; AvgPool2d: Average Pooling Layer; Split: Tensor Splitting Operation; C2f-CBAM: C2f Convolutional Block Attention Module; SiLU: SiLU Activation Function; ReLU: ReLU Activation Function; Sigmoid: Sigmoid Activation Function; CBAM: Convolutional Block Attention Module; ChannelAttention: Channel Attention Module;

[0063] SpatialAttention: Spatial Attention Module; BatchNorm2d: Batch Normalization Operation; Bbox.Loss: Bounding Box Loss Operation; Cls.Loss: Classification Loss Operation. The improvement steps of the YoloV8 neural network model are as follows:

[0064] Please refer to Figure 3 , Build the CBAM (Convolutional Block Attention Module) module: The CBAM module includes a channel attention module and a spatial attention module, guiding the network to focus on significant features and improving the detection performance of the model. The input feature F enters the CBAM module and passes through the channel attention module M c and the spatial attention module M s , to obtain the enhanced feature F out ; where represents element-wise multiplication;

[0065] Please refer to Figure 4 and Figure 5 , Build the CMAM-C2f module: To enhance the feature expression ability of the C2f module, especially to better capture small target information in the case of dealing with complex backgrounds, the CBAM module is added after the Bottleneck module of the C2f module and serves as the last layer of the CMAM-C2f module, further enhancing the features extracted by multiple Bottleneck modules through the CBAM module;

[0066] Embed the CMAM-C2f module into the YoloV8 neural network model, replacing the original C2f module, to obtain the improved YoloV8 neural network model.

[0067] Step 5: Use the labeled fish detection dataset in Step 3 to train the improved YoloV8 neural network model, and use the labeled coral detection dataset to train another improved YoloV8 neural network model. Set the number of training iterations to 500 and the batch size to 16; through data augmentation, loss calculation, and updating of model parameters, a coral recognition model and a fish recognition model are obtained respectively. By using the two models to identify corals and fish respectively, the misrecognition rate is significantly reduced.

[0068] The improved YoloV8 neural network model has a smaller neural network density, which not only improves the recognition speed but also enhances the real-time performance.

[0069] Step 6: As Figure 8 and Figure 9 shown, after saving the real-time monitored underwater video samples frame by frame as underwater images and performing the defogging and enhancement processing, if fish need to be recognized, the image is input into the fish recognition model to output the types and coordinates of the recognized fish; if corals need to be recognized, the image is input into the coral recognition model to output the types and coordinates of the recognized corals. At the same time, the appearance time and quantity information can also be counted and then packaged and saved in the database for scientific research personnel to analyze.

[0070] A method for identifying marine fish and corals based on deep learning in the present invention obtains a filtered image by performing minimum filtering and guided filtering on an underwater image, and then combines the filtered image with the atmospheric light value to perform defogging and enhancement processing on the original image, significantly improving the quality of the underwater image, enhancing the adaptability of the recognition model to underwater images under different environmental conditions, ensuring accurate recognition under various complex conditions, being applicable to various sea area environments, and having a wide range of application scenarios.

[0071] Based on the same inventive concept, the present invention also provides a device for identifying marine fish and corals based on deep learning.

[0072] Embodiment 2

[0073] A device for identifying marine fish and corals based on deep learning includes a memory storing an executable program and a processor, and the processor runs the program to execute the method steps of Embodiment 1. Since the device introduced in Embodiment 2 of the present invention is a hardware device for implementing the method of Embodiment 1 of the present invention, based on the method introduced in Embodiment 1 of the present invention, those skilled in the art can understand the specific implementation manner of the device, so it will not be elaborated here. All methods adopted in Embodiment 1 of the present invention fall within the scope of protection of the present invention.

[0074] The above are only specific embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent structural transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A method for identifying marine fish and corals based on deep learning, characterized in that: The steps include: Step 1, collecting underwater image samples of ocean corals as original images; Step 2, defogging and enhancing the original image, including: calculating the dark channel of the original image through a minimum filter; using the dark channel to calculate the guided filtering parameters, and then performing guided filtering on the original image to remove noise and maintain edge features to obtain a filtered image; determining the pixel area representing the source of atmospheric light from the dark channel, and then taking the maximum value of each RGB channel from the original image as the atmospheric light value based on the pixel area; using the filtered image and the atmospheric light value to calculate the original image to obtain a defogging and enhanced image; Step 3: Collect the dehazed and enhanced images as a detection dataset, and mark the coral species and their locations in the images; Step 4: Improvement based on the YoloV8 neural network model: Add the CBAM module to the C2f module of the YoloV8 neural network model to obtain the C2f-CMBA module, enhance the feature expression ability of the C2f module, and obtain the improved YoloV8 neural network model; Step 5: Use the detection data set labeled in step 3 to train the improved YoloV8 neural network model. After the training is completed, the marine coral recognition model is obtained.

2. The method for identifying marine fish and corals based on deep learning according to claim 1, characterized in that: The step 2 specifically performs the following steps: Step 21: Calculate the dark channel V1(x,y) of the input original image I(x,y) through the minimum filter: V1(x,y)=min c∈{r,g,b} (min (x',y')∈Ω(x,y) I c (x',y')); Where I represents the original image matrix, Ω(x,y) represents a sliding window of fixed size centered at (x,y), and I c (x', y') represents the pixel value of the original image matrix on channel c, where {r, g, b} are the red, green and blue pixel value channels; Step 22: Use the dark channel V1(x,y) to calculate the guided filter parameters, including the coefficient a and the constant b: Among them, cov(V1,I) and var(V1) are covariance and variance respectively: in, is the image matrix V1,V1 2 ,V1·I,I performs mean filtering to reduce the noise in the image by taking the average value of the neighborhood pixels, ∈ is the dielectric constant; Step 23: Perform guided filtering on the original image I(x, y) to obtain a filtered image V1'(x, y), the formula is: V'1(x, y) = a*I(x, y) + b; Step 24: Determine the pixel area representing the source of atmospheric light from the dark channel: Calculate the histogram of the dark channel V1(x,y), and set the number of pixel bins to K: hist(V1)=Historgram(V1,bins=K); Step 25, calculate the cumulative distribution function CDF of the pixel intensity value l: Determine the pixel intensity value l where the cumulative distribution function reaches 99.9% in the histogram max : l max =max{l|CDF(l)≤0.999}; Select the pixels in the dark channel V1 with values ​​higher than l max The positions of the pixels in the image are determined as the brightest 0.001% and are used to represent the sources of atmospheric light. Step 26, taking the maximum value of each RGB channel from the original image according to the pixel area as the atmospheric light value A; Step 27: Calculate the original image I(x, y) using the filtered image V1'(x, y) and the atmospheric light value A to obtain the defogging enhanced image I':

3. The method for identifying marine fish and corals based on deep learning according to claim 1, characterized in that: The step 4 is specifically performed as follows: Build CBAM module: The CBAM module includes a channel attention module and a spatial attention module to guide the network to focus on significant features. The input feature F enters the CBAM module and passes through the channel attention module M in turn. c With the spatial attention module M s , and get the enhanced feature F out ; in Expressed as element-wise multiplication; Build the CMAM-C2f module: Add the CBAM module after the Bottleneck module of the C2f module and as the last layer of the CMAM-C2f module. The features extracted by multiple Bottleneck modules are further enhanced through the CBAM module. The CMAM-C2f module is embedded into the YoloV8 neural network model to replace the original C2f module, thereby obtaining an improved YoloV8 neural network model.

4. A method for identifying marine fish and corals based on deep learning according to any one of claims 1 to 3, characterized in that: In step 1, underwater image samples of marine fish are also collected as original images; Execute step 2; In step 3, the dehazed enhanced images are collected as a detection data set, and the fish species in the images and their locations in the images are marked; Execute step 4; Execute step 6 and use the detection data set marked in step 3 to train another improved YoloV8 neural network model. After the training is completed, a marine fish recognition model is obtained.

5. A deep learning-based marine fish and coral identification device, characterized in that: The method comprises a memory storing an executable program and a processor, wherein the processor runs the program to execute the method steps of any one of claims 1 to 3.

6. The deep learning-based marine fish and coral identification device according to claim 5, characterized in that: The processor runs the program to perform the method steps described in claim 4.

Images