Large-scale fish image classification method based on improved ConvNext

By improving the ConvNext network and SIMAM attention module, and combining it with the Focal Loss loss function, the problems of species coverage and scientific rigor in fish image recognition algorithms were solved, achieving high-accuracy fish classification and supporting ecological research and conservation.

CN118887477APending Publication Date: 2024-11-01SHENYANG INST OF COMPUTING TECH CO LTD THE CHINESE ACAD OF SCI
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411141470.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

Existing fish image recognition algorithms suffer from limited species coverage and insufficiently scientific and professional classification, making it difficult to meet the needs of ecological research and biodiversity conservation.

Method used

An improved ConvNext network structure is adopted, which combines the SIMAM attention module and the Focal Loss loss function. Through data preprocessing, feature map fusion and fully connected layer processing, the accuracy and scientific nature of fish image classification are improved.

Benefits of technology

It improves the accuracy of fish classification, resulting in more scientific classifications that support ecological research and biodiversity conservation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118887477A_ABST
    Figure CN118887477A_ABST
Patent Text Reader

Abstract

The invention belongs to the field of computer vision image classification, and particularly relates to a large-scale fish image classification method based on improved ConvNext. According to the method, the local bias prior knowledge and the context information of the image are considered, the feature maps of the image are extracted through application of the ConvNext pure convolutional neural network to the local bias prior knowledge of the image, and meanwhile, the feature maps are fused and the context feature information is extracted for feature fusion in a mode of embedding an SIMAM attention module. By introducing a class balance training technology and a Focal Loss loss function, the fitting ability of the model to a long-tail distributed data set is improved, the problem that the fitting ability of the model to data with a small sample size is poor is relieved, and then the generalization ability of the model is enhanced. According to the improved model, the accuracy and the F1 value are remarkably improved, the calculation burden is not increased, the attention degree of the model on fish data with few samples is improved to a certain extent, and the performance of a fish classification algorithm model can be effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of computer vision image classification, in particular to a large-scale fish image classification method based on improved ConvNext. Background Art

[0002] Image-based aquatic organism recognition technology is one of the key technologies in the current ecological field. Through accurate identification of aquatic organisms, we can better understand the structure and function of aquatic ecosystems and provide a scientific basis for the protection and management of aquatic organisms. It also helps to timely discover and evaluate problems in aquatic ecosystems and provide support for taking effective ecological protection measures.

[0003] The application of aquatic organism identification technology can help us better understand the distribution and quantity of fishery resources, and provide a scientific basis for the rational use and management of fishery resources. At the same time, this technology can also improve the efficiency and accuracy of fishery fishing, reduce bycatch and waste, and promote the sustainable development of fisheries. Aquatic organism identification technology can also provide ecologists with a large amount of accurate data, which promotes the study of aquatic ecosystems. Through the analysis of these data, we can gain an in-depth understanding of the ecological habits, interspecies relationships, community structure, etc. of aquatic organisms, reveal the operating mechanism and evolutionary laws of the ecosystem, and provide a scientific basis for ecology and biodiversity conservation. Summary of the invention

[0004] In order to improve the accuracy and F1 value of fish classification and the scientificity of classification, the present invention proposes a fish classification method of improved ConvNext. The model data set is a public fish classification data set proposed in FishNet. The benchmark data classification of this data set is based on aquatic biological taxonomy, including 8 taxonomic classes, 83 orders, 463 families, 3826 genera, 17,357 species, and 94,532 images. The order and family levels are used as the target levels for fish classification. After the image is preprocessed, it passes through the ConvNext Block network to output the feature map, and then the feature map is fused with context features through the SIMAM attention module, and then through the fully connected feedforward neural network, the fused feature map after the attention mechanism is fully connected and mapped, and finally the fish classification result is obtained. This method can better classify fish scientifically.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0006] A large-scale fish image classification method based on improved ConvNext includes the following steps:

[0007] 1) Obtain image data of aquatic fish and pre-process it;

[0008] 2) Down-sampling the pre-processed image data;

[0009] 3) Use a convolutional neural network-based classification model to classify the downsampled image data;

[0010] 4) Use the loss function to optimize the classification model.

[0011] The step 1) comprises the following steps:

[0012] 1.1) Label the fish at the family and order level in the biological classification hierarchy as the category label of the image, and divide the image data of each category into a training set and a test set;

[0013] 1.2) Decode the image data in the training set and convert it into Tensor format;

[0014] 1.3) Randomly rotate the image data in Tensor format with a fixed probability for data enhancement;

[0015] 1.4) Add a fixed proportion of random salt and pepper noise to the enhanced image data;

[0016] 1.5) Perform center cropping on the processed image data;

[0017] 1.6) Scale the pixel values ​​of the cropped image data uniformly so that the pixel data is between [0,1];

[0018] 1.7) Normalize the image data after pixel value scaling, and convert the input image pixel values ​​into data with fixed mean and variance.

[0019] The step 3) comprises the following steps:

[0020] 3.1) Input the downsampled image data into the ConvNext Block network layer to obtain the feature map;

[0021] 3.2) Input the feature map into the SIMAM attention module for feature fusion to obtain a new feature map;

[0022] 3.3) Input the new feature map into the downsampling module for processing;

[0023] 3.4) Loop through steps 3.1) to 3.3), treat the ConvNext Block network layer, the SIMAM attention module, and the downsampling module as a whole module, stack the modules in a ratio of 3:3:9:3, pass the input image data to the stacked modules layer by layer, obtain the final feature map, and input it into the fully connected layer;

[0024] 3.5) Perform softmax processing on the output of the fully connected layer to obtain the classification result.

[0025] The step 3.2) comprises the following steps:

[0026] 3.2.1) Input the feature map data into the energy function and calculate all feature values ​​uniformly to obtain the energy value;

[0027] 3.2.2) The energy value is weighted as the attention score on the feature map to obtain the feature map modified after the context information is obtained through the attention mechanism.

[0028] The energy function is specifically:

[0029]

[0030] Among them, e t represents the energy of the neuron, λ represents the regularization parameter, t represents the value of the current neuron, μ and σ 2 are the mean and variance of all neuron values, respectively.

[0031] The step 4) is specifically as follows:

[0032] The classification results are trained with the Focal Loss function for error backpropagation, and the current Loss value is compared with the best historical Loss value after each round of training. If the result of this round of training is smaller than the best historical Loss value, the network parameters of this round of training are saved.

[0033] The Focal Loss function L f for:

[0034]

[0035] Among them, α represents the weight factor, γ represents the modulation coefficient, and y * The predicted output represents the classification probability.

[0036] A large-scale fish image classification system based on improved ConvNext, including:

[0037] A data preprocessing module is used to obtain image data of aquatic fish and preprocess it;

[0038] A data partitioning module, used for down-sampling the pre-processed image data;

[0039] An image classification module, used to classify the downsampled image data using a classification model based on a convolutional neural network;

[0040] The model optimization module is used to optimize the classification model using the loss function.

[0041] A large-scale fish image classification device based on improved ConvNext includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement the large-scale fish image classification method based on improved ConvNext when executing the computer program.

[0042] A computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a large-scale fish image classification method based on improved ConvNext is implemented.

[0043] The present invention has the following beneficial effects and advantages:

[0044] 1. The present invention helps to improve the classification accuracy of fish;

[0045] 2. The classification results obtained by the present invention are more scientific based on biological taxonomy;

[0046] 3. Accurate classification of fish helps ecological research and biodiversity conservation. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is the algorithm flow chart of the present invention;

[0048] Figure 2 It is the structural diagram of ConvNext Block of the present invention;

[0049] Figure 3 It is the structural diagram of the SIMAM attention module of the present invention. DETAILED DESCRIPTION

[0050] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0051] The existing image-based fish recognition algorithms have problems such as limited coverage of recognized species and insufficient scientific and professional classification. This paper uses the improved ConvNext deep learning technology to construct an efficient, multi-category and high-accuracy fish recognition method, which has important ecological protection, technological innovation and social value. Figure 1 As shown, the main steps of the method are as follows:

[0052] Step 1: Obtain image data of aquatic organisms, label them at the order and family level, and divide the data set. According to the frequency of each fish at the order and family level in the data set, divide the fish into majority and minority categories. For each category, use the sample frequency of the category to sample into the training set, and ensure that 80% of the data is the training set and 20% of the data is the test set;

[0053] Step 2: Preprocess the image data. The purpose of the preprocessing process is to enhance the generalization ability of the model and ensure that the size of the input to the model is consistent. Specifically, it includes image enhancement methods such as random rotation and random noise, as well as a method of center cropping to 224*224 images to maintain the consistency of the input image size.

[0054] Step 3: Perform a preliminary downsampling of the image data before inputting it into the main network structure, and input the preprocessed image data into a 4*4 convolution kernel with a step size of 4 for downsampling;

[0055] Step 4: Enter the main network structure, input the downsampled image data into the ConvNext Block network layer, and extract the feature map through the characteristics of the convolutional neural network;

[0056] Step 5: The feature map obtained by ConvNext Block is used as input to the SIMAM attention module. In the SIMAM attention module, the energy of each pixel of all feature maps is calculated by using the energy function, and then the energy of each pixel is used as a weight to weight the pixel of the feature map to obtain a new feature map. The feature map is a new feature map calculated by the attention mechanism that perceives the context information, ensuring that the new feature map contains the feature representation learned from the context information;

[0057] Step 6: The new feature map result calculated by the SIMAM attention mechanism is used as input to the downsampling module for downsampling to reduce the size of the feature map;

[0058] Step 7: Enter the loop stage, take the above steps 4-6 as a whole module, and stack the modules in a ratio of 3:3:9:3. The difference between the four blocks in the ratio here is the feature dimension of the final output and the feature dimension of the input. Then, the input is passed to the stacked modules layer by layer to obtain the final feature map and input it to the fully connected layer.

[0059] Step 8: After obtaining the final feature map, perform softmax processing on the result of the fully connected layer to obtain the classification result;

[0060] Step 9: Due to the problem of long-tail classification of samples, the classification results are trained with error backpropagation using the Focal Loss function, and the best existing Loss value is compared after each round of training. If the training result of this round reaches the best, the network parameters of this round of training are saved.

[0061] The model dataset is a public fish classification dataset proposed in FishNet. The benchmark data classification of this dataset is based on aquatic biological taxonomy, including 8 taxonomic classes, 83 orders, 463 families, 3826 genera, 17,357 species, and 94,532 images. This paper uses the order and family levels as the target levels for fish classification, that is, there are 463 classification results at the family level and 83 classification results at the order level. For the order and family levels, the frequency of each category of data images is calculated, and the first 80% is used as the majority class fish and the last 20% is used as the minority class fish. In the majority class and minority class, the dataset is sampled according to the frequency, 80% as the training set and 20% as the test set. In this way, four classification scenes are split for model training, and the majority class and minority class are split. Then the images need to be preprocessed for the four scenes. The specific steps to be performed include:

[0062] 1) Decode the image data in the training set and convert it into Tensor format;

[0063] 2) Randomly rotate the image data in Tensor format with a fixed probability to perform data augmentation;

[0064] 3) Add a fixed proportion of random salt and pepper noise to the image data in Tensor format;

[0065] 4) Perform center cropping on the processed image data to ensure that the image data size input to the backbone network is fixed;

[0066] 5) uniformly scale the pixel values ​​of the cropped image data so that the pixel data is between [0, 1];

[0067] 6) Normalize the image data after pixel value scaling, and convert the input image pixel values ​​into data with fixed mean and variance.

[0068] The ConvNext Block described contains three main parts. Depthwise Convolution: Depthwise separable convolution, which is a lightweight convolution operation that can effectively reduce the amount of calculation and the number of parameters and improve the model reasoning performance. LayerNorm: Used to normalize the input features to ensure the consistency of data distribution. Here, the Transformer structure is used to replace BachtNorm with LayerNorm, and the regularization layer is reduced to further improve the performance of model reasoning. MLP with GeLU: Similar to the multi-layer perceptron layer in Transformer, but ConNext uses GeLU as the activation function to replace the ReLU activation function in the activation function, and reduces the number of activation functions. GeLU improves the nonlinear expression ability of the module through smoother nonlinear features, and GeLU has a non-zero gradient when the input is negative, which effectively alleviates the neuron pseudo-death phenomenon caused by the ReLU activation function when the input is less than zero, and can accelerate the training of the model. The GeLU function formula is:

[0069] GeLU(x)=x·P(X≤x),X~N(0,1)

[0070] See the detailed structure of ConvNext Block for details. Figure 2 .

[0071] The Focal Loss loss function is specifically a loss function that performs well in solving the problem of class imbalance and difficult and easy sample learning in classification tasks. It aims to solve the problem of class imbalance between positive and negative samples, and emphasizes the model's attention to difficult samples. Focal Loss introduces two key modulation factors α and γ to achieve its unique functions. The weight factor α is used to balance the weights of positive and negative samples to alleviate the problems caused by class imbalance, and the modulation coefficient γ is used to control the weight difference between difficult and easy samples. The introduction of the Focal Loss loss function further alleviates the problem of class imbalance in fish classification datasets. The formula of Focal Loss is as follows:

[0072]

[0073] like Figure 3 As shown in the figure, the SIMAM attention module is a simple but very effective attention module, which is embedded into the backbone convolutional neural network in this paper to improve the global context information extraction ability of the model. Unlike channel-based and spatial-based attention modules, this module works by inferring the three-dimensional attention weights in the feature map without adding parameters to the original network. Based on some well-known neuroscience theories, this module proposes an optimized energy function to find the importance of each neuron. The energy function calculation formula is as follows:

[0074]

[0075] where e t represents the energy of the neuron, λ represents the regularization parameter, t represents the value of the current neuron, μ and σ 2 are the mean and variance of all neuron values ​​respectively. For each input neuron, SIMAM can calculate the energy of each neuron through the above formula, and then determine which neurons are more important to get the attention score, thereby helping the network to better understand the global information and enhance the model's expressiveness. Here, as long as the feature map data is input as the neuron in the formula, a feature map gathered in the key part after attention calculation will be obtained.

Claims

1. A large-scale fish image classification method based on improved ConvNext, It is characterized in that The following steps are involved: 1) Obtain image data of aquatic fish and pre-process it; 2) Down-sampling the pre-processed image data; 3) Use a convolutional neural network-based classification model to classify the downsampled image data; 4) Use the loss function to optimize the classification model.

2. A large-scale fish image classification method based on improved ConvNext according to claim 1, It is characterized in that The step 1) comprises the following steps: (1.1) Label the fish at the family and order level in the biological classification hierarchy as the category label of the image, and divide the image data of each category into a training set and a test set; (1.2) Decode the image data in the training set and convert it into Tensor format; (1.3) Randomly rotate the image data in Tensor format with a fixed probability to perform data enhancement; (1.4) Add a fixed proportion of random salt and pepper noise to the enhanced image data; (1.5) Perform center cropping on the processed image data; (1.6) uniformly scale the pixel values ​​of the cropped image data so that the pixel data is between [0,1]; (1.7) Normalize the image data after pixel value scaling, and convert the input image pixel values ​​into data with fixed mean and variance.

3. A large-scale fish image classification method based on improved ConvNext according to claim 1, It is characterized in that The step 3) comprises the following steps: (3.1) Input the downsampled image data into the ConvNext Block network layer to obtain the feature map; (3.2) Input the feature map into the SIMAM attention module for feature fusion to obtain a new feature map; (3.3) Input the new feature map into the downsampling module for processing; (3.4) Loop through steps 3.1) to 3.3), treat the ConvNext Block network layer, the SIMAM attention module, and the downsampling module as a whole module, stack the modules in a ratio of 3:3:9:3, pass the input image data to the stacked modules layer by layer, obtain the final feature map, and input it into the fully connected layer; (3.5) The output of the fully connected layer is processed by softmax to obtain the classification result.

4. A large-scale fish image classification method based on improved ConvNext according to claim 3, It is characterized in that The step (3.2) comprises the following steps: (3.2.1) Input the feature map data into the energy function and perform a unified calculation on all feature values ​​to obtain the energy value; (3.2.2) The energy value is weighted as the attention score on the feature map to obtain the feature map modified after the context information is obtained through the attention mechanism.

5. A large-scale fish image classification method based on improved ConvNext according to claim 4, It is characterized in that The energy function is specifically: Among them, e t represents the energy of the neuron, λ represents the regularization parameter, t represents the value of the current neuron, μ and σ 2 are the mean and variance of all neuron values, respectively.

6. A large-scale fish image classification method based on improved ConvNext according to claim 1, It is characterized in that The step 4) is specifically as follows: The classification results are trained with the Focal Loss function for error backpropagation, and the current Loss value is compared with the best historical Loss value after each round of training. If the result of this round of training is smaller than the best historical Loss value, the network parameters of this round of training are saved.

7. A large-scale fish image classification method based on improved ConvNext according to claim 6, It is characterized in that The Focal Loss function L f for: Among them, α represents the weight factor, γ represents the modulation coefficient, and y * The predicted output represents the classification probability.

8. A large-scale fish image classification system based on improved ConvNext, It is characterized in that include: A data preprocessing module is used to obtain image data of aquatic fish and preprocess it; A data partitioning module, used for down-sampling the pre-processed image data; An image classification module, used to classify the downsampled image data using a classification model based on a convolutional neural network; The model optimization module is used to optimize the classification model using the loss function.

9. A large-scale fish image classification device based on improved ConvNext, It is characterized in that It comprises a memory and a processor; the memory is used to store a computer program; the processor is used to implement a large-scale fish image classification method based on improved ConvNext as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, It is characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, a large-scale fish image classification method based on improved ConvNext as described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Fine-grained image classification method based on convolutional neural network

    CN115631369A

  • Fish identification method and system, computer equipment and storage medium

    CN116092005A

  • Agricultural greenhouse extraction method based on attention mechanism and lightweight full convolutional network

    CN116385907A

  • Grain semantic segmentation method based on multi-feature fusion and related equipment

    CN118470334A

  • Intelligent fish identification method and sorting system

    CN118470385A