Plankton image enhancement method based on deep learning model

Abstract

The invention discloses a plankton image enhancement method based on a deep learning model, which belongs to the technical field of image enhancement, is used for plankton image enhancement, and comprises the steps of inputting a plankton image into a local feature extraction module, obtaining a plankton local feature map, and adjusting parameters of a global feature extraction module. Inputting the plankton image into a global feature extraction module to obtain a plankton global feature map, carrying out feature fusion on the plankton local feature map and the plankton global feature map to obtain a locally enhanced plankton image and a globally enhanced plankton image, and inputting the locally enhanced plankton image and the globally enhanced plankton image into a classifier, and obtaining a plankton image enhancement result. According to the method, the key local features of the plankton are obtained, the local features and the global features of the plankton are extracted, the enhanced plankton image is obtained, and the image enhancement accuracy of the plankton is improved through the image.

Description

Technical Field

[0001] The invention discloses a plankton image enhancement method based on a deep learning model, and belongs to the technical field of image enhancement. Background Art

[0002] In recent years, image enhancement methods based on deep learning have shown great potential, especially the Swin Transformer network, which provides a new idea for intelligent identification of plankton. The Swin Transformer network, which is commonly used for image classification in deep learning, is applied to microscopic images to obtain rich contextual information and global features of plankton. The global feature map of plankton is obtained through the Swin Transformer network, and then the enhanced image is used for classification to improve the recognition accuracy of plankton. In the patent application with publication number CN111898677A, a plankton automatic detection method based on Faster R-CNN is disclosed. The plankton data set is divided into training, verification and testing in proportion, and then the training set and verification set are input into the Faster R-CNN model for training and optimization; the trained model is tested in the test set through indicators such as average accuracy, so that plankton can be correctly intelligently classified. However, this method does not take into account the rich contextual information of plankton in the long range during the image enhancement of plankton, which makes its image enhancement accuracy have certain errors. Summary of the invention

[0003] The purpose of the present invention is to provide a plankton image enhancement method based on a deep learning model to solve the problem in the prior art that plankton image enhancement does not take contextual information into account, resulting in low image enhancement accuracy.

[0004] A plankton image enhancement method based on a deep learning model comprises the following steps: inputting a plankton image into a local feature extraction module to obtain a plankton local feature map, adjusting parameters of a global feature extraction module, inputting the plankton image into a global feature extraction module to obtain a plankton global feature map, performing feature fusion on the plankton local feature map and the plankton global feature map to obtain a locally enhanced and globally enhanced plankton image, and inputting the locally enhanced and globally enhanced plankton image into a classifier to obtain a plankton image enhancement result.

[0005] The local feature extraction module includes, in sequence, a two-dimensional convolutional layer, an activation pooling module, three two-dimensional convolutional layers, an activation pooling module, a two-dimensional convolutional layer, an activation pooling module, three two-dimensional convolutional layers, and an activation pooling module, and the activation pooling module includes an activation function ReLU layer and a two-dimensional maximum pooling layer.

[0006] The local feature extraction module has three shortcut connections. The first one is connected from the first two-dimensional convolutional layer to the second activation pooling module, the second one is connected from the second two-dimensional convolutional layer to the sixth two-dimensional convolutional layer, and the third one is connected from the third two-dimensional convolutional layer to the eighth two-dimensional convolutional layer. Feature fusion is performed at the end of the shortcut connection.

[0007] The global feature extraction module includes an image segmentation module, a linear embedding module and three floating Transformer modules in sequence, the linear embedding module includes a linear embedding layer and a floating Transformer layer, and the floating Transformer module includes a patch merging layer and a floating Transformer layer.

[0008] The dimensions of the linear embedding module are Where H, W, and C are the height, width, and number of channels of the feature. The linear embedding module consists of two units, each of which consists of a linear embedding layer and a floating Transformer layer. The size of the first floating Transformer module is It consists of two units, each of which consists of a patch merging layer and a floating Transformer layer; The size of the second floating Transformer module is It consists of six units, each of which consists of a patch merging layer and a floating Transformer layer; The size of the third floating Transformer module is It consists of two units, each of which consists of a patch merging layer and a floating Transformer layer.

[0009] The floating Transformer layer includes a layer normalization module and a window-based multi-head self-attention mechanism module in sequence. The output of the window-based multi-head self-attention mechanism module is added to the input of the floating Transformer layer. The addition result is input into the layer normalization module and the multi-layer perceptron module in sequence to obtain texture features. The output result of the multi-layer perceptron module is input into the multi-head self-attention mechanism module to obtain a result with context information.

[0010] The classifier consists of a fully connected layer and a Softmax activation function.

[0011] The cross-loss entropy function and AdamW optimization algorithm are used to adjust the parameters of the global feature extraction module.

[0012] Cross loss entropy function H 1 for: In the formula, q(x i ) is the i-th category x i The probability of n 1 is the number of pixel label types, p(x i ) is the one-hot encoding of the true value of plankton.

[0013] The output of the image segmentation module is non-overlapping image patches, the output of the linear embedding layer is an image patch with encoded position information and classification information, the output of the patch merging layer is a large image patch after merging adjacent image patches, and the output of the third floating Transformer layer is an information map with global features.

[0014] Compared with the prior art, the present invention has the following beneficial effects: the present invention obtains the key local features of plankton, obtains the long-range contextual semantic information and global features of plankton using the offset window mechanism, extracts the local features and global features of plankton and obtains an enhanced plankton image, which improves the image enhancement accuracy of plankton. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a structural diagram of the deep learning model of the present invention; Figure 2 for Figure 1 The structural diagram of the local feature extraction module in ; Figure 3 It is a technical flow chart of the present invention; Figure 4 Schematic diagram of the processing principle of the deep learning model. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] A plankton image enhancement method based on a deep learning model comprises the following steps: inputting a plankton image into a local feature extraction module to obtain a plankton local feature map, adjusting parameters of a global feature extraction module, inputting the plankton image into a global feature extraction module to obtain a plankton global feature map, performing feature fusion on the plankton local feature map and the plankton global feature map to obtain a locally enhanced and globally enhanced plankton image, and inputting the locally enhanced and globally enhanced plankton image into a classifier to obtain a plankton image enhancement result.

[0018] The local feature extraction module includes, in sequence, a two-dimensional convolutional layer, an activation pooling module, three two-dimensional convolutional layers, an activation pooling module, a two-dimensional convolutional layer, an activation pooling module, three two-dimensional convolutional layers, and an activation pooling module, and the activation pooling module includes an activation function ReLU layer and a two-dimensional maximum pooling layer.

[0019] The local feature extraction module has three shortcut connections. The first one is connected from the first two-dimensional convolutional layer to the second activation pooling module, the second one is connected from the second two-dimensional convolutional layer to the sixth two-dimensional convolutional layer, and the third one is connected from the third two-dimensional convolutional layer to the eighth two-dimensional convolutional layer. Feature fusion is performed at the end of the shortcut connection.

[0020] The global feature extraction module includes an image segmentation module, a linear embedding module and three floating Transformer modules in sequence, the linear embedding module includes a linear embedding layer and a floating Transformer layer, and the floating Transformer module includes a patch merging layer and a floating Transformer layer.

[0021] The dimensions of the linear embedding module are Where H, W, and C are the height, width, and number of channels of the feature. The linear embedding module consists of two units, each of which consists of a linear embedding layer and a floating Transformer layer. The size of the first floating Transformer module is It consists of two units, each of which consists of a patch merging layer and a floating Transformer layer; The size of the second floating Transformer module is It consists of six units, each of which consists of a patch merging layer and a floating Transformer layer; The size of the third floating Transformer module is It consists of two units, each of which consists of a patch merging layer and a floating Transformer layer.

[0022] The floating Transformer layer includes a layer normalization module and a window-based multi-head self-attention mechanism module in sequence. The output of the window-based multi-head self-attention mechanism module is added to the input of the floating Transformer layer. The addition result is input into the layer normalization module and the multi-layer perceptron module in sequence to obtain texture features. The output result of the multi-layer perceptron module is input into the multi-head self-attention mechanism module to obtain a result with context information.

[0023] The classifier consists of a fully connected layer and a Softmax activation function.

[0024] The cross-loss entropy function and AdamW optimization algorithm are used to adjust the parameters of the global feature extraction module.

[0025] Cross loss entropy function H 1 for: In the formula, q(x i ) is the i-th category x i The probability of n 1 is the number of pixel label types, p(x i ) is the one-hot encoding of the true value of plankton.

[0026] The output of the image segmentation module is non-overlapping image patches, the output of the linear embedding layer is an image patch with encoded position information and classification information, the output of the patch merging layer is a large image patch after merging adjacent image patches, and the output of the third floating Transformer layer is an information map with global features.

[0027] The structure of the deep learning model of the present invention is as follows Figure 1 As shown, the plankton image is input into the local feature extraction module to obtain the local feature map of the plankton, the parameters of the global feature extraction module are adjusted, the plankton image is input into the global feature extraction module to obtain the global feature map of the plankton, the local feature map of the plankton and the global feature map of the plankton are feature fused to obtain the locally enhanced and globally enhanced plankton image, and the locally enhanced and globally enhanced plankton images are output into the classifier to obtain the plankton image enhancement result.

[0028] The global feature extraction module includes an image segmentation module Patch Partition, a linear embedding module and three floating Transformer modules in sequence. The linear embedding module includes a linear embedding layer Liner Embedding and a floating Transformer layer Swim Transformer Block. The floating Transformer module includes a patch merging layer Patch Merbing and a floating Transformer layer.

[0029] Figure 1 The structure of the local feature extraction module in Figure 2 As shown, the local feature extraction module includes a two-dimensional convolution layer Conv2d, an activation pooling module, three two-dimensional convolution layers, an activation pooling module, a two-dimensional convolution layer, an activation pooling module, three two-dimensional convolution layers, and an activation pooling module in sequence, and the activation pooling module includes an activation function ReLU layer and a two-dimensional maximum pooling layer MaxPool2d.

[0030] The local feature extraction module has three shortcut connections. The first one is connected from the first two-dimensional convolutional layer to the second activation pooling module, the second one is connected from the second two-dimensional convolutional layer to the sixth two-dimensional convolutional layer, and the third one is connected from the third two-dimensional convolutional layer to the eighth two-dimensional convolutional layer. Feature fusion is performed at the end of the shortcut connection.

[0031] The present invention first simultaneously inputs the plankton image into the local feature extraction branch and the global feature extraction branch, uses the plankton local feature extraction module to obtain the local feature map of the plankton, and uses the plankton global feature extraction module to obtain the global feature map of the plankton. In the local feature extraction branch, a module for extracting local features of plankton is constructed by using operations such as convolution, and then the local features of the plankton are extracted. In the global feature extraction branch, the plankton image data is first divided into multiple image patches (patches), and the offset window mechanism and hierarchical structure design are used to obtain long-range rich context information and reduce the computational complexity of the input image size. Then, the multi-head self-attention based on the shift window and the multi-head attention based on the window are used alternately to capture the global context information, and then the adjacent image patches in the mapping map of the global information of the plankton are merged into a larger image patch, and then sent to the next plankton Transformer layer to obtain more advanced global semantic features, and obtain the global feature enhanced image of the plankton. Finally, the local feature map of plankton and the global feature map obtained above are fused to obtain an enhanced plankton image, which is used for intelligent recognition and classification of plankton.

[0032] The technical process of the present invention is as follows Figure 3 As shown, it involves four functional parts: data preprocessing, local feature extraction, global feature extraction, feature fusion, model training and prediction. The data preprocessing part is used to randomly crop, randomly flip and normalize the plankton image data (training set, validation set and test set), and convert them to meet the input of the model. Before inputting into the Swin Transformer network (plankton Transformer layer), the image needs to be cut into blocks, and the preprocessed image is cut into a series of non-overlapping image blocks of fixed size to facilitate input into the Swin Transformer network.

[0033] In the training phase, public plankton data was used to construct a training set, which was input into the local plankton feature extraction branch based on convolution operation and the global plankton feature extraction branch based on Swin Transformer network. In the local feature extraction branch, the local features of plankton were obtained after processing with the local plankton feature extraction module and a series of batch normalization and activation functions. In the Patch Partition module (linear embedding layer) of the global feature extraction branch, a series of non-overlapping image blocks were obtained; these non-overlapping image blocks were input into LinearEmbedding to obtain image blocks with encoded position information and classification information; the above image blocks were input into the Swin Transformer network to obtain low-level information maps such as plankton texture; the maps were input into PatchMerging (patch merging layer) to merge adjacent image blocks to construct a larger image block; the newly constructed image blocks were input into the Swin Transformer network to obtain rich context information and global features. Finally, the local features, rich contextual information and global features are input into the feature fusion module to obtain the enhanced plankton image, which is then input into the classifier. The Softmax function is then used to obtain the plankton classification result, and the cross-loss entropy function is used with the AdamW optimization algorithm to train the model.

[0034] In the testing phase, the constructed plankton test set is input into the deep learning model to obtain the local and global features of the plankton. Then the local features and global features are fused to obtain an enhanced plankton image. The final plankton image enhanced with local features, rich contextual semantic information and global features is input into the classifier, and the plankton is classified using the Softmax function, and finally the optimal plankton classification result of the model is obtained.

[0035] The processing principle of the deep learning model is shown as follows Figure 4 As shown, the plankton image enhancement method based on local features, rich context information and global features includes the following steps: Step S100, preprocessing the public plankton image data, including data normalization and data enhancement, wherein data normalization maps the plankton image data to the interval [0, 1] to make it conform to the normal distribution and easy to converge; data enhancement includes random data cropping and random data horizontal flipping. Random cropping can randomly crop all plankton images into images of size 224*224, and random horizontal rotation determines whether the image is horizontally flipped by setting the probability. Data enhancement can make the data richer, so that the model has stronger generalization ability.

[0036] Step S200, extract local features using the plankton local feature extraction module, which is composed of a convolution layer, an activation layer, and a maximum pooling layer. In order to better capture the local features of plankton, the module applies a residual connection operation, and performs upsampling operations on the results after the second maximum pooling operation, the sixth convolution operation, and the eighth convolution operation, respectively, and then performs feature fusion with the results after the first convolution layer, the second convolution, and the third convolution, respectively, to obtain rich local features of plankton; in addition, in the plankton global feature extraction module, a two-dimensional convolution operation is used to segment the image obtained in step S100, and the entire image is segmented into tokens with a window size of patch*patch as sample data (for example, if the patch is 4, the sample set is The image blocks are as follows: Among them, X i represents the token after the image is segmented, 16 represents the feature dimension of each token of the plankton image (4×4×1), and n is the number of tokens after the image is segmented Each token is then encoded to obtain an embedding vector of dimension 96. The final token dimension is B×56×56×96, where B represents the number of images processed in each batch. The above overall operation can be implemented through a two-dimensional convolution operation, and then the output result of the two-dimensional convolution is flattened and transformed.

[0037] Step S300, applying a linear embedding layer on the original feature value to map it to an arbitrary feature dimension. The linear embedding layer can be used to perform relative position encoding on each position in the sequence obtained in step S200. Relative position encoding encodes the sequence according to the relative relationship between the positions. During the encoding process, the relative distance and relationship between different positions in the sequence are taken into account. The encoded sequence can capture the relative information between positions by calculating the offset or relative position difference between different positions, thereby better processing the position information in long sequences.

[0038] Step S400: Token with position coding information (z l-1 ) is input into the Swin Transformer network. The specific architecture is as follows Figure 4As shown in the figure, after layer normalization (LN), the normalized features of each token are obtained. The normalized features of the token are input into the window-based multi-head self-attention mechanism (W-MSA). W-MSA reduces the computational complexity of the model by a divide-and-conquer approach. The global features obtained by W-MSA are added to the token with position encoding information. It is the result of adding the output of the window-based multi-head self-attention mechanism module to the input of the floating Transformer layer. Input into LN and Multilayer Perceptron (MLP) to obtain texture and other features The input is fed into a ShiftedWindow based Multi-head Self Attention (SW-MSA) mechanism module to obtain rich context information. The W-MSA module is defined as follows: Where Q, K, and V represent query, key, and value matrices, respectively; B 1 represents the relative position bias; d represents the dimension of the query / key.

[0039] Step S500: Input the token feature map obtained above into the Patch Merging module to merge adjacent tokens. This module is equivalent to a downsampling operation, but it also expands the dimension of the token to twice its original size while reducing the resolution, and then reduces the number of channels through a linear layer.

[0040] In step S600, the merged token is input into a Swin Transformer network with different depths and different numbers of shifted window attentions. The Swin Transformer network here is consistent with the Swin Transformer network architecture in step S400.

[0041] Step S700, inputting the plankton local features obtained by the local feature extraction module and the rich contextual semantic information and global features extracted by SwinTransformer into the feature fusion module, thereby obtaining an enhanced plankton image.

[0042] Step S800, the enhanced plankton image is input into the classification module, and after being output by the fully connected network, it is connected to the Softmax function to generate the final classification result.

[0043] The above step S200 is implemented by a convolution operation, and steps S300 to S600 are implemented by a SwinTransformer network. The cross entropy loss function is optimized using the AdamW optimization algorithm, and steps S200 to S800 are repeated.

[0044] To verify the present invention, the microscopic image data of plankton is taken as an example for image enhancement, and then the enhanced plankton image is classified and tested. The spatial size of the plankton image data set is not fixed and is a grayscale image. First, the acquired plankton image data is classified into 7 categories, and the data size of each category remains basically the same, and then all images are cropped to a size of 224*224. In the constructed plankton image data set, the ratio of the training set to the test set is 9:1, and then the verification set is divided from the training set according to a certain ratio to verify the model training results of each epoch. The detailed division of the 7 categories of plankton is shown in Table 1.

[0045] Table 1 Number of samples for training, validation and testing of plankton dataset category Training set Validation set Test Set Appendicularian 1196 178 112 Chaetognath 1072 208 175 Copepod 1056 272 149 Hydromedusae 1184 244 157 Siphonophore 1017 190 106 Trichodesmium 1104 208 170 Tunicate l 152 160 130 total 7781 1460 999 .

[0046] Under the above sample conditions, the method of the present invention is compared with the VGG-16 and ResNet-34 methods. For fairness, both VGG-16 and ResNet-34 networks are pre-trained models. The accuracy under top-1 and top-5 indicators is recorded. The experimental results are shown in Table 2.

[0047] Table 2 Comparison of model classification performance

[0048] Among them, Acc@1 and Acc@5 are the accuracy rates under top-1 and top-5 indicators respectively. It can be concluded from Table 2 that the image enhancement method proposed in the present invention has better classification results, and the classification effect has been improved to varying degrees. In the case of top-1, the accuracy of the category predicted by the PlanktonTNet network of the present invention and the real category is better than the comparison method. It can be concluded from Table 3 that the method proposed in the present invention is better than the model based on convolutional neural network in performance.

[0049] Table 3 Model performance comparison method Number of parameters Operation Amount VGG-16 134.29 15.41 ResNet-34 21.28 3.60 PlanktonTNet 19.51 12.23 .

[0050] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features may be replaced by equivalents, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A plankton image enhancement method based on a deep learning model, characterized in that: The method includes inputting a plankton image into a local feature extraction module to obtain a plankton local feature map, adjusting parameters of a global feature extraction module, inputting the plankton image into a global feature extraction module to obtain a plankton global feature map, performing feature fusion on the plankton local feature map and the plankton global feature map to obtain a locally enhanced and globally enhanced plankton image, and inputting the locally enhanced and globally enhanced plankton image into a classifier to obtain a plankton image enhancement result.

2. The plankton image enhancement method based on a deep learning model according to claim 1, characterized in that: The local feature extraction module includes, in sequence, a two-dimensional convolutional layer, an activation pooling module, three two-dimensional convolutional layers, an activation pooling module, a two-dimensional convolutional layer, an activation pooling module, three two-dimensional convolutional layers, and an activation pooling module, and the activation pooling module includes an activation function ReLU layer and a two-dimensional maximum pooling layer.

3. The plankton image enhancement method based on a deep learning model according to claim 2, characterized in that: The local feature extraction module has three shortcut connections. The first one is connected from the first two-dimensional convolutional layer to the second activation pooling module, the second one is connected from the second two-dimensional convolutional layer to the sixth two-dimensional convolutional layer, and the third one is connected from the third two-dimensional convolutional layer to the eighth two-dimensional convolutional layer. Feature fusion is performed at the end of the shortcut connection.

4. The plankton image enhancement method based on a deep learning model according to claim 3, characterized in that: The global feature extraction module includes an image segmentation module, a linear embedding module and three floating Transformer modules in sequence, the linear embedding module includes a linear embedding layer and a floating Transformer layer, and the floating Transformer module includes a patch merging layer and a floating Transformer layer.

5. The plankton image enhancement method based on a deep learning model according to claim 4, characterized in that: The dimensions of the linear embedding module are Where H, W, and C are the height, width, and number of channels of the feature. The linear embedding module consists of two units, each of which consists of a linear embedding layer and a floating Transformer layer. The size of the first floating Transformer module is It consists of two units, each of which consists of a patch merging layer and a floating Transformer layer; The size of the second floating Transformer module is It consists of six units, each of which consists of a patch merging layer and a floating Transformer layer; The size of the third floating Transformer module is It consists of two units, each of which consists of a patch merging layer and a floating Transformer layer.

6. The method for enhancing plankton images based on a deep learning model according to claim 5, characterized in that: The floating Transformer layer includes a layer normalization module and a window-based multi-head self-attention mechanism module in sequence. The output of the window-based multi-head self-attention mechanism module is added to the input of the floating Transformer layer. The addition result is input into the layer normalization module and the multi-layer perceptron module in sequence to obtain texture features. The output result of the multi-layer perceptron module is input into the multi-head self-attention mechanism module to obtain a result with context information.

7. The method for enhancing plankton images based on a deep learning model according to claim 6, characterized in that: The classifier consists of a fully connected layer and a Softmax activation function.

8. The plankton image enhancement method based on a deep learning model according to claim 7, characterized in that: The cross-loss entropy function and AdamW optimization algorithm are used to adjust the parameters of the global feature extraction module.

9. The method for enhancing plankton images based on a deep learning model according to claim 8, characterized in that: Cross loss entropy function H 1 for: In the formula, q(x i ) is the i-th category x i The probability of n 1 is the number of pixel label types, p(x i ) is the one-hot encoding of the true value of plankton.

10. The plankton image enhancement method based on a deep learning model according to claim 9, characterized in that: The output of the image segmentation module is non-overlapping image patches, the output of the linear embedding layer is an image patch with encoded position information and classification information, the output of the patch merging layer is a large image patch after merging adjacent image patches, and the output of the third floating Transformer layer is an information map with global features.

Images