Fine-grained phytoplankton microscopic image enhancement classification method and model building method thereof

Abstract

The invention provides a fine-grained phytoplankton microscopic image enhancement classification method and a model building method thereof, and belongs to the technical field of underwater image enhancement classification, and the method comprises the following steps: firstly, collecting phytoplankton samples from a plurality of sea areas, and obtaining image data through a microscopic imaging technology; screening, preprocessing and data enhancement are carried out on the images, and a high-quality training set and a high-quality test set are constructed; according to the method, a double-branch deep learning model HySwinFormer is developed, the double-branch deep learning model HySwinFormer comprises a multi-scale context capture branch based on hierarchical vision TransFormer and a local detail enhancement branch designed by adopting a parallel convolution kernel, and joint expression of features is realized through a designed weighted fusion strategy; and training and optimizing the model to obtain a final model. Experimental results show that the classification performance of the HySwinFormer model on multiple data sets is superior to that of an existing mainstream deep learning model, and the classification effect is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of underwater image enhancement and classification, and in particular relates to a fine-grained phytoplankton microscopic image enhancement and classification method and a model building method thereof. Background Art

[0002] Marine phytoplankton plays an indispensable role in the global ecosystem. As primary producers, they fix carbon dioxide through photosynthesis, drive the global carbon cycle, and provide a basic energy source for the marine food web. In addition, phytoplankton also plays an important role in regulating the acid-base balance of the ocean and generating oxygen. Accurate identification and classification of phytoplankton is not only critical for understanding its functions and dynamics in the ecosystem, but also provides important support for assessing marine environmental changes, monitoring ecological health, and formulating scientific environmental protection strategies.

[0003] Traditional phytoplankton classification methods mainly rely on morphological analysis and manually designed image processing technology, which is based on observing and measuring the geometric shape, texture characteristics and optical properties of cells, combined with pattern recognition algorithms for classification. Although certain results have been achieved in early studies, these methods still have many limitations. First, morphological analysis is highly dependent on professional knowledge and experience, and the operation process is cumbersome and time-consuming, which makes it difficult to meet the needs of efficient processing of large-scale samples. Secondly, when faced with complex and changeable phytoplankton morphologies, manually designed feature extraction methods often find it difficult to fully capture tiny but critical morphological differences, resulting in limited classification accuracy.

[0004] Compared with traditional methods, deep learning-based classification methods can automatically learn and extract complex features in data, effectively improving classification efficiency. However, when dealing with fine-grained recognition tasks of different species within the same genus, the distinguishing ability of existing models is still insufficient. On the one hand, the morphological characteristics of species in the same genus are often highly similar, and the slight differences in texture details, edge structures, and local features are difficult to fully capture through traditional feature extraction methods; on the other hand, microscopic images are limited by the resolution of optical equipment and changes in lighting conditions during the acquisition process, and are often accompanied by noise, uneven focus, and other problems, which further increase the complexity of the image. In addition, current research lacks systematic consideration and customized design of microscopic image characteristics. Although some methods have attempted to capture subtle morphological differences by introducing attention mechanisms, feature fusion strategies, or more complex feature extraction structures, related research is still insufficient on phytoplankton, a target object that is highly similar and sensitive to imaging conditions. General deep learning architectures are prone to confusion when dealing with such fine-grained image classification tasks, and how to integrate customized features that adapt to the microscopic environment in the model structure and training process remains to be explored. Summary of the invention

[0005] In view of the above problems, the first aspect of the present invention provides a method for building a fine-grained phytoplankton microscopic image enhancement classification model, comprising the following process: Step 1: Obtain phytoplankton samples from multiple representative sea areas and obtain original microscopic image datasets through microscopic imaging technology; Step 2, preprocessing the acquired original microscopic image data set to construct a training set and a test set; Step 3, build an end-to-end fine-grained image enhancement classification model HySwinFormer under the deep learning framework. The model adopts a dual-branch architecture, including a multi-scale context capture branch and a local detail enhancement branch; the multi-scale context capture branch is based on the hierarchical visual Transformer architecture, which includes four feature extraction stages. Each stage obtains multi-scale global feature representation by performing block division, block merging and multi-head self-attention module processing on the input image; a cross-block residual fusion module is set in the third stage. After a predetermined number of hierarchical visual Transformer modules are passed through this stage, the main path features and the residual path features are fused; the local detail enhancement branch consists of four consecutive stages, each of which contains a dual-path convolution module, which performs refined processing on the input features through multi-directional convolution kernels, thereby enhancing the expression of local features; finally, the two branches merge the global features and local features through a weighted fusion strategy in each corresponding stage to generate feature representations suitable for subsequent classification tasks; Step 4: Use the training set and test set to train and test the constructed fine-grained image enhancement classification model HySwinFormer, and select the model with the best performance as the final model.

[0006] Preferably, the preprocessing in step 2 includes data sorting, screening and preprocessing of the original image, eliminating blurred, incomplete or impure images, and performing data enhancement operations on the training set: using random cropping with a cropping ratio range of 80% to 100%, and generating different image areas through cropping; using random horizontal flipping to flip the image horizontally with a probability of 50%; using random rotation with a rotation angle range of ±30 degrees; using color jittering to adjust the brightness and contrast of the image within a range of ±0.2, and adjust the saturation within a range of ±0.3.

[0007] Preferably, in the multi-scale context capture branch in step 3, a structure based on a hierarchical visual Transformer is used to perform multi-scale extraction of features; first, in the initial stage, the input image is divided into non-overlapping small blocks, and the features of these blocks are projected into a high-dimensional space through a linear embedding layer; in subsequent stages, the resolution of the feature map is gradually reduced by merging and linearly transforming adjacent feature blocks, and the number of channels is increased to form a feature representation with a multi-scale hierarchy; At each stage, the features are first calculated by the sliding window multi-head self-attention W-MSA or shifting window multi-head self-attention SW-MSA module; the sliding window multi-head self-attention performs self-attention calculation within a fixed window, while the shifting window multi-head self-attention realizes cross-window information interaction by cyclically shifting the window. The calculation process is as follows:

[0008]

[0009]

[0010]

[0011] in, , and Respectively expressed in sequence Second and The feature representation obtained after processing is and In the Second and The intermediate feature variables to be input into the self-attention or multi-layer perceptron module after normalization in the secondary processing; Representation layer normalization processing, is a multi-layer perceptron module; in the feature processing process, the residual connection mechanism is used to add the processing result to the feature representation of the previous stage to realize the gradual transfer of features. The computational complexity of the sliding window self-attention W-MSA is:

[0012] in, and Represent the height and width of the feature map respectively, is the number of channels, is the window size.

[0013] Preferably, the cross-block residual fusion module introduces an improved efficient multi-scale attention EMA module, which realizes joint processing of main path and residual path features by grouping, directionally and spatially weighting, and channel re-weighting the main path and residual path features; In the EMA module, channel fusion and feature reweighting are achieved through 1×1 convolution, and the use of 3×3 convolution is partially reduced. First, the input feature channel dimensions are grouped and divided into several subsets. For each subset feature, global average pooling operations are performed in the height direction and the width direction respectively to extract global information in two orthogonal directions. Then, the pooling results in the two directions are concatenated, and directional weighting factors are generated through 1×1 convolution and Sigmoid activation function. Another branch applies 3×3 convolution operation to the input features of each subset as an auxiliary branch to extract local features. The completion method After weighting, the weighted features are group normalized, and global average pooling and Softmax normalization are performed. The spatial correlation information between subset features is captured through matrix multiplication operations to generate spatial weighting factors. Subsequently, the directional weighted results are fused with the spatial weighted results, and nonlinear mapping is applied to further perform channel reweighting processing. Finally, the weighted results are applied element by element to the original features, so that the deep features are redistributed and expressed according to actual needs. Finally, the final fusion weights are obtained and applied through the following steps to complete the final fusion process of the main path features and the residual path features: S1, extracting main path features after every six hierarchical visual Transformer modules in the third stage and residual path characteristics , align the dimensions of the two to ensure that the features can directly participate in the subsequent fusion operation; S2, generates the fusion weights of the main path and the residual path respectively and , the weights are calculated based on a channel-by-channel weighting mechanism:

[0014]

[0015] in, and is a learnable parameter used to calculate the weight distribution of the main path and the residual path; the Softmax operation here refers to exponential mapping and normalization of each component of the input vector to obtain a weighted distribution; S3, according to the generated weights, the main path features and the residual path features are weighted and summed element by element to obtain the fused features :

[0016] in, represents an element-wise multiplication operation, The fused features.

[0017] Preferably, the local detail enhancement branch is used to enhance the feature expression of the local area, and the branch comprises four dual-path convolution modules, each module adopts convolution kernels of sizes 3×7 and 7×3 to extract detail information in the horizontal and vertical directions; wherein, the convolution layers of the first three dual-path convolution modules adopt a stride of 2 to downsample the feature map and expand the number of output channels to twice the number of input channels; the convolution layer of the last dual-path convolution module adopts a stride of 1 to keep the spatial size and number of channels of the feature map unchanged; each module processes the input features through parallel multi-directional convolution operations, and then fuses the convolution outputs of the two directions by element-by-element addition to generate an enhanced local detail feature map; all convolution layers adopt a padding method that can keep the output feature map consistent with the input feature map in spatial size.

[0018] Preferably, the weighted fusion strategy in step 3 is specifically: The features of the multi-scale context capture branch and the local detail enhancement branch are weighted fused in each stage, and each fusion is performed using the following formula:

[0019] in Represents the global features from the multi-scale context capture branch; Represents local features from the local detail enhancement branch; represents the fused feature representation; The value of is set to 0.6, that is, in the fusion process, the global features from the multi-scale context capture branch are given a high weight, while the local features from the local detail enhancement branch are reflected with supplementary weights. The specific steps of feature fusion are: S1, in the initial stage, local and global initial features are extracted through two branches respectively, and fused according to the above weighted formula to obtain the basic representation; S2, in the middle stage, after reducing the dimension of the features outputted in the previous stage through block merging operation, the global features and local features are fused again according to the weighted formula; S3, in the deep stage, the processed deep local and global features are finally fused, and the fused features are globally averaged pooled, and finally mapped to the target classification space through the fully connected layer to obtain the classification result.

[0020] Preferably, the fine-grained image enhancement classification model HySwinFormer includes three types; the network depth of the multi-scale context capture branch of the first type is 2, 2, 6, 2, the embedding dimensions are 96, 192, 384, 768 respectively, and the number of input channels and the number of output channels of the local detail enhancement branch are 96, 192, 384, 768 and 192, 384,768, 768 respectively; the network depth of the second type is 2, 2, 18, 2, the embedding dimensions are 96, 192, 384, 768, and the number of input channels and the number of output channels of the local detail enhancement branch are the same as the first type; the network depth of the third type is 2, 2, 18, 2, the embedding dimensions are increased to 128, 256, 512, 1024, and the number of input channels and the number of output channels of the local detail enhancement branch are 128, 256, 512, 1024 respectively. 1024 and 256, 512, 1024, 1024.

[0021] The second aspect of the present invention provides a fine-grained phytoplankton microscopic image enhancement classification method, comprising the following process: S1, real-time acquisition of high-resolution microscopic images of phytoplankton to be classified; S2, inputting the microscopic image of the phytoplankton to be classified into the fine-grained image enhancement classification model constructed by the construction method described in the first aspect; S3, outputs real-time classification results.

[0022] The third aspect of the present invention provides a fine-grained phytoplankton microscopic image enhancement classification device, the device includes at least one processor and at least one memory, the processor and the memory are coupled; the memory stores a computer execution program of the fine-grained image enhancement classification model built by the building method described in the first aspect; when the processor executes the computer execution program stored in the memory, the processor executes a fine-grained phytoplankton microscopic image enhancement classification method.

[0023] The fourth aspect of the present invention provides a computer-readable storage medium, which stores a computer execution program of the fine-grained image enhancement classification model constructed by the construction method described in the first aspect. When the computer execution program is executed by a processor, the processor executes a fine-grained phytoplankton microscopic image enhancement classification method.

[0024] Compared with the prior art, the present invention has the following beneficial effects: The present invention effectively solves the limitations of traditional classification methods and general deep learning models in fine-grained classification tasks by designing a deep learning model suitable for fine-grained phytoplankton microscopic image classification and its construction method. Aiming at the tiny morphological differences in phytoplankton microscopic images, the present invention proposes a dual-branch architecture HySwinFormer model, including a multi-scale context capture branch and a local detail enhancement branch, which realize the joint representation of global information and local detail features through a reasonable feature fusion strategy.

[0025] In the model design, the cross-block residual fusion module is introduced into the multi-scale context capture branch to integrate the residual path and main path features to ensure the effective transmission of deep features; the multi-path convolution module is used in the local detail enhancement branch to effectively capture the unique texture, edge and local structure features in the image through parallel multi-directional convolution operations. These two modules optimize different characteristics in their respective branches and jointly improve the fine-grained classification effect.

[0026] In the data processing link, the present invention combines the imaging characteristics of microscopic images and designs a set of customized data enhancement strategies, thereby enhancing the model's adaptability to different imaging conditions. For model training, the present invention formulates a training strategy for dynamic learning rate adjustment, and further optimizes the performance of the model through an adaptive parameter update method. In addition, the present invention provides comprehensive data support for model training by making a high-quality phytoplankton microscopic image dataset. These designs not only meet the actual needs of phytoplankton classification tasks, but also provide important inspiration and reference for other related fine-grained microscopic image classification tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a schematic diagram of the process of the fine-grained phytoplankton microscopic image enhancement classification method proposed in the present invention.

[0028] Figure 2 Schematic diagram of a sample microscopic image of phytoplankton.

[0029] Figure 3 Schematic diagram of the overall structure of the HySwinFormer model.

[0030] Figure 4 Schematic diagram of the structure of the cross-block residual fusion module.

[0031] Figure 5 Schematic diagram of the structure of the dual-path convolution module.

[0032] Figure 6 It is the learning rate decay curve.

[0033] Figure 7 Comparison of different models’ attention areas.

[0034] Figure 8 This is a schematic diagram of the simple structure of the fine-grained phytoplankton microscopic image enhancement classification device. DETAILED DESCRIPTION

[0035] The invention will be further described below in conjunction with specific embodiments.

[0036] This embodiment further illustrates the method proposed by the present invention through a specific experimental process. The overall process is as follows: Figure 1 shown.

[0037] 1. Raw data collection and preprocessing The data collection of this study covers multiple areas, among which the Bohai Sea, the Yellow Sea, the coastal areas of Qingdao, the waters near Zhoushan, and the offshore areas of Sanya are the key sea areas for collection, which are extremely representative. Samples were collected using a phytoplankton trawl (mesh diameter 20 microns) or a CTD (underwater temperature, salinity and depth measurement system) with its own water sampler to obtain phytoplankton samples at different depths and environmental conditions. The sampling depth does not exceed 200 meters. After separation and processing, the collected samples were imaged using a Nikon Ti2-U inverted fluorescence differential interference microscope (Inversed Fluorescent DIC Microscope) to obtain high-resolution microscopic images.

[0038] The collected original microscopic images need to undergo strict preprocessing operations before entering the model training. First, the image data is screened to remove blurred, incomplete, and impure images to ensure data quality. Subsequently, images that meet the quality standards are selected and standardized preprocessing operations are performed, including adjustments to the image brightness, contrast, and pixel value range to unify the input features of the image. In order to further optimize the model's adaptability to diverse data, data enhancement operations were performed on the training set data during the preprocessing stage, including random cropping, random horizontal flipping, random rotation, and color jittering. Among them, the random cropping ratio range is 80% to 100%, which is used to generate views of different areas; random horizontal flipping flips the image with a probability of 50%; the random rotation angle is controlled within the range of ±30 degrees; and the color jitter is adjusted within the range of ±0.2 for brightness and contrast and ±0.3 for saturation. In this embodiment, the data set is divided into a training set and a test set at a ratio of 7:3. The experiment was based on images of the genus Rhizosolenia in a self-made data set. There are multiple groups of species with very small morphological differences within the genus. Sample examples are shown in Figure 2 .

[0039] 2. Model building Based on the self-made microscopic image dataset, this embodiment designs and builds a deep learning model HySwinFormer, whose structure is as follows: Figure 3 As shown. The model adopts a dual-branch architecture, including a multi-scale context capture branch for global feature extraction and a local detail enhancement branch for capturing detail features. The two branches achieve joint representation through staged feature fusion to meet the needs of fine-grained phytoplankton image classification tasks. This embodiment also designs three model variants, namely Tiny version, Small version and Base version, to adapt to different computing resource conditions and data set sizes. The specific information is shown in Table 1.

[0040] Table 1 Parameter settings of different versions of the model

[0041] The multi-scale context capture branch is designed based on the Swin Transformer architecture. This branch first performs patch segmentation on the input image through a convolutional layer with a stride of 4 and a convolution kernel size of 4×4, divides the image into 4×4 non-overlapping regions, and embeds it into a high-dimensional feature space. In subsequent stages, the patches of the feature map are merged through the Patch Merging layer, and the number of channels is reduced by using the linear layer while reducing the spatial resolution, gradually building a multi-scale feature representation.

[0042] At each stage, the features are first calculated by the sliding window multi-head self-attention (W-MSA) or shifted window multi-head self-attention (SW-MSA) module. W-MSA performs self-attention calculations within a fixed window, while SW-MSA implements cross-window information interaction by cyclically shifting the window. The above calculation process is as follows:

[0043]

[0044]

[0045]

[0046] in, Indicates LayerNorm normalization, It is a multi-layer perceptron module, which realizes the gradual transfer of features through the residual connection mechanism. The computational complexity of the sliding window self-attention (W-MSA) is:

[0047] in, and Represent the height and width of the feature map respectively, is the number of channels, is the window size.

[0048] In order to solve the problem that the existing architecture may cause gradient disappearance or limited feature expression in the deeper stage of the network, this embodiment designs a cross-block residual fusion module in the multi-scale context capture branch, and its structure is as follows: Figure 4 As shown in the figure. This module combines residual connections and the task-driven Efficient Multi-Scale Attention (EMA) module to achieve the joint expression of the main path and residual path features. Specifically, after every six Swin Transformer Blocks in the third stage, the features of the main path and the residual path are integrated through residual connections.

[0049] The integrated features are simultaneously input into the EMA module for fusion. First, the main path features and residual path characteristics Reshape them to ensure that they have the same dimension. Then, the feature fusion weights are generated through the multi-scale feature interaction mechanism. The weight generation process includes channel-by-channel weighting operations and normalizing the weights using the Softmax function. The specific calculation process is as follows:

[0050]

[0051]

[0052] in, and are learnable parameters for generating the weights of the main path and residual path, respectively. represents an element-wise multiplication operation, The fused features.

[0053] In particular, this embodiment optimizes the original EMA module, mainly through 1×1 convolution to achieve efficient channel fusion and feature reweighting to enhance the original feature expression ability. Compared with the original version, this embodiment reduces the reliance on 3×3 convolution, thereby avoiding the problem of over-smoothing of detail information that may be caused by its larger receptive field.

[0054] To further enhance the model's ability to capture fine-grained features, this embodiment designs a local detail enhancement branch in the model. This branch focuses on capturing detail information in different directions in the image to more comprehensively extract texture and edge features. The local detail enhancement branch contains four dual-path convolution modules, whose structure is as follows: Figure 5As shown in the figure. Each module uses convolution kernels of size 3×7 and 7×3 respectively to effectively extract detail features in the horizontal and vertical directions. The convolution layers of the first three modules use a stride of 2 to downsample the feature map and make the number of output channels twice the number of input channels. The convolution layer of the last module uses a stride of 1 to keep the spatial size and number of channels of the feature map consistent. Each dual-path convolution module processes the input features through parallel multi-directional convolution operations, and then fuses the convolution outputs of the two directions by element-by-element addition to generate an enhanced detail feature map.

[0055] In order to achieve the effective joint expression of the multi-scale context capture branch and the local detail enhancement branch features, a staged feature weighted fusion mechanism is designed. This mechanism performs weighted fusion of global features and local features at each stage to give full play to the complementary advantages of the two types of features. The fusion calculation formula is as follows:

[0056] in, The value of is set to 0.6, and the contribution of global features (extracted by the multi-scale context capture branch) is emphasized during the fusion process, while local features (extracted by the local detail enhancement branch) are enhanced by The weight of is supplemented.

[0057] In the process of feature fusion, the initial global features and local features are first extracted through the multi-scale context capture branch and the local detail enhancement branch, and then fused according to the weighted formula to generate a basic representation. Subsequently, the Patch Merging operation is performed on the features output in the previous stage to reduce the dimension, and the global features and local features are weightedly fused again to further enhance the feature expression. In the deep level stage, the fused global features and local features are used to generate a high-level feature representation through global average pooling, and finally the fused features are mapped to the target classification space using the fully connected layer to complete the classification task.

[0058] 3. Model training The implementation platform of a fine-grained phytoplankton microscopic image classification method in this embodiment is based on the Ubuntu 22.04 operating system, the programming language is Python 3.12, the deep learning framework uses PyTorch 2.3.0, and the CUDA version is 12.1. The model was trained for 100 rounds on a system equipped with an NVIDIA GeForce RTX 3090 GPU (video memory 24GB), which lasted about 2 hours. The relevant settings of each training parameter are also recorded in detail in the experiment, as shown in Table 2.

[0059] Table 2 Experimental configuration details

[0060] The learning rate scheduling strategy uses the ReduceLROnPlateau method, where the decay factor is set to 0.5, the patience parameter is 10, and the minimum learning rate is set to 1e-6. This method can decay the learning rate to 0.5 times the original value when the performance of the validation set does not improve within 10 rounds, thereby preventing the model from falling into a local optimum. The learning rate decay curve is as follows Figure 6 shown.

[0061] The cross-entropy loss function is used in model training to calculate the average cross-entropy loss for each batch of samples. For a batch of samples, the average cross entropy loss function is calculated as shown in formula x, where Indicates The sample in The true label of the class, It is The sample is predicted by the model as The probability of the class.

[0062]

[0063] 4. Experimental Results In order to fully verify the effectiveness of the method proposed in this paper, two parts of experiments were designed and completed. The first part is a comparative experiment, which evaluates the fine-grained classification ability of the proposed HySwinFormer model by comparing its performance with existing methods. The second part is an ablation experiment, which analyzes the specific contribution of each module and design to the model performance.

[0064] The experiment used three classification evaluation indicators, including classification accuracy (Acc), Matthews correlation coefficient (MCC), and F1 score (F1). Among them, classification accuracy measures the overall classification ability of the model, Matthews correlation coefficient is suitable for evaluation scenarios with imbalanced categories, precision reflects the accuracy of positive class prediction, recall indicates the coverage of positive class samples, and F1 score combines the balance between precision and recall. The higher the value of each indicator, the better the performance of the model.

[0065] Comparative experiment: This example selects a variety of deep learning models with excellent classification effects in recent years as comparison models, including typical architectures such as ResNet, ViT, and ConvNeXt. The experiment is based on the image data of the genus Rhizocystis in the self-made data set to demonstrate the advantages of the present invention in the classification of fine-grained phytoplankton microscopic images. The specific comparison results are shown in Table 3.

[0066] Table 3 Comparison of performance indicators on self-made datasets

[0067] It can be seen that the HySwinFormer model proposed in the present invention is superior to similar models with similar parameter quantities in all evaluation indicators.

[0068] This embodiment also conducts comparative experiments on two representative open source datasets, Plankton Set 1.0 and IFCB. These two public datasets have different data scales and diversity, and can comprehensively evaluate the robustness and applicability of the model in fine-grained classification tasks. The specific comparison results are shown in Table 4.

[0069] Table 4 Comparison of performance indicators on Plankton Set 1.0 and IFCB datasets

[0070] In summary, compared with other advanced models, the performance of the present invention on two public datasets is more outstanding, and each performance index reaches or exceeds the current best level.

[0071] Ablation experiment: In order to evaluate the performance contribution of each innovative design in the fine-grained phytoplankton microscopic image classification task, this experiment gradually added cross-block residual fusion modules and local detail enhancement branches, and analyzed the specific improvement of each part on the model performance. The results are shown in Table 5.

[0072] Table 5 Ablation experiment

[0073] The ablation experiment results show that each of the above components contributes to the performance gain of the present invention.

[0074] 5. Focus on area visualization analysis To further verify the ability of HySwinFormer to capture feature information in the fine-grained phytoplankton image classification task, we used Gradient-weighted Class Activation Mapping (Grad-CAM) to compare the focus areas of each model. Grad-CAM is a gradient-based visualization technique that generates a heat map of the feature map to show the importance of different regions of the input image in the fine-grained classification results, allowing us to more intuitively understand the decision basis of the model.

[0075] This example compares the visualization results of the Tiny version of the present invention with those of ResNet50, DenseNet201, ConvNeXt-T, and Swin-T, and selects some typical images from the test set for display. To facilitate comparative analysis, the key feature areas judged from a biological perspective are marked with red boxes in the input images. Figure 7 As shown, the HySwinFormer model is more accurate in judging key areas and can effectively capture subtle differences such as cell width and cell apex morphology, which is more consistent with human judgment criteria. Other models also show their own characteristics. For example, Swin Transformer is better at capturing global features, while ResNet is good at extracting local features. However, in some cases, some models may mistake irrelevant information for important information. Overall, thanks to its unique architectural design, the present invention performs better in capturing subtle features in Rhizopus images.

[0076] 6. Summary and analysis Through the experiments in this embodiment, it can be seen that the method for constructing a fine-grained phytoplankton microscopic image classification model proposed in the present invention has the following overall process: Figure 1 As shown, the specific steps include: Step 1: Complete data collection, obtain phytoplankton samples from multiple representative sea areas, and obtain original microscopic image data sets through microscopic imaging technology; Step 2: sort, filter and preprocess the original images, remove blurry, incomplete or impure images, build training sets and test sets, and perform data enhancement on the training sets. Step 3: Design and build a classification model HySwinFormer under the deep learning framework. The model adopts a dual-branch design, in which the multi-scale context capture branch is based on the hierarchical visual Transformer architecture and is responsible for extracting global context information; the local detail feature enhancement branch captures multi-directional detail features through parallel convolution kernel design; the features of the two branches are jointly represented through a weighted fusion strategy; Step 4: Based on the data set processed in step 2, the HySwinFormer model constructed in step 3 is trained, and the model parameters are updated through iterative optimization; Step 5: Use the test set constructed in step 2 to evaluate the performance of the trained HySwinFormer model, calculate various key performance indicators and generate relevant visualization results; finally, select the model with the best performance as the final model.

[0077] like Figure 8As shown, the present invention also provides a fine-grained phytoplankton microscopic image enhancement classification device, the device includes at least one processor and at least one memory, and also includes a communication interface and an internal bus; a computer execution program is stored in the memory; a computer execution program of a fine-grained image enhancement classification model constructed by the construction method as described above is stored in the memory; when the processor executes the computer execution program stored in the memory, the processor can execute a fine-grained phytoplankton microscopic image enhancement classification method. The internal bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a U disk, a mobile hard disk, a read-only memory, a disk or an optical disk.

[0078] The device may be provided as a terminal, a server or other forms of device. In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0079] The present invention also provides a computer-readable storage medium, which stores a computer execution program of the fine-grained image enhancement classification model constructed by the construction method as described above. When the computer execution program is executed by a processor, the processor can execute a fine-grained phytoplankton microscopic image enhancement classification method.

[0080] Specifically, a system, device or equipment equipped with a readable storage medium may be provided, on which a software program code for implementing the functions of any of the above-mentioned embodiments is stored, and a computer or processor of the system, device or equipment reads and executes the instructions stored in the readable storage medium. In this case, the program code read from the readable medium itself can implement the functions of any of the above-mentioned embodiments, so the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of the present invention.

[0081] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0082] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for building a fine-grained phytoplankton microscopic image enhancement classification model, characterized in that: The process includes: Step 1: Obtain phytoplankton samples from multiple representative sea areas and obtain original microscopic image datasets through microscopic imaging technology; Step 2, preprocessing the acquired original microscopic image data set to construct a training set and a test set; Step 3: Build an end-to-end fine-grained image enhancement classification model HySwinFormer under the deep learning framework. The model adopts a dual-branch architecture, including a multi-scale context capture branch and a local detail enhancement branch. The multi-scale context capture branch is based on a hierarchical visual Transformer architecture and includes four feature extraction stages. Each stage obtains a multi-scale global feature representation by performing block division, block merging, and multi-head self-attention module processing on the input image. In the third stage, a cross-block residual fusion module is set up. After a predetermined number of layered visual Transformer modules, the main path features and the residual path features are fused. The local detail enhancement branch consists of four consecutive stages, each of which contains a dual-path convolution module. The input features are refined through multi-directional convolution kernels to enhance the expression of local features. Finally, the two branches merge the global features and local features through a weighted fusion strategy in each corresponding stage to generate feature representations suitable for subsequent classification tasks. Step 4: Use the training set and test set to train and test the constructed fine-grained image enhancement classification model HySwinFormer, and select the model with the best performance as the final model.

2. A method for building a fine-grained phytoplankton microscopic image enhancement classification model as claimed in claim 1, characterized in that: The preprocessing in step 2 includes data sorting, screening and preprocessing of the original images, eliminating blurred, incomplete or impure images, and performing data enhancement operations on the training set: using random cropping with a cropping ratio range of 80% to 100%, and generating different image areas through cropping; using random horizontal flipping to flip the image horizontally with a probability of 50%; using random rotation with a rotation angle range of ±30 degrees; using color jittering to adjust the brightness and contrast of the image within a range of ±0.2, and adjust the saturation within a range of ±0.

3.

3. A method for building a fine-grained phytoplankton microscopic image enhancement classification model as claimed in claim 1, characterized in that: In the multi-scale context capture branch in step 3, a structure based on a hierarchical visual Transformer is used to extract features at multiple scales. First, in the initial stage, the input image is divided into non-overlapping small blocks, and the features of these blocks are projected into a high-dimensional space through a linear embedding layer. In subsequent stages, the resolution of the feature map is gradually reduced by merging and linearly transforming adjacent feature blocks, and the number of channels is increased to form a feature representation with a multi-scale hierarchy. At each stage, the features are first calculated by the sliding window multi-head self-attention W-MSA or shifting window multi-head self-attention SW-MSA module; the sliding window multi-head self-attention performs self-attention calculation within a fixed window, while the shifting window multi-head self-attention realizes cross-window information interaction by cyclically shifting the window. The calculation process is as follows: in, , and Respectively expressed in sequence Second and The feature representation obtained after processing is and In the Second and The intermediate feature variables to be input into the self-attention or multi-layer perceptron module after normalization in the secondary processing; Representation layer normalization processing, is a multi-layer perceptron module; in the feature processing process, the residual connection mechanism is used to add the processing result to the feature representation of the previous stage to realize the gradual transfer of features. The computational complexity of the sliding window self-attention W-MSA is: in, and Represent the height and width of the feature map respectively, is the number of channels, is the window size.

4. A method for building a fine-grained phytoplankton microscopic image enhancement classification model as claimed in claim 1, characterized in that: The cross-block residual fusion module introduces an improved efficient multi-scale attention EMA module, which realizes the joint processing of the main path and residual path features by grouping, directionally and spatially weighting, and channel re-weighting the main path and residual path features; In the EMA module, channel fusion and feature reweighting are achieved through 1×1 convolution, and the use of 3×3 convolution is partially reduced; first, the input feature channel dimensions are grouped and divided into several subsets, and for each subset feature, global average pooling operations are performed in the height direction and the width direction respectively to extract global information in two orthogonal directions; then, the pooling results in the two directions are spliced, and the directional weighting factor is generated through 1×1 convolution and Sigmoid activation function; another branch applies 3×3 convolution operation to the input features of each subset as an auxiliary branch to extract local features; After completing the directional weighting, the weighted features are group normalized, and global average pooling and Softmax normalization are performed. The spatial correlation information between subset features is captured through matrix multiplication operations to generate spatial weighting factors. Subsequently, the directional weighted results are fused with the spatial weighted results, and nonlinear mapping is applied to further perform channel reweighting processing; finally, the weighted results are applied element by element to the original features, so that the deep features are redistributed and expressed according to actual needs; finally, the final fusion weights are obtained and applied through the following steps to complete the final fusion process of the main path features and the residual path features: S1, extracting main path features after every six hierarchical visual Transformer modules in the third stage and residual path characteristics , align the dimensions of the two to ensure that the features can directly participate in the subsequent fusion operation; S2, generates the fusion weights of the main path and the residual path respectively and , the weights are calculated based on a channel-by-channel weighting mechanism: in, and is a learnable parameter used to calculate the weight distribution of the main path and the residual path; the Softmax operation here refers to exponential mapping and normalization of each component of the input vector to obtain a weighted distribution; S3, according to the generated weights, the main path features and the residual path features are weighted and summed element by element to obtain the fused features : in, represents an element-wise multiplication operation, The fused features.

5. A method for building a fine-grained phytoplankton microscopic image enhancement classification model as claimed in claim 1, characterized in that: The local detail enhancement branch is used to enhance the feature expression of the local area. The branch includes four dual-path convolution modules, each of which uses convolution kernels of sizes 3×7 and 7×3 to extract detail information in the horizontal and vertical directions. The convolution layers of the first three dual-path convolution modules use a stride of 2 to downsample the feature map and expand the number of output channels to twice the number of input channels. The convolution layer of the last dual-path convolution module uses a stride of 1 to keep the spatial size and number of channels of the feature map unchanged. Each module processes the input features through parallel multi-directional convolution operations, and then fuses the convolution outputs of the two directions by element-by-element addition to generate an enhanced local detail feature map. All convolution layers use a padding method that can keep the output feature map consistent with the input feature map in spatial size.

6. A method for building a fine-grained phytoplankton microscopic image enhancement classification model according to claim 1, characterized in that: The weighted fusion strategy in step 3 is specifically: The features of the multi-scale context capture branch and the local detail enhancement branch are weighted fused in each stage, and each fusion is performed using the following weighted formula: in Represents the global features from the multi-scale context capture branch; Represents local features from the local detail enhancement branch; represents the fused feature representation; The value of is set to 0.6, that is, in the fusion process, the global features from the multi-scale context capture branch are given a high weight, while the local features from the local detail enhancement branch are reflected with supplementary weights. The specific steps of feature fusion are: S1, in the initial stage, local and global initial features are extracted through two branches respectively, and fused according to the weighted formula to obtain the basic representation; S2, in the middle stage, after reducing the dimension of the features outputted in the previous stage through block merging operation, the global features and local features are fused again according to the weighted formula; S3, in the deep stage, the processed deep local and global features are finally fused, and the fused features are globally averaged pooled, and finally mapped to the target classification space through the fully connected layer to obtain the classification result.

7. A method for building a fine-grained phytoplankton microscopic image enhancement classification model as claimed in claim 1, characterized in that: The fine-grained image enhancement classification model HySwinFormer includes three types; the network depth of the multi-scale context capture branch of the first type is 2, 2, 6, 2, the embedding dimensions are 96, 192, 384, 768 respectively, and the number of input channels and the number of output channels of the local detail enhancement branch are 96, 192, 384, 768 and 192, 384, 768, 768 respectively; the network depth of the second type is 2, 2, 18, 2, the embedding dimensions are 96, 192, 384, 768, and the number of input channels and the number of output channels of the local detail enhancement branch are the same as the first type; the network depth of the third type is 2, 2, 18, 2, the embedding dimensions are increased to 128, 256, 512, 1024, and the number of input channels and the number of output channels of the local detail enhancement branch are 128, 256, 512, 1024 respectively. 1024 and 256, 512, 1024, 1024.

8. A fine-grained phytoplankton microscopic image enhancement classification method, characterized in that: The process includes: S1, real-time acquisition of high-resolution microscopic images of phytoplankton to be classified; S2, inputting the microscopic image of the phytoplankton to be classified into the fine-grained image enhancement classification model constructed by the construction method according to any one of claims 1 to 7; S3, outputs real-time classification results.

9. A fine-grained phytoplankton microscopic image enhancement classification device, characterized in that: The device includes at least one processor and at least one memory, and the processor and the memory are coupled; the memory stores a computer execution program of a fine-grained image enhancement classification model constructed by the construction method described in any one of claims 1 to 7; when the processor executes the computer execution program stored in the memory, the processor executes a fine-grained phytoplankton microscopic image enhancement classification method.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer execution program of the fine-grained image enhancement classification model constructed by the construction method as described in any one of claims 1 to 7. When the computer execution program is executed by the processor, the processor executes a fine-grained phytoplankton microscopic image enhancement classification method.

Images