A feature extraction method and system for microscopic images

Abstract

The application provides a feature extraction method and system of a microscopic image, and belongs to the technical field of microscopic image recognition. The application comprises the following steps: constructing a microscopic image data set to form a data pool of single-channel images; pre-processing each single-channel image to obtain a training set; constructing a network model based on mask self-supervised learning, wherein the network model adopts an improved U-net structure, uses a cyclic convolution block to replace a traditional convolution block, and introduces style embedding in the connection process; inputting the training set into the network model based on mask self-supervised learning for training to obtain a trained network model based on mask self-supervised learning; and constructing a feature extractor based on the trained network model based on mask self-supervised learning, and using the constructed feature extractor to extract features of the microscopic image. The application provides an unbiased feature extraction method capable of processing various different types of microscopic images.

Description

Technical Field

[0001] This invention relates to the field of microscopic image recognition technology, and in particular to a method and system for feature extraction of microscopic images. Background Technology

[0002] Automated quantitative analysis of microscope images is becoming increasingly widespread, whether in small-scale or high-throughput experiments. The rich information contained in these images, also known as visual representation or image representation, has been successfully applied in basic and applied biological research, such as protein localization identification, cell cycle stage classification, mechanism of action prediction, and high-content drug screening. Furthermore, recent advances in space omics technologies have brought new demands for quantifying microscope images. For example, space proteomics can image more than 50 disease-related proteins in a single tissue section, and space transcriptomics can acquire both image and transcriptional data. However, due to the complexity and diversity of microscope images, capturing valuable and subtle biological information remains a significant challenge.

[0003] Currently, there are many methods available for representing microscopic images. The most classic approach is to extract predefined morphological features, such as intensity, shape, and texture. However, this engineered approach can discard a great deal of subtle information, and its implementation is susceptible to the effects of image quality.

[0004] With the development of computer vision and deep learning, learning-based image analysis methods are constantly being proposed. Some of these are task-oriented end-to-end networks, which, while achieving good performance, are limited to their specific tasks and difficult to generalize to other image analysis applications. Others are feature extractors, which are part of pre-trained networks and offer greater flexibility when applied to downstream analysis. A common approach is to use models trained on the natural image classification task ImageNet, which can be used in many downstream scenarios and was even used in cutting-edge multi-omics research until 2022. To extract features more closely aligned with biological applications, a recent study attempted to build CytoImageNet for microscope images, but it did not show a significant breakthrough compared to ImageNet-based methods. Furthermore, these methods not only require substantial manual annotation work, but the features learned through supervised classification tasks may also have potential biases for certain image categories.

[0005] Self-supervised learning offers new prospects for feature extraction from microscope images. This method extracts image features learned by the pixels themselves, without relying on predefined semantic scenes. Its agent task is to transform the original image and require the model to learn the mapping between the transformed and original images. In particular, mask-based visual representation learning has recently attracted much attention in natural image research. Furthermore, recent advances in cell segmentation algorithms demonstrate that a well-trained network can possess strong generalization capabilities. However, a breakthrough, general-purpose tool for representing microscope images is still lacking, highlighting the urgent need to investigate an unbiased feature extraction method capable of handling a wide variety of microscope image types. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a feature extraction method and system for microscopic images, comprising the following steps: Step S1, constructing a microscopic image dataset, wherein different types of images in the dataset have different numbers of channels; Step S2, separating the channels of each image in the dataset to form a data pool of single-channel images; Step S3, preprocessing each single-channel image to obtain a training set; the preprocessing includes randomly occluding image blocks of a preset proportion in each single-channel image; Step S4, constructing a network model based on mask self-supervised learning, wherein the network model adopts an improved U-net structure, which consists of an encoder, a decoder, and a connection layer, using recurrent convolutional blocks instead of traditional convolutional blocks, and introducing style embedding during the connection process; Step S5, inputting the training set into the mask self-supervised learning network model for training, obtaining a trained mask self-supervised learning network model; Step S6, constructing a feature extractor based on the trained mask self-supervised learning network model, and using the constructed feature extractor to extract features from the microscopic images. This invention provides an unbiased feature extraction method capable of processing various types of microscopic images.

[0007] This invention provides a feature extraction method for microscopic images, comprising the following steps:

[0008] Step S1: Construct a microscopic image dataset, in which different types of images have different numbers of channels;

[0009] Step S2: Separate the channels of each image in the dataset to form a data pool of single-channel images;

[0010] Step S3: Preprocess each single-channel image to obtain a training set; the preprocessing includes randomly occluding image blocks of a preset proportion in each single-channel image;

[0011] Step S4: Construct a network model based on mask self-supervised learning. The network model adopts an improved U-net structure, which consists of three structures: encoder, decoder, and connection layer. Recurrent convolutional blocks are used instead of traditional convolutional blocks, and style embedding is introduced during the connection process.

[0012] Step S5: Input the training set into the mask-based self-supervised learning network model for training, and obtain the trained mask-based self-supervised learning network model;

[0013] Step S6: Construct a feature extractor based on the trained mask-based self-supervised learning network model, and use the constructed feature extractor to extract features from the microscopic image.

[0014] The encoder is responsible for encoding the image into high-dimensional features, the decoder is responsible for decoding the high-dimensional features back into the image, and the connection layer is responsible for communication between the encoding and decoding layers. Recurrent convolutional blocks can improve the model's performance, and introducing style embeddings can make the model more sensitive to images of different styles.

[0015] Preferably, the microscopic image dataset in step S1 includes four types: fluorescence images, phase-contrast images, tissue images, and histopathological images.

[0016] Preferably, the fluorescence image and tissue image have two channels, and the phase contrast image and histopathological image have one channel.

[0017] Preferably, the histopathological image dataset contains three sub-datasets: the multi-organ nuclear segmentation dataset MoNuSeg, the multi-organ cell nuclear segmentation and classification challenge dataset MoNuSac, and the nuclear classification, localization, and segmentation dataset NuCLS.

[0018] Preferably, the histopathological image dataset includes three sub-datasets: MoNuSeg, MoNuSac, and NuCLS, which are used together and divided into training and validation sets according to a preset ratio.

[0019] Preferably, the preset ratio is 4:1.

[0020] Preferably, in the preprocessing of step S3, sampling and data augmentation are further included before randomly occluding image blocks of a preset proportion for each single-channel image.

[0021] Preferably, the sampling specifically involves randomly selecting a preset number of images from four types: fluorescence images, phase contrast images, tissue images, and histopathological images.

[0022] Preferably, the data augmentation specifically includes: randomly cropping a square region of a preset size from each image, then normalizing, randomly rotating and randomly scaling the region, and the processed image is the target image.

[0023] Preferably, if the image is smaller than the preset size, black pixels are used to fill that size.

[0024] Preferably, the preset size is 224x224.

[0025] Preferably, in step S4, the network model based on mask self-supervised learning learns and reconstructs the target image.

[0026] Preferably, gradient descent is used to iteratively optimize the model and reconstruct the target image.

[0027] Preferably, the network model training based on mask self-supervised learning uses MSE as the loss function and AdamW as the optimizer.

[0028] Preferably, constructing a feature extractor based on a trained mask-based self-supervised learning network model specifically involves adding an average pooling layer as a feature extractor to the trained mask-based self-supervised learning network model.

[0029] For each single-channel image, the trained encoder encodes the image into a (256, 28, 28) feature tensor, and then a global average pooling layer is used to reduce its dimensionality to a (256,) feature embedding.

[0030] This invention provides a feature extraction system for microscopic images, which uses any of the aforementioned microscopic image feature extraction methods to extract features from microscopic images; including:

[0031] The microscopic image dataset construction module collects multiple microscopic images to form a microscopic image dataset;

[0032] The image preprocessing module preprocesses the images to obtain the training set; the preprocessing includes randomly occluding image blocks of a preset proportion for each single-channel image;

[0033] The network model building module based on mask self-supervised learning adopts an improved U-net structure to build the network model. The improved U-net structure consists of three structures: encoder, decoder and connection layer. Recurrent convolutional blocks are used instead of traditional convolutional blocks, and style embedding is introduced in the connection process.

[0034] The network model training module based on mask self-supervised learning takes the training set as input to the network model based on mask self-supervised learning for training, and obtains the trained network model based on mask self-supervised learning.

[0035] The feature extractor construction and feature extraction module constructs a feature extractor based on a trained mask-based self-supervised learning network model, and uses the constructed feature extractor to extract features from microscopic images.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] (1) This invention organizes a training set containing four types of microscopic images, enabling our model to learn a general representation pattern to handle different types of microscopic images.

[0038] (2) During the preprocessing of this invention, a preset number of images are randomly selected from four types of microscopic images in sequence to alleviate the data imbalance problem caused by the different numbers of images of different types.

[0039] (3) The present invention uses mask self-supervised learning during training, which can ensure that the learned features are unbiased and not limited by specific semantic annotations.

[0040] (4) This invention unifies the number of channels of the input image to 1, which facilitates subsequent connection of the image at the feature level and provides an unbiased feature extraction method that can process various types of microscopic images. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the training set images according to an embodiment of the present invention, wherein a1 and a2 are fluorescence images (Cellpose), b1 and b2 are phase contrast images (LIVECell), c1 and c2 are tissue images (TissueNet), and d1 and d2 are examples of histopathological images (Histo).

[0042] Figure 2 This is a schematic diagram of a network model training method based on mask self-supervised learning according to an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram illustrating feature extraction using a microscopic image feature extractor according to an embodiment of the present invention.

[0044] Figure 4 This is a schematic diagram illustrating the performance of reconstructing a whole image with a 25% mask according to an embodiment of the present invention. In this diagram, e1, e2, and e3 are the mask image, the original image, and the reconstructed image of the fluorescence image, respectively; f1, f2, and f3 are the mask image, the original image, and the reconstructed image of the phase contrast image, respectively; g1, g2, and g3 are the mask image, the original image, and the reconstructed image of the tissue image, respectively; and h1, h2, and h3 are the mask image, the original image, and the reconstructed image of the histopathological image, respectively.

[0045] Figure 5This is a performance diagram illustrating the reconstruction of a single-cell image with a 25% mask, according to an embodiment of the present invention. In this diagram, i1, i2, and i3 represent the mask image, original image, and reconstructed image of a certain channel in the evaluation dataset, respectively; j1, j2, and j3 represent the mask image, original image, and reconstructed image of another channel, respectively; k1, k2, and k3 represent the mask image, original image, and reconstructed image of a certain channel in another evaluation dataset, respectively; and l1, l2, and l3 represent the mask image, original image, and reconstructed image of another channel, respectively.

[0046] Figure 6 This is a flowchart of a feature extraction method for microscopic images according to an embodiment of the present invention. Detailed Implementation

[0047] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0048] This invention provides a feature extraction method for microscopic images, comprising the following steps:

[0049] Step S1: Construct a microscopic image dataset, in which different types of images have different numbers of channels;

[0050] Step S2: Separate the channels of each image in the dataset to form a data pool of single-channel images;

[0051] Step S3: Preprocess each single-channel image to obtain a training set; the preprocessing includes randomly occluding image blocks of a preset proportion in each single-channel image;

[0052] Step S4: Construct a network model based on mask self-supervised learning. The network model adopts an improved U-net structure, which consists of three structures: encoder, decoder, and connection layer. Recurrent convolutional blocks are used instead of traditional convolutional blocks, and style embedding is introduced during the connection process.

[0053] Step S5: Input the training set into the mask-based self-supervised learning network model for training, and obtain the trained mask-based self-supervised learning network model;

[0054] Step S6: Construct a feature extractor based on the trained mask-based self-supervised learning network model, and use the constructed feature extractor to extract features from the microscopic image.

[0055] The encoder is responsible for encoding the image into high-dimensional features, the decoder is responsible for decoding the high-dimensional features back into the image, and the connection layer is responsible for communication between the encoding and decoding layers. Recurrent convolutional blocks can improve the model's performance, and introducing style embeddings can make the model more sensitive to images of different styles.

[0056] According to a specific embodiment of the present invention, the microscopic image dataset in step S1 includes four types: fluorescence images, phase contrast images, tissue images, and histopathological images.

[0057] According to one specific embodiment of the present invention, the fluorescence image and tissue image have two channels, and the phase contrast image and histopathological image have one channel.

[0058] According to a specific embodiment of the present invention, the histopathological image dataset comprises three sub-datasets: the multi-organ nuclear segmentation dataset MoNuSeg, the multi-organ cell nuclear segmentation and classification challenge dataset MoNuSac, and the nuclear classification, localization, and segmentation dataset NuCLS.

[0059] According to a specific embodiment of the present invention, the histopathological image dataset includes three sub-datasets, MoNuSeg, MoNuSac, and NuCLS, which are used together and divided into training set and validation set according to a preset ratio.

[0060] According to a specific embodiment of the present invention, the preset ratio is 4:1.

[0061] According to a specific embodiment of the present invention, in the preprocessing of step S3, sampling and data augmentation are further included before randomly occluding image blocks of a preset proportion for each single-channel image.

[0062] According to a specific embodiment of the present invention, the sampling specifically involves randomly selecting a preset number of images from four types: fluorescence images, phase contrast images, tissue images, and histopathological images.

[0063] According to a specific embodiment of the present invention, the data enhancement specifically includes: randomly cropping a square region of a preset size from each image, then normalizing, randomly rotating and randomly scaling the region, and the processed image is the target image.

[0064] According to one specific embodiment of the present invention, 25% of each target image is randomly occluded.

[0065] According to one specific embodiment of the present invention, if the image is smaller than a preset size, black pixels are filled to that size.

[0066] According to a specific embodiment of the present invention, the preset size is 224x224.

[0067] According to a specific embodiment of the present invention, in step S4, a network model based on mask self-supervised learning learns and reconstructs the target image.

[0068] According to a specific embodiment of the present invention, gradient descent is used to iteratively optimize the model and reconstruct the target image.

[0069] According to a specific embodiment of the present invention, in the downsampling stage, the feature dimensions of the four levels of the encoder are 32, 64, 128 and 256, respectively; in the upsampling stage, the decoder also contains four levels of sub-blocks, with the feature dimensions of the four levels decreasing in reverse order, namely 256, 128, 64 and 32, respectively; finally, a convolutional unit is used to process the feature map of the last sub-block to adapt to the final image reconstruction task.

[0070] According to a specific embodiment of the present invention, the network model training based on mask self-supervised learning uses MSE as the loss function and AdamW as the optimizer.

[0071] According to a specific embodiment of the present invention, during the training process, a total of 1000 epochs are randomly sampled, and in each epoch, a total of 12000 training images are sampled sequentially from four types of images.

[0072] According to a specific embodiment of the present invention, during the training process, the initial learning rate is 0.001, the learning rate decay rate is set to 0.05, the image patch size is set to 16*16, and the size of each batch is set to 16.

[0073] According to a specific embodiment of the present invention, the feature extractor is constructed based on a trained mask-based self-supervised learning network model by adding an average pooling layer as a feature extractor to the trained mask-based self-supervised learning network model.

[0074] For each single-channel image, the trained encoder encodes the image into a (256, 28, 28) feature tensor, and then a global average pooling layer is used to reduce its dimensionality to a (256,) feature embedding.

[0075] This invention provides a feature extraction system for microscopic images, which uses any of the aforementioned microscopic image feature extraction methods to extract features from microscopic images; including:

[0076] The microscopic image dataset construction module collects multiple microscopic images to form a microscopic image dataset;

[0077] The image preprocessing module preprocesses the images to obtain the training set; the preprocessing includes randomly occluding image blocks of a preset proportion for each single-channel image;

[0078] The network model building module based on mask self-supervised learning adopts an improved U-net structure to build the network model. The improved U-net structure consists of three structures: encoder, decoder and connection layer. Recurrent convolutional blocks are used instead of traditional convolutional blocks, and style embedding is introduced in the connection process.

[0079] The network model training module based on mask self-supervised learning takes the training set as input to the network model based on mask self-supervised learning for training, and obtains the trained network model based on mask self-supervised learning.

[0080] The feature extractor construction and feature extraction module constructs a feature extractor based on a trained mask-based self-supervised learning network model, and uses the constructed feature extractor to extract features from microscopic images.

[0081] Example 1

[0082] According to a specific embodiment of the present invention, the feature extraction method for microscopic images of the present invention will be described in detail.

[0083] This invention provides a feature extraction method for microscopic images, comprising the following steps:

[0084] Step S1: Construct a microscopic image dataset, in which different types of images have different numbers of channels;

[0085] Step S2: Separate the channels of each image in the dataset to form a data pool of single-channel images;

[0086] Step S3: Preprocess each single-channel image to obtain a training set; the preprocessing includes randomly occluding image blocks of a preset proportion in each single-channel image;

[0087] Step S4: Construct a network model based on mask self-supervised learning. The network model adopts an improved U-net structure, which consists of three structures: encoder, decoder, and connection layer. Recurrent convolutional blocks are used instead of traditional convolutional blocks, and style embedding is introduced during the connection process.

[0088] Step S5: Input the training set into the mask-based self-supervised learning network model for training, and obtain the trained mask-based self-supervised learning network model;

[0089] Step S6: Construct a feature extractor based on the trained mask-based self-supervised learning network model, and use the constructed feature extractor to extract features from the microscopic image.

[0090] Example 2

[0091] According to a specific embodiment of the present invention, the feature extraction method for microscopic images of the present invention will be described in detail.

[0092] This invention provides a feature extraction method for microscopic images, comprising the following steps:

[0093] Step S1: Construct a microscopic image dataset, in which different types of images have different numbers of channels;

[0094] Step S2: Separate the channels of each image in the dataset to form a data pool of single-channel images;

[0095] Step S3: Preprocess each single-channel image to obtain a training set; the preprocessing includes randomly occluding image blocks of a preset proportion in each single-channel image;

[0096] Step S4: Construct a network model based on mask self-supervised learning. The network model adopts an improved U-net structure, which consists of three structures: encoder, decoder, and connection layer. Recurrent convolutional blocks are used instead of traditional convolutional blocks, and style embedding is introduced during the connection process.

[0097] Step S5: Input the training set into the mask-based self-supervised learning network model for training, and obtain the trained mask-based self-supervised learning network model;

[0098] Step S6: Construct a feature extractor based on the trained mask-based self-supervised learning network model, and use the constructed feature extractor to extract features from the microscopic image.

[0099] The microscopic image dataset in step S1 includes four types: fluorescence images, phase-contrast images, tissue images, and histopathological images.

[0100] In step S3, the preprocessing includes sampling and data augmentation before randomly occluding image blocks of a preset proportion for each single-channel image.

[0101] In step S4, the network model based on mask self-supervised learning learns and reconstructs the target image.

[0102] Example 3

[0103] According to a specific embodiment of the present invention, the feature extraction method for microscopic images of the present invention will be described in detail.

[0104] This invention provides a feature extraction method for microscopic images, comprising the following steps:

[0105] Step S1: Construct a microscopic image dataset, in which different types of images have different numbers of channels;

[0106] Step S2: Separate the channels of each image in the dataset to form a data pool of single-channel images;

[0107] Step S3: Preprocess each single-channel image to obtain a training set; the preprocessing includes randomly occluding image blocks of a preset proportion in each single-channel image;

[0108] Step S4: Construct a network model based on mask self-supervised learning. The network model adopts an improved U-net structure, which consists of three structures: encoder, decoder, and connection layer. Recurrent convolutional blocks are used instead of traditional convolutional blocks, and style embedding is introduced during the connection process.

[0109] Step S5: Input the training set into the mask-based self-supervised learning network model for training, and obtain the trained mask-based self-supervised learning network model;

[0110] Step S6: Construct a feature extractor based on the trained mask-based self-supervised learning network model, and use the constructed feature extractor to extract features from the microscopic image.

[0111] The microscopic image dataset in step S1 includes four types: fluorescence images, phase-contrast images, tissue images, and histopathological images.

[0112] Fluorescence images and tissue images have two channels, while phase contrast images and histopathological images have one channel.

[0113] The histopathological image dataset consists of three sub-datasets: the multi-organ nuclear segmentation dataset MoNuSeg, the multi-organ cell nuclear segmentation and classification challenge dataset MoNuSac, and the nuclear classification, localization, and segmentation dataset NuCLS.

[0114] Specifically, the sampling involves randomly selecting a predetermined number of images from four types: fluorescence images, phase-contrast images, tissue images, and histopathological images.

[0115] like Figure 1 The diagram shows a schematic of the training set constructed in this embodiment, which is a large, high-quality microscopic image dataset. These images are divided into four types: fluorescence images (Cellpose), phase-contrast images (LIVECell), tissue images (TissueNet), and histopathological images (Histo). The Histo dataset further includes three sub-datasets: MoNuSeg, MoNuSac, and NuCLS. These datasets have different numbers of channels; Cellpose and TissueNet images have two channels, while LIVECell and Histo images have one channel.

[0116] Example 4

[0117] According to a specific embodiment of the present invention, the feature extraction method for microscopic images of the present invention will be described in detail.

[0118] This invention provides a feature extraction method for microscopic images, comprising the following steps:

[0119] Step S1: Construct a microscopic image dataset, in which different types of images have different numbers of channels;

[0120] Step S2: Separate the channels of each image in the dataset to form a data pool of single-channel images;

[0121] Step S3: Preprocess each single-channel image to obtain a training set; the preprocessing includes randomly occluding image blocks of a preset proportion in each single-channel image;

[0122] Step S4: Construct a network model based on mask self-supervised learning. The network model adopts an improved U-net structure, which consists of three structures: encoder, decoder, and connection layer. Recurrent convolutional blocks are used instead of traditional convolutional blocks, and style embedding is introduced during the connection process.

[0123] Step S5: Input the training set into the mask-based self-supervised learning network model for training, and obtain the trained mask-based self-supervised learning network model;

[0124] Step S6: Construct a feature extractor based on the trained mask-based self-supervised learning network model, and use the constructed feature extractor to extract features from the microscopic image.

[0125] The microscopic image dataset in step S1 includes four types: fluorescence images, phase-contrast images, tissue images, and histopathological images.

[0126] In step S3, the preprocessing includes sampling and data augmentation before randomly occluding image blocks of a preset proportion for each single-channel image.

[0127] Specifically, the data enhancement includes: randomly cropping a square region of a preset size from each image, then normalizing, randomly rotating and randomly scaling the region, and the processed image is the target image.

[0128] In step S4, the network model based on mask self-supervised learning learns and reconstructs the target image.

[0129] In this process, gradient descent is used to iteratively optimize the model and reconstruct the target image.

[0130] Example 5

[0131] According to a specific embodiment of the present invention, the feature extraction method for microscopic images of the present invention will be described in detail.

[0132] This invention provides a feature extraction method for microscopic images, comprising the following steps:

[0133] Step S1: Construct a microscopic image dataset, in which different types of images have different numbers of channels;

[0134] Step S2: Separate the channels of each image in the dataset to form a data pool of single-channel images;

[0135] Step S3: Preprocess each single-channel image to obtain a training set; the preprocessing includes randomly occluding image blocks of a preset proportion in each single-channel image;

[0136] Step S4: Construct a network model based on mask self-supervised learning. The network model adopts an improved U-net structure, which consists of three structures: encoder, decoder, and connection layer. Recurrent convolutional blocks are used instead of traditional convolutional blocks, and style embedding is introduced during the connection process.

[0137] Step S5: Input the training set into the mask-based self-supervised learning network model for training, and obtain the trained mask-based self-supervised learning network model;

[0138] Step S6: Construct a feature extractor based on the trained mask-based self-supervised learning network model, and use the constructed feature extractor to extract features from the microscopic image.

[0139] In the training of the network model based on mask self-supervised learning, MSE is used as the loss function and AdamW is used as the optimizer.

[0140] Specifically, the feature extractor is constructed based on the trained mask-based self-supervised learning network model by adding an average pooling layer as the feature extractor.

[0141] Example 6

[0142] According to a specific embodiment of the present invention, the feature extraction method for microscopic images of the present invention will be described in detail.

[0143] This invention provides a feature extraction method for microscopic images, comprising the following steps:

[0144] Step S1: Construct a microscopic image dataset, in which different types of images have different numbers of channels;

[0145] Step S2: Separate the channels of each image in the dataset to form a data pool of single-channel images;

[0146] Step S3: Preprocess each single-channel image to obtain a training set; the preprocessing includes randomly occluding image blocks of a preset proportion in each single-channel image;

[0147] Step S4: Construct a network model based on mask self-supervised learning. The network model adopts an improved U-net structure, which consists of three structures: encoder, decoder, and connection layer. Recurrent convolutional blocks are used instead of traditional convolutional blocks, and style embedding is introduced during the connection process.

[0148] Step S5: Input the training set into the mask-based self-supervised learning network model for training, and obtain the trained mask-based self-supervised learning network model;

[0149] Step S6: Construct a feature extractor based on the trained mask-based self-supervised learning network model, and use the constructed feature extractor to extract features from the microscopic image.

[0150] The microscopic image dataset in step S1 includes four types: fluorescence images, phase-contrast images, tissue images, and histopathological images.

[0151] Fluorescence images and tissue images have two channels, while phase contrast images and histopathological images have one channel.

[0152] The histopathological image dataset consists of three sub-datasets: the multi-organ nuclear segmentation dataset MoNuSeg, the multi-organ cell nuclear segmentation and classification challenge dataset MoNuSac, and the nuclear classification, localization, and segmentation dataset NuCLS.

[0153] Specifically, the sampling involves randomly selecting a predetermined number of images from four types: fluorescence images, phase-contrast images, tissue images, and histopathological images.

[0154] The histopathological image dataset includes three sub-datasets: MoNuSeg, MoNuSac, and NuCLS, which are used together and divided into training and validation sets according to a preset ratio.

[0155] In step S3, the preprocessing includes sampling and data augmentation before randomly occluding image blocks of a preset proportion for each single-channel image.

[0156] Specifically, the data enhancement includes: randomly cropping a square region of a preset size from each image, then normalizing, randomly rotating and randomly scaling the region, and the processed image is the target image.

[0157] If the image is smaller than the preset size, it is filled with black pixels to that size.

[0158] Example 7

[0159] According to a specific embodiment of the present invention, the feature extraction method for microscopic images of the present invention will be described in detail.

[0160] This invention provides a feature extraction method for microscopic images, comprising the following steps:

[0161] Step S1: Construct a microscopic image dataset, in which different types of images have different numbers of channels;

[0162] Step S2: Separate the channels of each image in the dataset to form a data pool of single-channel images;

[0163] Step S3: Preprocess each single-channel image to obtain a training set; the preprocessing includes randomly occluding image blocks of a preset proportion in each single-channel image;

[0164] Step S4: Construct a network model based on mask self-supervised learning. The network model adopts an improved U-net structure, which consists of three structures: encoder, decoder, and connection layer. Recurrent convolutional blocks are used instead of traditional convolutional blocks, and style embedding is introduced during the connection process.

[0165] Step S5: Input the training set into the mask-based self-supervised learning network model for training, and obtain the trained mask-based self-supervised learning network model;

[0166] Step S6: Construct a feature extractor based on the trained mask-based self-supervised learning network model, and use the constructed feature extractor to extract features from the microscopic image.

[0167] The microscopic image dataset in step S1 includes four types: fluorescence images, phase-contrast images, tissue images, and histopathological images.

[0168] Fluorescence images and tissue images have two channels, while phase contrast images and histopathological images have one channel.

[0169] The histopathological image dataset consists of three sub-datasets: the multi-organ nuclear segmentation dataset MoNuSeg, the multi-organ cell nuclear segmentation and classification challenge dataset MoNuSac, and the nuclear classification, localization, and segmentation dataset NuCLS.

[0170] The histopathological image dataset includes three sub-datasets: MoNuSeg, MoNuSac, and NuCLS, which are used together and divided into training and validation sets according to a preset ratio.

[0171] In step S3, the preprocessing includes sampling and data augmentation before randomly occluding image blocks of a preset proportion for each single-channel image.

[0172] Specifically, the sampling involves randomly selecting a predetermined number of images from four types: fluorescence images, phase-contrast images, tissue images, and histopathological images.

[0173] Specifically, the data enhancement includes: randomly cropping a square region of a preset size from each image, then normalizing, randomly rotating and randomly scaling the region, and the processed image is the target image.

[0174] If the image is smaller than the preset size, it is filled with black pixels to that size.

[0175] In step S4, the network model based on mask self-supervised learning learns and reconstructs the target image.

[0176] In this process, gradient descent is used to iteratively optimize the model and reconstruct the target image.

[0177] In the training of the network model based on mask self-supervised learning, MSE is used as the loss function and AdamW is used as the optimizer.

[0178] Specifically, the feature extractor is constructed based on the trained mask-based self-supervised learning network model by adding an average pooling layer as the feature extractor.

[0179] Example 8

[0180] According to a specific embodiment of the present invention, the feature extraction method for microscopic images of the present invention will be described in detail.

[0181] This invention provides a feature extraction method for microscopic images, comprising the following steps:

[0182] Step S1: Construct a microscopic image dataset, in which different types of images have different numbers of channels;

[0183] Step S2: Separate the channels of each image in the dataset to form a data pool of single-channel images;

[0184] Step S3: Preprocess each single-channel image to obtain a training set; the preprocessing includes randomly occluding image blocks of a preset proportion in each single-channel image;

[0185] Step S4: Construct a network model based on mask self-supervised learning. The network model adopts an improved U-net structure, which consists of three structures: encoder, decoder, and connection layer. Recurrent convolutional blocks are used instead of traditional convolutional blocks, and style embedding is introduced during the connection process.

[0186] Step S5: Input the training set into the mask-based self-supervised learning network model for training, and obtain the trained mask-based self-supervised learning network model;

[0187] Step S6: Construct a feature extractor based on the trained mask-based self-supervised learning network model, and use the constructed feature extractor to extract features from the microscopic image.

[0188] In this process, 25% of each target image is randomly occluded.

[0189] The microscopic image dataset in step S1 includes four types: fluorescence images, phase-contrast images, tissue images, and histopathological images.

[0190] Fluorescence images and tissue images have two channels, while phase contrast images and histopathological images have one channel.

[0191] The histopathological image dataset consists of three sub-datasets: the multi-organ nuclear segmentation dataset MoNuSeg, the multi-organ cell nuclear segmentation and classification challenge dataset MoNuSac, and the nuclear classification, localization, and segmentation dataset NuCLS.

[0192] The histopathological image dataset includes three sub-datasets: MoNuSeg, MoNuSac, and NuCLS, which are used together and divided into training and validation sets according to a preset ratio.

[0193] The preset ratio is 4:1.

[0194] In step S3, the preprocessing includes sampling and data augmentation before randomly occluding image blocks of a preset proportion for each single-channel image.

[0195] Specifically, the sampling involves randomly selecting a predetermined number of images from four types: fluorescence images, phase-contrast images, tissue images, and histopathological images.

[0196] Specifically, the data enhancement includes: randomly cropping a square region of a preset size from each image, then normalizing, randomly rotating and randomly scaling the region, and the processed image is the target image.

[0197] The preset size is 224x224.

[0198] If the image is smaller than the preset size, it is filled with black pixels to that size.

[0199] In step S4, the network model based on mask self-supervised learning learns and reconstructs the target image.

[0200] In this process, gradient descent is used to iteratively optimize the model and reconstruct the target image.

[0201] like Figure 2 The diagram shown illustrates the network model training method based on mask self-supervised learning in this embodiment.

[0202] To address the issue of varying channel counts in different images, the input image is set to have 1 channel to facilitate subsequent feature-level concatenation. In this embodiment, a target image with 25% random occlusion is used as input, allowing the model to learn and reconstruct it. This challenging task requires a deep understanding of microscopic images. The model uses a modified U-Net structure from Cellpose, primarily consisting of an encoder, decoder, and connection layers. Recurrent convolutional blocks replace traditional convolutional blocks, and style embedding is introduced during the connection process. In the downsampling stage, the feature dimensions of the encoder's four layers are 32, 64, 128, and 256, respectively. In the upsampling stage, the decoder also contains four layers of sub-blocks, with the feature dimensions decreasing in reverse order: 256, 128, 64, and 32. Finally, a convolutional unit processes the feature map of the last sub-block to adapt it to the final image reconstruction task.

[0203] During training, a total of 1000 epochs were randomly sampled, and in each epoch, 12000 training images were sampled sequentially from four types of images.

[0204] During training, MSE was used as the loss function, AdamW was used as the optimizer, the initial learning rate was 0.001, the learning rate decay rate was set to 0.05, the image patch size was set to 16*16, and the size of each batch was set to 16.

[0205] like Figure 4 and Figure 5 As shown, the performance of this embodiment in reconstructing a single-cell image with 25% random occlusion and an entire image is demonstrated. It can be seen that, apart from some minor texture details, the present invention can restore the original image very well. Even on single-cell images, the reconstructed image is difficult for the human eye to distinguish from the original image.

[0206] Specifically, the feature extractor is constructed based on the trained mask-based self-supervised learning network model by adding an average pooling layer as the feature extractor.

[0207] like Figure 3 The diagram shown illustrates the feature extraction process of the microscopic image feature extractor constructed in this embodiment. Using the previously trained encoder along with an average pooling layer as the feature extractor, it can then be applied to feature extraction from microscopic images. The extracted features can be used for downstream analysis.

[0208] Example 9

[0209] According to a specific embodiment of the present invention, the feature extraction system for microscopic images of the present invention will be described in detail.

[0210] This invention provides a feature extraction system for microscopic images, which uses any of the aforementioned microscopic image feature extraction methods to extract features from microscopic images; including:

[0211] The microscopic image dataset construction module collects multiple microscopic images to form a microscopic image dataset;

[0212] The image preprocessing module preprocesses the images to obtain the training set; the preprocessing includes randomly occluding image blocks of a preset proportion for each single-channel image;

[0213] The network model building module based on mask self-supervised learning adopts an improved U-net structure to build the network model. The improved U-net structure consists of three structures: encoder, decoder and connection layer. Recurrent convolutional blocks are used instead of traditional convolutional blocks, and style embedding is introduced in the connection process.

[0214] The network model training module based on mask self-supervised learning takes the training set as input to the network model based on mask self-supervised learning for training, and obtains the trained network model based on mask self-supervised learning.

[0215] The feature extractor construction and feature extraction module constructs a feature extractor based on a trained mask-based self-supervised learning network model, and uses the constructed feature extractor to extract features from microscopic images.

[0216] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A feature extraction method of a microscopic image, characterized by, The method comprises the following steps: Step S1, constructing a microscopic image dataset, different types of images in the dataset have different channel numbers; Step S2, separating the channels of each image in the dataset to form a data pool of single-channel images; Step S3, preprocessing each single-channel image to obtain a training set; the preprocessing includes randomly obscuring a preset proportion of image blocks of each single-channel image; Step S4, constructing a network model based on mask self-supervised learning, the network model based on mask self-supervised learning learns and reconstructs a target image, the network model adopts an improved U-net structure, the improved U-net structure is composed of an encoder, a decoder and a connection layer, a cyclic convolution block is used to replace a traditional convolution block, and a style embedding is introduced in the connection process; Step S5, inputting the training set into the network model based on mask self-supervised learning for training to obtain a trained network model based on mask self-supervised learning; Step S6, constructing a feature extractor by adding an average pooling layer to the trained network model based on mask self-supervised learning, and using the constructed feature extractor to extract features of the microscopic image.

2. The feature extraction method of a microscopic image according to claim 1, characterized in that, The microscopic image dataset in step S1 includes four types: fluorescence images, phase contrast images, tissue images and histopathological images.

3. The feature extraction method of a microscopic image according to claim 2, characterized in that, The fluorescence images and the tissue images have two channels, and the phase contrast images and the histopathological images have one channel.

4. The feature extraction method of a microscopic image according to claim 2, characterized in that, The histopathological image dataset includes three sub-datasets, namely a multi-organ nucleus segmentation dataset MoNuSeg, a multi-organ nucleus segmentation and classification challenge dataset MoNuSac, and a nucleus classification, localization and segmentation dataset NuCLS.

5. The method of claim 2, wherein, In the preprocessing in step S3, sampling and data enhancement are further included before randomly obscuring a preset proportion of image blocks of each single-channel image.

6. The method of claim 5, wherein, The sampling specifically comprises sequentially and randomly selecting a preset number of images from the four types of fluorescence images, phase contrast images, tissue images and histopathological images.

7. The method of claim 6, wherein, The data enhancement specifically includes: randomly cropping a square region of a preset size from each image, then performing normalization, random rotation and random scaling, and the processed image is a target image.

8. The method of claim 7, wherein, If the image is less than the preset size, black pixels are filled to the size.

9. The method of feature extraction of a microscopic image according to any one of claims 7-8, characterized in that, MSE is used as a loss function in the training of the network model based on mask self-supervised learning, and AdamW is used as an optimizer.

10. A feature extraction system of a microscopic image, characterized by, The feature extraction method of the microscopic image according to any one of claims 1-9 is used for feature extraction of the microscopic image; comprising: a microscopic image dataset construction module, which acquires a plurality of microscopic images to construct a microscopic image dataset; an image preprocessing module, which pre-processes the images to obtain a training set; the preprocessing includes randomly obscuring a preset proportion of image blocks of each single-channel image; a network model construction module based on mask self-supervised learning, which constructs the network model based on mask self-supervised learning using an improved U-net structure, the improved U-net structure is composed of an encoder, a decoder and a connection layer, a cyclic convolution block is used to replace a traditional convolution block, and a style embedding is introduced in the connection process; The network model training module based on the mask self-supervised learning trains the training set in the network model based on the mask self-supervised learning, and obtains the trained network model based on the mask self-supervised learning. The feature extractor construction and feature extraction module constructs the feature extractor based on the trained network model based on the mask self-supervised learning, and extracts the features of the microscopic image by using the constructed feature extractor.

Images