Fine adjustment method for cell segmentation
By generating high-quality synthetic cell images and multimodal feature fusion technology, the problem of low cell segmentation accuracy caused by limited labeling data and single imaging methods in the existing technology is solved, which improves the generalization ability and adaptability of the model, and significantly improves the accuracy and reliability of cell segmentation.
Patent Information
- Application Number
- CN202411783143.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-06
- Publication Date
- 2025-03-14
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, due to the limited labeling data, the model is prone to overfitting, and only a single imaging method is used, and the complementary information provided by different imaging methods cannot be fully utilized, resulting in low segmentation accuracy of the cell segmentation task, and the model performs well on specific data sets but has poor generalization ability on other data sets.
By collecting different types of cell image data and pre-processing, high-quality synthetic cell images are generated using the GAN model to expand the training set; the U-Net model is used to migrate the basic model to the cell segmentation task, and fine-tune it with a small amount of labeled data; multimodal feature extraction method is used to extract features from light mirror images and electron microscope images, and feature fusion technology is introduced to fuse features of different modes.
By generating high-quality synthetic images and fusion of multimodal features, the training set is enriched, the generalization ability and adaptability of the model are improved, and the accuracy and reliability of cell segmentation are significantly improved.
Smart Images
Figure CN119625727A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of cell segmentation fine-tuning, and in particular to a cell segmentation fine-tuning method. Background Art
[0002] Cell segmentation fine-tuning refers to further optimizing and adjusting the model in the cell image segmentation task by using a pre-trained deep learning model (such as U-Net) and combining it with a small amount of labeled data to improve its performance in specific cell segmentation tasks. However, in the prior art, due to limited labeled data, the model is prone to overfitting, resulting in poor performance on unseen data. In addition, the prior art usually only uses a single imaging method (such as light microscopy images), which cannot fully utilize the complementary information provided by different imaging methods, affecting the segmentation accuracy. At the same time, the models in the prior art often perform well on specific data sets, but have poor generalization ability and low adaptability on other data sets. Summary of the invention
[0003] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.
[0004] To achieve the above object, the present invention provides the following technical solutions:
[0005] A cell segmentation fine-tuning method includes collecting different types of cell image data and preprocessing them; using a small amount of labeled data and a large amount of unlabeled data to generate high-quality synthetic cell images through a GAN model to expand a training set; using a U-Net model to migrate the basic model to the cell segmentation task and fine-tune it using a small amount of labeled data; using a multimodal feature extraction method to extract features from light microscope images and electron microscope images respectively, and introducing feature fusion technology to fuse features of different modalities.
[0006] As a further solution of the present invention: the collection and preprocessing of different types of cell image data includes collecting different types of cell image data from multiple sources, including optical images and electron microscope images; using high-resolution optical microscope and electron microscope equipment to collect images of multiple cell types, including normal cells, diseased cells and cell cycle images at different stages; using a median filter to remove salt and pepper noise in the image, applying a Gaussian filter to smooth the image, using the Otsu algorithm to binarize the image, removing the background area, and unifying the image size to 512x512 pixels, and converting the format to PNG format.
[0007] As a further solution of the present invention: the method uses a small amount of labeled data and a large amount of unlabeled data to generate high-quality synthetic cell images through a GAN model, and the training set is expanded, including selecting 10% of the images from the preprocessed cell image dataset for labeling as a labeled dataset, and the remaining 90% of the images as an unlabeled dataset; constructing a generative adversarial network model, including a generator and a discriminator; initializing the parameters of the generator and the discriminator, and in each training iteration, randomly selecting a batch of images from the labeled dataset as real images, and generating a batch of synthetic images from a random noise vector; calculating the loss function of the discriminator for real images and synthetic images, and updating the parameters of the discriminator; calculating the loss function of the generator, and updating the parameters of the generator.
[0008] As a further solution of the present invention: the generator and the discriminator need to be optimized during the training process after being generated, and the expression is:
[0009]
[0010] Among them, x is the real image, z is the random noise vector, and p data (x) is the probability distribution of the real image, p z (z) is the probability distribution of the random noise vector, D(x) is the output of the discriminator for the real image, G(z) is the synthetic image generated by the generator, D(G(z) is the output of the discriminator for the synthetic image, E is the expected value operator, and log is the natural logarithm function.
[0011] As a further solution of the present invention: the U-Net model is used to migrate the basic model to the cell segmentation task, and fine-tuning is performed using a small amount of labeled data, including selecting a U-Net model that performs well in medical image segmentation tasks from a public pre-trained model library, such as the U-Net model that won the ISBI cell tracking challenge; retaining the basic structure of the U-Net model, including the encoder and decoder parts; adding additional convolutional layers and batch normalization layers to the decoder part to enhance the feature extraction capability of the model; and using a sigmoid activation function in the output layer to output the probability that each pixel belongs to a cell.
[0012] As a further solution of the present invention: the U-Net model is used to migrate the basic model to the cell segmentation task, and fine-tuning is performed using a small amount of labeled data, which also includes data preparation and fine-tuning strategies, specifically, from the preprocessed cell image dataset, 10% of the images are selected for annotation as the labeled dataset; ensuring that the images in the labeled dataset cover different cell types and imaging conditions; using the labeled dataset to fine-tune the U-Net model, and setting a learning rate decay strategy; introducing mixed precision training technology, and using a combination of Dice loss function and cross entropy loss function.
[0013] As a further solution of the present invention: the fine-tuning process of optimizing the U-Net model is expressed as:
[0014]
[0015] Among them, y is the true label, is the output predicted by the model, indicating the probability that each pixel belongs to a cell, α is the balance factor, and its value range is [0,1]. is the Dice loss function, defined as:
[0016]
[0017] Among them, ∈ is the smoothing factor;
[0018] is the binary cross entropy loss function, defined as:
[0019]
[0020] Where N is the total number of pixels.
[0021] As a further solution of the present invention: the multimodal feature extraction method is used to extract features from the light microscope image and the electron microscope image respectively, and the feature fusion technology is introduced to fuse and summarize the features of different modalities. From the preprocessed cell image data set, the light microscope image and the electron microscope image of the same cell sample are selected; the high-level features of the light microscope image and the electron microscope image are extracted using a convolutional neural network; the extracted feature map is pooled to generate a feature vector of a fixed size; the features extracted from the light microscope image and the electron microscope image are fused to generate a comprehensive feature representation, and the expression is:
[0022]
[0023] Among them, F light is the feature vector extracted from the light microscope image, F EM is the feature vector extracted from the electron microscope image, A is the attention weight, ranging from [0,1], and F fused is the comprehensive feature vector after fusion.
[0024] The present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of a cell segmentation fine-tuning method as described in the first aspect of the present invention is implemented.
[0025] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of a cell segmentation fine-tuning method as described in the first aspect of the present invention is implemented.
[0026] Compared with the prior art, the present invention has the following beneficial effects:
[0027] By collecting image data of different types of cells from multiple sources and performing preprocessing, including noise reduction, smoothing, binarization and other operations, the quality and consistency of the data are ensured, making subsequent data processing and model training more efficient and accurate. By preprocessing the images, noise interference can be effectively reduced and image contrast can be improved, thereby improving the accuracy of cell detection and segmentation. Synthetic images are generated through the GAN model, and a small amount of real labeled data is used to guide the generation process. This not only solves the problems of high cost and time-consuming data annotation in the biomedical field, but also enriches the training set by generating diverse synthetic images, improving the generalization and adaptability of the model. By fusing multimodal information, the model's ability to understand complex cell structures is significantly improved, thereby improving the accuracy and reliability of cell segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0029] Figure 1 Schematic diagram of a process for fine-tuning cell segmentation. DETAILED DESCRIPTION
[0030] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, the specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.
[0031] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0032] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.
[0033] Example
[0034] See also Figure 1 , is an embodiment of the present invention, which provides a cell segmentation fine-tuning method, including:
[0035] Different types of cell image data were collected from multiple hospitals and laboratories, including light microscope images and electron microscope images. The images covered normal cells, diseased cells, and cell cycle images at different stages. High-resolution optical microscopes and electron microscopes were used for image acquisition to ensure high image quality. A median filter was used to remove salt and pepper noise in the image, and a Gaussian filter was applied to smooth the image. The Otsu algorithm was used to binarize the image and remove the background area. The image size was unified to 512x512 pixels and converted to PNG format.
[0036] From the preprocessed cell image dataset, 10% of the images are selected for annotation as the annotated dataset, and the remaining 90% of the images are used as the unlabeled dataset. A generative adversarial network model is constructed, including a generator and a discriminator. The generator uses a deep convolutional neural network (DCGAN) structure, and the discriminator also uses a deep convolutional neural network structure. The parameters of the generator and the discriminator are initialized. In each training iteration, a batch of images are randomly selected from the annotated dataset as real images, and a batch of synthetic images are generated from random noise vectors. The loss function of the discriminator for real images and synthetic images is calculated, and the parameters of the discriminator are updated. The loss function of the generator is calculated, and the parameters of the generator are updated.
[0037] Select a U-Net model that performs well in medical image segmentation tasks from a public pre-trained model library, such as the U-Net model that won the ISBI cell tracking challenge. Keep the basic structure of the U-Net model, including the encoder and decoder parts, add additional convolutional layers and batch normalization layers to the decoder part to enhance the feature extraction ability of the model, use the sigmoid activation function in the output layer, and output the probability that each pixel belongs to a cell. From the pre-processed cell image dataset, select 10% of the images for annotation as the annotation dataset, and ensure that the images in the annotation dataset cover different cell types and imaging conditions. Use the annotation dataset to fine-tune the U-Net model, set the learning rate decay strategy, introduce mixed precision training technology, and use a combination of Dice loss function and cross entropy loss function.
[0038] The U-Net model is fine-tuned using annotated datasets, the learning rate decay strategy is set, mixed precision training technology is introduced, a combination of Dice loss function and cross entropy loss function is used, and convolutional neural networks are used to extract high-level features of light microscope images and electron microscope images. The extracted feature maps are pooled to generate fixed-size feature vectors, and the features extracted from light microscope images and electron microscope images are fused to generate a comprehensive feature representation.
[0039] This embodiment also provides a computer device suitable for the cell segmentation fine-tuning method, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the cell segmentation fine-tuning method proposed in the above embodiment.
[0040] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.
[0041] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for fine-tuning cell segmentation proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0042] Tabular data analysis:
[0043] Parameter name Prior art 1 Prior Art 2 Method of the present invention unit Training set size 1200 1800 9500 open Annotation data ratio 10％ 10％ 10％ Composite image quality good generally excellent Segmentation accuracy 84.5％ 86.8％ 92.1％ ％ Dice coefficient 0.77 0.80 0.87 Jaccard Index 0.64 0.68 0.74 Training time 13 hours 11 hours 8.5 hours Hour Generalization generally better excellent
[0044] It can be seen from the above table data that the method of the present invention is superior to the prior art in many aspects, and the specific analysis is as follows:
[0045] The training set sizes of the prior arts 1 and 2 are 1,200 and 1,800 images respectively, while the method of the present invention generates high-quality synthetic cell images through the GAN model, expanding the training set to 9,500 images, increasing the diversity of the training set and improving the generalization ability of the model.
[0046] The quality of the synthetic images generated by the prior arts 1 and 2 is good and general respectively, while the quality of the synthetic images generated by the method of the present invention is excellent. The high-quality synthetic images help the model to better learn the characteristics of cells and improve the accuracy of segmentation.
[0047] The segmentation accuracy of the prior arts 1 and 2 is 84.5% and 86.8% respectively, while the segmentation accuracy of the method of the present invention reaches 92.1%, indicating that the method of the present invention has higher accuracy and reliability in the cell segmentation task.
[0048] The Dice coefficients of the prior arts 1 and 2 are 0.77 and 0.80 respectively, while the Dice coefficient of the method of the present invention reaches 0.87. The Dice coefficient is an important indicator for measuring the segmentation effect. A higher Dice coefficient means that the segmentation result has a higher degree of overlap with the true label and the segmentation effect is better.
[0049] The generalization capabilities of the existing technologies 1 and 2 are average and good, respectively, while the generalization capability of the method of the present invention is excellent. Through multimodal feature extraction and fusion technology, the method of the present invention can better adapt to different types of cell images and improve the robustness and generalization capability of the model.
[0050] In summary, by collecting image data of different types of cells from multiple sources and performing preprocessing, including noise reduction, smoothing, binarization and other operations, the quality and consistency of the data are ensured, making subsequent data processing and model training more efficient and accurate. By preprocessing the images, noise interference can be effectively reduced and image contrast can be improved, thereby improving the accuracy of cell detection and segmentation. By generating synthetic images through the GAN model and using a small amount of real labeled data to guide the generation process, it not only solves the problems of high cost and time-consuming data annotation in the biomedical field, but also can enrich the training set by generating diverse synthetic images, improve the generalization and adaptability of the model, and significantly improve the model's ability to understand complex cell structures by fusing multimodal information, thereby improving the accuracy and reliability of cell segmentation.
[0051] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A cell segmentation fine-tuning method, characterized in that: include, Collect and pre-process different types of cell image data; Using a small amount of labeled data and a large amount of unlabeled data, the GAN model is used to generate high-quality synthetic cell images and expand the training set. The U-Net model is used to migrate the basic model to the cell segmentation task and fine-tune it using a small amount of labeled data; A multimodal feature extraction method is used to extract features from light microscope images and electron microscope images respectively, and feature fusion technology is introduced to fuse features of different modalities.
2. A cell segmentation fine-tuning method according to claim 1, characterized in that: The collecting and preprocessing of different types of cell image data includes: Collect different types of cell image data from multiple sources, including light microscopy and electron microscopy images; Use high-resolution optical and electron microscopy equipment to capture images of a variety of cell types, including normal cells, diseased cells, and images of different stages of the cell cycle; A median filter was used to remove salt and pepper noise from the image, a Gaussian filter was applied to smooth the image, the Otsu algorithm was used to binarize the image, the background area was removed, and the image size was unified to 512x512 pixels and converted to PNG format.
3. A cell segmentation fine-tuning method according to claim 1, characterized in that: The method uses a small amount of labeled data and a large amount of unlabeled data to generate high-quality synthetic cell images through a GAN model. The expanded training set includes: From the preprocessed cell image dataset, select 10% of the images for annotation as the annotated dataset, and the remaining 90% of the images as the unlabeled dataset; Build a generative adversarial network model, including a generator and a discriminator; Initialize the parameters of the generator and discriminator. In each training iteration, randomly extract a batch of images from the labeled dataset as real images, and generate a batch of synthetic images from the random noise vector. Calculate the loss function of the discriminator for real images and synthetic images, and update the parameters of the discriminator; Calculate the loss function of the generator and update the parameters of the generator.
4. A cell segmentation fine-tuning method according to claim 3, characterized in that: After the generator and discriminator are generated, they need to be optimized during the training process. The expression is: Among them, x is the real image, z is the random noise vector, and p data (x) is the probability distribution of the real image, p z (z) is the probability distribution of the random noise vector, D(x) is the output of the discriminator for the real image, G(z) is the synthetic image generated by the generator, D(G(z) is the output of the discriminator for the synthetic image, E is the expected value operator, and log is the natural logarithm function.
5. A cell segmentation fine-tuning method according to claim 1, characterized in that: The U-Net model is used to migrate the basic model to the cell segmentation task, and fine-tuning is performed using a small amount of labeled data, including: Select a U-Net model from the public pre-trained model library that performs well in medical image segmentation tasks, such as the U-Net model that won the ISBI Cell Tracking Challenge; Keep the basic structure of the U-Net model, including the encoder and decoder parts; Add additional convolutional layers and batch normalization layers in the decoder part to enhance the feature extraction ability of the model; The sigmoid activation function is used in the output layer to output the probability that each pixel belongs to a cell.
6. A cell segmentation fine-tuning method according to claim 1, characterized in that: The U-Net model is used to migrate the basic model to the cell segmentation task, and fine-tuning using a small amount of labeled data also includes data preparation and fine-tuning strategies, as follows: From the preprocessed cell image dataset, select 10% of the images for annotation as the annotation dataset; Ensure that images in the annotated dataset cover different cell types and imaging conditions; Fine-tune the U-Net model using the labeled dataset and set the learning rate decay strategy; Introducing mixed precision training technology, using a combination of Dice loss function and cross entropy loss function.
7. A cell segmentation fine-tuning method according to claim 6, characterized in that: The fine-tuning process of optimizing the U-Net model is expressed as: Among them, y is the true label, is the output predicted by the model, indicating the probability that each pixel belongs to a cell, α is the balance factor, and its value range is [0,1]. is the Dice loss function, defined as: Among them, ∈ is the smoothing factor; is the binary cross entropy loss function, defined as: Where N is the total number of pixels.
8. A cell segmentation fine-tuning method according to claim 1, characterized in that: The multimodal feature extraction method is used to extract features from the light microscope image and the electron microscope image respectively, and the feature fusion technology is introduced to fuse the features of different modes, including: Select light microscope images and electron microscope images of the same cell sample from the preprocessed cell image dataset; Use convolutional neural networks to extract high-level features from light microscopy and electron microscopy images; Perform pooling operation on the extracted feature map to generate a feature vector of fixed size; The features extracted from the light microscope image and the electron microscope image are fused to generate a comprehensive feature representation, which is expressed as: Among them, F ligh t is the feature vector extracted from the light microscope image, F EM is the feature vector extracted from the electron microscope image, A is the attention weight, ranging from [0,1], and F fused is the comprehensive feature vector after fusion.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of a cell segmentation fine-tuning method according to any one of claims 1 to 8 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a cell segmentation fine-tuning method according to any one of claims 1 to 8 are implemented.