A U-Net++ cell segmentation network system, method, device and terminal
By introducing multi-scale and multi-feature fusion modules into the U-Net++ network, combining data augmentation and watershed algorithms, the category imbalance and noise problems in medical cell image segmentation are solved, and segmentation accuracy and diagnostic efficiency are improved.
Patent Information
- Application Number
- CN202111351003.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-15
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-11-15
AI Technical Summary
The existing medical cell image segmentation methods have problems such as high time and energy consumption of manual diagnosis, subjective influence of the results, and the problem of accurate segmentation caused by category imbalance and noise in medical image segmentation.
U-Net++ network based on multi-scale and multi-feature fusion modules is adopted to improve the segmentation accuracy of the model and avoid missegment through data augmentation, image preprocessing and watershed algorithms.
It improves the accuracy and robustness of medical cell image segmentation, reduces missegment and information loss, and enhances diagnostic efficiency.
Smart Images

Figure CN114240961B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a U-Net++ cell segmentation network system, method, device and terminal. Background Art
[0002] At present, medical cell images are usually obtained through microscopes or more advanced optical instruments. Then, relevant pathologists need to use their professional knowledge to qualitatively annotate and analyze these biological cell images and finally draw conclusions. However, this manual diagnosis method not only consumes a lot of time and energy, but may also be affected by personal experience. Different people will get different results when parsing the same cell image. Therefore, this method of diagnosis and analysis will make the results more or less affected by subjective factors. In addition, with the accumulation of a large number of modern medical images, if superior image processing algorithms can be applied to medical images for diagnosis and analysis, it can not only greatly reduce the workload and working time of doctors, but also greatly improve the objectivity of the results. Therefore, how to apply image processing algorithms in the fields of tumor segmentation, multi-organ segmentation, cell segmentation, etc., and use these algorithms to accurately realize the automation of medical image analysis is imminent.
[0003] With the development of Convolutional Neural Network (CNN) in the fields of computer vision and medical image processing, deep learning has become the mainstream method in medical image segmentation tasks and can realize automatic segmentation of medical images. Traditional medical image segmentation methods mainly use threshold segmentation algorithms, clustering technology segmentation algorithms and deformable model segmentation algorithms. Kotropoulous et al. proposed a method for segmenting chest X-ray images. This method first filters the image, then segments it through a threshold algorithm, and finally sets a kernel function in the SVM to accurately segment the image; Kumbhar et al. use the trained K-means clustering method for MRI segmentation. The LM-K-means method is used to segment white matter and gray matter in MR images. Compared with the classic K-means method, this method has higher accuracy; the Snake method is the first deformable model applied to medical image segmentation. Xu et al. analyzed the reason why the Snake method has poor convergence on large curvature boundaries and used a gradient vector field (GVF) instead of a gradient field.
[0004] With the emergence of Fully Convolutional Network (FCN), Ronneberger et al. proposed U-Net for biomedical image segmentation. U-Net has demonstrated strong segmentation performance in the segmentation tasks of neural structures and electron microscope images. It has become the most widely used neural network architecture in medical image segmentation tasks. However, due to the problems of class imbalance and noise in medical images, obtaining accurate segmentation results is still a difficult problem.
[0005] Through the above analysis, the problems and defects of the prior art are as follows:
[0006] (1) Manual diagnosis methods not only consume a lot of time and energy, but may also be affected by personal experience. Different people will obtain different results when analyzing the same cell image. Therefore, this method of diagnosis and analysis will make the results more or less affected by subjective factors.
[0007] (2) In the segmentation tasks of U-Net for neural structures and electron microscope images, it is still a difficult task to obtain accurate segmentation results due to the problems of class imbalance and noise in medical images.
[0008] The difficulty in solving the above problems and defects is that there are often high-density cell groups and a lot of useless interference information in medical cell images, such as unclear images, microbial impurities, and uneven cell staining. It is precisely the existence of these problems that makes it more difficult to design appropriate cell foreground features, resulting in manual annotation and traditional segmentation algorithms being unable to accurately segment target cells.
[0009] The significance of solving the above problems and defects is: solving the above problems can avoid the mis-segmentation caused by the imbalance of categories in cell images, avoid the loss of edge information of large targets and information of small targets in cell images, and the impact of differences in receptive fields on network performance, greatly improving the accuracy of segmentation. Effective medical image segmentation is an indispensable part of modern auxiliary medical systems, which greatly improves the diagnostic and treatment efficiency of radiologists and clinicians. Summary of the invention
[0010] In view of the problems existing in the prior art, the present invention provides a U-Net++ cell segmentation network system, method, device and terminal, and more particularly, relates to a U-Net++ cell segmentation network system, method, device and terminal based on a multi-scale and multi-feature fusion module.
[0011] The present invention is implemented as follows: a U-Net++ cell segmentation method, the U-Net++ cell segmentation method comprising:
[0012] The input image is initialized and preprocessed through data enhancement, data normalization and regularization of image size operations; the U-Net++ cell segmentation model based on multi-scale and multi-feature fusion modules is trained. Based on the U-Net++ network, multi-scale modules and multi-feature fusion modules are added to segment the input image and obtain preliminary segmentation results; the watershed algorithm is used to accurately segment the adhesion area in the cell segmentation image; the segmentation results are saved and analyzed, and the images in the verification set are segmented using instance methods and existing technologies, and comparative experiments and analysis are carried out.
[0013] Further, the U-Net++ cell segmentation method comprises the following steps:
[0014] Step 1: Image preprocessing, including data enhancement, image size regularization and data normalization: Since there is less cell image data, the above operations not only increase the diversity of training samples and improve the generalization ability of the model, but also improve the robustness of the model, reduce the sensitivity of the model to the image, and alleviate the overfitting phenomenon.
[0015] Step 2: Train the U-Net++ cell segmentation model based on the multi-scale and multi-feature fusion module: This improves the segmentation accuracy of the model and solves the segmentation difficulty problem caused by category imbalance in medical image segmentation.
[0016] Step three: Use the watershed algorithm to accurately segment the cell adhesion area in the image: accurately segment the cell adhesion area in the image, while avoiding over-segmentation, ensuring the subsequent quality analysis of the cell interior.
[0017] Step 4: Save and analyze the segmentation results: Output and save the segmentation results and corresponding performance indicators for later analysis and inspection.
[0018] Furthermore, in step 1, the image preprocessing includes initializing the input image by data enhancement, data normalization and regularizing the image size:
[0019] (1) Data enhancement
[0020] 1) Contrast enhancement: Map the grayscale points in the image so that the grayscale of the entire image roughly conforms to a uniform distribution, thereby enhancing the contrast of the image;
[0021] 2) Horizontal flip: randomly select images from each training batch and perform a 180° mirror flip;
[0022] 3) Upside-down flip: randomly select images in each training batch and perform 180° vertical flip;
[0023] 4) Image rotation: Randomly rotate the images in each training batch by 0 to 180 degrees, and fill the missing parts with zeros;
[0024] 5) Image translation: Randomly translate the images in each training batch upward, downward, left, and right, and fill the missing parts with zeros;
[0025] 6) Gaussian noise: Gaussian noise is randomly added to each training batch of images. The probability density function of Gaussian noise is as follows:
[0026]
[0027] Where σ is the standard deviation of z, is the mean.
[0028] (2) Image size regularization and data normalization
[0029] 1) Regularize the size of the input image and unify the image size of the training set and the test set to 96×96;
[0030] 2) Normalize the input cell image data and normalize the value range of the input image pixels to [0,1] by scaling. The calculation method is as follows:
[0031]
[0032] Among them, x ij is the pixel value of a point in the image, min is the minimum pixel value, and max is the maximum pixel value.
[0033] Further, in step 2, the training of the U-Net++ cell segmentation model based on the multi-scale and multi-feature fusion module includes:
[0034] (1) Input the preprocessed image into the U-Net++ network to train the model
[0035] The U-net++ network is an improvement based on the U-Net network. The network can be divided into two parts: feature extraction and image restoration. The feature extraction part is mainly composed of convolutional layers and pooling layers connected in series. The convolutional layer uses a 3×3 convolution kernel, and the pooling layer uses the maximum pooling method to participate in the calculation. The convolutional layer and the pooling layer both use the nonlinear correction function ReLu as the activation function; the image restoration part completes pixel filling alternately through upsampling layers and jump connections. U-Net++ improves the jump connection part of the U-Net network, introduces feature extraction units in the jump connection stage to capture features at different levels, and integrates these features through feature superposition.
[0036] (2) Adding a multi-feature fusion module to the U-Net++ network structure
[0037] The U-Net++ network is extended into two parallel architectures, adding different feature fusion modules in the upsampling stage and the downsampling stage.
[0038] 1) The feature fusion module in the downsampling stage performs feature fusion on the outputs of the two encoders after maximum pooling and average pooling respectively. The calculation steps are as follows:
[0039] x mel =Maxpooling(x el );
[0040] x mer =Avgpooling(x er );
[0041] x dl =x dr =Cat(x mel , x mer );
[0042] Among them, x el 、x er is the input feature of the feature fusion module, x dl 、x dr is the output feature of the feature fusion module.
[0043] 2) The feature fusion module in the upsampling stage includes: deconvolution operation on the input features, fusion after upsampling the features to the appropriate size and dimension; feature fusion of the fused upsampled features with the two features transmitted by the jump connection; dimensionality reduction operation of the fused features by 1×1 convolution and feature fusion again; the calculation steps are as follows:
[0044] x del =Deconv(x el );
[0045] x der =Deconv(x er );
[0046] x de =Cat(x del , x der );
[0047] x l =w l (Cat(x skipl , x de ))+b l ;
[0048] x r =w r (Cat(x skipr, x de ))+b r ;
[0049] x dl =x dr =Cat(x l , x r );
[0050] Among them, x el 、x er 、x skipl and x skipr is the input feature of the feature fusion module, x del and x der represents the up-sampled features, x de is x del and x der The fused features, x l and x r is the feature after 1×1 convolution dimension reduction, x dl 、x dr is the output feature of the feature fusion module.
[0051] (3) The feature extraction module in the network is replaced with a multi-scale module. The multi-scale module fuses the input features with convolution kernels with receptive field sizes of 3×3 and 7×7 for two consecutive operations, and then performs 1×1 convolution to reduce the dimension of the fused features and output them. Among them, 3×3 and 7×7 use ordinary convolution, the padding of 3×3 convolution is set to 1, the padding of 7×7 convolution is set to 3, and the step size is 1.
[0052] Among them, the calculation steps of the multi-scale module are as follows:
[0053] x1=w 32 (w 31 x+b 31 )+b 32 ;
[0054] x2=w 72 (w 71 x+b 71 )+b 72 ;
[0055] X = Cat(x1, x2);
[0056] F=w f X+b f ;
[0057] Among them, x represents the input feature, w 31 and w 32 is the convolution kernel weight with a receptive field size of 3×3, w 71 and w72 is the convolution kernel weight with a receptive field size of 7×7, w f is the convolution kernel weight with a receptive field size of 1×1, and F is the output of the multi-scale module.
[0058] Further, in step three, the method of accurately segmenting the cell adhesion area in the image using a watershed algorithm includes:
[0059] Through the improved U-Net++ network training model, each pixel in the image is divided into two categories: foreground and background; then the watershed algorithm is used to accurately segment the cell adhesion area in the image.
[0060] The watershed segmentation method is a mathematical morphology segmentation method based on topological theory. The basic idea is to regard the image as a topological landform in geodesy. The gray value of each pixel in the image represents the altitude of the point. Each local minimum and its influence area is called a catchment basin, and the boundary of the catchment basin forms a watershed. The calculation process of the watershed includes a sorting process and a flooding process; after sorting the gray level of each pixel from low to high, in the flooding process from low to high, the first-in-first-out FIFO structure is used to judge and mark the influence domain of each local minimum at the h-order height.
[0061] The watershed algorithm obtains the water basin image of the input image. The boundary points between the water basins are the watersheds, which represent the maximum points of the input image. Therefore, in order to obtain the edge information of the image, the gradient image is usually used as the input image, that is:
[0062]
[0063] Among them, f(x, y) represents the original image, and grad{.} represents the gradient operation.
[0064] The gradient image is thresholded to eliminate over-segmentation caused by slight changes in grayscale, namely:
[0065] g(x,y)=max(grad{f(x,y)},g(θ));
[0066] Where g(θ) represents the threshold.
[0067] Furthermore, in step 4, the storage and analysis of the segmentation results include:
[0068] The segmentation results and corresponding performance indicators are output and saved for later analysis and inspection. The images in the validation set are segmented using the example method and the existing technology, and the intersection over union (IoU) and segmentation value are used as evaluation indicators. The evaluation indicator calculation formula is as follows:
[0069]
[0070]
[0071] Among them, TP, TN, FP, and FN are positive samples of cell pixels predicted by the model as positive, negative samples predicted by the model as negative, negative samples predicted by the model as positive, and positive samples predicted by the model as negative; X and Y are the true value and the predicted value, respectively. IoU is a standard metric for semantic segmentation, which is the ratio of the intersection and union of the true value and the predicted value. When the value is 1, it means that the two are exactly the same; Dice Score is an important metric for medical image segmentation, which indicates the similarity between the true value and the predicted value. The larger the value, the higher the similarity.
[0072] Another object of the present invention is to provide a U-Net++ cell segmentation network system using the U-Net++ cell segmentation method, wherein the U-Net++ cell segmentation network system comprises:
[0073] Image preprocessing module, used for data enhancement, image size regularization and data normalization;
[0074] Cell segmentation model training module, used to train the U-Net++ cell segmentation model based on multi-scale and multi-feature fusion modules;
[0075] The cell adhesion region segmentation module is used to accurately segment the cell adhesion region in the image using the watershed algorithm;
[0076] The segmentation result saving and analysis module is used to output and save the segmentation results and corresponding performance indicators for later analysis and inspection.
[0077] Another object of the present invention is to provide a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and when the computer program is executed by the processor, the processor performs the following steps:
[0078] The input image is initialized and preprocessed through data enhancement, data normalization and regularization of image size operations; the U-Net++ cell segmentation model based on multi-scale and multi-feature fusion modules is trained. Based on the U-Net++ network, multi-scale modules and multi-feature fusion modules are added to segment the input image and obtain preliminary segmentation results; the watershed algorithm is used to accurately segment the adhesion area in the cell segmentation image; the segmentation results are saved and analyzed, and the images in the verification set are segmented using instance methods and existing technologies, and comparative experiments and analysis are carried out.
[0079] Another object of the present invention is to provide an information data processing terminal, which is used to implement the U-Net++ cell segmentation network system.
[0080] Another object of the present invention is to provide an application of the U-Net++ cell segmentation network system in medical cell image segmentation, tumor segmentation and multi-organ segmentation.
[0081] Combining all the above technical solutions, the advantages and positive effects of the present invention are as follows: The U-Net++ cell segmentation network provided by the present invention involves a cell segmentation method based on deep learning, which is directly related to the reliability of diagnosis and is also a difficult problem in medical image processing. The present invention effectively solves the segmentation problem of medical cell images and proposes a U-Net++ cell segmentation algorithm based on a multi-scale and multi-feature fusion module: wherein the multi-scale module can integrate features of different levels in medical images, so that the network can obtain more diverse semantic information from feature maps of the same scale; during downsampling and upsampling, feature fusion and sharing enable the network to classify pixels more accurately, thereby improving the accuracy of segmentation; at the same time, the watershed algorithm is used to accurately segment the cell adhesion area in the image. Compared with traditional cell segmentation methods, the accuracy and practicality of this method have been significantly improved. The present invention is suitable for auxiliary medical diagnosis, smart medical care and other fields.
[0082] The key technology of the present invention is a U-Net++ image segmentation algorithm based on multi-scale and multi-feature fusion; in terms of image preprocessing, a variety of data enhancement methods are first used to expand the data; secondly, before the cell image is input into the convolutional neural network, the image size is regularized and the data is normalized; in terms of image segmentation, a multi-scale and multi-feature fusion module is added to the U-Net++ network, which overcomes the problems that the traditional segmentation network cannot optimize the segmentation of different data sets and the segmentation difficulties caused by the imbalance of categories in the cell data samples; for the adhered cells in the image, the watershed algorithm is used to separate the mutually adhered cells to obtain the final segmentation result; finally, the segmentation result is saved and output for subsequent viewing and analysis.
[0083] The present invention uses the U-Net++ network as the basis, and improves the accuracy of the segmentation results by integrating features at different scales. The flexible network structure is combined with the deep supervision method to greatly reduce the number of parameters of the network with a large number of parameters and ensure stable segmentation accuracy.
[0084] The present invention extends U-Net++ in parallel and combines different downsampling methods to alleviate the problem of semantic information loss during downsampling. In addition, different feature fusion modules are designed between downsampling and upsampling to achieve feature fusion and sharing in the network, which improves the segmentation accuracy of the model and solves the impact of class imbalance in medical image segmentation on image segmentation results.
[0085] The present invention uses a multi-scale module to replace the feature extraction module in the network, so that the network can capture features at different levels by combining receptive fields of different sizes, increasing the diversity of features. In addition, the network can better determine the category of pixels in the image through receptive fields of different sizes, overcome the segmentation difficulties caused by the imbalance of categories in the sample, and improve the accuracy and robustness of the model. The watershed algorithm is used to accurately segment the cell adhesion area in the image, while avoiding the phenomenon of over-segmentation, ensuring that the quality analysis of the cell interior can be performed later.
[0086] The improvement of the downsampling stage of the present invention can not only retain the background information and texture information of the image, reduce the loss of semantic information, but also provide more complete feature information for subsequent convolution. The improved method of the upsampling stage of the present invention effectively integrates high-dimensional features with low-dimensional features, improves the utilization rate of semantic information in the network, reduces the loss of semantic information, and shares information in the network.
[0087] The invention introduces the multi-scale module, which has the following advantages: firstly, embedding multiple convolution kernels in the network can obtain feature information at multiple different levels, which not only improves the segmentation performance of the network, but also facilitates the learning of the network during the training process; secondly, the multi-scale module uses different convolution kernels to extract different semantic information. This structure enables the encoder of the network to better extract features and the decoder to better restore features. BRIEF DESCRIPTION OF THE DRAWINGS
[0088] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0089] Figure 1 It is a flow chart of the U-Net++ cell segmentation method provided by an embodiment of the present invention.
[0090] Figure 2 It is a schematic diagram of the U-Net++ cell segmentation method provided by an embodiment of the present invention.
[0091] Figure 3It is a structural block diagram of the U-Net++ cell segmentation network system provided by an embodiment of the present invention;
[0092] In the figure: 1. Image preprocessing module; 2. Cell segmentation model training module; 3. Cell adhesion area segmentation module; 4. Segmentation result storage and analysis module.
[0093] Figure 4 It is a U-Net++ network structure diagram provided by an embodiment of the present invention.
[0094] Figure 5 It is a structural diagram of a feature fusion module in the downsampling stage provided in an embodiment of the present invention.
[0095] Figure 6 It is a structural diagram of the feature fusion module in the upsampling stage provided in an embodiment of the present invention.
[0096] Figure 7 It is a multi-scale module structure diagram provided by an embodiment of the present invention.
[0097] Figure 8-Figure 9 It is a schematic diagram of the segmentation result provided by the embodiment of the present invention. DETAILED DESCRIPTION
[0098] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0099] In view of the problems existing in the prior art, the present invention provides a U-Net++ cell segmentation network system, method, device and terminal. The present invention is described in detail below with reference to the accompanying drawings.
[0100] like Figure 1 As shown, the U-Net++ cell segmentation method provided by the embodiment of the present invention includes the following steps:
[0101] S101, image preprocessing, including data enhancement, image size regularization and data normalization, increases the diversity of training samples, improves the generalization ability of the model, improves the robustness of the model, reduces the sensitivity of the model to the image, and alleviates the overfitting phenomenon;
[0102] S102, training a U-Net++ cell segmentation model based on multi-scale and multi-feature fusion modules, adding different feature fusion modules in the upsampling and downsampling stages of the network, using multi-scale modules to replace the feature extraction modules in the network, and combining receptive fields of different sizes to enable the network to capture features at different levels, increase feature diversity, improve the segmentation accuracy of the model, and solve the segmentation difficulty problem caused by category imbalance in medical image segmentation;
[0103] S103, using the watershed algorithm to accurately segment the cell adhesion area in the image, while avoiding the over-segmentation phenomenon, ensuring the subsequent quality analysis of the cell interior.
[0104] S104, storage and analysis of the segmentation results, outputting and storing the segmentation results and corresponding performance indicators for later analysis and inspection.
[0105] The principle diagram of the U-Net++ cell segmentation method provided by the embodiment of the present invention is as follows Figure 2 shown.
[0106] like Figure 3 As shown, the U-Net++ cell segmentation network system provided by the embodiment of the present invention includes:
[0107] Image preprocessing module 1, used for data enhancement, image size regularization and data normalization;
[0108] Cell segmentation model training module 2, used to train the U-Net++ cell segmentation model based on multi-scale and multi-feature fusion modules;
[0109] The cell adhesion region segmentation module 3 is used to accurately segment the cell adhesion region in the image using a watershed algorithm;
[0110] The segmentation result storage and analysis module 4 is used to output and store the segmentation results and corresponding performance indicators for later analysis and inspection.
[0111] The technical solution of the present invention is further described below in conjunction with specific embodiments.
[0112] Reference Figure 2 , the implementation steps of the present invention are as follows:
[0113] Step 1: Image preprocessing: Initialize and preprocess the input image through operations such as data augmentation, data normalization, and regularizing the image size.
[0114] 1.1) Data augmentation: Since there is less cell image data, data augmentation not only increases the diversity of training samples and improves the generalization ability of the model, but also improves the robustness of the model, reduces the sensitivity of the model to images, and alleviates overfitting. The specific steps of data augmentation are as follows:
[0115] a) Contrast enhancement: Map the grayscale points in the image so that the grayscale of the entire image roughly conforms to a uniform distribution, thereby enhancing the contrast of the image.
[0116] b) Horizontal flip: Randomly select images from each training batch and perform a 180° mirror flip.
[0117] c) Upside-down flip. Randomly select images from each training batch and perform a 180° vertical flip.
[0118] d) Image rotation: Randomly rotate the images in each training batch by 0-180° and fill the missing parts with zeros.
[0119] e) Image translation: Randomly translate the images in each training batch upward, downward, left, or right, and fill the missing parts with zeros.
[0120] f) Gaussian noise. Gaussian noise is randomly added to each training batch of images. The probability density function of Gaussian noise is as follows:
[0121]
[0122] Where σ is the standard deviation of z, is the mean.
[0123] 1.2) Image size regularization and data normalization
[0124] a) Normalize the size of the input image and change the image size of the training set and test set to 96×96.
[0125] b) Normalize the input cell image data and normalize the value range of the input image pixels to [0,1] by scaling. The specific calculation method is as follows:
[0126]
[0127] Among them, x ij is the pixel value of a point in the image, min is the minimum pixel value, and max is the maximum pixel value.
[0128] Step 2: Train the segmentation model - U-Net++ cell segmentation model based on multi-scale and multi-feature fusion modules
[0129] 2.1) Input the preprocessed images into the U-Net++ network to train the model. The U-Net++ network structure is shown in the figure below: Figure 4 shown.
[0130] The U-net++ network is an improvement based on the U-Net network. The network can be divided into two parts: feature extraction and image restoration. The feature extraction part is mainly composed of convolutional layers and pooling layers connected in series. The convolutional layer uses a 3×3 convolution kernel, and the pooling layer uses the maximum pooling method to participate in the calculation. The convolutional layer and the pooling layer both use the nonlinear correction function ReLu as the activation function; the image restoration part completes pixel filling alternately through upsampling layers and jump connections. U-Net++ improves the jump connection part of the U-Net network, introduces feature extraction units in the jump connection stage to capture features at different levels, and integrates these features through feature superposition.
[0131] 2.2) Adding a multi-feature fusion module to the U-Net++ network structure
[0132] The U-Net++ network is extended into two parallel architectures, adding different feature fusion modules in the upsampling stage and the downsampling stage.
[0133] a) If Figure 5 As shown in the figure, the feature fusion module in the downsampling stage performs feature fusion on the outputs of the two encoders after maximum pooling and average pooling respectively. This improvement can not only retain the background information and texture information of the image, reduce the loss of semantic information, but also provide more complete feature information for subsequent convolution. The specific calculation steps are as follows:
[0134] x mel =Maxpooling(x el )
[0135] x mer =Avgpooling(x er )
[0136] x dl =x dr =Cat(x mel , x mer )
[0137] Among them, x el 、x er is the input feature of the feature fusion module, x dl 、x dr is the output feature of the feature fusion module.
[0138] b) If Figure 6As shown in the figure, the feature fusion module in the upsampling stage is mainly divided into the following parts: First, the input features are deconvolved, and the features are upsampled to the appropriate size and dimension before fusion. Secondly, the fused upsampled features are fused with the two features transmitted by the jump connection respectively. Finally, the fused features are reduced in dimension using 1×1 convolution and then fused again. This improved method effectively integrates high-dimensional features with low-dimensional features, improves the utilization rate of semantic information in the network, reduces the loss of semantic information, and shares information in the network. The specific calculation steps are as follows:
[0139] x del =Deconu(x el )
[0140] x der =Deconv(x er )
[0141] x de =Cat(x del , x der )
[0142] x l =w l (Cat(x skipl , x de ))+b l
[0143] x r =w r (Cat(x skipr , x de ))+b r
[0144] x dl =x dr =Cat(x l , x r )
[0145] Among them, x el 、x er 、x skipl and x skipr is the input feature of the feature fusion module, x del and x der represents the up-sampled features, x de is x del and x der The fused features, x l and x r is the feature after 1×1 convolution dimension reduction, x dl 、x dr is the output feature of the feature fusion module.
[0146] 2.3) Replace the feature extraction module in the network with a multi-scale module, such as Figure 7 The multi-scale module shown in the figure fuses the input features with convolution kernels of receptive field size 3×3 and 7×7 for two consecutive operations, and then performs 1×1 convolution to reduce the dimension of the fused features and output them. Among them, 3×3 and 7×7 use ordinary convolutions, the padding of 3×3 convolution is set to 1, the padding of 7×7 convolution is set to 3, and the step size is 1. The introduction of the multi-scale module has the following main advantages: First, embedding multiple convolution kernels in the network can obtain feature information at multiple different levels, which not only improves the segmentation performance of the network, but also facilitates the learning of the network during training; secondly, the multi-scale module uses different convolution kernels to extract different semantic information. This structure enables the encoder of the network to better extract features and the decoder to better restore features. The specific calculation steps of this module are as follows:
[0147] x1=w 32 (w 31 x+b 31 )+b 32
[0148] x2=w 72 (w 71 x+b 71 )+b 72
[0149] X = Cat(x1, x2)
[0150] F=w f X+b f
[0151] Among them, x represents the input feature, w 31 and w 32 is the convolution kernel weight with a receptive field size of 3×3, w 71 and w 72 is the convolution kernel weight with a receptive field size of 7×7, w f is the convolution kernel weight with a receptive field size of 1×1, and F is the output of the multi-scale module.
[0152] Step 3: Watershed algorithm
[0153] Through the improved U-Net++ network training model, each pixel in the image is divided into two categories: foreground and background. However, in biological cell images, there are usually cells that adhere to each other. In order to ensure the subsequent quality analysis of the inside of the cell, the watershed algorithm is used to accurately segment the cell adhesion area in the image.
[0154] The watershed segmentation method is a mathematical morphology segmentation method based on topological theory. Its basic idea is to regard the image as a topological landform in geodesy. The gray value of each pixel in the image represents the altitude of the point. Each local minimum and its influence area are called a watershed, and the boundary of the watershed forms a watershed. The calculation process of the watershed is divided into two steps: one is the sorting process and the other is the flooding process. First, the gray level of each pixel is sorted from low to high. Then, in the process of flooding from low to high, the first-in-first-out (FIFO) structure is used to judge and mark the influence domain of each local minimum at the h-order height.
[0155] The watershed algorithm obtains the water basin image of the input image, and the boundary points between the water basins are the watershed. Obviously, the watershed represents the maximum point of the input image. Therefore, in order to obtain the edge information of the image, the gradient image is usually used as the input image, that is:
[0156]
[0157] Among them, f(x, y) represents the original image, and grad{.} represents the gradient operation.
[0158] The watershed algorithm has a good response to weak edges. Noise in the image and slight grayscale changes on the surface of the object will cause over-segmentation. In order to eliminate the over-segmentation caused by the watershed algorithm, the gradient image is thresholded to eliminate the over-segmentation caused by slight changes in grayscale, that is,
[0159] g(x,y)=max(grad{f(x,y)},g(θ))
[0160] Where g(θ) represents the threshold.
[0161] Step 4: Saving and analyzing segmentation results
[0162] The segmentation results and corresponding performance indicators are output and saved for later analysis and inspection. In order to better illustrate the implementation effect of the present invention, the images in the validation set are segmented using the method of the present invention and the prior art, and the intersection over union (IoU) and segmentation value (Dice Score) are used as evaluation indicators. The evaluation indicator calculation formula is as follows:
[0163]
[0164]
[0165] Among them, TP, TN, FP, and FN are positive samples of cell pixels predicted by the model as positive, negative samples predicted by the model as negative, negative samples predicted by the model as positive, and positive samples predicted by the model as negative. X and Y are the true value and the predicted value, respectively. IoU is a standard metric for semantic segmentation, which is the ratio of the intersection and union of the true value and the predicted value. When its value is 1, it means that the two are exactly the same. Dice Score is an important metric for medical image segmentation, which indicates the similarity between the true value and the predicted value. The larger the value, the higher the similarity.
[0166] As shown in Table 1, the segmentation results of five segmentation algorithms, namely, the threshold segmentation Otsu algorithm, the U-Net model, the U-Net++ model and the model in the embodiment of the present invention, are shown in turn.
[0167] Table 1 Segmentation results of five segmentation algorithms
[0168] method IoU Dice Score Otsu 0.2351 0.3782 FCN-8s 0.5349 0.7984 U-Net 0.7421 0.8509 U-Net++ 0.8393 0.9035 Examples of the present invention 0.8415 0.9148
[0169] Figure 8 , Fig. 9 The third and fourth images in the first row and the second row of images in the figure respectively show the use of threshold segmentation Otsu algorithm, FCN model, U-Net model, U-Net++ model and the model in the embodiment of the present invention. Figure 8 and Fig. 9 The first row shows the segmentation result of the first image.
[0170] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When the use is implemented in whole or in part in the form of a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL) or wireless (e.g., infrared, wireless, microwave, etc.) mode) to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.
[0171] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.
Claims
1. A U-Net++ cell segmentation method, characterized in that: The U-Net++ cell segmentation method performs initialization preprocessing on the input image through data enhancement, data normalization and regularization of image size operations; trains the U-Net++ cell segmentation model based on the multi-scale and multi-feature fusion module, adds the multi-scale module and the multi-feature fusion module based on the U-Net++ network, segments the input image, and obtains a preliminary segmentation result; uses the watershed algorithm to accurately segment the adhesion area in the cell segmentation image; completes the preservation and analysis of the segmentation results, respectively uses the example method and the existing technology to segment the images in the verification set, and conducts comparative experiments and analysis; The U-Net++ cell segmentation method comprises the following steps: Step 1: Image preprocessing, including data enhancement, image size regularization and data normalization; Step 2: Train the U-Net++ cell segmentation model based on multi-scale and multi-feature fusion modules; Step 3, using the watershed algorithm to accurately segment the cell adhesion area in the image; Step 4: Save and analyze the segmentation results. The segmentation results and corresponding performance indicators are output and saved for later analysis and inspection. In step 2, the training is based on a U-Net++ cell segmentation model based on a multi-scale and multi-feature fusion module, including: (1) Input the preprocessed image into the U-Net++ network to train the model The U-net++ network is an improvement based on the U-Net network. The network can be divided into two parts: feature extraction and image restoration. The feature extraction part is mainly composed of convolutional layers and pooling layers connected in series. The convolutional layer uses a 3×3 convolution kernel, and the pooling layer uses the maximum pooling method to participate in the calculation. The convolutional layer and the pooling layer both use the nonlinear correction function ReLu as the activation function. The image restoration part completes pixel filling alternately through upsampling layers and jump connections. U-Net++ improves the jump connection part of the U-Net network. In the jump connection stage, feature extraction units are introduced to capture features at different levels, and these features are integrated by feature superposition. (2) Adding a multi-feature fusion module to the U-Net++ network structure The U-Net++ network is extended into two parallel architectures, adding different feature fusion modules in the upsampling stage and the downsampling stage; 1) The feature fusion module in the downsampling stage performs feature fusion on the outputs of the two encoders after maximum pooling and average pooling respectively. The calculation steps are as follows: x mel =Maxpooling(x el ); x mer =Avgpooling(x er ); x dl =x dr =Cat(x mel ,x mer ); Among them, x el 、x er is the input feature of the feature fusion module, x dl 、x dr is the output feature of the feature fusion module; 2) The feature fusion module in the upsampling stage includes: deconvolution operation on the input features, fusion after upsampling the features to the appropriate size and dimension; feature fusion of the fused upsampled features with the two features transmitted by the jump connection; dimensionality reduction operation of the fused features by 1×1 convolution and feature fusion again; the calculation steps are as follows: x del =Deconv(x el ); x der =Deconv(x er ); x de =Cat(x del ,x der ); x l =w l (Cat(x skipl ,x de ))+b l ; x r =w r (Cat(x skipr ,x de ))+b r ; x dl =x dr =Cat(x l ,x r ); Among them, x el 、x er 、x skipl and x skipr is the input feature of the feature fusion module, x del and x der represents the up-sampled features, x de is x del and x der The fused features, x l and x r is the feature after 1×1 convolution dimension reduction, x dl 、x dr is the output feature of the feature fusion module; (3) The feature extraction module in the network is replaced with a multi-scale module. The multi-scale module performs two consecutive operations on the input features with convolution kernels of receptive field sizes of 3×3 and 7×7, fuses them, and then performs 1×1 convolution to reduce the dimension of the fused features and outputs them. Among them, 3×3 and 7×7 use ordinary convolution, the padding of 3×3 convolution is set to 1, the padding of 7×7 convolution is set to 3, and the step size is 1. Among them, the calculation steps of the multi-scale module are as follows: x1=w 32 (w 31 x+b 31 )+b 32 ; x2=w 72 (w 71 x+b 71 )+b 72 ; X = Cat(x1, x2); F=w f X+b f ; Among them, x represents the input feature, w 31 and w 32 is the convolution kernel weight with a receptive field size of 3×3, w 71 and w 72 is the convolution kernel weight with a receptive field size of 7×7, w f is the convolution kernel weight with a receptive field size of 1×1, and F is the output of the multi-scale module.
2. The U-Net++ cell segmentation method according to claim 1, characterized in that: In step 1, the image preprocessing includes initializing the input image through data enhancement, data normalization and regularizing the image size operations: (1) Data enhancement 1) Contrast enhancement: Map the grayscale points in the image so that the grayscale of the entire image roughly conforms to a uniform distribution, thereby enhancing the contrast of the image; 2) Horizontal flip: randomly select images from each training batch and perform a 180° mirror flip; 3) Upside-down flip: randomly select images in each training batch and perform 180° vertical flip; 4) Image rotation: Randomly rotate the images in each training batch by 0 to 180 degrees, and fill the missing parts with zeros; 5) Image translation: Randomly translate the images in each training batch upward, downward, left, and right, and fill the missing parts with zeros; 6) Gaussian noise: Gaussian noise is randomly added to each training batch of images. The probability density function of Gaussian noise is as follows: Where σ is the standard deviation of z, is the mean; (2) Image size regularization and data normalization 1) Regularize the size of the input image and unify the image size of the training set and the test set to 96×96; 2) Normalize the input cell image data and normalize the value range of the input image pixels to [0,1] by scaling. The calculation method is as follows: Among them, x ij is the pixel value of a point in the image, min is the minimum pixel value, and max is the maximum pixel value.
3. The U-Net++ cell segmentation method according to claim 1, characterized in that: In step 3, the method of accurately segmenting the cell adhesion area in the image using a watershed algorithm includes: Through the improved U-Net++ network training model, each pixel in the image is divided into two categories: foreground and background; then the watershed algorithm is used to accurately segment the cell adhesion area in the image; The watershed segmentation method is a mathematical morphology segmentation method based on topological theory. The basic idea is to regard the image as a topological landform in geodesy. The gray value of each pixel in the image represents the altitude of the point. Each local minimum and its influence area is called a watershed, and the boundary of the watershed forms a watershed. The calculation process of the watershed includes a sorting process and a flooding process. After sorting the gray level of each pixel from low to high, in the flooding process from low to high, the influence domain of each local minimum at the h-order height is judged and marked using a first-in-first-out FIFO structure. The watershed algorithm obtains the water basin image of the input image. The boundary points between the water basins are the watersheds, which represent the maximum points of the input image. Therefore, in order to obtain the edge information of the image, the gradient image is used as the input image, that is: Where f(x, y) represents the original image, and grad{·} represents the gradient operation; The gradient image is thresholded to eliminate over-segmentation caused by slight changes in grayscale, namely: g(x,y)=max(grad{f(x,y)},g(θ)); Where g(θ) represents the threshold.
4. The U-Net++ cell segmentation method according to claim 1, characterized in that: In step 4, the storage and analysis of the segmentation results include: The segmentation results and corresponding performance indicators are output and saved for later analysis and inspection; the images in the validation set are segmented using the instance method and the existing technology, and the intersection over union (IoU) and segmentation value are used as evaluation indicators. The evaluation indicator calculation formula is as follows: Among them, TP, TN, FP, and FN are positive samples of cell pixels predicted by the model as positive, negative samples predicted by the model as negative, negative samples predicted by the model as positive, and positive samples predicted by the model as negative; X and Y are the true value and predicted value, respectively; IoU is a standard metric for semantic segmentation, which is the ratio of the intersection and union of the true value and the predicted value. When the value is 1, it means that the two are exactly the same; DiceScore is an important metric for medical image segmentation, which indicates the similarity between the true value and the predicted value. The larger the value, the higher the similarity.
5. A U-Net++ cell segmentation network system for implementing the U-Net++ cell segmentation method according to any one of claims 1 to 4, characterized in that: The U-Net++ cell segmentation network system includes: Image preprocessing module, used for data enhancement, image size regularization and data normalization; Cell segmentation model training module, used to train the U-Net++ cell segmentation model based on multi-scale and multi-feature fusion modules; The cell adhesion region segmentation module is used to accurately segment the cell adhesion region in the image using the watershed algorithm; The segmentation result saving and analysis module is used to output and save the segmentation results and corresponding performance indicators for later analysis and inspection.
6. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the U-Net++ cell segmentation method according to any one of claims 1 to 4.
7. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the U-Net++ cell segmentation network system as described in claim 5.
Citation Information
Patent Citations
Cancer cell image detection and segmentation method based on Mask R-CNN algorithm
CN111489327A
Blood leukocyte image segmentation method based on UNet++ and ResNet
CN112070772A
Automatic tumor segmentation system and method, and electronic device
WO2021104056A1
