A 3D cell image instance segmentation method based on depth vector field regression
Through the depth vector field regression method, an example segmentation deep regression neural network is constructed to learn the Euclidean distance and direction vector from cells to background, solving the problem of tight cell adhesion in three-dimensional cell image segmentation, significantly improving segmentation accuracy.
Patent Information
- Application Number
- CN202111461795.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-11-17
- Filing Date
- 2021-12-02
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-12-02
AI Technical Summary
The prior art has the problem of tight adhesion in three-dimensional cell image segmentation, which leads to low segmentation accuracy and affects subsequent recognition accuracy.
Using a method based on deep vector field regression, an instance segmented deep regression neural network is constructed. Through the convolutional neural network model and loss function design, the Euclidean distance and direction vector from cells to background pixels are learned, and converted into a regression task to improve segmentation accuracy.
By converting the segmentation task into a regression task, learning prospects and background information and including cell direction information, the segmentation accuracy of three-dimensional cells is significantly improved and the segmentation process is simplified.
Smart Images

Figure CN114155249B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of cell image processing, and in particular to a three-dimensional cell image instance segmentation method based on depth vector field regression. Background Art
[0002] At present, life science has gradually entered the era of single-cell biology. The Human Cell Atlas Project, launched in 2016, also aims at omics data at the single-cell level. The lineage and development state of each cell determines how it reacts to surrounding cells and the environment. Applying gene expression and sequencing to the single-cell level plays a vital role in understanding the origin, function, and variation of cells, as well as finding answers to a series of important biological problems in cancer research, stem cell biology, immunology, developmental biology, and neuroscience. The three-dimensional cell lineage and cell distribution remain basically unchanged, and it has become possible to accurately track and analyze their gene expression at the single-cell level by image computing. As the gold standard for cell lineage tracing, three-dimensional cells are the only multicellular model organisms that have obtained a complete cell lineage from fertilized eggs to adults and have a fixed cell lineage.
[0003] Automatic segmentation and recognition of objects of interest are two very challenging tasks in the fields of computer vision, medicine, and biological image processing. The research most relevant to 3D cell segmentation and recognition is mainly distributed in the semantic segmentation, instance segmentation, target detection and recognition of computer vision, and the segmentation and recognition of specific organs, tissues, and cells in biomedicine. Aside from the differences in specific methods and segmentation results, the purpose of segmentation and recognition in these related studies is generally to extract a class of objects of interest from the background, such as people, cars, cells, etc. This is different from the segmentation and recognition of 3D cells, which requires segmenting different individuals of the same type of cells.
[0004] Detection and segmentation of cells in 3D images is the first step in studying cell expression. The mainstream idea of existing algorithms is to segment cells first and then perform subsequent recognition based on the results of cell segmentation. Therefore, the accuracy of segmentation directly affects the accuracy of subsequent recognition. In the field of biomedical image processing, the main difficulties in accurately segmenting 3D cell images come from the following three aspects: (1) low signal-to-noise ratio of images caused by image acquisition, staining defects and sample preparation; (2) heterogeneity of cell size, shape and brightness distribution between and within cells; (3) close contact or partial overlap of cells caused by dense distribution.
[0005] Therefore, how to solve the existing challenges faced by cell segmentation and improve the accuracy of cell segmentation has become a technical problem that needs to be solved urgently. Summary of the invention
[0006] The purpose of the present invention is to solve the defects of cells closely adhering and difficult to separate in the prior art of three-dimensional cell image segmentation, and to provide a three-dimensional cell image instance segmentation method based on depth vector field regression to solve the above problem.
[0007] In order to achieve the above object, the technical solution of the present invention is as follows:
[0008] A three-dimensional cell image instance segmentation method based on depth vector field regression comprises the following steps:
[0009] 11) Acquisition and preprocessing of original data and label data of three-dimensional cell images: Acquisition of original three-dimensional cell images and label images, cutting of original three-dimensional cell images as training sets; vector processing of label images and acquisition of vector diagrams of label images;
[0010] 12) Construction of deep regression neural network for instance segmentation: Build a deep regression neural network for instance segmentation based on the convolutional neural network model, and design the loss function and parameter settings for network training;
[0011] 13) Training of instance segmentation deep regression neural network: input the preprocessed three-dimensional cell image and vector graph of the label image into the instance segmentation deep regression neural network for training;
[0012] 14) Obtaining a three-dimensional cell distance map: obtaining an original three-dimensional cell image to be segmented, and inputting the original three-dimensional cell image to be segmented into a trained instance segmentation deep regression neural network to output a three-channel three-dimensional cell vector map; performing a Euclidean distance transformation on the three-channel three-dimensional cell vector map to convert it into a three-channel three-dimensional cell distance map to be segmented;
[0013] 15) Obtaining the three-dimensional cell instance segmentation result: Post-processing the three-channel three-dimensional cell distance map to be segmented to obtain the segmented three-dimensional cell instance result.
[0014] The construction of the instance segmentation deep regression neural network includes the following steps:
[0015] 21) The instance segmentation deep regression neural network is set to be based on a convolutional neural network, the basic network of the convolutional neural network is a U-Net architecture, and the input of the convolutional neural network is a one-channel three-dimensional original image;
[0016] 22) Set the output of the convolutional neural network to the same size as the input image, but the output is a three-channel 3D image stitched together, where the three channels represent the x, y, and z coordinates respectively;
[0017] 23) Set the loss function of the convolutional neural network to the distance loss L distand vector direction constrained color loss L color , the total loss function is as follows:
[0018] L total =αL dist +βL color ,
[0019] Among them, α and β are weight coefficients;
[0020] 24) For the weight coefficient in the total loss function, the weight coefficient ratio is set to 7:1, and the total loss function is designed as follows:
[0021] L total =7*L dist +L color ;
[0022] 25) Set the last softmax layer of the U-Net architecture to be removed.
[0023] The training of the instance segmentation deep regression neural network includes the following steps:
[0024] 31) Perform image preprocessing on the vector map of the label image to obtain the position of the nearest background pixel corresponding to each pixel in the cell; the image preprocessing method is as follows:
[0025] 311) Subtract the position of each pixel in the cell from the coordinates of the nearest background pixel corresponding to the pixel in the cell to obtain a position vector, which represents the direction vector from the pixel in the cell to its nearest background pixel;
[0026] 312) Set the pixel position in the cell to C ij =(X ij ,Y ij ,Z ij ), the coordinates of the nearest background pixel corresponding to the pixel are The position vector is then expressed as:
[0027] ΔP ij =B ij -C ij ;
[0028] 32) Inputting the preprocessed label image and the three-dimensional cell original image into the instance segmentation deep regression neural network;
[0029] 33) Using the depth regression technique, the direction vector from the pixel to its nearest background pixel is regressed;
[0030] 34) The regressed direction vector and the direction vector of the label image are input into the loss function for gradient feedback to update the model parameters;
[0031] 35) When the loss function Loss value drops to a relatively optimal level, the instance segmentation deep regression neural network training is completed.
[0032] The acquisition of the three-dimensional cell instance segmentation result comprises the following steps:
[0033] 41) The three-channel three-dimensional cell distance map to be segmented is converted into a distance map, and the formula is as follows:
[0034]
[0035] Among them, x, y, z are vector values, and d is the distance value;
[0036] 42) Process the distance map into a probability map, and process the probability map into uint8 type, and invert the image. The inversion formula is as follows:
[0037] S = 255 – S;
[0038] Where S is the probability map;
[0039] 43) Given parameter h as 7, specify the minimum value of the probability map plus h compared with 255 as the seed, use the probability map as the mask, and reconstruct the image by erosion;
[0040] 44) Find the peak value of the reconstructed image and find all the local maximum values in the image;
[0041] 45) Perform connected domain processing on the image after the local maximum, use the reconstructed image as the image, the peak image as the marker, and use the watershed algorithm to separate the cells;
[0042] 46) For the image after watershed, set all backgrounds to 0 to obtain a segmentation mask;
[0043] 47) Finally, fill the holes in the segmentation mask to obtain the final 3D cell segmentation result.
[0044] Beneficial Effects
[0045] Compared with the prior art, the present invention discloses a three-dimensional cell image instance segmentation method based on deep vector field regression. By converting the segmentation task into a regression task, the regression task is used to learn the Euclidean distance from the foreground pixel to the background pixel, and the direction vector from the foreground pixel to the nearest background pixel is used to better learn the foreground and background information. Meanwhile, the direction information of the cell is also included, and color loss is used to further strengthen the learning of the direction information, thereby simply and efficiently improving the segmentation accuracy of the three-dimensional cell. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1is a method sequence diagram of the present invention;
[0047] Figure 2 This is a framework diagram of a deep regression neural network for instance segmentation involved in the present invention;
[0048] Figure 3a A vector transformation calculation graph of all pixels including three cells in the present invention;
[0049] Figure 3b The vector transformation calculation diagram of the present invention is a vector transformation calculation diagram of the number of pixels of the present invention, which is ten times the number of pixels of all the pixels of three cells diluted;
[0050] Figure 4a A three-dimensional cell original image involved in the present invention;
[0051] Figure 4b A label image corresponding to the selected three-dimensional cell original image involved in the present invention;
[0052] Figure 4c The result graph after distance transformation of the label graph involved in the present invention;
[0053] Figure 5a This is the result of segmenting three-dimensional cells using the traditional IFT-WATERSHED method;
[0054] Figure 5b This is the result of segmenting three-dimensional cells using Unet;
[0055] Figure 5c This is the segmentation result of three-dimensional cells using the Dist method;
[0056] Figure 6a This is a graph showing the results of learning a three-dimensional cell model using the method of the present invention;
[0057] Figure 6b It is the final segmentation result image after post-processing the output image using the method described in the present invention. DETAILED DESCRIPTION
[0058] In order to have a further understanding and recognition of the structural features and the effects achieved by the present invention, a preferred embodiment and accompanying drawings are used for detailed description as follows:
[0059] like Figure 1 As shown, the three-dimensional cell image instance segmentation method based on depth vector field regression described in the present invention comprises the following steps:
[0060] The first step is to obtain and preprocess the original data and label data of the 3D cell image. The original 3D cell image and label image are obtained. Since the 3D cell image is too large, the original 3D cell image is cut into blocks as a training set. The label image is vectorized and the vector image of the label image is obtained. In the laboratory stage, the image is cut into 128×128×80 blocks, and the data set is divided into a training set and a test set, which are randomly divided in a ratio of 4:1.
[0061] The second step is to build a deep regression neural network for instance segmentation: Figure 2 As shown in the figure, we build a deep regression neural network for instance segmentation based on the convolutional neural network model, and design the loss function and parameter settings for network training. The specific steps are as follows:
[0062] (1) The instance segmentation deep regression neural network is set based on the convolutional neural network. The basic network of the convolutional neural network is the U-Net architecture, and the input of the convolutional neural network is a one-channel three-dimensional original image.
[0063] (2) The output of the convolutional neural network is set to have the same size as the input image, but the output is a three-channel three-dimensional image stitched together, where the three channels represent the x, y, and z coordinates respectively.
[0064] (3) Set the loss function of the convolutional neural network to the distance loss L dist and vector direction constrained color loss L color , the total loss function is as follows:
[0065] L total =αL dist +βL color ,
[0066] Among them, α and β are weight coefficients.
[0067] (4) For the weight coefficient in the total loss function, the weight coefficient ratio is set to 7:1, and the total loss function is designed as follows:
[0068] L total =7*L dist +L color .
[0069] (5) Set to remove the last softmax layer of the U-Net architecture for regression tasks. Usually, when solving multi-classification problems, we will add a softmax layer to the last layer of the network and set n output nodes. The output value of each node in the softmax layer becomes a probability distribution with a probability sum of 1. In the present invention, we use deep regression to learn features, so we need to remove the last softmax layer.
[0070] The third step is to train the instance segmentation deep regression neural network: the preprocessed three-dimensional cell image and the vector graph of the label image are input into the instance segmentation deep regression neural network for training.
[0071] Existing methods for instance segmentation of cells are usually trained and learned on the basis of classification. Therefore, when there are many cells and the cells are closely connected, it is difficult for existing methods to distinguish these closely connected cells well. The three-dimensional cell instance segmentation based on deep regression proposed in the present invention draws on the idea of regression. Specifically, the present invention uses regression task learning to solve this problem. The advantage of regression learning is that it can clearly distinguish the boundaries of densely contacted cells, and further considers using directional information to distinguish different cells.
[0072] The specific steps for training the deep regression neural network for instance segmentation are as follows:
[0073] (1) Perform image preprocessing on the vector map of the label image to obtain the position of the nearest background pixel corresponding to each pixel in the cell. The direction vector from the foreground pixel to the background pixel of the original three-dimensional cell is learned through a deep neural network. Therefore, the label processing needs to be completed in advance to facilitate the later learning of the network. The image preprocessing method is as follows:
[0074] A1) Subtract the position of each pixel in the cell from the coordinates of the nearest background pixel corresponding to the pixel in the cell to obtain a position vector, which represents the direction vector from the pixel in the cell to its nearest background pixel;
[0075] A2) Set the pixel position in the cell to C ij =(X ij ,Y ij ,Z ij ), the coordinates of the nearest background pixel corresponding to the pixel are The position vector is then expressed as:
[0076] ΔP ij =B ij -C ij .
[0077] The generated position vector display effect is as follows Figure 3a and Figure 3b shown.
[0078] (2) The preprocessed label image and the original three-dimensional cell image are input into the instance segmentation deep regression neural network.
[0079] (3) Using deep regression technology, the direction vector from the pixel to its nearest background pixel is regressed.
[0080] (4) The regressed direction vector and the direction vector of the label image are input into the loss function for gradient feedback to update the model parameters.
[0081] (5) When the loss function Loss value drops to a relatively optimal level, the instance segmentation deep regression neural network training is completed.
[0082] Here, the maximum number of training times can be determined through experiments based on the selected network (deep regression neural network model) and data set, and can be determined in combination with parameter adjustment experience, with the goal of successfully training the network (deep regression neural network model). The sign of successful network (deep regression neural network model) training is usually that the loss function value no longer decreases, but overfitting must also be prevented. The final judgment of the network (deep regression neural network model) performance requires testing the test set, and judging based on the results of the test set can use some quantitative indicators, such as the F1 score or the AJI index.
[0083] Step 4: Obtaining the three-dimensional cell distance map. Obtain the original three-dimensional cell image to be segmented, and input the original three-dimensional cell image to be segmented into the trained instance segmentation deep regression neural network to output a three-channel three-dimensional cell vector map; perform Euclidean distance transformation on the three-channel three-dimensional cell vector map to convert it into a three-channel three-dimensional cell distance map to be segmented.
[0084] Step 5: Obtaining the 3D cell instance segmentation result: Post-processing the three-channel 3D cell distance map to be segmented to obtain the segmented 3D cell instance result.
[0085] (1) Convert the three-channel three-dimensional cell distance map to be segmented into a distance map using the following formula:
[0086]
[0087] Among them, x, y, z are vector values, and d is the distance value;
[0088] (2) Process the distance map into a probability map, and process the probability map into uint8 type, and invert the image. The inversion formula is as follows:
[0089] S = 255 – S;
[0090] Where S is the probability map;
[0091] (3) Given parameter h as 7, specify the minimum value of the probability map plus h compared with 255 as the seed, use the probability map as the mask, and reconstruct the image by erosion;
[0092] (4) Find the peak value of the reconstructed image and find all the local maximum values in the image;
[0093] (5) Perform connected domain processing on the image after the local maximum, use the reconstructed image as the image, the peak image as the marker, and use the watershed algorithm to separate the cells;
[0094] (6) For the image after watershed, set all backgrounds to 0 to obtain a segmentation mask;
[0095] (7) Finally, fill the holes in the segmentation mask to obtain the final 3D cell segmentation result.
[0096] In the laboratory, when post-processing the distance map of 3D cells, there is a set of parameters for morphological reconstruction in the post-processing process. In the experimental test, based on the quantitative results of the F1 score and AJI indicator, we set the parameter threshold to 10 to obtain the final 3D cell instance segmentation result.
[0097] Figure 4a and Figure 4b are the original image and label image of the three-dimensional cell image, Figure 4c It is a distance map display obtained by performing distance transformation on the label image. The goal is to segment the cell instances in the original image based on the label image. Figure 5a The segmentation results obtained by using the traditional method IFT-WATERSHED to perform cell instance segmentation on three-dimensional cell images are Figure 5b Unet is used to segment cells by learning three categories: cell boundary, cell interior and background. Figure 5b This is the result obtained using this method. Figure 5c The Dist mentioned above performs cell segmentation by directly regressing the distance. Figure 6a is the distance map learned under the deep model trained in the present invention, Figure 6b This is the final result obtained by using the present invention. By comparison, it can be intuitively found that the segmentation effect of the method proposed by the present invention is better.
[0098] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.
Claims
1. A 3D cell image instance segmentation method based on depth vector field regression, characterized in that: The following steps are involved: 11) Acquisition and preprocessing of original data and label data of three-dimensional cell images: Acquisition of original three-dimensional cell images and label images, cutting of original three-dimensional cell images as training sets; vector processing of label images and acquisition of vector diagrams of label images; 12) Construction of deep regression neural network for instance segmentation: Build a deep regression neural network for instance segmentation based on the convolutional neural network model, and design the loss function and parameter settings for network training; The construction of the instance segmentation deep regression neural network includes the following steps: 121) The instance segmentation deep regression neural network is set to be based on a convolutional neural network, the basic network of the convolutional neural network is a U-Net architecture, and the input of the convolutional neural network is a one-channel three-dimensional original image; 122) Set the output of the convolutional neural network to have the same size as the input image, but the output is a three-channel three-dimensional image stitched together, where the three channels represent the x, y, and z coordinates respectively; 123) Set the loss function of the convolutional neural network to the distance loss L dist and vector direction constrained color loss L color , the total loss function is as follows: L total =αL dist +βL color , Among them, α and β are weight coefficients; 124) For the weight coefficient in the total loss function, the weight coefficient ratio is set to 7:1, and the total loss function is designed as follows: L total =7*L dist +L color ; 125) Set to remove the last softmax layer of the U-Net architecture; 13) Training of instance segmentation deep regression neural network: input the preprocessed three-dimensional cell image and vector graph of the label image into the instance segmentation deep regression neural network for training; 14) Obtaining a three-dimensional cell distance map: obtaining an original three-dimensional cell image to be segmented, and inputting the original three-dimensional cell image to be segmented into a trained instance segmentation deep regression neural network to output a three-channel three-dimensional cell vector map; performing a Euclidean distance transformation on the three-channel three-dimensional cell vector map to convert it into a three-channel three-dimensional cell distance map to be segmented; 15) Obtaining the three-dimensional cell instance segmentation result: Post-processing the three-channel three-dimensional cell distance map to be segmented to obtain the segmented three-dimensional cell instance result.
2. A three-dimensional cell image instance segmentation method based on depth vector field regression according to claim 1, characterized in that: The training of the instance segmentation deep regression neural network includes the following steps: 21) Perform image preprocessing on the vector map of the label image to obtain the position of the nearest background pixel corresponding to each pixel in the cell; the image preprocessing method is as follows: 211) Subtract the position of each pixel in the cell from the coordinates of the nearest background pixel corresponding to the pixel in the cell to obtain a position vector, which represents the direction vector from the pixel in the cell to its nearest background pixel; 212) Set the pixel position in the cell to C ij =(X ij ,Y ij ,Z ij ), the coordinates of the nearest background pixel corresponding to the pixel are The position vector is then expressed as: ΔP ij =B ij -C ij ; 22) Inputting the preprocessed label image and the three-dimensional cell original image into the instance segmentation deep regression neural network; 23) Using the instance segmentation deep regression neural network, regress the direction vector from the pixel to its nearest background pixel; 24) The regressed direction vector and the direction vector of the label image are input into the loss function for gradient feedback to update the model parameters; 25) When the loss function Loss value drops to a relatively optimal level, the instance segmentation deep regression neural network training is completed.
3. The method for three-dimensional cell image instance segmentation based on depth vector field regression according to claim 1, characterized in that: The acquisition of the three-dimensional cell instance segmentation result comprises the following steps: 31) Convert the three-channel three-dimensional cell distance map to be segmented into a distance map using the following formula: Among them, x, y, z are vector values, and d is the distance value; 32) Process the distance map into a probability map, and process the probability map into uint8 type, and invert the image. The inversion formula is as follows: S = 255–S; Where S is the probability map; 33) Given parameter h as 7, specify the minimum value of the probability map plus h compared with 255 as the seed, use the probability map as the mask, and reconstruct the image by erosion; 34) Find the peak value of the reconstructed image and find all the local maximum values in the image; 35) Perform connected domain processing on the image after the local maximum, use the reconstructed image as the image, the peak image as the marker, and use the watershed algorithm to separate the cells; 36) For the image after watershed, set all backgrounds to 0 to obtain a segmentation mask; 37) Finally, fill the holes in the segmentation mask to obtain the final 3D cell segmentation result.
Citation Information
Patent Citations
Semantic segmentation and edge fused high-resolution remote sensing target accurate detection method
CN112084872A
Method for segmenting and counting adipocyte image based on deep learning model
CN113570633A
