Cell nucleus weak supervision instance segmentation method and system based on Fourier guidance
Through Fourier-guided weak supervision method and contour modeling, the problems of high labeling cost and poor generalization ability in nucleus pathological images are solved, and high-quality nucleus instance segmentation effect is achieved under a small number of labels.
Patent Information
- Application Number
- CN202411152191.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-12-13
AI Technical Summary
The prior art has problems in the example segmentation of nucleus pathological images with high data annotation cost, poor generalization ability, and inability to effectively deal with nucleus overlap. In particular, traditional methods require large amounts of annotation and high-quality data.
Adopt the weakly supervised instance segmentation method based on Fourier guidance, and train it through the fully automatic Fourier prior information boot model to reduce the annotation cost, and handle intensive segmentation problems through outline modeling to achieve close to the full with a small number of annotations. The segmentation effect of data supervision.
It greatly reduces the annotation cost of training data, improves the accuracy of nucleus segmentation and the generalization ability of the model, and can realize high-quality nucleus instance segmentation on a small number of annotated data sets to adapt to incomplete or noisy data.
Smart Images
Figure CN119131791A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cell image analysis, and more specifically, to a Fourier-guided weakly supervised instance segmentation method and system for cell nuclei. Background Art
[0002] With the successful application of deep learning in the field of computer vision, especially in the progress of semantic segmentation and instance segmentation tasks, deep learning models such as convolutional neural networks (CNN) and their variants have become the mainstream tools for processing cell images. CNN has made significant progress in the analysis, classification and diagnosis of pathological images with its excellent feature extraction ability and recognition ability of complex patterns. These networks have greatly improved the accuracy of lesion detection and classification in pathological images by automatically learning hierarchical features in images. However, most of the existing algorithms need to rely on a large amount of data for full supervision training. The fully supervised learning of deep neural networks in these methods requires a large amount of training data, and these data are annotated pixel by pixel. These data need to be accurately labeled at the instance level, that is, the boundaries and categories of each cell nucleus are labeled. This is a time-consuming and expensive task, and usually requires professional medical domain knowledge and manual annotation. It is difficult and costly to collect such data sets. In this case, these existing algorithms cannot solve the instance segmentation problem in cell nucleus pathological images well.
[0003] In addition, existing cell nucleus instance segmentation schemes are mainly divided into three categories: 1) Segmentation methods based on traditional image processing, which rely heavily on complex post-processing operations and have poor generalization; 2) Detection-based or detection-first-segmentation-based methods, which take a long time to process and are difficult to apply to the problem of cell nucleus instance segmentation with a small number of annotations; 3) Direct segmentation-based methods, which directly predict multiple types of segmentation masks and then obtain cell nucleus segmentation instances using post-processing (such as clustering). This type of method does not directly model the instance and is difficult to handle the dense segmentation problem when the cell nucleus overlap is large.
[0004] Therefore, in order to solve the above problems, the present invention proposes an algorithm that introduces Fourier prior guidance to complete weakly supervised cell nucleus instance segmentation. The algorithm uses fully automatically generated prior guidance information as a guide and integrates the prior information into the training process of the model, achieving a cell nucleus instance segmentation effect close to full data supervision with only a small amount of annotation. Summary of the invention
[0005] The present invention aims to overcome at least one defect (shortcoming) of the above-mentioned prior art, and provides a method and system for weakly supervised instance segmentation of cell nuclei based on Fourier guidance, which is used to solve the problems in the prior art of difficulty in obtaining large-scale data sets for cell nucleus instance segmentation and high annotation cost, as well as the inability of existing algorithms to solve the instance segmentation problem in cell nucleus pathology images, and achieve a cell nucleus instance segmentation effect close to full data supervision with only a small amount of annotation.
[0006] In a first aspect, the technical solution adopted by the present invention is a Fourier-guided weakly supervised instance segmentation method for nuclei, the method comprising:
[0007] S1: Input the cell nuclear pathology image into the model for feature extraction to obtain multi-level image features;
[0008] S2: Extract the 0th layer image features from the multi-level image features, and perform Fourier guided mask segmentation on them to obtain the predicted guided information segmentation mask;
[0009] S3: Segment the mask according to the predicted guidance information, update the multi-level image features, and obtain the guided enhanced image features;
[0010] S4: Extract the first layer of image features from the guided enhanced image features and input them into the cell nucleus instance segmentation head to predict each instance and obtain the final cell nucleus instance segmentation result.
[0011] In this scheme, the fully automatically generated Fourier prior guidance information is used as a guide, and the Fourier prior guidance information is integrated into the training process of the model. Weakly supervised learning is used for instance segmentation. Compared with traditional pixel-level annotation, weakly supervised learning greatly reduces the annotation cost of training data and improves training efficiency. Moreover, through instance-level annotation and combined with instance segmentation methods, more accurate cell nucleus segmentation results can be obtained, which has higher accuracy and reliability for cell analysis tasks. It can also adapt to data with incomplete or noisy annotations, making the model have stronger generalization ability, thus solving the problem of high-quality annotation required in traditional cell segmentation methods, and realizing that only a small amount of annotated cell nucleus image data sets are needed to train and obtain cell nucleus instance segmentation effects close to full data supervision.
[0012] Preferably, in the step S2, a Fourier mask segmentation head composed of deconvolution is used to perform Fourier guided mask segmentation on the cell nucleus guidance information present in the 0th layer image features to obtain a predicted guidance information segmentation mask.
[0013] Since the cell nucleus guidance information can effectively limit the segmentation area, by performing Fourier guided mask segmentation on it, the model can focus on processing the cell nucleus area of interest, thereby saving resources and time and improving the processing efficiency of the model. In addition, the introduction of the predicted guidance information segmentation mask can reduce the interference of complex background, uneven lighting and other problems that traditional image segmentation algorithms may face, thereby improving the accuracy of segmentation, so that the model only needs a small amount of annotated cell nucleus image data set to be trained to obtain cell nucleus instance segmentation effects close to full data supervision.
[0014] Preferably, in step S3, the updating of multi-level image features includes: traversing image features of all levels, first adjusting the size of the predicted guidance information segmentation mask to a size consistent with the image features of the current level, then performing a dot product of the resized predicted guidance information segmentation mask and the image features of the current level, and repeating the above operation for image features of all levels until image features of all levels are updated.
[0015] Therefore, by updating the multi-level image features, the prior information based on the cell nucleus' own characteristics is updated to the image features of all levels, and enhanced image features are obtained, thereby solving the problem of existing cell nucleus instance segmentation algorithms that need to rely on a large amount of data for fully supervised training.
[0016] Preferably, in step S4, before outputting the final cell nucleus instance segmentation result, it also includes screening the segmented cell nucleus instances in combination with the predicted guidance information segmentation mask, firstly cropping the area corresponding to the cell nucleus instance from the predicted guidance information segmentation mask, and then comparing the regional guidance mask obtained after cropping with a preset threshold value, if the maximum value of the regional guidance mask is less than the preset threshold value, then retain the cell nucleus instance, otherwise discard the cell nucleus instance, until the above-mentioned screening operation traverses all cell nucleus instances, and the retained cell nucleus instance is used as the final cell nucleus instance segmentation result.
[0017] Therefore, the predicted guidance information segmentation mask can be used to screen out cell nucleus instances with high accuracy, further improving the accuracy of model detection.
[0018] Further preferably, in step S4, the prediction of each instance includes contour coordinate regression prediction and category prediction, and the contour coordinate regression prediction includes: sampling 128 coordinate points from the edge contour of each cell nucleus, and using the coordinates to represent each cell nucleus instance.
[0019] In this scheme, the use of contours for modeling can effectively enable the model to learn the explicit expression information of the cell nucleus instance, so as to handle the instance segmentation of overlapping cell nuclei in dense situations and improve the accuracy of the model in performing cell nucleus instance segmentation.
[0020] Preferably, the preset threshold value range is any floating point number between 0 and 1. The threshold value can be set according to the actual situation, and by comparing the set threshold value with the regional guide mask, the cell nucleus instances that meet the conditions can be screened out, ensuring that the final cell nucleus instance segmentation result has a high accuracy rate.
[0021] Preferably, in step S1, the model includes a feature extraction backbone network composed of several residual modules connected end to end, the residual modules are divided into a main path and a branch path, the main path features are output through at least two convolutional layers in the main path, and the branch features are obtained by performing dimensionality reduction processing through one convolutional layer in the branch, the main path features and the branch features have the same dimension, and finally the main path features and the branch features are directly added to extract multi-level image features.
[0022] In the residual module, by introducing branches to jump connections, the network layer can learn residual mapping, and can also enable the network to extract complex features in the input data more deeply and efficiently, thereby improving the network's expressiveness and generalization capabilities, and can also enable gradients to propagate faster, accelerate the training process, and reduce training time.
[0023] On the other hand, the present invention also provides a Fourier-guided weakly supervised instance segmentation system for cell nuclei, the system comprising:
[0024] Feature extraction module: input the cell nuclear pathology image into the model for feature extraction to obtain multi-level image features;
[0025] Guided mask segmentation module: extracts the 0th layer image features from the multi-level image features extracted by the feature extraction module, and performs Fourier guided mask segmentation on the 0th layer image features to obtain a predicted guided information segmentation mask;
[0026] Image feature update module: updates the multi-level image features according to the predicted guidance information segmentation mask to obtain the guided enhanced image features;
[0027] Instance segmentation module: extract the first layer image features from the guided enhanced image features, and input the first layer image features into the cell nucleus instance segmentation head to predict each instance, so as to obtain the final cell nucleus instance segmentation result.
[0028] In this system, the Fourier prior guidance information is integrated into the training process of the model by setting up a guided mask segmentation module, thereby guiding the model to perform instance segmentation of the cell nucleus based on the prior information of the cell nucleus's own characteristics, realizing the ability to perform instance segmentation of the cell nucleus with a small amount of annotations and with a certain degree of generalization.
[0029] Preferably, the instance segmentation module also includes a cell nucleus instance screening unit, in which the segmented cell nucleus instances are screened using the predicted guidance information segmentation mask. First, the area corresponding to the cell nucleus instance is cropped from the predicted guidance information segmentation mask, and then the regional guidance mask obtained after cropping is compared with a preset threshold. If the maximum value of the regional guidance mask is less than the preset threshold, the cell nucleus instance is retained, otherwise the cell nucleus instance is discarded until the above screening operation has traversed all cell nucleus instances.
[0030] By setting a cell nucleus instance screening unit to further screen the segmented cell nucleus instances, it is ensured that the final output cell nucleus instance segmentation result has a high accuracy.
[0031] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned Fourier-guided weakly supervised instance segmentation method for cell nuclei when executing the program.
[0032] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the Fourier-guided weakly supervised instance segmentation method for cell nuclei as described above is implemented.
[0033] The present invention also provides a computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, the Fourier-guided weakly supervised instance segmentation method for cell nuclei as described above is implemented.
[0034] Compared with the prior art, the present invention has the following beneficial effects:
[0035] 1. Guided by Fourier prior, the characteristics of cell nucleus images are injected into the training process of deep learning models in the form of prior information, and weakly supervised learning is used for instance segmentation, which greatly reduces the labeling cost of training data and improves training efficiency. It solves the problem that traditional cell segmentation methods require high-quality labeling, and realizes that only a small amount of labeled cell nucleus image data sets are needed to train and obtain cell nucleus instance segmentation effects close to full data supervision.
[0036] 2. Using contours for modeling can help the model learn explicit expression information of cell nucleus instances, so that it can handle instance segmentation of overlapping cell nuclei in dense situations. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 The present invention provides a flow chart of the method.
[0038] Figure 2 Detailed flow chart of the algorithm provided by the present invention.
[0039] Figure 3 This is a schematic diagram of the residual module structure provided by the present invention.
[0040] Figure 4 Schematic diagram of the Fourier-guided mask segmentation process provided by the present invention.
[0041] Figure 5 This is a schematic diagram of the cell nucleus contour modeling provided by the present invention.
[0042] Figure 6 This is a schematic diagram of the system structure provided by the present invention.
[0043] Figure 7 A schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0044] The drawings of the present invention are only for illustrative purposes and should not be construed as limiting the present invention. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.
[0045] Example 1
[0046] like Figure 1 and Figure 2 As shown, Figure 1 The flowchart of this method is shown in FIG. Figure 2 Detailed flowchart of the algorithm of this method. In this embodiment, a method for weakly supervised instance segmentation of cell nuclei based on Fourier guidance is provided, and the method includes:
[0047] Step S1: Input the cell nuclear pathology image into the model for feature extraction to obtain multi-level image features;
[0048] In this embodiment, the ResNet CNN deep learning model is used as the feature extraction backbone network. Figure 3 As shown, Figure 3A schematic diagram of the residual module structure provided for this embodiment, in the step S1, the feature extraction backbone network is composed of a number of residual modules connected end to end, and the residual modules are divided into a main path and a branch path, when the input feature is of a dimension of 56×56×64, it passes through 128 3×3 convolutional layers with a stride of 2 on the main path, and the feature dimension at this time is 28×28×128, and then passes through a relu activation function, 128 3×3 convolutional layers with a stride of 1, to obtain the main path feature; the branch directly inputs the input 56×56×64 feature dimension into 128 1×1 convolutional layers with a stride of 2 to obtain the branch feature, and the dimension of the branch feature is also 28×28×128, the main path feature and the branch feature are directly added to obtain the output feature of the current residual module, and finally each residual module is connected end to end to extract the final multi-level image feature, wherein the higher the level, the smaller the resolution, and the higher the level of feature information contained.
[0049] In the residual module described in this embodiment, branches are introduced to perform jump connections so that the network layer can learn residual mapping, and the network model can extract complex features in the input data more deeply and efficiently, thereby improving the network's expressiveness and generalization capabilities, and enabling the gradient to propagate faster, accelerating the training process and reducing training time.
[0050] Preferably, step S2 is: extracting the 0th layer image features from the multi-level image features, and performing Fourier guided mask segmentation on the features to obtain a predicted guided information segmentation mask;
[0051] like Figure 4 As shown, Figure 4 A schematic diagram of the Fourier guided mask segmentation process provided for this embodiment. In step S2 of this embodiment, for the features extracted by the backbone network, a Fourier mask segmentation head composed of deconvolution is used in this embodiment to perform Fourier guided mask segmentation on the cell nucleus guidance information present in the 0th layer image features to obtain a predicted guidance information segmentation mask. The input feature dimension of the Fourier mask segmentation head constructed in this embodiment is 192×192×64, and the feature aggregation is performed through 64 7×7 convolution layers with a step of 1, and then a relu activation function and a 1×1 convolution layer with a step of 1 are exchanged to obtain a feature with a dimension of 192×192×1, and finally a predicted guidance information segmentation mask with a dimension of 384×384×1 is obtained through an upsampling using simple linear interpolation. Wherein, the predicted guidance information segmentation mask is a floating point number with each pixel value between 0 and 1, and its resolution is increased by a simple linear interpolation upsampling process to increase the size of the image so that it is consistent with the input cell nucleus pathology image.
[0052] Since the cell nucleus guidance information can effectively limit the segmentation area, by performing Fourier guided mask segmentation on it, the model can focus on processing the cell nucleus area of interest, thereby saving resources and time and improving the processing efficiency of the model. In addition, the introduction of the predicted guidance information segmentation mask can reduce the interference of complex background, uneven lighting and other problems that traditional image segmentation algorithms may face, thereby improving the accuracy of segmentation, so that the model only needs a small amount of annotated cell nucleus image data set to be trained to obtain cell nucleus instance segmentation effects close to full data supervision.
[0053] Preferably, in the present embodiment, step S3 is: updating the multi-level image features according to the predicted guidance information segmentation mask to obtain the guided enhanced image features;
[0054] In step S3, the updating of the multi-level image features includes: traversing the image features of all levels, first adjusting the size of the predicted guidance information segmentation mask to the same size as the image features of the current level, then performing a dot multiplication of the resized predicted guidance information segmentation mask and the image features of the current level, and repeating the above operation for the image features of all levels until the image features of all levels are updated. Figure 1 and Figure 2 As shown in the figure, taking the 0th layer image features and the 1st layer image features as examples, the predicted guidance information segmentation mask with a dimension of 384×384×1 is resized to obtain the resized guidance information segmentation mask with a dimension of 96×96×1, and then multiplied with the 1st layer image features with a dimension of 96×96×128, and finally the guided enhanced image features with a dimension of 96×96×128 are obtained. The next step is performed after the image features of all levels are updated.
[0055] Therefore, by updating the multi-level image features, the prior information based on the cell nucleus' own characteristics is updated to the image features of all levels to obtain enhanced image features, thereby solving the problem that the existing cell nucleus instance segmentation algorithm needs to rely on a large amount of data for full supervision training. A small amount of annotated cell nucleus image datasets can be used to train and obtain a cell nucleus instance segmentation effect close to full data supervision.
[0056] Preferably, in the present embodiment, step S4 is: extracting the first layer of image features from the image features after guided enhancement, and inputting them into the cell nucleus instance segmentation head to perform prediction on each instance, so as to obtain the final cell nucleus instance segmentation result.
[0057] More preferably, if Figure 2As shown, in the step S4, before outputting the final cell nucleus instance segmentation result, the segmented cell nucleus instance is also screened in combination with the predicted guidance information segmentation mask. First, the area corresponding to the cell nucleus instance is cropped from the predicted guidance information segmentation mask, and then the regional guidance mask obtained after cropping is compared with the preset threshold. In this embodiment, the threshold is set to 0.95. If the maximum value of the regional guidance mask is less than the preset threshold of 0.95, the cell nucleus instance is retained, otherwise the cell nucleus instance is discarded until the above screening operation traverses all cell nucleus instances, and the retained cell nucleus instance is used as the final cell nucleus instance segmentation result.
[0058] Therefore, by using the predicted guidance information segmentation mask, it is possible to screen out cell nucleus instances with high accuracy, further improving the accuracy of model detection. In addition, the design of the guided mask also provides pathologists with a model visualization of the mask area of interest, which has a certain degree of interpretability.
[0059] More preferably, in step S4, the prediction of each instance includes contour coordinate regression prediction and category prediction, and the contour coordinate regression prediction includes: sampling 128 coordinate points from the edge contour of each cell nucleus, and using the coordinates to represent each cell nucleus instance, such as Figure 5 As shown, Figure 5 The schematic diagram of cell nucleus contour modeling provided for this embodiment uses contour modeling to enable the model to learn explicit expression information of cell nucleus instances, so as to handle instance segmentation of overlapping cell nuclei in dense situations and improve the accuracy of the model in performing cell nucleus instance segmentation.
[0060] Preferably, in this embodiment, the preset threshold range is any floating point number between 0 and 1. The threshold can be set according to actual conditions, and by comparing the set threshold with the regional guide mask, the cell nucleus instances that meet the conditions can be screened out, ensuring that the final cell nucleus instance segmentation result has a high accuracy rate.
[0061] In addition, this solution also compares three classic algorithms based on detection-first-segmentation, regression-based, and segmentation-based solutions, namely Mask R-CNN, CPN, and TopoSeg, and uses the classic evaluation indicators PQ and AJI of cell nucleus instance segmentation to evaluate the test results. The test results are shown in Table 1. It can be seen from Table 1 that the Fourier-guided weakly supervised instance segmentation method for cell nuclei provided in this embodiment has better performance.
[0062]
[0063] Table 1
[0064] In summary, in this embodiment, the fully automatically generated Fourier prior guidance information is used as a guide, the Fourier prior guidance information is integrated into the training process of the model, and weakly supervised learning is used for instance segmentation. Compared with traditional pixel-level annotation, weakly supervised learning greatly reduces the annotation cost of training data and improves training efficiency. Moreover, through instance-level annotation and combined with instance segmentation methods, more accurate cell nucleus segmentation results can be obtained, which has higher accuracy and reliability for cell analysis tasks, and can also adapt to data with incomplete or noisy annotations, so that the model has stronger generalization ability, thereby solving the problem of high-quality annotation required in traditional cell segmentation methods, and realizing that only a small amount of annotated cell nucleus image data sets are needed to train to obtain cell nucleus instance segmentation effects close to full data supervision.
[0065] Example 2
[0066] like Figure 6 As shown, Figure 6 This is a schematic diagram of the system structure provided in this embodiment. In this embodiment, a Fourier-guided weakly supervised instance segmentation system for cell nuclei is provided, and the system includes:
[0067] Feature extraction module: The cell nuclear pathology image is input into the model for feature extraction to obtain multi-level image features; the feature extraction module includes the ResNet CNN deep learning model as the feature extraction backbone network, and the introduction of branches in the backbone network for jump connection allows the network layer to learn residual mapping, and also enables the network model to extract complex features in the input data more deeply and efficiently, thereby improving the network's expressiveness and generalization capabilities, and also enables the gradient to propagate faster, accelerate the training process, and reduce training time.
[0068] Guided mask segmentation module: extracts the 0th layer image features from the multi-level image features extracted by the feature extraction module, and performs Fourier guided mask segmentation on the 0th layer image features to obtain a predicted guided information segmentation mask;
[0069] In this module, the Fourier mask segmentation head composed of deconvolution is used to perform Fourier guided mask segmentation on the cell nucleus guidance information present in the 0th layer image features to obtain the predicted guidance information segmentation mask, so that the model can focus on processing the cell nucleus area of interest, thereby saving resources and time and improving the processing efficiency of the model. In addition, the introduction of the predicted guidance information segmentation mask can reduce the interference of complex background, uneven lighting and other problems that traditional image segmentation algorithms may face, thereby improving the accuracy of segmentation, so that the model only needs a small amount of annotated cell nucleus image data sets to be trained to obtain a cell nucleus instance segmentation effect close to full data supervision.
[0070] Image feature update module: According to the predicted guidance information segmentation mask, the multi-level image features are updated to obtain the guided enhanced image features; after introducing Fourier prior information, the image feature update module is used to update it to all levels of images, so that all levels of images can learn the key feature information, thereby solving the problem of existing cell nucleus instance segmentation algorithms that need to rely on a large amount of data for fully supervised training.
[0071] Instance segmentation module: extract the first layer image features from the guided enhanced image features, and input the first layer image features into the cell nucleus instance segmentation head to predict each instance, so as to obtain the final cell nucleus instance segmentation result.
[0072] In the system described in the embodiment, the Fourier prior guidance information is integrated into the training process of the model by setting a guided mask segmentation module, thereby guiding the model to perform instance segmentation of the cell nucleus based on the prior information of the cell nucleus' own characteristics, thereby achieving the ability to perform instance segmentation of the cell nucleus with a small amount of annotation and having a certain degree of generalization.
[0073] Preferably, the instance segmentation module also includes a cell nucleus instance screening unit, in which the segmented cell nucleus instances are screened using the predicted guidance information segmentation mask. First, the area corresponding to the cell nucleus instance is cropped from the predicted guidance information segmentation mask, and then the regional guidance mask obtained after cropping is compared with a preset threshold. In this embodiment, the threshold of the cell nucleus instance screening unit is set to 0.95. If the maximum value of the regional guidance mask is less than the preset threshold of 0.95, the cell nucleus instance is retained, otherwise the cell nucleus instance is discarded until the above screening operation traverses all cell nucleus instances, and the retained cell nucleus instance is used as the final cell nucleus instance segmentation result.
[0074] Therefore, in this embodiment, a cell nucleus instance screening unit is provided to further screen the segmented cell nucleus instances, thereby ensuring that the cell nucleus instance segmentation result finally outputted has a high accuracy.
[0075] Example 3
[0076] In this embodiment, Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the Fourier-guided weakly supervised instance segmentation method for the cell nucleus, which includes: S1: inputting the cell nucleus pathology image into the model for feature extraction to obtain multi-level image features; S2: extracting the 0th layer image features from the multi-level image features, and performing Fourier-guided mask segmentation on them to obtain a predicted guided information segmentation mask; S3: updating the multi-level image features according to the predicted guided information segmentation mask to obtain guided enhanced image features; S4: extracting the 1st layer image features from the guided enhanced image features, and inputting them into the cell nucleus instance segmentation head to perform prediction of each instance to obtain the final cell nucleus instance segmentation result.
[0077] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0078] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned Fourier-guided weakly supervised instance segmentation method for cell nuclei when executing the program.
[0079] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the Fourier-guided weakly supervised instance segmentation method for cell nuclei as described above is implemented.
[0080] The present invention also provides a computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, the Fourier-guided weakly supervised instance segmentation method for cell nuclei as described above is implemented.
[0081] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.
Claims
1. A Fourier-guided weakly supervised instance segmentation method for nuclei, characterized in that: The method comprises: S1: Input the cell nuclear pathology image into the model for feature extraction to obtain multi-level image features; S2: Extract the 0th layer image features from the multi-level image features, and perform Fourier guided mask segmentation on them to obtain the predicted guided information segmentation mask; S3: Segment the mask according to the predicted guidance information, update the multi-level image features, and obtain the guided enhanced image features; S4: Extract the first layer of image features from the guided enhanced image features and input them into the cell nucleus instance segmentation head to predict each instance and obtain the final cell nucleus instance segmentation result.
2. The method for cell nucleus weakly supervised instance segmentation based on Fourier guidance according to claim 1, characterized in that: In the step S2, a Fourier mask segmentation head composed of deconvolution is used to perform Fourier guided mask segmentation on the cell nucleus guidance information present in the 0th layer image features to obtain a predicted guidance information segmentation mask.
3. The method for cell nucleus weakly supervised instance segmentation based on Fourier guidance according to claim 1, characterized in that: In step S3, the updating of multi-level image features includes: traversing the image features of all levels, first adjusting the size of the predicted guidance information segmentation mask to a size consistent with the image features of the current level, then performing a dot product between the resized predicted guidance information segmentation mask and the image features of the current level, and repeating the above operation for the image features of all levels until the image features of all levels are updated.
4. The method for cell nucleus weakly supervised instance segmentation based on Fourier guidance according to claim 1, characterized in that: In the step S4, before outputting the final cell nucleus instance segmentation result, it also includes screening the segmented cell nucleus instances in combination with the predicted guidance information segmentation mask. First, the area corresponding to the cell nucleus instance is cropped from the predicted guidance information segmentation mask, and then the regional guidance mask obtained after cropping is compared with the preset threshold. If the maximum value of the regional guidance mask is less than the preset threshold, the cell nucleus instance is retained, otherwise the cell nucleus instance is discarded, until the above screening operation traverses all cell nucleus instances, and the retained cell nucleus instance is used as the final cell nucleus instance segmentation result.
5. The method for Fourier-guided weakly supervised instance segmentation of cell nuclei according to claim 4, characterized in that: In step S4, the prediction of each instance includes contour coordinate regression prediction and category prediction. The contour coordinate regression prediction includes: sampling 128 coordinate points from the edge contour of each cell nucleus, and using the coordinates to represent each cell nucleus instance.
6. The method for Fourier-guided weakly supervised instance segmentation of cell nuclei according to claim 5, characterized in that: The preset threshold value range is any floating point number between 0 and 1.
7. A Fourier-guided weakly supervised instance segmentation method for nuclei according to any one of claims 1 to 6, characterized in that: In step S1, the model includes a feature extraction backbone network composed of several residual modules connected end to end, the residual modules are divided into a main path and a branch path, the main path features are output through at least two convolutional layers in the main path, and the branch features are obtained by performing dimensionality reduction processing through one convolutional layer in the branch. The dimensions of the main path features and the branch features are the same. Finally, the main path features and the branch features are directly added to extract multi-level image features.
8. A Fourier-guided weakly supervised instance segmentation system for cell nuclei, characterized in that: The system comprises: Feature extraction module: input the cell nuclear pathology image into the model for feature extraction to obtain multi-level image features; Guided mask segmentation module: extracts the 0th layer image features from the multi-level image features extracted by the feature extraction module, and performs Fourier guided mask segmentation on the 0th layer image features to obtain a predicted guided information segmentation mask; Image feature update module: updates the multi-level image features according to the predicted guidance information segmentation mask to obtain the guided enhanced image features; Instance segmentation module: extract the first layer image features from the guided enhanced image features, and input the first layer image features into the cell nucleus instance segmentation head to predict each instance, so as to obtain the final cell nucleus instance segmentation result.
9. The Fourier-guided weakly supervised instance segmentation system for nuclei according to claim 8, characterized in that: The instance segmentation module also includes a cell nucleus instance screening unit. In the cell nucleus instance screening unit, the segmented cell nucleus instances are screened using the predicted guidance information segmentation mask. First, the area corresponding to the cell nucleus instance is cropped from the predicted guidance information segmentation mask, and then the regional guidance mask obtained after cropping is compared with a preset threshold. If the maximum value of the regional guidance mask is less than the preset threshold, the cell nucleus instance is retained, otherwise the cell nucleus instance is discarded until the above screening operation traverses all cell nucleus instances.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the Fourier-guided weakly supervised instance segmentation method for cell nuclei is implemented as described in any one of claims 1 to 7.