Nuclei instance segmentation method and system with cross-graph context awareness

Abstract

The present application relates to the technical field of image processing, and discloses a cross-image context-aware nucleus instance segmentation method and system, comprising: acquiring a cell image and segmenting it into a slice group, and constructing a nucleus instance segmentation model comprising a feature extraction module, a context injection module, a prompt encoding module, a mask decoder and a texture encoder; the feature extraction module extracts prior features and domain features of the slice, refines the prior features through a multi-level feature refinement block, extracts and fuses missing features from the domain features to obtain enhanced features; the context injection module generates image features by integrating context information according to the enhanced features, the prompt encoding module generates point embedding, the mask decoder obtains a mask of each nucleus according to the image features and the point embedding, and the instance segmentation result is obtained by combining the masks; the texture encoder encodes the enhanced features and the instance segmentation result into texture features, which are stored into the context injection module together with the enhanced features. The present application can improve the instance segmentation effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a cross-image context-aware cell nucleus instance segmentation method and system. BACKGROUND

[0002] Cell nucleus instance segmentation is a basic step for extracting meaningful biological information, and the recognition of the relative topology, size and shape of cell nuclei is crucial for downstream tasks such as pathological image diagnosis and biological cell research. Therefore, cell nucleus instance segmentation has been studied more and more. Cell nucleus instance segmentation is usually based on digitized cell images, such as Whole Slide Image (WSI), and digitized cell images have extremely high resolution and contain a large number of cell nuclei, making instance segmentation difficult.

[0003] In the prior art, there are methods for cell nucleus instance segmentation using deep learning technology, such as HoVer-Net, U-Net, nn-UNet, etc. These models use an encoder to extract features from an image and a decoder to process these features into the final segmentation result. However, these models extract features from the current input image and cannot consider context information (including cell nucleus spatial distribution and tissue structure) from the surrounding image, and relying only on local features in the input image cannot achieve the best segmentation result.

[0004] SAM 2 is a basic model for video segmentation, containing a stream memory for real-time video processing and instant propagation, and showing excellent performance and generalization ability on large-scale datasets (such as Segment Anything Video, SA-V). Therefore, SAM 2 has application potential in instance segmentation. However, there are some problems when applying SAM 2 to cell images. First, the dataset that SAM 2 faces is not a cell image, and it cannot extract specific domain features that exist in it. Directly using SAM 2 will reduce the segmentation performance. Second, although SAM 2 supports instant propagation and can segment the entire video using only a hint from the first frame, the distribution of cell nuclei in cell images is relatively dense, and it is difficult to capture all cell nucleus instances through instant propagation, which will affect the segmentation performance and cause missed detection. SUMMARY

[0005] To this end, the technical problem to be solved by the present application is to overcome the deficiencies in the prior art and provide a cross-image context-aware cell nucleus instance segmentation method and system that can effectively extract domain features, integrate historical context information, and improve the effect of cell nucleus instance segmentation.

[0006] To solve the above technical problems, the present application provides a cross-graph context-aware cell nucleus instance segmentation method, comprising:

[0007] acquiring a cell image and scanning and segmenting into a slice group comprising multiple slices, constructing a cell nucleus instance segmentation model, the cell nucleus instance segmentation model comprising a feature extraction module, a context injection module, a prompt encoding module, a mask decoder and a texture encoder;

[0008] The feature extraction module extracts prior features and domain features of the slice through a multi-layer encoder, refines the prior features of each layer through a multi-level feature refinement block, extracts and fuses missing features from the domain features of each layer, and obtains enhanced features;

[0009] The context injection module generates image features according to the enhanced features and context information from the surrounding segmented slices, the prompt encoding module generates point embeddings of each cell nucleus according to the domain features, and the mask decoder obtains the mask of each cell nucleus according to the image features and the point embeddings;

[0010] Combining the mask of each cell nucleus obtains the instance segmentation result of the slice, and combining the instance segmentation results of all slices obtains the instance segmentation result of the cell image;

[0011] The instance segmentation result of the slice and the enhanced features are input into the texture encoder to generate texture features, and the texture features and the enhanced features are stored in the context injection module to provide context information for the next slice.

[0012] Further, the feature extraction module comprises a CNN encoder and a ViT encoder, the CNN encoder is a multi-layer convolutional neural network encoder, the ViT encoder comprises a Patch Embedded module and a plurality of feature refinement layers, the number of layers of the feature refinement layer is the same as the number of layers of the convolutional neural network encoder, each layer of the feature refinement layer comprises a plurality of feature refinement blocks, and each feature refinement block comprises a multi-level feature refinement block and a ViT block.

[0013] Further, the feature extraction module extracts prior features and domain features of the slice through a multi-layer encoder, refines the prior features of each layer through a multi-level feature refinement block, extracts and fuses missing features from the domain features of each layer, and obtains enhanced features, specifically:

[0014] When each slice is input into the feature extraction module, the i-th layer of the CNN encoder outputs the i-th layer of the domain features;

[0015] The ViT encoder first extracts the first layer prior feature through the Patch Embedded module, fuses the i-th layer prior feature and the i-th layer domain feature through the i-th layer feature refinement layer to obtain the i+1-th layer prior feature, and takes the output of the last feature refinement layer as the enhanced feature.

[0016] Further, the calculation method of the i+1-th layer prior feature is:

[0017] ,

[0018] , is the i+1-th layer prior feature, ViT( ) is the operation of the ViT block, is the i-th layer prior feature, MFRB( ) is the operation of the multi-level feature refinement block, [ ] n indicates the operation of n times of ViT(MFRB( )), n is the number of feature refinement blocks in the feature refinement layer;

[0019] In the i-th feature refinement layer, the feature input into the j-th feature refinement block is denoted as , j = 1, 2, …, n, , that is, , the feature obtained after the multi-level feature refinement block in the j-th feature refinement block is denoted as , and the operation of the multi-level feature refinement block is specifically:

[0020] ,

[0021] , is the cross-attention operation, LN( ) is the layer normalization operation, is the i-th layer domain feature, is the refined domain feature, , The result obtained after the operation of the ViT block is .

[0022] Further, the calculation method of the is:

[0023] ,

[0024] , wherein Adapter( ) is the Adapter fine-tuning operation.

[0025] Further, the context injection module generates the image feature according to the enhanced feature and the context information from the surrounding segmented slices, specifically:

[0026] The context injection module includes an environment memory and a texture memory. For the i-th slice of the current input, the environment memory stores the enhanced features corresponding to the previous i-1 slices, denoted as... Let j = 1, 2, ..., i-1; and let the enhancement feature corresponding to the i-th slice be . ,when When inputting a context injection module, select the closest one from the environment memory. The M1 enhanced features are used to construct the shortest distance memory bank, denoted as M1. ;application Injecting information about the surrounding environment The features containing environmental information are obtained and denoted as ,Will Input texture memory;

[0027] The texture memory stores the texture features of Mt slices that have undergone instance segmentation and have the greatest similarity difference. The texture features from the texture memory are selected... The M2 closest texture features are used to construct a maximum similarity memory, denoted as M2. ;application Inject foreground information Obtain image features, denoted as .

[0028] Furthermore, the shortest distance memory bank is: , For the closest The k-th feature among the M1 enhanced features;

[0029] The The calculation method is as follows: ,in, For memory and attention operations;

[0030] The maximum similarity memory is: , For the closest The q-th feature among the M2 texture features;

[0031] The The calculation method is as follows: .

[0032] Furthermore, when the texture memory stores the texture features of the Mt slices that have completed instance segmentation and have the greatest similarity difference, the storage method is as follows:

[0033] Let the texture feature corresponding to the current i-th slice be denoted as ,Will The confidence level is denoted as ;

[0034] Let the texture features of the divided slices of the Mt completed instances stored in the texture memory be denoted as , j = 1, 2, …, Mt; calculate The sum of the similarity between any one of the texture features and all the remaining texture features, select the one corresponding to the maximum sum, denoted as , the confidence of is denoted as ;

[0035] If is greater than , delete from the texture memory and store , otherwise the texture memory remains unchanged.

[0036] Further, the prompt encoding module generates a point embedding of each nucleus according to the domain feature, specifically:

[0037] The prompt encoding module includes a prediction head and a prompt encoder, and the prediction head includes a regression head and a classification head.

[0038] The prediction head predicts the point of each nucleus according to the domain feature, and the prompt encoder encodes the point into a point embedding.

[0039] The application also provides a cross-graph context-aware nucleus instance segmentation system, comprising:

[0040] An image acquisition module acquires a cell image and scans and divides the cell image into a slice group comprising multiple slices;

[0041] A model construction module constructs a nucleus instance segmentation model, wherein the nucleus instance segmentation model comprises a feature extraction module, a context injection module, a prompt encoding module, a mask decoder, and a texture encoder; the feature extraction module extracts prior features and domain features of the slices through a multi-layer encoder, refines the prior features of each layer through a multi-level feature refinement block, extracts and fuses missing features from the domain features of each layer to obtain enhanced features; the context injection module generates image features according to the enhanced features and integrates context information from surrounding segmented slices to generate image features, the prompt encoding module generates a point embedding of each nucleus according to the domain feature, and the mask decoder obtains a mask of each nucleus according to the image features and the point embedding;

[0042] An instance segmentation module combines the masks of each nucleus to obtain an instance segmentation result of the slice, combines the instance segmentation results of all the slices to obtain an instance segmentation result of the cell image; the texture encoder is input with the instance segmentation result of the slice and the enhanced features to generate texture features, and the texture features and the enhanced features are stored in the context injection module to provide context information for the next slice.

[0043] The above technical solutions of the present application have the following beneficial effects compared with the prior art:

[0044] The present application extracts the prior features and field features of the cell image through the feature extraction module, and refines the prior features and integrates the field features using the multi-level feature refinement block, thereby realizing effective extraction of the field features. On this basis, the historical context information around each cell nucleus in the cell image is integrated through the context injection module, so that the model is more suitable for the cell image, and the effect of cell nucleus instance segmentation is improved. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to make the content of the present application more easily understood, the present application will be further described in detail below according to specific embodiments of the present application and in conjunction with the drawings, in which:

[0046] Figure 1 The flowchart of the method in the preferred embodiment of the present application.

[0047] Figure 2 The structure diagram of the cell nucleus instance segmentation model in the preferred embodiment of the present application.

[0048] Figure 3 The structure diagram of the feature refinement layer in the preferred embodiment of the present application.

[0049] Figure 4 The structure diagram of the multi-level feature refinement block in the preferred embodiment of the present application.

[0050] Figure 5 The structure diagram of the context injection module in the preferred embodiment of the present application. DETAILED DESCRIPTION

[0051] The present application will be further described below in conjunction with the drawings and specific embodiments, so that those skilled in the art can better understand the present application and implement it, but the embodiments are not limiting to the present application.

[0052] Referring to Figure 1 The present application discloses a cross-graph context-aware cell nucleus instance segmentation method, comprising the following steps:

[0053] S1: Obtain a cell image and scan and segment it into a slice group comprising a plurality of slices, the entire cell image obtained by scanning is regarded as a video, and each slice in the slice group can be regarded as a video frame. The cell image is taken as an input image, denoted as I, in this embodiment, a sliding window is used to scan I from the center in a counterclockwise direction, to generate a slice group, denoted as P, P={P1,…,P i ,…,P N}P idenotes the slice of the i-th input, and N denotes the number of slices in the slice group.

[0054] S2: Construct a cell nucleus instance segmentation model as shown in Figure 2 The cell nucleus instance segmentation model in the application is an improvement based on the existing SAM 2 model, and the cell nucleus instance segmentation model includes a feature extraction module, a context injection module, a prompt encoding module, a mask decoder, and a texture encoder.

[0055] S3: The feature extraction module extracts prior features and domain features of the slice through a multi-layer encoder, refines the prior features of each layer through a multi-level feature refinement block (MFRB), extracts and fuses missing features from the domain features of each layer, and obtains enhanced features, denoted as ; thereby effectively extracting specific domain features in the slice.

[0056] The feature extraction module includes a CNN encoder and a ViT encoder. The CNN encoder is a multi-layer convolutional neural network encoder, and the CNN encoder in the embodiment uses a common ConvNeXt encoder with a 4-layer structure. The ViT encoder is based on Hiera and includes a Patch Embedded module and a multi-layer feature refinement layer. The number of layers of the feature refinement layer is the same as the number of layers of the convolutional neural network encoder, and the feature refinement layer also has 4 layers in the embodiment. Each layer of the feature refinement layer includes a plurality of feature refinement blocks, as shown in Figure 3 Each feature refinement block includes a multi-level feature refinement block and a ViT block, and the multi-level feature refinement block is used to facilitate the flow of information between the CNN encoder and the ViT encoder (i.e., the ViT branch and the CNN branch). In the embodiment, the 4-layer feature refinement layer has K1, K2, K3, and K4 feature refinement blocks, respectively. The values of K1, K2, K3, and K4 are adjusted according to actual conditions. The Patch Embedded module is a Patch Embedded module in the ViT model, and the ViT block is a single-layer Transformer Encoder in the ViT model.

[0057] When each slice is input into the feature extraction module, the i-th layer of the CNN encoder outputs the i-th layer domain feature, denoted as , i=L1, L2, L3, L4, respectively representing the 1st layer, the 2nd layer, the 3rd layer, and the 4th layer.

[0058] The ViT encoder first extracts the 1st layer prior feature through the Patch Embedded module, denoted as , and fuses the i-th layer prior feature and the i-th layer domain feature through the i-th layer feature refinement layer to obtain the i+1 layer prior feature. The output of the last layer feature refinement layer is used as the enhanced feature, i.e. .

[0059] The calculation method of the (i+1)th layer prior feature is:

[0060] ,

[0061] Wherein, is the (i+1)th layer prior feature, ViT() is the operation of the ViT block, is the ith layer prior feature, MFRB() is the operation of the multi-level feature refinement block, [ ] n represents the operation of n times of ViT(MFRB()), and n is the number of feature refinement blocks in the feature refinement layer. In this embodiment, the 4-layer feature refinement layer performs K1, K2, K3 and K4 times of ViT(MFRB()) operations respectively;

[0062] In the ith feature refinement layer, the feature input into the jth feature refinement block is denoted as , j=1,2…,n, is , as shown in Figure 4 , the operation of the multi-level feature refinement block is specifically:

[0063] ,

[0064] Wherein, is the cross-attention operation, LN() is the layer normalization operation, is the , After the operation of the ViT block, the result is ;

[0065] The calculation method of the (i+1)th layer prior feature is:

[0066] ,

[0067] Wherein, Adapter() is the Adapter fine-tuning operation, which is used for initial fine-tuning to include domain knowledge.

[0068] S4: The context injection module generates image features according to the context information from the surrounding segmented slices based on the enhanced features, denoted as .

[0069] To help the model acquire contextual information from surrounding slices when segmenting the current slice, this invention designs a Context Injection Module (CIM). The CIM extends the memory module of SAM2 (the memory bank and memory attention in SAM2) and can combine contextual information from surrounding slices during model prediction, thereby achieving more accurate mask prediction. Figure 5 As shown, the context injection module includes an environment memory (EMB) and a texture memory (TMB). The environment memory is used to supplement the surrounding tissue microenvironment information (i.e., background information), and the texture memory provides additional foreground information (texture, size, etc. of cell nuclei). Therefore, CIM can effectively extract and utilize the surrounding context information.

[0070] For the i-th slice of the current input, the environment memory stores the enhanced features corresponding to the previous i-1 slices, denoted as . Let j = 1, 2, ..., i-1; and let the enhancement feature corresponding to the i-th slice be . ,when When inputting a context injection module, select the closest one from the environment memory. The M1 enhanced features are used to construct the shortest distance memory bank, denoted as M1. , , For the closest The k-th feature among the M1 enhanced features; in this embodiment, the closest one is selected. When calculating the M1 enhanced features, Euclidean distance is used to determine the distance, i.e., calculating... and The Euclidean distance between them is denoted as ,pass The value determines distance. Applications Injecting information about the surrounding environment The features containing environmental information are obtained and denoted as ,Will Input texture memory. The calculation method is as follows: ,in, ( ) represents the memory attention operation. Since the environmental memory bank is used to store and provide augmenting features from the same image, it is cleared whenever the input image is changed.

[0071] In order to better provide and The texture memory stores texture features from Mt slices that have undergone instance segmentation and have the greatest similarity difference, based on features similar to cell nuclei in the texture memory. The texture memory is then used to select texture features from the texture memory. The M2 closest texture features are used to construct a maximum similarity memory, denoted as M2. , , For the closest The q-th feature among the M2 texture features; in this embodiment, the closest one is selected. Cosine similarity is used to determine proximity when dealing with M2 texture features. Applications Inject foreground information Obtain image features, denoted as , The calculation method is as follows: .

[0072] When storing the texture features of Mt slices that have undergone instance segmentation with the greatest similarity difference in the texture memory, the storage method is as follows: The texture feature corresponding to the current i-th slice is denoted as... ,Will The confidence level is denoted as The texture features of the Mt slices that have been segmented into instances and stored in the texture memory are denoted as... j=1,2,…,Mt, where Mt is the length of the texture memory; calculate The sum of similarities between any one texture feature and all other texture features is used to select the texture feature with the largest sum, denoted as . ,Will The confidence level is denoted as ;like Greater than Then delete in the texture memory. And store Otherwise, the texture memory remains unchanged. Reliability and diversity of features are achieved by storing the texture features of the Mt slices with the greatest similarity differences that have undergone instance segmentation in the texture memory.

[0073] The existing SAM 2 model's memory module utilizes foreground information from already segmented slices to aid in segmenting the current block; however, the first-in-first-out (FIFO) access strategy cannot effectively provide the necessary foreground information. Furthermore, contextual information (such as the tissue microenvironment) plays a crucial role in identifying cell nucleus instances, and SAM 2 also fails to provide this contextual information. To address these issues, this invention designs a CIM to supplement foreground information (e.g., cell nucleus texture and size) and contextual information.

[0074] S5: The prompting encoding module generates a point embedding for each cell nucleus based on domain features to reduce the number of missed detections.

[0075] In this embodiment, the prompt encoding module includes a prediction head and a prompt encoder, and the prompt encoder used in this embodiment is the prompt encoder in SAM 2. The prediction head includes a regression head and a classification head.

[0076] The prediction head predicts the points of each nucleus according to the field features, and the prompt encoder encodes the points into point embeddings, denoted as The points of the nucleus refer to the nuclei that can be identified on the current slice, and a point is generated at the position of the nucleus on the current slice after the nucleus is identified. In this embodiment, some points will be pre-sprayed on the slice first, and then the regression head will predict the offset of the points according to the features of the slice, so as to change the positions of all the points; then, the classification head will judge whether the points belong to the nucleus or the background, and finally the positions of each nucleus can be located, that is, a point is generated on each nucleus.

[0077] S6: The mask decoder predicts the mask of the nucleus corresponding to each point, i.e., the mask of each nucleus, according to the image features and the point embeddings .

[0078] S7: The instance segmentation result of the slice is obtained by combining the mask of each nucleus, and the instance segmentation result of the cell image is obtained by combining the instance segmentation results of all the slices.

[0079] S8: The instance segmentation result of the slice and the enhanced features are input into the texture encoder to generate texture features, and the texture features and the enhanced features are stored in the context injection module to provide context information for the next slice.

[0080] In this embodiment, the instance segmentation result of the slice and the enhanced features are obtained by the texture encoder to obtain texture features, denoted as The texture features are stored in the texture memory bank in the context injection module, and the enhanced features (denoted as ) are stored in the environment memory bank in the context injection module to provide context information for the next slice.

[0081] The application also discloses a cross-map context-aware nucleus instance segmentation system, which comprises:

[0082] An image acquisition module acquires a cell image and scans and divides the cell image into a slice group comprising a plurality of slices;

[0083] The model construction module constructs a cell nucleus instance segmentation model, the cell nucleus instance segmentation model comprising a feature extraction module, a context injection module, a prompt encoding module, a mask decoder and a texture encoder; the feature extraction module extracts prior features and domain features of a slice through a multi-layer encoder, refines the prior features of each layer through a multi-level feature refinement block, extracts and fuses missing features from the domain features of each layer to obtain enhanced features; the context injection module generates image features according to the enhanced features and integrates context information from surrounding segmented slices, the prompt encoding module generates point embeddings of each nucleus according to the domain features, and the mask decoder obtains a mask of each nucleus according to the image features and the point embeddings;

[0084] The instance segmentation module combines the mask of each nucleus to obtain an instance segmentation result of the slice, combines instance segmentation results of all slices to obtain an instance segmentation result of the cell image, inputs the instance segmentation result of the slice and the enhanced features into the texture encoder to generate texture features, and stores the texture features and the enhanced features into the context injection module to provide context information for the next slice.

[0085] The application further discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the cell nucleus instance segmentation method.

[0086] The application further discloses a device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to realize the cell nucleus instance segmentation method.

[0087] Compared with the prior art, the application has the following advantages:

[0088] 1. The feature extraction module extracts prior features and domain features of a cell image, and uses a multi-level feature refinement block to refine the prior features and integrate the domain features, thereby effectively extracting specific domain features.

[0089] 2. The context injection module integrates historical context information around each nucleus in the cell image, so that the model is more suitable for the cell image and the effect of cell nucleus instance segmentation is improved.

[0090] 3. The prediction head comprising a regression head and a classification head is added after the convolution branch to automatically generate points of the nucleus, thereby improving the efficiency and reducing the labor cost compared with the process of manually annotating points in SAM 2.

[0091] To further prove the beneficial effects of the present application, in this embodiment, the present application method and the prior art U-Net, U-Net, Mask R-CNN, HoVer-Net, SMILE (for details see the paper "Pan, Peng, et al. "SMILE: Cost-sensitive multi-task learning for nuclear segmentation and classification with imbalanced annotations." Medical Image Analysis 88 (2023): 102867."), PointNu-Net (for details see the paper "Yao, Kai, et al. "Pointnu-net: Keypoint-assisted convolutional neural network for simultaneous multi-tissue histology nuclei segmentation and classification." IEEE Transactions on Emerging Topics in Computational Intelligence 8.1 (2023): 802-813."), SAM(Zero-Shot) (for details see the paper "Kirillov, Alexander, et al. "Segment anything." Proceedings of the IEEE / CVF international conference on computer vision. 2023."), MedSA (for details see the paper "Wu J, Ji W, Liu Y, et al. Medical SAMAdapter: Adapting Segment Anything Model for Medical Image Segmentation[J]. 2023."), HQ-SAM (for details see the paper "Ke, Lei, et al. "Segment anything in high quality." Advances in Neural Information Processing Systems 36 (2023): 29914-29934.") are used on the CPM-17 dataset and the MoNuSeg dataset, respectively.CellViT (see the paper “Hrst F, Rempe M, Heine L, et al. CellViT: Vision Transformers for precise cell segmentation and classification [J]. Medical Image Analysis, 2024, 94(000). DOI:10.1016 / j.media.2024.103143.”), SAM2, MedSAM2 (see the paper “Zhu J, Hamdi A, Qi Y, et al. Medical SAM 2: Segment medical images as video via Segment Anything Model 2 [J]. 2024.”) are used to simulate the cell nucleus instance segmentation experiment. Among these methods, U-Net, U-Net, Mask R-CNN, HoVer-Net, SMILE, PointNu-Net, SAM (Zero-Shot), MedSA, HQ-SAM and CellViT do not generate point embeddings, and SAM2 and MedSAM2 generate point embeddings by manually annotating points and then encoding them through a prompt encoder.

[0092] The Dice coefficient (Dice), the aggregated Jaccard index (AJI), the detection quality (DQ), the segmentation quality (SQ) and the panoramic quality (PQ) are used as evaluation indexes to evaluate the effect of cell nucleus instance segmentation. The experimental results are shown in Tables 1 and 2.

[0093] Table 1: Cell nucleus instance segmentation results of different methods on the CPM-17 dataset

[0094]

[0095] Table 2: Cell nucleus instance segmentation results of different methods on the MoNuSeg dataset

[0096]

[0097] From Tables 1 and 2, it can be seen that the indexes of the present application are all better than those of the prior art methods, thereby proving that the present application can significantly improve the performance of cell nucleus instance segmentation.

[0098] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0099] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0100] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0102] Obviously, the above-described embodiments are only examples and are not intended to limit the present application. Other variations and modifications can be made to the above-described embodiments without departing from the scope of the present application. The scope of the present application is not limited to the above-described embodiments and should be determined by the appended claims.

Claims

1. A cross-graph context-aware cell nucleus instance segmentation method, characterized in that, include: Cell images are acquired and scanned into slice groups comprising multiple slices. A cell nucleus instance segmentation model is constructed, which includes a feature extraction module, a context injection module, a cue encoding module, a mask decoder, and a texture encoder. The feature extraction module extracts prior features and domain features of slices through a multi-layer encoder, refines the prior features of each layer through multi-level feature refinement blocks, and extracts and fuses missing features from the domain features of each layer to obtain enhanced features. The context injection module integrates contextual information from surrounding segmented slices to generate image features based on the enhanced features; the cue encoding module generates point embeddings for each cell nucleus based on the neighborhood features; and the mask decoder obtains a mask for each cell nucleus based on the image features and point embeddings. The instance segmentation result of the slice is obtained by combining the mask of each cell nucleus, and the instance segmentation result of the cell image is obtained by combining the instance segmentation results of all slices. The instance segmentation result and enhancement features of the slice are input into the texture encoder to generate texture features, and the texture features and enhancement features are stored in the context injection module to provide context information for the next slice; The context injection module generates image features by integrating contextual information from surrounding segmented slices based on the enhanced features, specifically: The context injection module includes an environment memory and a texture memory. For the i-th slice of the current input, the environment memory stores the enhanced features corresponding to the previous i-1 slices, denoted as E. j Let j = 1, 2, ..., i-1; and let E be the enhancement feature corresponding to the i-th slice. i , when E i When inputting a context injection module, select the one closest to E in the environment memory. i The shortest distance memory is constructed from the M1 enhanced features, denoted as M. sd Application M sd Inject surrounding environmental information into E i The features containing environmental information are obtained and denoted as Will Input texture memory; The texture memory stores the texture features of Mt slices that have undergone instance segmentation and have the greatest similarity difference. The texture features from the texture memory are selected... The M2 closest texture features are used to construct a maximum similarity memory, denoted as M. ms Application M ms Inject foreground information Obtain image features, denoted as 2. The cross-graph context-aware cell nucleus instance segmentation method according to claim 1, characterized in that: The feature extraction module includes a CNN encoder and a ViT encoder. The CNN encoder is a multi-layer convolutional neural network encoder. The ViT encoder includes a patch embedded module and multiple feature refinement layers. The number of layers in the feature refinement layers is the same as the number of layers in the convolutional neural network encoder. Each feature refinement layer includes multiple feature refinement blocks, and each feature refinement block includes the multi-level feature refinement block and the ViT block.

3. The cross-graph context-aware cell nucleus instance segmentation method according to claim 2, characterized in that: The feature extraction module extracts prior features and domain features of slices through a multi-layer encoder, refines the prior features of each layer through multi-level feature refinement blocks, and extracts and fuses missing features from the domain features of each layer to obtain enhanced features, specifically: When each slice is input into the feature extraction module, the i-th layer of the CNN encoder outputs the i-th layer's neighborhood features; The ViT encoder first extracts the first-layer prior features through the Patch Embedded module, then fuses the first-layer prior features and the first-layer neighborhood features through the first-layer feature refinement layer to obtain the (i+1)th-layer prior features, and finally uses the output of the last feature refinement layer as the enhanced features.

4. The cross-graph context-aware cell nucleus instance segmentation method according to claim 3, characterized in that: The calculation method for the (i+1)th layer prior features is as follows: in, For the (i+1)th layer prior features, ViT() is the operation of the ViT block. Let be the prior features of the i-th layer, and MFRB() be the operation for refining multi-level feature blocks. n This indicates that the ViT(MFRB()) operation is performed n times, where n is the number of feature refinement blocks in the feature refinement layer; In the i-th feature refinement layer, the features of the input j-th feature refinement block are denoted as... j = 1, 2, ..., n That is The features obtained after passing through multiple feature refinement blocks in the j-th feature refinement block are denoted as follows: The specific operation of the multi-level feature refinement block is as follows: Among them, A c () represents the cross-attention operation, and LN() represents the layer normalization operation. For the i-th layer of the domain features, To refine the domain feature representation The result obtained after the ViT block operation is 5. The cross-graph context-aware cell nucleus instance segmentation method according to claim 4, characterized in that: The The calculation method is as follows: Among them, Adapter() is the Adapter fine-tuning operation.

6. The cross-graph context-aware cell nucleus instance segmentation method according to claim 1, characterized in that: The shortest distance memory bank is: M sd ={E k ,k=1,...,M1},E k For the closest to E i The k-th feature among the M1 enhanced features; The The calculation method is as follows: Among them, A e () represents memory and attention operations; The maximum similarity memory is: M ms ={E q ,q=1,...,M2},E q For the closest The q-th feature among the M2 texture features; The The calculation method is as follows:

7. The cross-graph context-aware cell nucleus instance segmentation method according to claim 1, characterized in that: When the texture memory stores the texture features of Mt slices that have undergone instance segmentation with the greatest similarity difference, the storage method is as follows: Let the texture feature corresponding to the current i-th slice be denoted as Will The confidence level is denoted as I. i ; Let the texture features of the Mt slices that have been segmented into instances and stored in the texture memory be denoted as... j = 1, 2, ..., Mt; Calculate The sum of the similarities between any one texture feature and all other texture features is used to select the texture feature with the largest sum, denoted as E. max E max The confidence level is denoted as I. max ; If I i Greater than I max Then delete E in the texture memory. max And store Otherwise, the texture memory remains unchanged.

8. The cross-graph context-aware cell nucleus instance segmentation method according to claim 1, characterized in that: The prompting encoding module generates a point embedding for each cell nucleus based on domain features, specifically: The cue encoding module includes a prediction head and a cue encoder, wherein the prediction head includes a regression head and a classification head; The prediction head predicts points for each cell nucleus based on domain features, and the cue encoder encodes the points as point embeddings.

9. A cross-graph context-aware cell nucleus instance segmentation system, characterized in that, include: The image acquisition module acquires cell images and scans and segments them into a group of slices containing multiple slices; The model building module constructs a cell nucleus instance segmentation model, which includes a feature extraction module, a context injection module, a cue encoding module, a mask decoder, and a texture encoder. The feature extraction module extracts prior features and domain features of slices through a multi-layer encoder, refines the prior features of each layer through multi-level feature refinement blocks, and extracts and fuses missing features from the domain features of each layer to obtain enhanced features. The context injection module integrates contextual information from surrounding segmented slices to generate image features based on the enhanced features; the cue encoding module generates point embeddings for each cell nucleus based on the neighborhood features; and the mask decoder obtains a mask for each cell nucleus based on the image features and point embeddings. The instance segmentation module combines the mask of each cell nucleus to obtain the instance segmentation result of the slice, and combines the instance segmentation results of all slices to obtain the instance segmentation result of the cell image. The instance segmentation result and enhancement features of the slice are input into the texture encoder to generate texture features, and the texture features and enhancement features are stored in the context injection module to provide context information for the next slice; The context injection module generates image features by integrating contextual information from surrounding segmented slices based on the enhanced features, specifically: The context injection module includes an environment memory and a texture memory. For the i-th slice of the current input, the environment memory stores the enhanced features corresponding to the previous i-1 slices, denoted as E. j Let j = 1, 2, ..., i-1; and let E be the enhancement feature corresponding to the i-th slice. i , when E i When inputting a context injection module, select the one closest to E in the environment memory. i The shortest distance memory is constructed from the M1 enhanced features, denoted as M. sd Application M sd Inject surrounding environmental information into E i The features containing environmental information are obtained and denoted as Will Input texture memory; The texture memory stores the texture features of Mt slices that have undergone instance segmentation and have the greatest similarity difference. The texture features from the texture memory are selected... The M2 closest texture features are used to construct a maximum similarity memory, denoted as M. ms Application M ms Inject foreground information Obtain image features, denoted as

Images