Transformer-based end-to-end cell segmentation and tracking method and system

Abstract

This paper discloses a Transformer-based end-to-end cell segmentation and tracking method and system. This method employs an improved model based on the Mask DINO model to implement cell segmentation and tracking. The method uses a top-K query selection strategy to select the K feature points with the highest probability in the output feature map as object queries, and uses the trajectory from the previous frame t-1 as the tracking query in the current frame t; additionally generates a splitting query representing mitosis; uses a Transformer decoder to decode the object query, tracking query, and splitting query; and uses a classification head to predict the classification labels, rotated bounding boxes, and segmentation masks of the cells in image I of the current frame t. This method aims to address challenges in cell segmentation and tracking scenarios and improve the accuracy of cell segmentation and tracking.

Description

Technical Field

[0001] The present invention relates to the technical field of cell segmentation and tracking, and in particular to a Transformer-based end-to-end cell segmentation and tracking method and system. Background Art

[0002] Cell segmentation and tracking are fundamental challenges in biomedical applications. They are crucial for understanding cell morphology, structure, and function, and are fundamental to life science research. With the success of deep learning in image segmentation and object detection, researchers are increasingly applying these techniques to cell tracking.

[0003] Most existing cell segmentation and tracking methods utilize deep neural networks such as U-Net for semantic segmentation. Subsequently, instance segmentation is addressed using the watershed algorithm and other techniques based on markers, followed by cell association using methods such as nearest neighbor linking and graph matching. Despite advances in neural networks, their reliance on traditional pre- and post-processing methods has exposed significant limitations. This reliance leads to complex and inefficient workflows, hindering the development of seamless end-to-end systems. This fragmented process not only limits the potential for integration and automation but also introduces error propagation across multiple steps, further degrading overall performance.

[0004] Recently developed Transformer-based DETR models have gained popularity due to their simplicity, efficiency, and end-to-end trainability, and have made significant progress in object detection and tracking tasks. As an end-to-end detector, DETR does not require hand-crafted modules and outperforms traditional detectors. In multi-object tracking, Transformer-based methods introduce tracking queries to regress object locations, thereby omitting the data association step and avoiding the complexity of additional graph optimization or motion and appearance modeling. In addition, thanks to the attention mechanism, these methods can effectively understand the complex relationship between objects and the environment, achieving state-of-the-art (SOTA) results on multiple multi-object tracking benchmarks. Despite the success of Transformer-based models in object detection and tracking, their application in cell tracking remains unexplored, which is attributed to several challenges: (1) Cell tracking is a comprehensive task that includes not only detection and tracking but also segmentation. (2) Cells have diverse shapes and are densely distributed. The same cell can show significant morphological changes during tracking. This diversity poses a challenge to Transformer-based models, which may find it difficult to effectively capture and represent such changes. (3) Specific cellular behaviors, such as cell division, introduce additional complexity to the tracking process and require specialized techniques to address these unique dynamics. Summary of the Invention

[0005] Technical problem to be solved by the present invention: In response to the above-mentioned problems in the prior art, a Transformer-based end-to-end cell segmentation and tracking method and system are provided. The present invention aims to solve the challenges in cell segmentation and tracking scenarios and improve the accuracy of cell segmentation and tracking.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A Transformer-based end-to-end cell segmentation and tracking method includes implementing cell segmentation and tracking using an improved model based on the Mask DINO model. The improved model based on the Mask DINO model, like the Mask DINO model, includes a CNN backbone network, a Transformer encoder, a Transformer decoder, and a classification head. The improved model implements cell segmentation and tracking by:

[0008] S1, use the CNN backbone network to extract multi-scale feature maps of the image I of the current frame t;

[0009] S2, uses the Transformer encoder to refine the multi-scale feature map and extract the output feature map;

[0010] S3 uses a top-K query selection strategy to select the K feature points with the highest probability in the output feature map as the object query, and uses the trajectory from the previous frame t-1 as the tracking query in the current frame t; calculates the distance between the centroids of the object query and the tracking query, and generates an additional division query representing mitosis if the distance between the centroids may lead to cell division; the object query, tracking query, and division query all contain a content query embedding represented by a feature embedding and a position query embedding represented by an anchor box;

[0011] S4, uses the Transformer decoder to decode object queries, tracking queries, and split queries;

[0012] S5, use the classification head to predict the classification label, rotation bounding box and segmentation mask of the cell in the image I of the current frame t based on the decoding results output by the Transformer decoder.

[0013] Optionally, the format of the anchor box is defined as ,in represents the anchor box, represents the center position of the anchor box, Represent the width and height of the anchor box respectively, Represents the rotation angle of the anchor frame; the function expression of the position query embedding represented by the anchor frame is:

[0014] ,

[0015] In the above formula, is the position query embedding represented by the anchor box, is the connection function, is the cosine position code, It is a multi-layer perceptron.

[0016] Optionally, the improved model based on the Mask DINO model includes position attention adopted by the mutual attention layer in the Transformer decoder layer of the Mask DINO model:

[0017] ,

[0018] The reference point of the encoder feature map Improved to the following improved reference point :

[0019] ,

[0020] ,

[0021] in, is position attention, represents the center position of the anchor box, is the reference point of the encoder feature map, is the cosine position code, is the scaling result value.

[0022] Optionally, the improved model based on the Mask DINO model includes cross attention adopted in the mutual attention layer in the Transformer decoder layer of the Mask DINO model:

[0023] ,

[0024] Offset the sample position learned relative to the center position The improved sampling position offset is: :

[0025] ,

[0026] ,

[0027] in, For queries with location priors Cross-attention, For query The center position of the anchor box, is the feature map output by the Transformer encoder, is the number of selected feature points, is the normalized attention weight, The transformation matrix is ​​used to obtain the attention value, is the feature map On point The feature code at represents the sampling position offset learned relative to the center position, To standardize operations, and Also the original sampling position offset in the X and Y axis directions, represent the width and height of the anchor box respectively.

[0028] Optionally, the improved model based on the Mask DINO model includes improving the segmentation branch for generating the segmentation mask in the classification head of the Mask DINO model to:

[0029] ,

[0030] In the above formula, query embeddings for content represented by feature embeddings, is the position query embedding represented by the anchor box, is element-wise multiplication, is the split head of the split branch, is the convolutional layer, is a two-fold upsampling operation, is the 1 / 4 original resolution feature map from the CNN backbone network, is the output feature map from the Transformer encoder upsampled to 1 / 8 of the original resolution, is the cosine position embedding of the pixel embedding map, which is a 1 / 4 original resolution feature map from the CNN backbone network. and the output feature map from the Transformer encoder upsampled to 1 / 8 the original resolution Add together to get.

[0031] Optionally, the distance between the centroids in step S3 may cause cell division, which refers to determining whether the distance between the centroids is less than a preset threshold t d , if the preset threshold t d It is judged that it may cause cell division, otherwise it is judged that it will not cause cell division.

[0032] Optionally, before implementing cell segmentation and tracking using the improved model based on the Mask DINO model, the method further includes training an end-to-end cell segmentation and tracking model based on Transformer, and the function expression of the loss function used in training the end-to-end cell segmentation and tracking model based on Transformer is:

[0033] ,

[0034] In the above formula, is the loss function, is the number of queries, each query is responsible for detecting a cell in the current frame t, and the queries include object queries Q object , Tracking Query Q track and split query Q mitosis Three types, is the classification loss, is the frame loss, is the segmentation loss, Is it the label in the real label data GT of the current frame t? Matching queries, Indicates that the query is successfully matched with the label in the real label data GT in the Hungarian algorithm Match, is the classification probability, is the rotation bounding box, is the segmentation mask, is the label index of the rotated bounding box, is the label index of the segmentation mask, For the i-th query The corresponding prediction output is:

[0035] ,

[0036] In the above formula, and is the weight parameter, is the exponential parameter;

[0037] ,

[0038] In the above formula, and is the weight parameter, for The L1 loss between for KFIoU loss between;

[0039] ,

[0040] In the above formula, and is the weight parameter, for The cross entropy loss between for The dice loss between.

[0041] In addition, the present invention also provides a Transformer-based end-to-end cell segmentation and tracking system, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method.

[0042] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method through a processor.

[0043] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method through a processor.

[0044] Compared with the existing technology, the present invention has the following main advantages: the present invention includes the use of an improved model based on the Mask DINO model to realize cell segmentation and tracking, including using a top-K query selection strategy to select the K feature points with the highest probability in the output feature map as object queries, and using the trajectory from the previous frame t-1 as the tracking query in the current frame t; additionally generating a splitting query representing mitosis; using a Transformer decoder to decode the object query, tracking query and splitting query; using a classification head to predict the classification label, rotated bounding box and segmentation mask of the cells in the image I of the current frame t. The present invention can solve the challenges in cell segmentation and tracking scenarios and improve the accuracy of cell segmentation and tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of the basic process of the method of the embodiment of the present invention.

[0046] Figure 2 Schematic diagram of the structure of the improved model (Cell DINO) in an embodiment of the present invention.

[0047] Figure 3Figure 3 is a cell tracking example of a horizontal bounding box (HBB) and a rotated bounding box (RBB) in an embodiment of the present invention, wherein (a) is a cell tracking example of a position attention map of a horizontal bounding box (HBB); (b) is a cell tracking example of a horizontal bounding box (HBB) and its position attention map in frame t; (c) is a cell tracking example of a position attention map of a rotated bounding box (RBB); (d) is a cell tracking example of a rotated bounding box (RBB) and its position attention map in frame t; (e) is a cell tracking example of a rotated bounding box (RBB) and its position attention map in frame t+1; (f) is a cell tracking example of a rotated bounding box (RBB) and its position attention map in frame t+2; and (g) is a cell tracking example of a rotated bounding box (RBB) and its position attention map in frame t+3.

[0048] Figure 4 Schematic diagram comparing different cell segmentation methods in an embodiment of the present invention, where (a) is the result of merging multiple cells using content query for comparison, and (b) is the result of improving segmentation using content and position query in this embodiment.

[0049] Figure 5 : These are examples of multiple selected datasets and corresponding Cell DINO segmentation results in an embodiment of the present invention, where (a) is the original image on the Fluo-N2DH-GOWT1 dataset, (b) is the original image on the PhC-C2DH-U373 dataset, (c) is the original image on the DIC-C2DH-HeLa dataset, (d) is the mask annotation of the original image (a) on the Fluo-N2DH-GOWT1 dataset, and (e) is the mask annotation of the original image (b) on the PhC-C2DH-U373 dataset.

[0050] Figure 6 These are the training curves of the horizontal bounding box (HBB) and the rotation bounding box (RBB) on the Fluo-N2DH-GOWT1 dataset in an embodiment of the present invention.

[0051] Figure 7 These are the training curves of the horizontal bounding box (HBB) and the rotation bounding box (RBB) on the PhC-C2DH-U373 dataset in an embodiment of the present invention.

[0052] Figure 8 These are the training curves of the horizontal bounding box (HBB) and the rotation bounding box (RBB) on the DIC-C2DH-HeLa dataset in an embodiment of the present invention.

[0053] Figure 9 is the preset threshold value t for different mitosis detections in the embodiment of the present invention d Impact on tracking performance. DETAILED DESCRIPTION

[0054] The end-to-end cell segmentation and tracking method based on Transformer in this embodiment includes using an improved model based on the MaskDINO model (named Cell DINO in this embodiment) to implement cell segmentation and tracking. The improved model based on the MaskDINO model and the MaskDINO model both include a CNN backbone network, a Transformer encoder, a Transformer decoder, and a classification head. Figure 1 and Figure 2 As shown in the figure, the improved model (Cell DINO) for cell segmentation and tracking includes:

[0055] S1, use the CNN backbone network to extract multi-scale feature maps of the image I of the current frame t;

[0056] S2, uses the Transformer encoder to refine the multi-scale feature map and extract the output feature map;

[0057] S3, using the top-K query selection strategy to select the K feature points with the highest probability in the output feature map as the object query, and using the trajectory from the previous frame t-1 as the tracking query in the current frame t to calculate the distance between the center of mass of the object query and the tracking query. If the distance between the center of mass may lead to cell division, an additional division query representing mitosis is generated; the object query, tracking query, and division query all contain a content query embedding represented by a feature embedding and a position query embedding represented by an anchor box;

[0058] S4, uses the Transformer decoder to decode object queries, tracking queries, and split queries;

[0059] S5, use the classification head to predict the classification label, rotation bounding box and segmentation mask of the cell in the image I of the current frame t based on the decoding results output by the Transformer decoder.

[0060] In order to adapt to the scene of diverse cell shapes and dense distribution, this embodiment uses a rotating bounding box (RBB) for cell detection. Specifically, the format of the anchor box in this embodiment is defined as ,in represents the anchor box, represents the center position of the anchor box, Represent the width and height of the anchor box respectively, Represents the rotation angle of the anchor frame; the function expression of the position query embedding represented by the anchor frame is:

[0061] ,

[0062] In the above formula, is the position query embedding represented by the anchor box, is the connection function, is the cosine position code, For the three types of queries, object query, tracking query and split query, each query q will generate a corresponding anchor box, corresponding to a final generated rotation bounding box and segmentation mask. In this embodiment, the rotation angle of the anchor box is added to the format definition of the anchor box. , so that the rotation angle of the anchor frame can be Positional attention in the Transformer decoder layer is introduced to better adapt to cells of different shapes. Figure 3 Figure 3 is a cell tracking example of a horizontal bounding box (HBB) and a rotated bounding box (RBB) in an embodiment of the present invention, wherein (a) is a cell tracking example of a position attention map of a horizontal bounding box (HBB); (b) is a cell tracking example of a horizontal bounding box (HBB) and its position attention map in frame t; (c) is a cell tracking example of a position attention map of a rotated bounding box (RBB); (d) is a cell tracking example of a rotated bounding box (RBB) and its position attention map in frame t; (e) is a cell tracking example of a rotated bounding box (RBB) and its position attention map in frame t+1; (f) is a cell tracking example of a rotated bounding box (RBB) and its position attention map in frame t+2; and (g) is a cell tracking example of a rotated bounding box (RBB) and its position attention map in frame t+3. Figure 3 Both position attention maps (a) and (c) are computed by taking the dot product between the position query and the position embedding in the feature map.

[0063] In this embodiment, object queries, tracking queries, and split queries all include a content query embedding represented by a feature embedding and a location query embedding represented by an anchor box. Figure 4 Schematic diagram comparing different cell segmentation methods, where (a) shows the merging of multiple cells using content query, and (b) shows the improved segmentation result using content and location query in this embodiment. The location query embedding represented by an anchor box can improve the query accuracy.

[0064] In this embodiment, the improved model based on the Mask DINO model includes position attention adopted by the mutual attention layer in the Transformer decoder layer of the Mask DINO model:

[0065] ,

[0066] The reference point of the encoder feature map Improved to the following improved reference point :

[0067] ,

[0068] ,

[0069] in, is position attention, represents the center position of the anchor box, is the reference point of the encoder feature map, is the cosine position code, To scale the result value, a more appropriate position attention that is more suitable for the cell shape can be obtained through the above-mentioned transformation of the reference point position in the encoder feature map.

[0070] In this embodiment, the improved model based on the Mask DINO model includes the cross attention adopted in the mutual attention layer in the Transformer decoder layer of the Mask DINO model:

[0071] ,

[0072] Offset the sample position learned relative to the center position The improved sampling position offset is: :

[0073] ,

[0074] ,

[0075] in, For queries with location priors Cross-attention, For query The center position of the anchor box, is the feature map output by the Transformer encoder, is the number of selected feature points, is the normalized attention weight, The transformation matrix is ​​used to obtain the attention value, is the feature map On point The feature code at represents the sampling position offset learned relative to the center position, To standardize operations, and Also the original sampling position offset in the X and Y axis directions, Represent the width and height of the anchor box respectively. Through the above steps, the deformable attention mechanism is used to accelerate the convergence of the improved model (Cell DINO).

[0076] As a unified model for image segmentation, Mask DINO constructs a pixel embedding map to achieve mask classification for all segmentation tasks. The pixel embedding map is obtained by integrating the 1 / 4 original resolution feature map from the CNN backbone network. Output feature map from the Transformer encoder upsampled to 1 / 8 the original resolution Then, the embedding is queried by the content represented in the feature embedding The output mask is obtained by performing a dot product operation between and the pixel embedding map, as shown below:

[0077] ,

[0078] In the above formula, is the segmentation mask obtained by the segmentation branch, query embeddings for content represented by feature embeddings, is element-wise multiplication, is the split head of the split branch, is the convolutional layer, is a two-fold upsampling operation, is the 1 / 4 original resolution feature map from the CNN backbone network, is the output feature map from the Transformer encoder upsampled to 1 / 8 of the original resolution. Figure 4 It can be seen that in cell instance segmentation applications, adjacent cells are often incorrectly segmented into a single entity. This embodiment assumes that this problem stems from the semantic nature of content queries, which prioritize cell semantics over precise spatial details. To alleviate this challenge, this embodiment integrates location information into the segmentation process. Therefore, the improved model based on the Mask DINO model in this embodiment includes an improvement to the segmentation branch for generating segmentation masks in the classification head of the Mask DINO model as follows:

[0079] ,

[0080] In the above formula, query embeddings for content represented by feature embeddings, is the position query embedding represented by the anchor box, is element-wise multiplication, is the split head of the split branch, is the convolutional layer, is a two-fold upsampling operation, is the 1 / 4 original resolution feature map from the CNN backbone network, is the output feature map from the Transformer encoder upsampled to 1 / 8 of the original resolution, is the cosine position embedding of the pixel embedding map, which is a 1 / 4 original resolution feature map from the CNN backbone network. and the output feature map from the Transformer encoder upsampled to 1 / 8 the original resolution Add together to get.

[0081] In this embodiment, the distance between the centroids may cause cell division in step S3, which refers to determining whether the distance between the centroids is less than a preset threshold t d , if the preset threshold t d Then the judgment may lead to cell division, otherwise the judgment will not lead to cell division. In models like DETR, the object query is responsible for detecting the object in the current frame, but it cannot track the object across the video sequence. Therefore, a tracking query carrying identity information is needed to track objects in the video sequence. The tracking query is initialized using the embedding and position information of the object detected in the previous frame t-1. However, each tracking query can only track a single object and cannot handle cell division events. To solve this problem, this embodiment proposes the concept of mitosis query to manage cell division scenarios. During the training process of Cell DINO, this embodiment uses iterative prediction to optimize the output of each layer in the decoder. This process includes the input of the decoder, which corresponds to the output of the last layer of the encoder. Therefore, even the initially assigned object queries only contain rough detection results. As Figure 3 As shown, this embodiment selects K object queries Q from the encoder object , and then further filter out high confidence object queries Q′ object As a potential detection. Then, this embodiment calculates Q' object and trajectory query Q track The distance between the centers of mass. object For each query in , this embodiment only retains the association with the nearest trajectory query. If a cell (trajectory query) is associated with two object queries, and the centroid distance between the trajectory query and each object query is less than a preset threshold t d , it is considered likely to have divided. For each cell that may have divided (trajectory query), this embodiment generates a corresponding mitosis query. The content of the mitosis query is aligned with the content query of the associated trajectory query. This embodiment then uses the location information of the two object queries linked to the trajectory query to update the location query of the trajectory and mitosis query. Subsequently, detection, tracking, and mitosis queries are collectively input into the Transformer decoder for processing. Finally, the classification head is used to determine whether division ultimately occurred.

[0082] In order to promote the effective interaction between trajectory query and object query, this embodiment implements an end-to-end training method by randomly sampling adjacent frames of unit DINO. At the current frame t, this embodiment generates N = N object + N track queries, where N object is the number of object queries (detection queries), N track Query Q for the trajectory track the number of N track depends on the detection state from t−k to frame t−1. This embodiment ignores the consideration of split scenes during training. Each query Produces a prediction output ,in is the classification probability, is the rotation bounding box, is the segmentation mask.

[0083] First, this embodiment uses trajectory identification to match predictions (expressed as ):

[0084] ,

[0085] and the true label data GT (represented as ):

[0086] ,

[0087] in, is the prediction result of the i-th test data, N is the number of test data; in the above formula, is the true label of the i-th sample, is the label of the classification probability of the i-th sample, is the label of the rotated bounding box of the i-th sample, is the label of the segmentation mask of the i-th sample, is the number of samples of the true label data GT. Then, hybrid matching is used to match the remaining predictions and the true label data GT. This embodiment assumes Representation and prediction results Matched real label data GT, index Represents the best matching arrangement of labels. When this embodiment jointly trains detection, segmentation and tracking tasks, this embodiment uses a combined loss function To train Cell DINO. Before using the improved model based on the Mask DINO model to implement cell segmentation and tracking, this embodiment also includes training an end-to-end cell segmentation and tracking model based on Transformer. The function expression of the loss function used in training the end-to-end cell segmentation and tracking model based on Transformer is:

[0088] ,

[0089] In the above formula, is the loss function, is the number of queries, each query is responsible for detecting a cell in the current frame t, and the queries include object queries Q object , Tracking Query Q track and split query Q mitosis Three types, is the classification loss, is the frame loss, is the segmentation loss, Is it the label in the real label data GT of the current frame t? Matching queries, Indicates that the query is successfully matched with the label in the real label data GT in the Hungarian algorithm Match, is the classification probability, is the rotation bounding box, is the segmentation mask, is the label index of the rotated bounding box, is the label index of the segmentation mask, For the i-th query The corresponding prediction output is:

[0090] ,

[0091] In the above formula, and is the weight parameter, is an index parameter; as an optional implementation, in this embodiment 、 and ;

[0092] ,

[0093] In the above formula, and is the weight parameter, for The L1 loss between for As an optional implementation, in this embodiment and ;

[0094] ,

[0095] In the above formula, and is the weight parameter, for The cross entropy loss between for As an optional implementation, in this embodiment and .

[0096] In order to verify the effectiveness, robustness and feasibility of the end-to-end cell segmentation and tracking method based on Transformer in this example, this example selected three datasets from the Cell Tracking Challenge: Fluo-N2DH-GOWT1, PhC-C2DH-U373 and DIC-C2DH-HeLa. Figure 5As shown, (a) is the original image on the Fluo-N2DH-GOWT1 dataset, (b) is the original image on the PhC-C2DH-U373 dataset, (c) is the original image on the DIC-C2DH-HeLa dataset, (d) is the mask annotation of the original image (a) on the Fluo-N2DH-GOWT1 dataset, (e) is the mask annotation of the original image (b) on the PhC-C2DH-U373 dataset, and (f) is the mask annotation of the original image (c) on the DIC-C2DH-HeLa dataset. For training and testing purposes, each dataset contains two sequences, labeled 01 and 02, in the training set and test set, respectively. The details of the selected datasets are as follows: (1) The Fluo-N2DH-GOWT1 dataset contains 368 images of GFP-transfected GOWT1 mouse embryonic stem cells, each image is 1024×1024 pixels in size. The main challenges of this dataset are the low contrast of some cells and the limited number of cells that pass through the imaging area axially. (2) The PhC-C2DH-U373 dataset contains 460 images depicting glioblastoma-gastric U373 cell samples, each with a size of 636×520 pixels. This dataset poses significant challenges because the cell shapes are highly variable and in some cases have a similar appearance to the background. (3) The DIC-C2DH-HeLa dataset contains 398 images of HeLa cells on flat glass, each with a size of 512×512 pixels. A significant difficulty inherent in this dataset is that the cells are densely distributed and have variable morphologies. For the training sequences, both high-quality (GoldTruth) and sub-high-quality (SilverTruth) annotations are included. The high-quality (GoldTruth) annotations are manually annotated and include tracking information using point-by-point annotations as well as segmentation annotations for some cells of interest. On the other hand, the sub-high-quality annotations provide segmentation annotations for all images, which are obtained by fusing the segmentation results of high-performance methods previously demonstrated on the cell tracking challenge (CTC). Because the high-quality annotations only contain segmentation annotations for some cells, this example refers to the existing network model EmbedTrack during training. By merging the segmentation masks in the lower-quality annotations with the point-by-point tracking annotations in the high-quality annotations, this example creates fully labeled training data with reasonable annotation accuracy.

[0097] The improved model Cell DINO used in this embodiment follows the basic architecture of ResNet50 CNN feature extraction and Transformer encoder-decoder of Mask DINO, which serves as the basis for improvement. During training, for each training set, this embodiment allocates 90% of the images for training and the remaining images for validation. The model weights of this embodiment are initialized using the best-performing Mask DINO model on the COCO dataset. Subsequently, for each dataset, this embodiment trains the model for 40,000 iterations with a batch size of 1. The initial learning rate is set to 1×10 -4 , and then reduced by a factor of 0.1 at 15,000 and 30,000 iterations. This embodiment also uses a variety of data augmentation techniques, including flipping, cropping, scaling, Gaussian noise, grayscale mapping, and image blur. In addition, this embodiment maintains consistency with the training hyperparameters outlined in Mask DINO. All experiments were performed on PyTorch1.8.1, using an NVIDIA GeForce RTX 2080 Ti GPU, running on Ubuntu 18.04 LTS (64-bit). During the tracking process, this embodiment filters out detections of newly identified cells with confidence scores below 0.2. For cells that have been tracked, cells with detection confidence scores below 0.1 are temporarily considered lost. To ensure the continuity of cell trajectories and prevent premature termination, when they are lost, this embodiment retains their trajectories for five frames and sets k =5. When processing cell division events, this embodiment sets a preset threshold t for the Fluo-N2DH-GOWT1 dataset. d = 0.01, set the preset threshold t for DIC-C2DH-HeLa d =0.11. Since there are few cell division events in the PhC-C2DH-U373 dataset, this example sets a preset threshold t d = 0 and the cell division events in this dataset are not considered. In addition, this embodiment uses the indicators provided by the cell tracking challenge CTC to evaluate the algorithm of this embodiment. The cell tracking challenge CTC provides four indicators: detection accuracy DET, segmentation accuracy SEG, tracking accuracy TRA and tracking overall performance OP CTB , with values ​​between [0,1]. The higher the value, the better the performance. Detection accuracy DET is used to evaluate the accuracy of cell detection, and its function expression is as follows:

[0098] ,

[0099] In the above formula, is the normalized directed acyclic graph matching, is the cost of transforming the computed nodes into the reference node set, The cost of creating a reference node set from scratch.

[0100] Segmentation accuracy SEG evaluates segmentation by comparing the intersection and union between the true value and the output mask. Its function expression is as follows:

[0101] ,

[0102] In the above formula, Indicates intersection, Indicates mutuality, is the true value, is the output mask.

[0103] Tracking accuracy TRA is used to evaluate the quality of tracking results, and its function expression is as follows:

[0104] ,

[0105] In the above formula, The AOGM values ​​needed to create a reference image from scratch.

[0106] Tracking overall performance OP CTB Used to evaluate the overall performance of the method, its function expression is as follows:

[0107] .

[0108] To evaluate the performance of the method in this example, the method in this example and other existing methods (including KTHG-SE, AC, KIT-GE, MU-US, MU-CZ, BGU-IL, UVA-NL, and PR-GE) were tested on the CTC cell tracking benchmark (CTB). The evaluation results on CTB are shown in Table I, which details the overall tracking performance. CTB , the scores, rankings and the gap with the first place on segmentation accuracy SEG and tracking accuracy TRA.

[0109] Table I: Results of Cell Tracking Benchmark (CTB)

[0110]

[0111] As shown in Table 1, the method of this embodiment performs very well on the DIC-C2DH-HeLa dataset, and has an excellent overall performance in tracking OP CTBThis success can be attributed to the dense and diverse distribution of cells in this dataset, which is very suitable for rotation bounding box (RBB) detection. In addition, on the Fluo-N2DH-GOWT1 dataset, although it ranks 6th in segmentation accuracy SEG and 5th in tracking accuracy TRA, due to the balanced performance of the improved model Cell DINO in segmentation and tracking, this embodiment has a better overall tracking performance than OP. CTB It achieved second place in the 2018 World Cell Tracking Competition. On the PhC-C2DH-U373 dataset, this example's performance was relatively average for two main reasons: the lack of cell division events limited the method's ability to handle mitosis; and the sparse distribution of cells meant that rotated bounding box (RBB) detection had little advantage over horizontal bounding box (HBB) detection. Despite this, the results of this example's method were still impressive, with a small performance gap compared to the best-performing methods. Overall, Cell DINO is suitable for a variety of cell tracking scenarios. In contrast, other methods (such as KIT-GE) performed well on the Fluo-N2DH-GOWT1 and PhC-C2DH-U373 datasets but struggled with the densely packed cells in the DIC-C2DH-HeLa dataset.

[0112] In order to analyze the impact of each component of the method in this embodiment on the model performance and select key parameters, this embodiment conducted a series of experiments on the training set and the validation set. Figure 6 、 Figure 7 and Figure 8In this example, experiments were conducted using horizontal bounding box (HBB) and rotated bounding box (RBB) configurations on three different training sets. Performance metrics on the validation set were monitored every 1000 iterations. HBB_DET represents the detection accuracy (DET) curve for horizontal bounding boxes, HBB_SEG represents the segmentation accuracy (SEG) curve for horizontal bounding boxes, RBB_DET represents the detection accuracy (DET) curve for rotated bounding boxes, and RBB_SEG represents the segmentation accuracy (SEG) curve for rotated bounding boxes. Initially, the results of this example showed that horizontal bounding boxes exhibited superior convergence speed and effectiveness compared to rotated bounding boxes. The early advantage of horizontal bounding boxes can be attributed to their simpler learning process in the initial stages of training, while rotated bounding boxes face greater complexity, resulting in slower convergence. However, an interesting shift occurred after approximately 4000 iterations, where rotated bounding boxes began to significantly outperform horizontal bounding boxes. This improvement was reflected not only in faster convergence but also in higher scores in the detection accuracy (DET) and segmentation accuracy (SEG) metrics. This example attributes this shift to the model's increasing ability to more effectively utilize rotated bounding boxes during training. Specifically, as the improved model, Cell DINO, progresses, it becomes increasingly adept at accurately delineating cell boundaries using rotated bounding boxes, thereby improving localization accuracy. On the PhC-C2DH-U373 dataset, rotated and horizontal bounding boxes exhibit similar performance in cell detection, likely due to the dataset's low cell density, typically containing only 6-8 cells per frame. Table II shows the impact of positional information on segmentation performance across different datasets.

[0113] Table II: Comparison of segmentation results with and without location information

[0114]

[0115] The results in Table II show that the denser the cell distribution, the greater the impact of position information, and the improvement is most significant for the DIC-C2DH-HeLa dataset. In contrast, datasets such as PhC-C2DH-U373 are characterized by sparse and widespread cell distribution, so the impact is minimal. In addition, in order to determine the preset threshold t of mitosis distance suitable for each dataset, d , this embodiment tests the training set of each data set. The test results are as follows Figure 9 As shown. When the preset threshold t d When set to 0, no mitosis query is generated and mitosis events are not considered. Test results show that for the Fluo-N2DH-GOWT1 dataset, the preset threshold t d The best tracking performance is achieved when the threshold t is 0.01. In contrast, in the DIC-C2DH-HeLa dataset, the preset threshold td The best performance is achieved when t is 0.11. It is worth noting that increasing the number of mitosis queries in the PhC-C2DH-U373 dataset leads to a decrease in tracking performance, which may be due to the extremely low incidence of mitosis events in this dataset. In addition, if the preset threshold t d Setting it too high may misidentify neighboring cells or newly emerged cells as cells arising from mitosis, adversely affecting cell tracking performance.

[0116] In summary, the Transformer-based end-to-end cell segmentation and tracking method of this embodiment employs an improved model based on the Mask DINO model (named Cell DINO in this embodiment) for cell segmentation and tracking. This improved model (Cell DINO) is a Transformer-based end-to-end cell tracking method based on the unified segmentation model Mask DINO. This method addresses the challenges of simultaneously detecting, segmenting, and tracking cells through three improvements. First, this method addresses the unique characteristics of cells compared to other objects such as pedestrians or cars. Cells exhibit a variety of shapes, making traditional horizontal bounding box (HBB) detection methods ineffective. Typically, each HBB may contain multiple cells or significant background noise. To overcome this limitation, this embodiment proposes the use of rotated bounding boxes (RBBs) for detection, as they better align with cell shapes and reduce noise interference. This embodiment uses rotated bounding boxes (RBBs) for cell detection and segmentation, effectively adapting to diverse cell shapes while minimizing interference from background elements and neighboring cells. Furthermore, normalizing the position query to align with the RBB provides more precise spatial prior information for cross-attention in the Transformer decoder. In addition, the method of this embodiment introduces a 5-dimensional representation of location query , which provides a precise spatial prior for the cross-attention module, enabling queries to focus on the local region of the corresponding object. Secondly, inspired by the Transformer-based tracking model in MOT (Multi-Object Tracking), this embodiment method uses tracking queries for cell tracking. Third, to handle cell mitosis, this additional query ensures accurate tracking results even in scenarios involving cell division. This embodiment method introduces a mitosis query to determine whether cells undergoing potential division have actually divided during tracking. This effectively identifies cell division events during cell tracking. This new approach improves the accuracy and robustness of cell tracking by specifically addressing the challenging task of detecting cell division behavior. This embodiment validates the effectiveness of Cell DINO in the Cell Tracking Challenge (CTC) and achieves excellent performance on multiple datasets, with outstanding performance in segmenting and tracking cells in microscopy images. In particular, this embodiment method achieves first place on the DIC-C2DH-HeLa dataset, which features diverse cell morphologies and dense populations, and second place on the Fluo-N2DH-GOWT1 dataset. Experimental results on the Cell Tracking Challenge dataset and related research convincingly demonstrate the effectiveness of the Transformer-based end-to-end cell segmentation and tracking method of this embodiment.

[0117] In addition, this embodiment also provides a Transformer-based end-to-end cell segmentation and tracking system, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method.

[0118] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instructions, and the computer program or instructions are programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method through a processor.

[0119] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method through a processor.

[0120] Those skilled in the art should understand that the technical solutions provided by the embodiments of the present application may be in the form of methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0121] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A Transformer-based end-to-end cell segmentation and tracking method, characterized by: The improved model based on the Mask DINO model is used to implement cell segmentation and tracking. The improved model based on the Mask DINO model, like the Mask DINO model, includes a CNN backbone network, a Transformer encoder, a Transformer decoder, and a classification head. The improved model implements cell segmentation and tracking by: S1, use the CNN backbone network to extract multi-scale feature maps of the image I of the current frame t; S2, uses the Transformer encoder to refine the multi-scale feature map and extract the output feature map; S3 uses a top-K query selection strategy to select the K feature points with the highest probability in the output feature map as the object query, and uses the trajectory from the previous frame t-1 as the tracking query in the current frame t; calculates the distance between the centroids of the object query and the tracking query, and generates an additional division query representing mitosis if the distance between the centroids may lead to cell division; the object query, tracking query, and division query all contain a content query embedding represented by a feature embedding and a position query embedding represented by an anchor box; S4, uses the Transformer decoder to decode object queries, tracking queries, and split queries; S5, use the classification head to predict the classification label, rotation bounding box and segmentation mask of the cell in the image I of the current frame t based on the decoding results output by the Transformer decoder.

2. The end-to-end cell segmentation and tracking method based on Transformer according to claim 1, characterized in that The format of the anchor box is defined as ,in represents the anchor box, represents the center position of the anchor box, Represent the width and height of the anchor box respectively, represents the rotation angle of the anchor frame; the function expression of the position query embedded in the anchor frame is: ， In the above formula, is the position query embedding represented by the anchor box, is the connection function, is the cosine position code, It is a multi-layer perceptron.

3. The Transformer-based end-to-end cell segmentation and tracking method according to claim 2, characterized in that The improved model based on the Mask DINO model includes positional attention adopted by the mutual attention layer in the Transformer decoder layer of the Mask DINO model: ， The reference point of the encoder feature map Improved to the following improved reference point : ， ， in, is position attention, represents the center position of the anchor box, is the reference point of the encoder feature map, is the cosine position code, is the scaling result value.

4. The Transformer-based end-to-end cell segmentation and tracking method according to claim 3, characterized in that The improved model based on the Mask DINO model includes the cross attention adopted in the mutual attention layer in the Transformer decoder layer of the Mask DINO model: ， Offset the sample position learned relative to the center position The improved sampling position offset is: : ， ， in, For queries with location priors Cross-attention, For query The center position of the anchor box, is the feature map output by the Transformer encoder, is the number of selected feature points, is the normalized attention weight, is the transformation matrix used to obtain the attention value, is the feature map On point The feature code at represents the sampling position offset learned relative to the center position, To standardize operations, and is the original sampling position offset in the X and Y axis directions, represent the width and height of the anchor box respectively.

5. The Transformer-based end-to-end cell segmentation and tracking method according to claim 4, characterized in that The improved model based on the Mask DINO model includes an improvement to the segmentation branch for generating segmentation masks in the classification head of the Mask DINO model: ， In the above formula, query embeddings for content represented by feature embeddings, is the position query embedding represented by the anchor box, is element-wise multiplication, is the split head of the split branch, is the convolutional layer, is a two-fold upsampling operation, is the 1 / 4 original resolution feature map from the CNN backbone network, is the output feature map from the Transformer encoder upsampled to 1 / 8 of the original resolution, is the cosine position embedding of the pixel embedding map, which is a 1 / 4 original resolution feature map from the CNN backbone network. and the output feature map from the Transformer encoder upsampled to 1 / 8 the original resolution Add together to get.

6. The Transformer-based end-to-end cell segmentation and tracking method according to claim 1, characterized in that The distance between the centroids in step S3 may cause cell division, which means judging whether the distance between the centroids is less than a preset threshold t d , if it is less than the preset threshold t d It is judged that it may cause cell division, otherwise it is judged that it will not cause cell division.

7. The Transformer-based end-to-end cell segmentation and tracking method according to claim 1, wherein: Before using the improved model based on the Mask DINO model to implement cell segmentation and tracking, the method also includes training an end-to-end cell segmentation and tracking model based on Transformer, and the function expression of the loss function used in training the end-to-end cell segmentation and tracking model based on Transformer is: ， In the above formula, is the loss function, is the number of queries, each query is responsible for detecting a cell in the current frame t, and the queries include object queries Q object , Tracking Query Q track and split query Q mitosis Three types, is the classification loss, is the frame loss, is the segmentation loss, Is the label in the real label data GT of the current frame t Matching queries, Indicates that the query is successfully matched with the label in the real label data GT in the Hungarian algorithm Match, is the classification probability, is the rotation bounding box, is the segmentation mask, is the label index of the rotated bounding box, is the label index of the segmentation mask, For the i-th query The corresponding prediction output is: ， In the above formula, and is the weight parameter, is the exponential parameter; ， In the above formula, and is the weight parameter, for The L1 loss between for KFIoU loss between; ， In the above formula, and is the weight parameter, for The cross entropy loss between for The dice loss between.

8. A Transformer-based end-to-end cell segmentation and tracking system, comprising an interconnected microprocessor and a memory, characterized in that: The microprocessor is programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instructions are programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method according to any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instructions are programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method according to any one of claims 1 to 7 through a processor.

Images