Transform-based end-to-end cell segmentation and tracking method and system

Abstract

The invention discloses a Transform-based end-to-end cell segmentation and tracking method and system, and the method comprises the steps: employing an improved model based on a Mask DINO model to achieve cell segmentation and tracking, and employing a top-K query selection strategy to select a K feature point with the highest probability from an output feature map for object query, taking the track from the previous frame t-1 as a tracking query in the current frame t; additionally generating a split query representing mitosis; decoding the object query, the tracking query and the split query by using a Transform decoder; and performing prediction by using a classification head to obtain a classification label, a rotating bounding box and a segmentation mask of a cell in the image I of the current frame t. The invention aims to solve challenges in a cell segmentation and tracking scene and improve the accuracy of cell segmentation and tracking.

Description

Technical Field

[0001] The present invention relates to the technical field of cell segmentation and tracking, and in particular to a Transformer-based end-to-end cell segmentation and tracking method and system. Background Art

[0002] Cell segmentation and tracking are fundamental challenges in biomedical applications, essential for understanding cell morphology, structure, and function, and fundamental to life science research. With the success of deep learning in image segmentation and object detection, researchers are increasingly applying these techniques to cell tracking.

[0003] Most existing cell segmentation and tracking methods utilize deep neural networks such as U-Net for semantic segmentation. Subsequently, instance segmentation is handled by marker-based watershed algorithms and other techniques, followed by cell association through methods such as nearest neighbor linking and graph matching. Despite the progress made by neural networks, the reliance on traditional pre- and post-processing methods has also exposed significant limitations. This reliance has led to complex and inefficient workflows, hindering the development of seamless end-to-end systems. The fragmented process not only limits the potential for integration and automation of the process, but also introduces error propagation in multiple steps, further reducing the overall performance.

[0004] Recently developed Transformer-based DETR models have gained popularity due to their simplicity, efficiency, and end-to-end trainability, and have made significant progress in object detection and tracking tasks. As an end-to-end detector, DETR does not require handcrafted modules and outperforms traditional detectors. In multi-object tracking, Transformer-based methods introduce tracking queries to regress object locations, thereby omitting the data association step and avoiding the complexity of additional graph optimization or motion and appearance modeling. In addition, thanks to the attention mechanism, these methods can effectively understand the complex relationship between objects and the environment, achieving state-of-the-art (SOTA) results on multiple multi-object tracking benchmarks. Despite the success of Transformer-based models in object detection and tracking, their application in cell tracking remains unexplored, which is attributed to several challenges: (1) Cell tracking is a comprehensive task that includes not only detection and tracking, but also segmentation. (2) Cells are diverse in shape and densely distributed. The same cell can show significant morphological changes during tracking. This diversity poses a challenge to Transformer-based models, which may find it difficult to effectively capture and represent such changes. (3) Specific cellular behaviors, such as cell division, introduce additional complexity to the tracking process and require specialized techniques to address these unique dynamics. Summary of the invention

[0005] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a Transformer-based end-to-end cell segmentation and tracking method and system are provided. The present invention aims to solve the challenges in cell segmentation and tracking scenarios and improve the accuracy of cell segmentation and tracking.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: A Transformer-based end-to-end cell segmentation and tracking method, comprising: implementing cell segmentation and tracking using an improved model based on a Mask DINO model; the improved model based on the Mask DINO model and the Mask DINO model both include a CNN backbone network, a Transformer encoder, a Transformer decoder, and a classification head; the improved model implements cell segmentation and tracking including: S1, use the CNN backbone network to extract multi-scale feature maps from the image I of the current frame t; S2, uses the Transformer encoder to refine the multi-scale feature map and extract the output feature map; S3, using the top-K query selection strategy to select the K feature points with the highest probability in the output feature map as object queries, and using the trajectory from the previous frame t-1 as the tracking query in the current frame t; calculating the distance between the centroids of the object query and the tracking query, and if the distance between the centroids may lead to cell division, additionally generating a division query representing mitosis; the object query, tracking query, and division query all contain a content query embedding represented by a feature embedding and a location query embedding represented by an anchor box; S4, uses the Transformer decoder to decode object queries, tracking queries, and split queries; S5, using the classification head to predict the classification label, rotation bounding box and segmentation mask of the cells in the image I of the current frame t according to the decoding results output by the Transformer decoder.

[0007] Optionally, the format of the anchor box is defined as ,in represents the anchor box, represents the center position of the anchor box, Respectively represent the width and height of the anchor box, represents the rotation angle of the anchor frame; the function expression of the position query embedding represented by the anchor frame is: , In the above formula, is the location query embedding represented by the anchor box, is the connection function, is the cosine position code, It is a multi-layer perceptron.

[0008] Optionally, the improved model based on the Mask DINO model includes position attention adopted by the mutual attention layer in the Transformer decoder layer of the Mask DINO model: , The reference point of the encoder feature map Improved to the following improved reference point : , , in, is the position attention, represents the center position of the anchor box, is the reference point of the encoder feature map, is the cosine position code, is the scaling result value.

[0009] Optionally, the improved model based on the Mask DINO model includes a cross attention adopted in a mutual attention layer in a Transformer decoder layer of the Mask DINO model: , The sample positions learned relative to the center position are offset The improved sampling position offset is: : , , in, For queries with location priors Cross-attention, For query The center position of the anchor box, is the feature map output by the Transformer encoder, is the number of selected feature points, is the normalized attention weight, The transformation matrix is ​​used to obtain the attention value, The feature map On point The feature encoding at represents the sampling position offset learned relative to the center position, To standardize operations, and Also the original sampling position offset in the X and Y axis directions, Represent the width and height of the anchor box respectively.

[0010] Optionally, the improved model based on the Mask DINO model includes improving a segmentation branch for generating a segmentation mask in a classification head of the Mask DINO model as follows: , In the above formula, Query embeddings for content represented by feature embeddings, is the location query embedding represented by the anchor box, is element-wise multiplication, is the split head of the split branch, is the convolutional layer, is a two-fold upsampling operation, is the 1 / 4 original resolution feature map from the CNN backbone network, is the output feature map from the Transformer encoder upsampled to 1 / 8 of the original resolution, is the cosine position embedding of the pixel embedding map, which is a 1 / 4 original resolution feature map from the CNN backbone network. and the output feature map from the Transformer encoder upsampled to 1 / 8 the original resolution Add together.

[0011] Optionally, the distance between the centroids in step S3 may lead to cell division, which refers to determining whether the distance between the centroids is less than a preset threshold value t d , if the preset threshold t d It is judged that it may cause cell division, otherwise it is judged that it will not cause cell division.

[0012] Optionally, before the improved model based on the Mask DINO model is used to implement cell segmentation and tracking, it also includes training an end-to-end cell segmentation and tracking model based on Transformer, and the function expression of the loss function used when training the end-to-end cell segmentation and tracking model based on Transformer is: , In the above formula, is the loss function, is the number of queries, each of which is responsible for detecting a cell in the current frame t. The queries include object queries Q object , Tracking Query Q track and split query Q mitosis Three types, is the classification loss, is the frame loss, is the segmentation loss, Is it the label in the real label data GT of the current frame t? Matching queries, Indicates that the query is successfully matched in the Hungarian algorithm with the label in the real label data GT Match, is the classification probability, is the rotation bounding box, is the segmentation mask, is the label index of the rotated bounding box, is the label index of the segmentation mask, For the i-th query The corresponding prediction output is: , In the above formula, and is the weight parameter, is the index parameter; , In the above formula, and is the weight parameter, for The L1 loss between for KFIoU loss between; , In the above formula, and is the weight parameter, for The cross entropy loss between for The dice loss between.

[0013] In addition, the present invention also provides a Transformer-based end-to-end cell segmentation and tracking system, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method.

[0014] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method through a processor.

[0015] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method through a processor.

[0016] Compared with the prior art, the present invention mainly has the following advantages: the present invention includes adopting an improved model based on the Mask DINO model to realize cell segmentation and tracking, including using a top-K query selection strategy to select K feature points with the highest probability in the output feature map for use as object query, and using the trajectory from the previous frame t-1 as the tracking query in the current frame t; additionally generating a split query representing mitosis; using a Transformer decoder to decode the object query, tracking query and split query; using a classification head to predict the classification label, rotated bounding box and segmentation mask of the cell in the image I of the current frame t. The present invention can solve the challenges in cell segmentation and tracking scenarios and improve the accuracy of cell segmentation and tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the basic flow of the method of the embodiment of the present invention.

[0018] Figure 2 Schematic diagram of the structure of the improved model (Cell DINO) in an embodiment of the present invention.

[0019] Figure 3 : The figures are cell tracking examples of the horizontal bounding box (HBB) and the rotated bounding box (RBB) in the embodiments of the present invention, wherein (a) is a cell tracking example of the position attention map of the horizontal bounding box (HBB); (b) is a cell tracking example of the horizontal bounding box (HBB) and its position attention map in frame t; (c) is a cell tracking example of the position attention map of the rotated bounding box (RBB); (d) is a cell tracking example of the rotated bounding box (RBB) and its position attention map in frame t; (e) is a cell tracking example of the rotated bounding box (RBB) and its position attention map in frame t+1; (f) is a cell tracking example of the rotated bounding box (RBB) and its position attention map in frame t+2; (g) is a cell tracking example of the rotated bounding box (RBB) and its position attention map in frame t+3.

[0020] Figure 4 Schematic diagram for comparing different cell segmentation methods in an embodiment of the present invention, wherein (a) is the result of merging multiple cells using content query for comparison, and (b) is the result of improving segmentation using content and position query in this embodiment.

[0021] Figure 51 is an example of multiple selected data sets and corresponding Cell DINO segmentation results in an embodiment of the present invention, wherein (a) is the original image on the Fluo-N2DH-GOWT1 data set, (b) is the original image on the PhC-C2DH-U373 data set, (c) is the original image on the DIC-C2DH-HeLa data set, (d) is the mask annotation of the original image (a) on the Fluo-N2DH-GOWT1 data set, and (e) is the mask annotation of the original image (b) on the PhC-C2DH-U373 data set.

[0022] Figure 6 1 is the training curve of the horizontal bounding box (HBB) and the rotation bounding box (RBB) on the Fluo-N2DH-GOWT1 dataset in an embodiment of the present invention.

[0023] Figure 7 ] are training curves of the horizontal bounding box (HBB) and the rotation bounding box (RBB) on the PhC-C2DH-U373 dataset in an embodiment of the present invention.

[0024] Figure 8 ] are training curves of the horizontal bounding box (HBB) and the rotation bounding box (RBB) on the DIC-C2DH-HeLa dataset in an embodiment of the present invention.

[0025] Fig. 9 is the preset threshold value t for different mitosis detections in the embodiment of the present invention d Impact on tracking performance. DETAILED DESCRIPTION

[0026] The end-to-end cell segmentation and tracking method based on Transformer in this embodiment includes using an improved model based on the MaskDINO model (named Cell DINO in this embodiment) to achieve cell segmentation and tracking. The improved model based on the Mask DINO model and the Mask DINO model both include a CNN backbone network, a Transformer encoder, a Transformer decoder, and a classification head. Figure 1 and Figure 2 As shown, the improved model (Cell DINO) realizes cell segmentation and tracking including: S1, use the CNN backbone network to extract multi-scale feature maps from the image I of the current frame t; S2, uses the Transformer encoder to refine the multi-scale feature map and extract the output feature map; S3, using the top-K query selection strategy to select the K feature points with the highest probability in the output feature map as object queries, using the trajectory from the previous frame t-1 as the tracking query in the current frame t to calculate the distance between the centroids of the object query and the tracking query, and if the distance between the centroids may lead to cell division, an additional division query representing mitosis is generated; the object query, tracking query, and division query all contain a content query embedding represented by a feature embedding and a location query embedding represented by an anchor box; S4, uses the Transformer decoder to decode object queries, tracking queries, and split queries; S5, using the classification head to predict the classification label, rotation bounding box and segmentation mask of the cells in the image I of the current frame t according to the decoding results output by the Transformer decoder.

[0027] In order to adapt to the scene with diverse cell shapes and dense distribution, this embodiment uses a rotated bounding box (RBB) for cell detection. Specifically, the format of the anchor box in this embodiment is defined as ,in represents the anchor box, represents the center position of the anchor box, Represent the width and height of the anchor box respectively, represents the rotation angle of the anchor frame; the function expression of the position query embedding represented by the anchor frame is: , In the above formula, is the location query embedding represented by the anchor box, is the connection function, is the cosine position code, For the three types of queries, object query, tracking query and split query, each query q will generate a corresponding anchor box, corresponding to a final generated rotation bounding box and segmentation mask. In this embodiment, the rotation angle of the anchor box is added to the format definition of the anchor box. , so that the rotation angle of the anchor frame can be Positional attention in the Transformer decoder layer is introduced to better adapt to cells of different shapes. Figure 3: The figures are cell tracking examples of the horizontal bounding box (HBB) and the rotated bounding box (RBB) in the embodiments of the present invention, wherein (a) is a cell tracking example of the position attention map of the horizontal bounding box (HBB); (b) is a cell tracking example of the horizontal bounding box (HBB) and its position attention map in frame t; (c) is a cell tracking example of the position attention map of the rotated bounding box (RBB); (d) is a cell tracking example of the rotated bounding box (RBB) and its position attention map in frame t; (e) is a cell tracking example of the rotated bounding box (RBB) and its position attention map in frame t+1; (f) is a cell tracking example of the rotated bounding box (RBB) and its position attention map in frame t+2; (g) is a cell tracking example of the rotated bounding box (RBB) and its position attention map in frame t+3. Figure 3 Both position attention maps (a) and (c) are computed by taking the dot product between the position query and the position embedding in the feature map.

[0028] In this embodiment, object query, tracking query and split query all include a content query embedding represented by a feature embedding and a location query embedding represented by an anchor box. Figure 4 Schematic diagram for comparing different cell segmentation methods, where (a) uses content query to merge multiple cells, and (b) uses content and position query to improve segmentation results in this embodiment. The position query embedding represented by an anchor box can improve the query accuracy.

[0029] In this embodiment, the improved model based on the Mask DINO model includes the position attention adopted by the mutual attention layer in the Transformer decoder layer of the Mask DINO model: , The reference point of the encoder feature map Improved to the following improved reference point : , , in, is the position attention, represents the center position of the anchor box, is the reference point of the encoder feature map, is the cosine position code, To scale the resulting values, a more appropriate position attention that is more suitable for the cell shape can be obtained through the above-mentioned transformation of the reference point positions in the encoder feature map.

[0030] In this embodiment, the improved model based on the Mask DINO model includes the cross attention adopted in the mutual attention layer in the Transformer decoder layer of the Mask DINO model: , The sample positions learned relative to the center position are offset The improved sampling position offset is: : , , in, For queries with location priors Cross-attention, For query The center position of the anchor box, is the feature map output by the Transformer encoder, is the number of selected feature points, is the normalized attention weight, The transformation matrix is ​​used to obtain the attention value, The feature map On point The feature encoding at represents the sampling position offset learned relative to the center position, To standardize operations, and Also the original sampling position offset in the X and Y axis directions, Represent the width and height of the anchor box respectively. Through the above steps, the deformable attention mechanism is used to accelerate the convergence of the improved model (Cell DINO).

[0031] As a unified model for image segmentation, Mask DINO constructs a pixel embedding map to achieve mask classification for all segmentation tasks. The pixel embedding map is obtained by integrating the 1 / 4 original resolution feature map from the CNN backbone network. Output feature map from Transformer encoder upsampled to 1 / 8 original resolution Then, the embedding is queried by looking at the content represented in the feature embeddings The output mask is obtained by performing a dot product operation between the pixel embedding map and the pixel embedding map, as shown in the following formula: , In the above formula, is the segmentation mask obtained by the segmentation branch, Query embeddings for content represented by feature embeddings, is element-wise multiplication, is the split head of the split branch, is the convolutional layer, is a two-fold upsampling operation, is the 1 / 4 original resolution feature map from the CNN backbone network, is the output feature map from the Transformer encoder upsampled to 1 / 8 of the original resolution. Figure 4 It can be seen that in cell instance segmentation applications, adjacent cells are often incorrectly segmented into a single entity. In this embodiment, it is assumed that this problem stems from the semantic nature of content queries, which prioritizes cell semantics rather than precise spatial details. In order to alleviate this challenge, this embodiment integrates location information into the segmentation process. Therefore, the improved model based on the Mask DINO model in this embodiment includes an improvement to the segmentation branch for generating segmentation masks in the classification head of the Mask DINO model as follows: , In the above formula, Query embeddings for content represented by feature embeddings, is the location query embedding represented by the anchor box, is element-wise multiplication, is the split head of the split branch, is the convolutional layer, is a two-fold upsampling operation, is the 1 / 4 original resolution feature map from the CNN backbone network, is the output feature map from the Transformer encoder upsampled to 1 / 8 of the original resolution, is the cosine position embedding of the pixel embedding map, which is a 1 / 4 original resolution feature map from the CNN backbone network. and the output feature map from the Transformer encoder upsampled to 1 / 8 the original resolution Add together.

[0032] In this embodiment, the distance between the centroids may lead to cell division in step S3, which means judging whether the distance between the centroids is less than a preset threshold value t d , if the preset threshold t dThen the judgment may lead to cell division, otherwise the judgment will not lead to cell division. In models like DETR, the object query is responsible for detecting objects in the current frame, but it is impossible to track objects across video sequences. Therefore, a tracking query carrying identity information is needed to track objects in video sequences. The tracking query is initialized using the embedding and position information of the object detected in the previous frame t-1. However, each tracking query can only track a single object and cannot handle cell division events. To solve this problem, this embodiment proposes the concept of mitosis query to manage cell division scenarios. During the training process of Cell DINO, this embodiment uses iterative prediction to optimize the output of each layer in the decoder. This process includes the input of the decoder, which corresponds to the output of the last layer of the encoder. Therefore, even the initially assigned object queries only contain rough detection results. As Figure 3 As shown, this embodiment selects K object queries Q from the encoder object , and then further filter out high confidence object queries Q′ object As a potential detection. Then, this embodiment calculates Q' object and trajectory query Q track The distance between the centroids. object For each query in , this embodiment only retains the association with the nearest trajectory query. If a cell (trajectory query) is associated with two object queries, and the centroid distance between the trajectory query and each object query is less than a preset threshold t d , it is considered likely to divide. For each cell that may divide (trajectory query), the present embodiment generates a corresponding mitosis query. The content of the mitosis query is aligned with the content query of the associated trajectory query. Then, the present embodiment uses the location information of the two object queries linked to the trajectory query to update the location query of the trajectory and mitosis query. Subsequently, the detection, tracking, and mitosis query are uniformly input into the Transformer decoder for processing. Finally, the classification head is used to determine whether the division finally occurs.

[0033] In order to promote effective interaction between trajectory query and object query, this embodiment implements an end-to-end training method by randomly sampling adjacent frames of unit DINO. At the current frame t, this embodiment generates N = N object + N track queries, where N object is the number of object queries (detection queries), N track Query Q for the trajectory track The number of N trackdepends on the detection state from t−k to frame t−1. This embodiment ignores the consideration of split scenes during training. Each query Produces a prediction output ,in is the classification probability, is the rotation bounding box, is the segmentation mask.

[0034] First, this embodiment uses trajectory identification to match predictions (expressed as ): , and the true label data GT (expressed as ): , in, is the prediction result of the i-th test data, N is the number of test data; in the above formula, is the true label of the i-th sample, is the label of the classification probability of the i-th sample, is the label of the rotated bounding box of the i-th sample, is the label of the segmentation mask of the i-th sample, is the number of samples of the true label data GT. Then, hybrid matching is used to match the remaining predictions and the true label data GT. This embodiment assumes Representation and prediction results Matching real label data GT, index Represents the best matching arrangement of labels. When this embodiment jointly trains detection, segmentation and tracking tasks, this embodiment uses a combined loss function To train Cell DINO. In this embodiment, before using the improved model based on the Mask DINO model to implement cell segmentation and tracking, it also includes training an end-to-end cell segmentation and tracking model based on Transformer, and the function expression of the loss function used when training the end-to-end cell segmentation and tracking model based on Transformer is: , In the above formula, is the loss function, is the number of queries, each of which is responsible for detecting a cell in the current frame t. The queries include object queries Q object , Tracking Query Q track and split query Q mitosis Three types, is the classification loss, is the frame loss, is the segmentation loss, Is it the label in the real label data GT of the current frame t? Matching queries, Indicates that the query is successfully matched in the Hungarian algorithm with the label in the real label data GT Match, is the classification probability, is the rotation bounding box, is the segmentation mask, is the label index of the rotated bounding box, is the label index of the segmentation mask, For the i-th query The corresponding prediction output is: , In the above formula, and is the weight parameter, is an index parameter; as an optional implementation, in this embodiment , and ; , In the above formula, and is the weight parameter, for The L1 loss between for As an optional implementation, in this embodiment and ; , In the above formula, and is the weight parameter, for The cross entropy loss between for As an optional implementation, in this embodiment and .

[0035] In order to verify the effectiveness, robustness and feasibility of the end-to-end cell segmentation and tracking method based on Transformer in this example, this example selected three datasets from the cell tracking challenge: Fluo-N2DH-GOWT1, PhC-C2DH-U373 and DIC-C2DH-HeLa. Figure 5As shown, (a) is the original image on the Fluo-N2DH-GOWT1 dataset, (b) is the original image on the PhC-C2DH-U373 dataset, (c) is the original image on the DIC-C2DH-HeLa dataset, (d) is the mask annotation of the original image (a) on the Fluo-N2DH-GOWT1 dataset, (e) is the mask annotation of the original image (b) on the PhC-C2DH-U373 dataset, and (f) is the mask annotation of the original image (c) on the DIC-C2DH-HeLa dataset. For training and testing purposes, each dataset contains two sequences, labeled 01 and 02, in the training set and test set, respectively. The details of the selected datasets are as follows: (1) The Fluo-N2DH-GOWT1 dataset contains 368 images of GFP-transfected GOWT1 mouse embryonic stem cells, each with a size of 1024×1024 pixels. The main challenges of this dataset are the low contrast of some cells and the limited number of cells that axially pass through the imaging area. (2) The PhC-C2DH-U373 dataset contains 460 images depicting glioblastoma-gastric U373 cell samples, each with a size of 636×520 pixels. This dataset poses significant challenges as the cell shapes are highly variable and in some cases have a similar appearance to the background. (3) The DIC-C2DH-HeLa dataset contains 398 images of HeLa cells on flat glass, each with a size of 512×512 pixels. A significant difficulty inherent in this dataset is that the cells are densely distributed and have variable morphologies. For the training sequences, both high-quality (GoldTruth) and sub-high-quality (SilverTruth) annotations are included. The high-quality (GoldTruth) annotations are manually annotated and include tracking information using point-by-point annotations as well as segmentation annotations for some cells of interest. On the other hand, the sub-high-quality annotations provide segmentation annotations for all images, which are obtained by fusing the segmentation results of high-performance methods previously demonstrated on the cell tracking challenge CTC (cell tracking challenge). Since the high-quality annotations only contain segmentation annotations for some cells, this embodiment refers to the existing network model EmbedTrack during training. By merging the segmentation masks in the sub-high-quality annotations with the point-by-point tracking annotations in the high-quality annotations, this embodiment creates fully labeled training data with reasonable annotation accuracy.

[0036] The improved model Cell DINO used in this embodiment follows the basic architecture of ResNet50 CNN feature extraction and Transformer encoder-decoder of Mask DINO, which is the basis for improvement. During training, for each training set, this embodiment allocates 90% of the images for training and the remaining images for validation. The model weights of this embodiment are initialized using the best performing Mask DINO model on the COCO dataset. Subsequently, for each dataset, this embodiment trains the model for 40,000 iterations with a batch size of 1. The initial learning rate is set to 1×10 -4 , and then decreased by a factor of 0.1 at 15,000 and 30,000 iterations. This embodiment also uses a variety of data augmentation techniques, including flipping, cropping, scaling, Gaussian noise, grayscale mapping, and image blur. In addition, this embodiment maintains consistency with the training hyperparameters outlined in Mask DINO. All experiments were performed on PyTorch1.8.1, using an NVIDIA GeForce RTX 2080 Ti GPU, running on Ubuntu 18.04 LTS (64-bit). During the tracking process, this embodiment filters out detections with confidence scores below 0.2 for newly identified cells. For cells that have already been tracked, cells with detection confidence scores below 0.1 are temporarily considered lost. To ensure the continuity of cell trajectories and prevent premature termination, when they are lost, this embodiment retains their trajectories for five frames and sets k =5. When processing cell division events, this embodiment sets a preset threshold t for the Fluo-N2DH-GOWT1 dataset. d = 0.01, set the preset threshold t for DIC-C2DH-HeLa d =0.11. Since there are few cell division events in the PhC-C2DH-U373 dataset, this example sets a preset threshold t d = 0 and the cell division events in this data set are not considered. In addition, this embodiment uses the indicators provided by the cell tracking challenge CTC to evaluate the algorithm of this embodiment. The cell tracking challenge CTC provides four indicators: detection accuracy DET, segmentation accuracy SEG, tracking accuracy TRA and tracking overall performance OP CTB , the value is between [0,1]. The higher the value, the better the performance. Detection accuracy DET is used to evaluate the accuracy of cell detection, and its function expression is as follows: , In the above formula, is the normalized directed acyclic graph matching, is the cost of transforming the computed nodes into the reference node set, The cost of creating a reference node set from scratch.

[0037] Segmentation accuracy SEG evaluates segmentation by comparing the intersection and union between the true value and the output mask. Its function expression is as follows: , In the above formula, Indicates intersection, Indicates mutuality, is the true value, is the output mask.

[0038] Tracking accuracy TRA is used to evaluate the quality of tracking results, and its function expression is as follows: , In the above formula, AOGM values ​​required to create a reference image from scratch.

[0039] Tracking overall performance OP CTB It is used to evaluate the overall performance of the method. Its function expression is as follows: .

[0040] In order to evaluate the performance of the method of this embodiment, the method of this embodiment and other existing methods (including KTHG-SE, AC, KIT-GE, MU-US, MU-CZ, BGU-IL, UVA-NL and PR-GE) were tested on the CTC cell tracking benchmark (CTB). The evaluation results on CTB are shown in Table I, which lists the comprehensive performance of tracking OP in detail. CTB , the scores, rankings and the gap with the first place on segmentation accuracy SEG and tracking accuracy TRA.

[0041] Table I: Results of Cell Tracking Benchmark (CTB)

[0042] As shown in Table 1, the method of this embodiment performs very well on the DIC-C2DH-HeLa dataset, and has a good tracking performance. CTB This success can be attributed to the dense and diverse distribution of cells in this dataset, which is very suitable for rotated bounding box (RBB) detection. In addition, on the Fluo-N2DH-GOWT1 dataset, although it ranks 6th in segmentation accuracy SEG and 5th in tracking accuracy TRA, due to the balanced performance of the improved model Cell DINO in segmentation and tracking, this embodiment has a better overall tracking performance than OP. CTBIt took second place in the . On the PhC-C2DH-U373 dataset, the performance of this embodiment is relatively average, mainly for two reasons: the lack of cell division events, which limits the ability of the method to handle mitosis; the cells are sparsely distributed, and the advantage of rotated bounding box (RBB) detection over horizontal bounding box (HBB) detection is small. Despite this, the results of the method in this embodiment are still impressive, with a small performance gap compared to the best performing methods. Overall, Cell DINO is suitable for a variety of cell tracking scenarios. In contrast, other methods (such as KIT-GE) perform well on the Fluo-N2DH-GOWT1 and PhC-C2DH-U373 datasets, but struggle when dealing with dense cells in the DIC-C2DH-HeLa dataset.

[0043] In order to analyze the impact of each component of the method in this embodiment on the model performance and select key parameters, this embodiment conducted a series of experiments on the training set and the validation set. Figure 6 , Figure 7 and Figure 8 In this embodiment, experiments are conducted using horizontal bounding box (HBB) and rotated bounding box (RBB) configurations on three different training sets, and the performance indicators on the validation set are monitored every 1000 iterations, where HBB_DET represents the curve of the detection accuracy DET of the horizontal bounding box, HBB_SEG represents the curve of the segmentation accuracy SEG of the horizontal bounding box, RBB_DET represents the curve of the detection accuracy DET of the rotated bounding box, and RBB_SEG represents the curve of the segmentation accuracy SEG of the rotated bounding box. Initially, the results of this embodiment show that the horizontal bounding box exhibits superior convergence speed and effectiveness compared to the rotated bounding box. The early advantage of the horizontal bounding box can be attributed to its simpler learning process in the initial stage of training, while the rotated bounding box faces greater complexity, resulting in slower convergence. However, an interesting shift occurs after about 4000 iterations, where the rotated bounding box begins to significantly outperform the horizontal bounding box. This improvement is not only reflected in accelerated convergence, but also in obtaining higher scores in the detection accuracy DET and segmentation accuracy SEG indicators. This embodiment believes that this shift stems from the model's increasing ability to more effectively utilize rotated bounding boxes during training. Specifically, as the improved model Cell DINO progresses, it becomes increasingly adept at using rotated bounding boxes to accurately depict cell boundaries, thereby improving localization accuracy. On the PhC-C2DH-U373 dataset, rotated bounding boxes and horizontal bounding boxes show similar performance in cell detection, which may be because the cell density of this dataset is low, typically containing only 6-8 cells per frame. Table II shows the impact of position information on the segmentation performance of different datasets.

[0044] Table II: Comparison of segmentation results with and without location information

[0045] The results in Table II show that the denser the cell distribution, the greater the impact of the position information, and the DIC-C2DH-HeLa dataset has the most significant improvement. In contrast, datasets such as PhC-C2DH-U373 are characterized by sparse and widespread cell distribution, so the impact is minimal. In addition, in order to determine the preset threshold t of the mitotic distance suitable for each dataset d , this embodiment tests the training set of each data set. The test results are as follows Fig. 9 As shown. When the preset threshold t d When set to 0, no mitosis query is generated and mitosis events are not considered. Test results show that for the Fluo-N2DH-GOWT1 dataset, the preset threshold t d The best tracking performance can be obtained with a threshold of 0.01. In contrast, in the DIC-C2DH-HeLa dataset, the preset threshold t d The best performance is achieved when t is 0.11. It is worth noting that increasing the number of mitotic queries in the PhC-C2DH-U373 dataset leads to a decrease in tracking performance, which may be due to the extremely low incidence of mitotic events in this dataset. In addition, if the preset threshold t d Setting it too high may misidentify neighboring cells or newly emerged cells as cells arising from mitosis, which may adversely affect cell tracking performance.

[0046] In summary, the end-to-end cell segmentation and tracking method based on Transformer in this embodiment includes using an improved model based on the Mask DINO model (named Cell DINO in this embodiment) to achieve cell segmentation and tracking. The improved model (Cell DINO) is a Transformer-based end-to-end cell tracking method based on the unified segmentation model Mask DINO. The method solves the challenge of simultaneously detecting, segmenting and tracking cells through three improvements. First, the method addresses the unique characteristics of cells compared with other objects such as pedestrians or cars. Cells exhibit a variety of shapes, making traditional horizontal bounding box (HBB) detection methods ineffective. Typically, each horizontal bounding box (HBB) may contain multiple cells or significant background noise. To overcome this limitation, the method of this embodiment recommends using rotated bounding boxes (RBBs) for detection because they can better align with cell shapes and reduce noise interference. This embodiment uses rotated bounding boxes (RBBs) for cell detection and segmentation, effectively adapting to different cell shapes while minimizing interference from background elements and neighboring cells. In addition, normalizing the position query to align with the RBB provides more accurate spatial prior information for cross attention in the Transformer decoder. In addition, the method of this embodiment introduces a 5-dimensional representation of location query , which provides accurate spatial priors for the cross-attention module, enabling queries to focus on local areas of corresponding objects. Secondly, inspired by the Transformer-based tracking model in MOT (Multi-Object Tracking), the method of this embodiment uses tracking queries for cell tracking. Third, in order to handle the mitotic behavior of cells, this additional query ensures accurate tracking results even in scenes involving cell division. The method of this embodiment introduces a mitotic query to determine whether cells undergoing potential division actually divide during tracking. It can effectively identify cell division events during cell tracking. This new method improves the accuracy and robustness of cell tracking by specifically solving the challenging task of detecting cell division behavior. This embodiment verifies the effectiveness of Cell DINO in the Cell Tracking Challenge (CTC) and achieves excellent performance on multiple datasets, and performs well in segmenting and tracking cells in microscope images. In particular, the method of this embodiment achieved first place on the DIC-C2DH-HeLa dataset with variable cell morphology and dense distribution, and second place on the Fluo-N2DH-GOWT1 dataset. Experimental results on the Cell Tracking Challenge dataset and related studies convincingly demonstrate the effectiveness of the Transformer-based end-to-end cell segmentation and tracking method of this embodiment.

[0047] In addition, this embodiment also provides a Transformer-based end-to-end cell segmentation and tracking system, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method.

[0048] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method through a processor.

[0049] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method through a processor.

[0050] Those skilled in the art should understand that the technical solutions provided by the embodiments of the present application may be in the form of methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0051] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A Transformer-based end-to-end cell segmentation and tracking method, characterized in that: The method comprises adopting an improved model based on the Mask DINO model to realize cell segmentation and tracking. The improved model based on the Mask DINO model and the Mask DINO model both include a CNN backbone network, a Transformer encoder, a Transformer decoder and a classification head. The improved model realizes cell segmentation and tracking including: S1, use the CNN backbone network to extract multi-scale feature maps from the image I of the current frame t; S2, uses the Transformer encoder to refine the multi-scale feature map and extract the output feature map; S3, using the top-K query selection strategy to select the K feature points with the highest probability in the output feature map as object queries, and using the trajectory from the previous frame t-1 as the tracking query in the current frame t; calculating the distance between the centroids of the object query and the tracking query, and if the distance between the centroids may lead to cell division, additionally generating a division query representing mitosis; the object query, tracking query, and division query all contain a content query embedding represented by a feature embedding and a location query embedding represented by an anchor box; S4, uses the Transformer decoder to decode object queries, tracking queries, and split queries; S5, using the classification head to predict the classification label, rotation bounding box and segmentation mask of the cells in the image I of the current frame t according to the decoding results output by the Transformer decoder.

2. The Transformer-based end-to-end cell segmentation and tracking method according to claim 1, characterized in that: The format of the anchor box is defined as ,in represents the anchor box, represents the center position of the anchor box, Respectively represent the width and height of the anchor box, represents the rotation angle of the anchor frame; the function expression of the position query embedding represented by the anchor frame is: ， In the above formula, is the location query embedding represented by the anchor box, is the connection function, is the cosine position code, It is a multi-layer perceptron.

3. The Transformer-based end-to-end cell segmentation and tracking method according to claim 2, characterized in that: The improved model based on the Mask DINO model includes position attention adopted by the mutual attention layer in the Transformer decoder layer of the Mask DINO model: ， The reference point of the encoder feature map Improved to the following improved reference point : ， ， in, is the position attention, represents the center position of the anchor box, is the reference point of the encoder feature map, is the cosine position code, is the scaling result value.

4. The Transformer-based end-to-end cell segmentation and tracking method according to claim 3, characterized in that: The improved model based on the Mask DINO model includes the cross attention adopted in the mutual attention layer in the Transformer decoder layer of the Mask DINO model: ， The sample positions learned relative to the center position are offset The improved sampling position offset is: : ， ， in, For queries with location priors Cross-attention, For query The center position of the anchor box, is the feature map output by the Transformer encoder, is the number of selected feature points, is the normalized attention weight, The transformation matrix is ​​used to obtain the attention value, The feature map On point The feature encoding at represents the sampling position offset learned relative to the center position, To standardize operations, and Also the original sampling position offset in the X and Y axis directions, Represent the width and height of the anchor box respectively.

5. The Transformer-based end-to-end cell segmentation and tracking method according to claim 4, characterized in that: The improved model based on the Mask DINO model includes improving the segmentation branch for generating the segmentation mask in the classification head of the Mask DINO model as follows: ， In the above formula, Query embeddings for content represented by feature embeddings, is the location query embedding represented by the anchor box, is element-wise multiplication, is the split head of the split branch, is the convolutional layer, is a two-fold upsampling operation, is the 1 / 4 original resolution feature map from the CNN backbone network, is the output feature map from the Transformer encoder upsampled to 1 / 8 of the original resolution, is the cosine position embedding of the pixel embedding map, which is a 1 / 4 original resolution feature map from the CNN backbone network. and the output feature map from the Transformer encoder upsampled to 1 / 8 the original resolution Add together.

6. The Transformer-based end-to-end cell segmentation and tracking method according to claim 1, characterized in that: The distance between the centroids in step S3 may lead to cell division, which means judging whether the distance between the centroids is less than a preset threshold value t d , if the preset threshold t d It is judged that it may cause cell division, otherwise it is judged that it will not cause cell division.

7. The Transformer-based end-to-end cell segmentation and tracking method according to claim 1, characterized in that: Before the improved model based on the Mask DINO model is used to realize cell segmentation and tracking, it also includes training an end-to-end cell segmentation and tracking model based on Transformer, and the function expression of the loss function used in training the end-to-end cell segmentation and tracking model based on Transformer is: ， In the above formula, is the loss function, is the number of queries, each of which is responsible for detecting a cell in the current frame t. The queries include object queries Q object , Tracking Query Q track and split query Q mitosis Three types, is the classification loss, is the frame loss, is the segmentation loss, Is it the label in the real label data GT of the current frame t? Matching queries, Indicates that the query is successfully matched in the Hungarian algorithm with the label in the real label data GT Match, is the classification probability, is the rotation bounding box, is the segmentation mask, is the label index of the rotated bounding box, is the label index of the segmentation mask, For the i-th query The corresponding prediction output is: ， In the above formula, and is the weight parameter, is the index parameter; ， In the above formula, and is the weight parameter, for The L1 loss between for KFIoU loss between; ， In the above formula, and is the weight parameter, for The cross entropy loss between for The dice loss between.

8. A Transformer-based end-to-end cell segmentation and tracking system, comprising an interconnected microprocessor and a memory, characterized in that: The microprocessor is programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method according to any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the Transformer-based end-to-end cell segmentation and tracking method according to any one of claims 1 to 7 through a processor.

Images