A medical image cell segmentation and tracking method

Abstract

The application belongs to the technical field of image recognition segmentation, and discloses a medical image cell segmentation and tracking method, which comprises the following steps: step 1: data processing; feature extraction: the backbone part in the model is used to extract features from the preprocessed image, and the CSPDarknet structure is adopted in YOLOv8; step 3: FPN-PAN multi-scale feature fusion; step 4: Head prediction according to multi-scale features. The application realizes real-time tracking of the motion trajectory of cells by combining with a tracking algorithm such as deepsort. The method is mainly based on the YOLOv8 framework, and the Simam attention mechanism and the multi-scale proto method are adopted to optimize the model, so that the detection effect of YOLOv8 is further improved. The application can automatically complete the analysis and detection of medical images, is high in convenience and easy to use.

Description

Technical Field

[0001] This invention belongs to the field of image recognition and segmentation technology, and particularly relates to a method for segmenting and tracking cells in medical images. Background Technology

[0002] Imagine you have a medical image containing densely packed cells, and you need to label all the cells in that image (including the bounding boxes and their corresponding masks). Without specialized knowledge, you wouldn't be able to complete this labeling task. Even with relevant expertise, dealing with such a large number of cells would require a significant investment of time and effort. What if you were processing multiple images or even video data simultaneously? This would be an extremely challenging task. Therefore, cell segmentation in medical images is a very tedious and repetitive task; doing it manually would be incredibly time-consuming, labor-intensive, and yield poor results.

[0003] With advancements in computer technology, relying on algorithms and machine vision to accomplish this task has become increasingly common. These methods are not only highly efficient but also accurately segment cells. Even with high-resolution images and densely packed cells, the results are remarkably good, often surpassing manual annotation. Early on, this technology primarily relied on image processing algorithms such as watershed and thresholding algorithms. While these algorithms were fast, their accuracy was lacking, exhibiting oversegmentation or undersegmentation issues. With the development of deep learning technology and advancements in computing power, cell segmentation based on machine learning models has achieved remarkable results. It not only achieves good segmentation performance with high detection rates and segmentation accuracy but also maintains high efficiency.

[0004] In cell segmentation, UNet, Mask R-CNN, and Hovernet are among the more effective networks. UNet is a semantic segmentation method; to obtain segmentation results for each cell, post-processing operations such as watershed segmentation are required, making the process cumbersome and affecting detection accuracy. Mask R-CNN is a two-stage instance segmentation network with many model parameters and a complex detection process; although it has high accuracy, its speed is limited. Hovernet achieves good detection results; however, it involves numerous pre- and post-processing operations, taking approximately 20 seconds to detect a 1536*1536 medical image on an NVIDIA GeForce RTX 3090 graphics card, making it relatively slow. In object detection, the YOLO series of models, as a single-stage detection model, offers both high detection accuracy and speed, making it highly valuable in practical applications. Building upon previous experience in algorithm design, Ultralytics proposed the YOLOv8 model, including five variants: yolov8n, yolov8s, yolov8m, yolov8l, and yolov8x. These variants progressively increase in the number of parameters, leading to corresponding improvements in accuracy. Users can choose the appropriate model based on their application needs. YOLOv8 can simultaneously perform bounding box prediction, classification, and instance segmentation, enabling it to handle cell segmentation tasks. However, during experiments, we found that YOLOv8's performance on cell segmentation datasets is still not ideal, particularly exhibiting a low DICE metric.

[0005] Cell tracking is an often overlooked problem. It is an important topic in cell biology and medical image processing. The purpose of cell tracking is to locate and track the position and movement trajectory of specific cells in an image sequence, thereby revealing dynamic information about cells during physiological and pathological processes such as cell growth and division.

[0006] Currently, cell tracking technology is mainly divided into manual tracking and automatic tracking. The accuracy of manual observation depends on the observer's experience level and cannot handle large amounts of data. Automatic tracking technology has become a major research direction, effectively mining data information and automatically tracking cell position and trajectory. Machine learning techniques, such as random forests and deep learning, are increasingly being adopted for cell object detection and tracking. However, current algorithms and models are not very accurate or efficient in cell tracking. Therefore, improving accuracy and efficiency through reasonable algorithms and models has become a key research focus. Summary of the Invention

[0007] The purpose of this invention is to provide a method for medical image cell segmentation and tracking to solve the above-mentioned technical problems.

[0008] To solve the above-mentioned technical problems, the specific technical solution of the medical image cell segmentation and tracking method of the present invention is as follows:

[0009] A method for cell segmentation and tracking in medical images includes the following steps:

[0010] Step 1: Data processing: including image normalization, image denoising, image segmentation, and image enhancement;

[0011] Step 2: Feature extraction: Feature extraction is performed on the image after scaling to 640*640. The backbone of the model is used to extract features from the preprocessed image. YOLOv8 adopts the CSPDarknet structure, which is a Darknet network with a CSP structure.

[0012] Step 3: FPN-PAN multi-scale feature fusion: FPN conveys semantic features from top to bottom, and PAN conveys strong localization features from bottom to top. PAN and FPN work together to aggregate parameters from different backbone layers to different detection layers.

[0013] Step 4: Head predicts based on multi-scale features: In the detection head, the two sub-modules Proto and cv4 predict the discrimination criteria of the mask and the confidence feature matrix of the mask, respectively. Then, the output results of these two modules are multiplied by matrix to finally obtain the mask confidence matrix corresponding to each grid. After processing by the sigmoid function, this matrix is ​​the probability value of each pixel belonging to the cell.

[0014] Furthermore, step 2 uses the CSPDarknet53 backbone network to extract target features. It contains 5 CSP modules. The downsampling of each CSP module is achieved by a 3×3 convolutional kernel. The YOLOv8 network model defines the size of the input image as 640×640. After feature extraction by the five CSP models in the backbone network, the feature map size changes five times, and finally changes from 640×640 to a 20×20 feature map, thus achieving rapid dimensionality reduction of the feature map.

[0015] Furthermore, in step 3, YOLOv8 adopts the design concept of a decoupled-head, that is, classification and localization use two branches that do not interfere with each other.

[0016] Furthermore, step 4 includes the following specific steps:

[0017] Step 4.1: Set up a separate Proto for each scale, then the corresponding Proto matrix prediction for each scale can be expressed as:

[0018] protos[i](x[i])

[0019] Where x represents the feature set extracted by the model at different scales, and protos represents the set of Proto modules corresponding to different scales;

[0020] Step 4.2: Apply the Simam attention mechanism to the features of the input Segment module;

[0021] Step 4.3: Add a layer to the PAN-FPN module to bring in the output of the second layer in the backbone network, so that the model can better incorporate the low-level features;

[0022] Step 4.4: In the segmentation part, the loss function of BCE+LovaszHingeLoss is adopted;

[0023] Step 4.5: Combine YOLOv8 with the deepsort tracking algorithm to track the movement trajectory of cells.

[0024] Furthermore, in the classification branch, the binary cross-entropy loss is used, and its calculation formula is as follows:

[0025]

[0026] Where the predicted category is the same as the true category, y i It is 1, otherwise it is 0.

[0027] Furthermore, in the regression branch, CIOU+DFL is used as the loss function. CIOU is an improvement on IOU, which stands for Intersection over Union, and its calculation formula is as follows:

[0028]

[0029] We propose GIOU, DIOU, and CIOU as loss functions.

[0030] GIOU directly sets IOU as the regression loss, and its calculation formula is as follows:

[0031]

[0032] Among them, A c Let be the area of ​​the smallest closure region between the two boxes, that is, the area of ​​the smallest box that simultaneously contains both the predicted box and the ground truth box. U is the area of ​​the union region of the two boxes. This refers to the proportion of the closed region that does not belong to either of the two bounding boxes, and ultimately 1-GIOU is used as the loss function.

[0033] The formula for calculating DIOU is:

[0034]

[0035] Where, b, b gt ρ and c represent the center points of the predicted and ground truth boxes, respectively, where ρ represents the Euclidean distance between the two center points, and c represents the diagonal distance of the smallest closure region that can simultaneously contain both the predicted and ground truth boxes. The CIOU loss function, based on the GIOU and DIOU functions, also considers the aspect ratio of the three elements of bounding box regression in the calculation, adding a penalty term, as shown in the following formula:

[0036]

[0037] Where α is the weighting function, and v is used to measure the similarity of aspect ratio, defined as

[0038] The complete CIOU loss function is defined as follows:

[0039] L CfOU =1-(IOU-R) CfOU )

[0040] The gradient of CIOU loss is similar to that of DIOU loss, but the gradient of v must also be considered. When the width and height are [0,1], w... 2 +h 2 The value of is usually very small, which can lead to gradient explosion, therefore in When implemented, it will be replaced with 1;

[0041] DFL loss, short for Distance-based Focal Loss, is a loss function used for classification tasks. It's an improvement on Focal Loss and Euclidean distance, designed to better handle imbalanced classifications and uneven difficulty levels. Its formula is:

[0042]

[0043] Where y_true is the true label vector, y_pred is the predicted label vector, N is the number of samples, γ is the hyperparameter for adjusting the difficulty weights, and d is the Euclidean distance.

[0044] Furthermore, in the segmentation branch, the LovaszHingeLoss loss function is added using the binary cross-entropy loss function. The LovaszHingeLoss loss function is also a loss function used for classification tasks, designed to minimize the average ranking error in multi-label classification tasks. Its formula is defined as follows:

[0045]

[0046] Where n is the number of labels, and y is the actual label vector. For predicting the label vector, π is a permutation function, and y π(i) Indicates the position of the i-th label after sorting, l i This represents the hinge loss function at that location;

[0047] The specific hinge loss function for each label is:

[0048]

[0049] Where y i ∈{-1,1} represents the true value of the i-th label. This represents the predicted value for the i-th label.

[0050] Furthermore, step 4 includes the following specific steps:

[0051] In the Head section, i.e., the Segment module, the Simam attention mechanism is added. Simam infers 3D attention weights for the feature map without increasing the original network parameters, and calculates the minimum energy for each neuron.

[0052]

[0053] Since the lower the energy, the greater the difference between neuron t and surrounding neurons, and the higher its importance, the importance of a neuron increases through 1 / e * Based on the definition of the attention mechanism, the features are enhanced using the following formula:

[0054]

[0055] YOLOv8 has abandoned the anchor-based design method used in previous models and adopted an anchor-free approach. During training, YOLOv8 has abandoned the static positive sample allocation strategy and adopted the dynamic positive sample allocation strategy of TaskAlignedAssigner.

[0056] The matching strategy of TaskAlignedAssigner is as follows: positive samples are selected based on the weighted scores of classification and regression. For each label box, for all predicted boxes, an alignment score (alignment_metrics) is obtained by weighting the classification score corresponding to the label box category with the IOU between the predicted box and the label box.

[0057] For each bounding box, the top K values ​​based on the alignment_metrics are selected as positive samples. Furthermore, based on instance segmentation, the deepsort tracking method is combined to track the movement trajectory of cells in streaming media. Deepsort, short for DeepSimpleOnlineandRealtimeTracking, is used to track targets in video and identify them with consecutive IDs. The specific steps of the cell tracking are as follows:

[0058] (1) Data preprocessing: Input the streaming media containing cell motion trajectories frame by frame into YOLOv8;

[0059] (2) Target detection and segmentation: The YOLOv8 algorithm is used to detect cells and obtain information such as their location and size;

[0060] (3) Target assignment: The Deepsort algorithm is used to match each detected cell with the previously tracked cells and assign a unique ID number to it;

[0061] (4) Continuous tracking: Deepsort uses Kalman filtering for tracking, fusing previous and current cell position information to more accurately estimate future motion;

[0062] (5) Cell attribute extraction: Based on the tracking results, various cell attributes can be extracted, such as movement speed and growth rate;

[0063] (6) Data visualization: Visualize the tracking results to facilitate further analysis and study of cell behavior.

[0064] To obtain the training dataset for the model, LabelMe software was used to label the cells in the images, resulting in segmentation mask data for each cell. Each cell in each image was uniquely identified by an integer. OpenCV was then used to obtain the bounding box corresponding to each cell. After filtering and segmentation of this data, each small image was cleaned to obtain the final cell segmentation dataset. To accurately evaluate the model's performance, a triple cross-validation method was used. The dataset was divided into three parts, with two parts used as the training set and the other as the validation set. Training and validation were performed sequentially, and the average test metric on the validation set was taken as the final test result.

[0065] Furthermore, it includes the YOLOv8 inference and post-processing workflow, which comprises the following steps:

[0066] (1) Converting the bounding box integral form to the 4D bounding box format: Convert the bounding box branch output from the Head, and use Softmax and Conv to calculate and convert the integral form to the 4D bounding box format;

[0067] (2) Dimension transformation: YOLOv8 outputs three feature maps with scales of 80x80, 40x40 and 20x20. The Head part outputs feature maps with a total of 6 scales for classification and regression. The three different scales of the class prediction branch and the bounding box prediction branch are concatenated and the dimension transformation is performed. For the convenience of subsequent processing, the original channel dimension is replaced at the end. The shapes of the class prediction branch and the bounding box prediction branch are (b,80x80+40x40+20x20,80)=(b,8400,80) and (b,8400,4), respectively.

[0068] (3) Decode and restore to the original image scale: The classification prediction branch performs Sigmoid calculation, while the bbox prediction branch needs to be decoded and restored to the original image in the xyxy format after decoding.

[0069] (4) Threshold filtering: Traverse each image in the batch and use score_thr for threshold filtering. In this process, multi_label and nms_pre also need to be considered to ensure that the number of detected boxes after filtering is no more than nms_pre.

[0070] (5) Restore to the original image scale and NMS: Based on the preprocessing, restore the remaining detection boxes to the original image scale before the network output, and then perform NMS. The final output detection boxes cannot be more than max_per_img.

[0071] The medical image cell segmentation and tracking method of the present invention has the following advantages:

[0072] (1) The present invention has high timeliness. The cell segmentation method based on convolutional neural networks has high efficiency.

[0073] (2) This invention has strong anti-interference performance. During the data processing stage, we filtered and preprocessed the data, and during training, we also used numerous data augmentation methods to reduce the risk of model overfitting. Furthermore, when using the model to predict test images, it can also guarantee high generalization ability.

[0074] (3) The present invention has high performance indicators. Due to the use of Simam attention mechanism and multi-scale proto method to optimize the model, the model has good segmentation effect and efficiency.

[0075] (4) This invention can simultaneously and efficiently complete the prediction, segmentation and classification of cell bounding boxes in medical images. At the same time, because it meets the requirements of real time, by combining the improved YOLOv8 model with tracking algorithms such as Deepsort, it can also realize the tracking of cell motion trajectories in streaming media.

[0076] (5) This invention can automatically complete the analysis and detection of medical images, is highly convenient and easy to use. Relying on the portability of Python, users can install and run the entire network on terminal systems such as laptops, desktops, and servers. At the same time, YOLOv8 can be easily accelerated using TensorRT, which can greatly speed up the inference speed of the model during user operation. Attached Figure Description

[0077] Figure 1 A diagram illustrating the limitations of IOU;

[0078] Figure 2 This is a schematic diagram of the Head section of the YOLOv8 model.

[0079] Figure 3 This is a schematic diagram of the Head section of the improved YOLOv8 model;

[0080] Figure 4 A flowchart for inference performed on the model;

[0081] Figure 5 Example graph of the dataset used during testing;

[0082] Figure 6 This is a schematic diagram of the inference results from the Qupath software.

[0083] Figure 7 This is a schematic diagram of the inference results of the YOLOv8-simam+mproto model.

[0084] Figure 8 This is a schematic diagram of a set of image inference results in a cell tracking video stream.

[0085] Figure 9 This is a schematic diagram of another set of image inference results in a cell tracking video stream. Detailed Implementation

[0086] To better understand the purpose, structure, and function of this invention, a medical image cell segmentation and tracking method of this invention will be described in further detail below with reference to the accompanying drawings.

[0087] An efficient medical image cell segmentation and tracking method based on the improved lightweight model YOLOv8 is proposed. The method includes a data processing part, a feature extraction part, an FPN-PAN multi-scale feature fusion part, and a head prediction based on multi-scale features (bounding box regression, classification, and segmentation).

[0088] Step 1: Data Processing: This mainly includes image normalization, image denoising, image segmentation, and image enhancement. Since the cell segmentation dataset is relatively small, these processes are crucial for enhancing the training effect of the subsequent model. Furthermore, to improve the model's generalization ability, we used pre-trained weights from the official COCO dataset; however, due to differences in parameters and some structures, we only selectively loaded them.

[0089] Step 2: Feature Extraction: Feature extraction is performed on the image scaled to 640*640. The backbone of the model is used to extract features from the preprocessed image, and the quality of feature extraction will affect the final prediction result. To improve the model's expressive power, YOLOv8 adopts the CSPDarknet structure, i.e., a Darknet network with a CSP structure. It is a cross-stage local network architecture. After adding the CSP structure, the problem of information duplication in the backbone network is solved, especially in the gradient optimization process of some large neural networks. This can significantly reduce the number of model parameters and the number of FLOPS (floating point operations), and improve the inference speed and performance of the final model. The main idea of ​​the CSP structure is to divide the feature map output by the convolutional layer into two parts. One part continues to undergo convolution, and the other part is fused with the feature map after the previous convolution operation. The CSP module can quickly reduce the dimensionality of the output feature map. Due to the addition of this idea, the size of the YOLOv4 network model can be reduced, improving the detection speed without sacrificing detection accuracy. Currently, mainstream networks using the CSP (Convolutional Spinning Process) structure have achieved certain results, such as DenseNet and ResNet used in graphs. This design is likely to be retained in future network architecture development. The input and output parameters of a model, as well as the number of convolutional layers, all affect the model's detection performance. The CSPDarknet53 backbone network is used to extract target features and contains five CSP modules. Downsampling of each CSP module can be achieved using a 3×3 convolutional kernel. The YOLOv8 network model defines the input image size as 640×640. After feature extraction by the five CSP models in the backbone network, the feature map size changes five times, ultimately transforming from 640×640 to a 20×20 feature map, thus achieving rapid feature map dimensionality reduction. The advantages of using the CSPDarknet53 network structure as the backbone in YOLOv8 include two aspects: firstly, it improves the feature extraction capability of the convolutional network without sacrificing detection accuracy, thereby increasing detection speed; secondly, it reduces the overall computational cost of the model, enabling training of the YOLOv4 model on a simple CPU. It is worth noting that, unlike the previous CSPDarknet structure, the C3 module has been replaced with the C2f module, which allows for further weight reduction. The final part of the YOLOv8 model uses three branches—classification, regression, and segmentation—to complete the corresponding tasks. This multi-task training mode can integrate information from multiple tasks, providing supervision and mitigating overfitting for each sub-task.Furthermore, YOLOv8 employs the popular Decoupled-Head design, which reduces interference between different tasks during parameter optimization while preserving the collaborative optimization effect among multiple tasks as much as possible. YOLOv8 also retains the SPPF module used in architectures such as YOLOv5; subsequently, FPN-PAN fuses features from different depths, effectively improving the model's detection performance for targets of different sizes. Ultimately, we obtain feature maps at three different scales.

[0090] Step 3: FPN-PAN Multi-scale Feature Fusion: PAN-FPN can perform multi-scale feature fusion. FPN stands for Feature Pyramid Network, which can convey semantic features from top to bottom, while PAN stands for Pyramid Attention Network, which can convey strong localization features from bottom to top. PAN and FPN work together to aggregate parameters from different backbone layers to different detection layers, so that the location and category information of the target are preserved to the greatest extent. This structural design improves the classification and localization accuracy of the model.

[0091] Unlike previous YOLO architectures, YOLOv8 adopts a decoupled-head design, where classification and localization use two independent branches, which can improve the model's performance on these two tasks.

[0092] Step 4: Head Prediction Based on Multi-Scale Features: In the head detection, to achieve cell segmentation, we also need to use a segmentation branch. This part uses two sub-modules, Proto and cv4, to predict the discrimination criteria and confidence feature matrix of the mask, respectively. Then, the outputs of these two modules are multiplied to obtain the mask confidence matrix corresponding to each grid. After processing by the sigmoid function, this matrix becomes the probability value of each pixel belonging to a cell.

[0093] When using the original YOLOv8 model for cell segmentation, we found that the segmentation performance of YOLOv8 was still unsatisfactory. In order to further improve the segmentation performance of the model, especially the DICE index, we propose the following improvement ideas:

[0094] (1) The FPN structure outputs multi-scale feature maps, such as Figure 2The YOLOv8 model structure is shown below. Here, k represents the kernel size, s represents the stride, p represents the padding value, c represents the number of channels, and Conv2d represents the convolutional layer. In the Head section, specifically the Segment module, the model uses the proto and cv4 sub-modules to predict the mask's discrimination criteria and confidence feature matrix, respectively. The outputs of these two modules are then multiplied to obtain the mask confidence matrix for each grid cell. This matrix, after processing with the sigmoid function, represents the probability that each pixel belongs to a cell. However, when processing these features in the Segment module, the same Proto sub-module is used to generate the mask's discrimination criteria matrix. While this reduces the number of parameters and speeds up inference, for multi-scale feature maps, a multi-scale Proto sub-module should be used to generate a separate Proto matrix at each scale, which should improve performance. Therefore, to improve the model's fitting effect, a separate Proto is set for each scale. The predicted Proto matrix for each scale can be represented as follows:

[0095] protos[i](x[i])

[0096] Where x represents the feature set extracted by the model at different scales, and protos represents the set of Proto modules corresponding to different scales.

[0097] The improved YOLOv8 Head section can be used Figure 3 In this diagram, k represents the kernel size, s represents the stride, p represents the padding value, c represents the number of channels, and Conv2d represents the convolutional layer.

[0098] (2) Meanwhile, in order to enhance the expressive power of features and enable the model to focus more on effective information, we used the Simam attention mechanism on the features input to the Segment module. This is a parameterless attention module that allows the network to learn more discriminative neurons, and it is flexible and effective in improving the representation ability of convolutional networks.

[0099] (3) Since the YOLOv8 model requires an input image resolution of 640*640, we need to scale the high-resolution medical images to 640*640, which may result in some targets being smaller, increasing the difficulty of detection. Furthermore, the YOLOv8 model uses a large downsampling factor. As discussed above, deeper feature maps have difficulty learning the feature information of small targets. Therefore, we propose adding a small target detection layer, which involves adding the shallower feature map to the deeper feature map before detection. This can improve the model's attention to small targets, thereby improving the detection performance. Therefore, we added another layer to the PAN-FPN module to introduce the output of the second layer in the backbone network, allowing the model to better incorporate the lower-level features.

[0100] (4) In order to improve the segmentation performance of the model, we adopted the loss function of BCE+LovaszHingeLoss in the segmentation part. The latter is suitable for the case of label imbalance and can effectively handle the problem of class imbalance.

[0101] In the classification branch, the binary cross-entropy loss is used, and its calculation formula is as follows:

[0102]

[0103] Where the predicted category is the same as the true category, y i It is 1 if it is true, otherwise it is 0. The cross-entropy loss function can accelerate model convergence and parameter updates during training, and can avoid the problem of decreased learning rate of the mean squared error loss function, because if the sigmoid activation function is used, the gradient vanishing problem will occur when using the mean squared error.

[0104] In the regression branch, CIOU+DFL is used as the loss function.

[0105] CIOU is an improvement on IOU. IOU stands for Intersection over Union, and its calculation formula is...

[0106]

[0107] It can reflect the degree of matching between the predicted bounding box and the ground truth bounding box, thus reflecting the quality of the detection; it also has a good property of scale invariance, meaning it is insensitive to scale. However, using it as a loss function has two main drawbacks:

[0108] (1) If two boxes do not intersect, according to the definition, IOU = 0, which cannot reflect the distance between them (overlap). At the same time, because loss = 0, there is no gradient backpropagation, and learning and training cannot be performed;

[0109] (2) IOU cannot accurately reflect the degree of overlap between the two. For example Figure 1 As shown, the IOU between the two bounding boxes is equal in all three cases, but their GIOU from left to right is 0.33, 0.24 and -0.1 respectively. When the directions between the two bounding boxes are more consistent, the GIOU value will also be higher. Therefore, GIOU can better reflect the degree of overlap between these bounding boxes.

[0110] To address these two issues, GIOU, DIOU, and CIOU were proposed in succession.

[0111] (1) GIOU believes that IOU can be directly set as the regression loss, and its calculation formula is as follows:

[0112]

[0113] Among them, A c Let be the area of ​​the smallest closure region between the two boxes, that is, the area of ​​the smallest box that simultaneously contains both the predicted box and the ground truth box. U is the area of ​​the union region of the two boxes. This refers to the proportion of the closure region that does not belong to either of the two bounding boxes. Ultimately, we can use 1-GIOU as the loss function. Compared to IOU, it not only focuses on the overlapping region but also on other non-overlapping regions, thus better reflecting the degree of overlap between the two bounding boxes.

[0114] (2) DIOU is more consistent with the object bounding box regression mechanism than GIOU, taking into account the distance, overlap, and scale between the object and the anchor, making the object bounding box regression more stable and avoiding the divergence problems that occur during training, as seen with IOU and GIOU. Its calculation formula is as follows:

[0115]

[0116] Where, b, b gt These represent the center points of the predicted and ground truth bounding boxes, respectively. ρ represents the Euclidean distance between the two center points, and c represents the diagonal distance of the smallest closure region that simultaneously contains both the predicted and ground truth bounding boxes. DIOU can also replace the ordinary IOU evaluation strategy in NMS, making the results obtained by NMS more reasonable and effective.

[0117] (3) The CIOU loss function, based on the aforementioned function, also considers the aspect ratio of the three elements of the bounding box regression in the calculation and adds a penalty term, as shown in the following formula.

[0118]

[0119] Where α is the weighting function, and v is used to measure the similarity of aspect ratio, defined as

[0120] The complete CIOU loss function is defined as follows:

[0121] L CfOU =1-(IOU-R) CfOU )

[0122] It's important to note that the gradient of CIOU loss is similar to that of DIOU loss, but the gradient of v must also be considered. When the width and height are [0,1], w... 2 +h 2 The value of is usually very small, which can lead to gradient explosion, therefore in When implemented, it will be replaced with 1.

[0123] On the other hand, DFL loss, short for Distance-based Focal Loss, is a loss function used for classification tasks. Based on improvements to Focal Loss and Euclidean distance, it aims to better handle imbalanced classifications and uneven difficulty levels. Its formula is:

[0124]

[0125] Where y_true is the true label vector, y_pred is the predicted label vector, N is the number of samples, γ is the hyperparameter for adjusting the difficulty weights, and d is the Euclidean distance.

[0126] For correctly predicted classes, DFL disregards the loss value and focuses only on misclassified samples. Furthermore, DFL utilizes Euclidean distance to calculate the loss for misclassified samples, providing a more accurate reflection of the degree of classification error. Additionally, the introduction of the γ parameter allows for varying sample difficulty levels, thus better addressing the issues of imbalanced classification and uneven difficulty.

[0127] In summary, DFL is better able to handle uneven difficulty in classification tasks and can more accurately measure the degree of classification error, thereby improving model performance.

[0128] In the splitting branch, we not only used the binary cross-entropy loss function, but also added the LovaszHingeLoss, which is also a loss function used for classification tasks. It is designed to minimize the average ranking error in multi-label classification tasks and, compared to the BCE loss function, handles class imbalance and label noise better. The latter's formula is defined as follows:

[0129]

[0130] Where n is the number of labels, and y is the actual label vector. For predicting the label vector, π is a permutation function, and y π(i) This indicates the position of the i-th label after sorting, e i This represents the hinge loss function at that location.

[0131] The specific hinge loss function for each label is:

[0132]

[0133] Where y i ∈{-1,1} represents the true value of the i-th label. This represents the predicted value for the i-th label.

[0134] Compared to the cross-entropy loss function, the Lovasz Hinge Loss has the advantage of being optimized for training on multi-label data, taking into account the mutual influence between labels, and is applicable to imbalanced label scenarios. Furthermore, Lovasz Hinge Loss is based on a permutation function, eliminating the need to set weights for each label, thus improving its ease of implementation.

[0135] (5) Combining YOLOv8 with tracking algorithms such as deepsort can enable the tracking of cell movement trajectories.

[0136] like Figure 2 As shown in the YOLOv8 model structure, in the Head part, i.e., the Segment module, the model needs to use feature maps of multiple scales from the PAN structure output to generate the final prediction result. To improve the distribution of these features, we added the Simam attention mechanism. Simam can infer three-dimensional attention weights for the feature maps without increasing the original network parameters. It proposes to optimize the energy function to mine the importance of each neuron, and derives a fast analytical solution for the energy function, which is easy to implement and can improve the model performance with a small increase in computation without increasing the number of parameters. After a series of formula derivations, we can finally obtain the minimum energy of each neuron as follows:

[0137]

[0138] Since the lower the energy, the greater the difference between neuron t and surrounding neurons, and the higher its importance, the importance of a neuron can be expressed as 1 / e * We obtain the following formula for feature enhancement based on the definition of attention mechanisms:

[0139]

[0140] YOLOv8 abandoned the anchor-based design approach used in previous models and instead adopted the popular anchor-free approach.

[0141] In object detection, anchor-free and anchor-based algorithms are two common methods. They differ in the following ways:

[0142] (1) Anchor-free algorithms do not require prior information, meaning they do not require pre-defined anchor boxes, while anchor-based algorithms require pre-defined anchor boxes as references. This can be achieved through clustering algorithms or two-stage models like Mask R-CNN, using RegionProposalNetwork in the first stage.

[0143] (2) Anchor-free algorithms are generally faster because they do not require calculating classification and regression information for each anchor box. Anchor-based algorithms, on the other hand, require calculating classification and regression information for each anchor box.

[0144] (3) Anchor-based algorithms perform better in detecting both small and large objects because they can use anchor boxes of different sizes to adapt to targets of different sizes. Anchor-free algorithms, on the other hand, may be limited when dealing with small targets.

[0145] (4) Anchor-free algorithms are more robust because they are not affected by prior information and can adaptively detect targets of different sizes and shapes. Meanwhile, the performance of anchor-based algorithms is also affected by the setting of the anchor boxes.

[0146] In summary, anchor-free algorithms are suitable for applications that emphasize speed and robustness, while anchor-based algorithms are suitable for applications with high requirements on target size and shape.

[0147] During training, YOLOv8 abandoned the static positive sample allocation strategy and instead adopted the dynamic positive sample allocation strategy of TaskAlignedAssigner.

[0148] (1) The TaskAlignedAssigner strategy dynamically allocates positive samples. It dynamically determines which predicted boxes are positive samples based on the IOU value between the predicted boxes and the ground truth boxes output by the current network. In contrast, the static positive sample allocation strategy selects a fixed proportion of predicted boxes as positive samples.

[0149] (2) The advantage of the TaskAlignedAssigner strategy is that it can dynamically adapt to different target sizes and proportions, thus better reducing overfitting. The static positive sample assignment strategy does not have this advantage.

[0150] (3) The disadvantage of the TaskAlignedAssigner strategy is its relatively slow speed, because it requires calculating the IOU value between each predicted box and the ground truth box. In contrast, the static positive sample assignment strategy can be determined before training and is relatively fast.

[0151] (4) In the experiment, the TaskAlignedAssigner strategy performed better than the static positive sample assignment strategy, and under the same amount of computation, TaskAlignedAssigner could obtain a higher AP (Average Precision) value.

[0152] In summary, the TaskAlignedAssigner strategy offers better performance and robustness compared to the static positive sample assignment strategy, but it is slower. Therefore, the choice should be made based on the specific scenario and requirements in practical applications.

[0153] The matching strategy of TaskAlignedAssigner can be simply summarized as: selecting positive samples based on a weighted average of the classification and regression scores.

[0154] (1) For each label box, for all predicted boxes, an alignment score (alignment_metrics) for association classification and regression is obtained by weighting the classification score corresponding to the label box category and the IOU between the predicted box and the label box.

[0155] (2) For each label box, select the top K largest alignment scores directly based on alignment_metrics as positive samples.

[0156] Because YOLOv8 has a fast inference speed, meeting real-time requirements, we can further achieve cell trajectory tracking in streaming media by combining it with tracking methods such as deepsort, based on instance segmentation. Deepsort, short for DeepSimpleOnlineandRealtimeTracking, can be used to track targets in video and identify them with consecutive IDs. The specific steps of cell tracking are as follows:

[0157] (1) Data preprocessing: Input the streaming media containing cell motion trajectories frame by frame into YOLOv8;

[0158] (2) Target detection and segmentation: The YOLOv8 algorithm is used to detect cells and obtain information such as their location and size;

[0159] (3) Target assignment: The Deepsort algorithm is used to match each detected cell with the previously tracked cells and assign a unique ID number to it;

[0160] (4) Continuous tracking: Deepsort uses Kalman filtering for tracking, fusing previous and current cell position information to more accurately estimate future motion;

[0161] (5) Cell attribute extraction: Based on the tracking results, various cell attributes can be extracted, such as movement speed and growth rate;

[0162] (6) Data visualization: Visualize the tracking results to facilitate further analysis and study of cell behavior.

[0163] To obtain the dataset for training the model, we need to use LabelMe software to label the cells in the images, ultimately obtaining segmentation mask data for each cell. Each cell in each image is uniquely identified by an integer. Then, we can use OpenCV to obtain the bounding box corresponding to each cell. After filtering (such as removing cells with small areas) and segmenting these data, we further clean each small image (such as removing cells that are only a very small portion remaining at the image edges due to segmentation). Finally, we obtain the cell segmentation dataset. To accurately evaluate the model's performance, we also need to use a triple cross-validation method. The dataset is divided into three parts, with two parts used as the training set and the other as the validation set each time. Training and validation are performed sequentially, and the average test metric on the validation set is taken as the final test result.

[0164] This invention's model enables end-to-end training and achieves high detection accuracy and detection rate while maintaining fast inference speed. Its application is highly convenient due to its small number of parameters, lightweight architecture, and the ease of deployment of YOLOv8 using TensorRT. TensorRT is a C++ library that facilitates high-performance inference on NVIDIA GPUs. It works in a complementary manner to training frameworks such as TensorFlow, Caffe, PyTorch, and MXNet, specifically designed for fast and efficient network inference on GPUs. TensorRT can compress, optimize, and deploy networks at runtime without framework overhead. It improves network latency, throughput, and efficiency by combining layers, optimizing kernel selection, and performing normalization and transformation to the optimal matrixmath method based on specified precision. Since YOLOv8 requires an input image size of 640*640, we uniformly scaled the images to 640*640 during training and testing.

[0165] This model is a single-stage object detection model. Compared to two-stage object detection models such as Mask R-CNN, YOLO can complete both bounding box detection and classification tasks with just one network. Furthermore, by adding a Segment module, YOLO can simultaneously perform cell segmentation. YOLO stands for YouonlylookOnce, meaning it can identify objects in an image with only one look. Its detection process is simpler and clearer, with fewer model parameters and faster detection speed, meeting the requirements for real-time object detection.

[0166] The inference and post-processing flow of YOLOv8 is as follows: Figure 4 As shown, it includes the following steps:

[0167] (1) Converting the bounding box integral form to the 4D bounding box format: Convert the bounding box branch output from the Head, and use Softmax and Conv to calculate and convert the integral form to the 4D bounding box format;

[0168] (2) Dimension Transformation: YOLOv8 outputs three feature maps with scales of 80x80, 40x40, and 20x20. The Head part outputs feature maps at six scales for classification and regression. The class prediction branch and bounding box prediction branch at three different scales are concatenated and their dimensions are transformed. For easier processing later, the original channel dimensions are moved to the end, and the shapes of the class prediction branch and bounding box prediction branch are (b, 80x80 + 40x40 + 20x20, 80) = (b, 8400, 80) and (b, 8400, 4), respectively.

[0169] (3) Decode and restore to the original image scale: The classification prediction branch performs Sigmoid calculation, while the bbox prediction branch needs to be decoded and restored to the original image in the xyxy format after decoding.

[0170] (4) Threshold filtering: Traverse each image in the batch and use score_thr for threshold filtering. During this process, multi_label and nms_pre also need to be considered to ensure that the number of filtered bounding boxes does not exceed nms_pre;

[0171] (5) Restoring to the original image scale and NMS: Based on the preprocessing, the remaining detection boxes are restored to the original image scale before network output, and then NMS is performed. The final output detection boxes cannot exceed max_per_img.

[0172] To improve the performance of YOLOv8 on cell segmentation tasks, this method makes some improvements to the Head part. The performance evaluation of these improvements will be presented below. The experimental dataset we use is a private immunohistochemistry segmentation dataset. Figure 5 The image shown is an example of the dataset used.

[0173] Example 1:

[0174] The following example illustrates how to segment cells from a medical image. Figure 5 As can be seen, this image contains a large number of cells. To label each cell with a unique bounding box would be extremely time-consuming and labor-intensive if done manually. Therefore, we used the open-source software QuPath to analyze this image, and the results are as follows... Figure 6 As shown in the results, the detection results of QuPath are not satisfactory. In contrast, the detection results using the improved YOLOv8 method are as follows: Figure 7 As shown in the figure below, our method can achieve better results.

[0175] Example 2:

[0176] For the Simam strategy and multi-scale proto strategy mentioned above, we conducted related ablation experiments using the Monuseg dataset. The Monuseg dataset contains four types of cell nuclei: epithelial cell nuclei, inflammatory cell nuclei, fibroblast cell nuclei, and fibroblast cell nuclei, with a total of 37 1000*1000 training images and 14 validation images of the same resolution.

[0177] The experiments yielded the comparison results shown in the table below. It can be seen that compared to the YOLOv8m model, adding Simam and multi-scale proto modules respectively improved the model's performance. Ultimately, YOLOv8m-simam-mproto achieved the best detection results.

[0178]

[0179] Example 3:

[0180] Cell tracking is valuable in various applications, from stem cell therapy to cancer metastasis research. It is of great significance for studying cell origin, understanding the development process of tissues and organs, and exploring the mechanisms of disease occurrence and development, and helps doctors to detect diseases more effectively and efficiently.

[0181] Deepsort is a multi-object tracking algorithm based on object detection. The process of combining it with YOLOv8 for cell tracking is roughly as follows:

[0182] (1) Obtain the original video frames;

[0183] (2) Use YOLOv8 to detect and segment cells in each frame of the image;

[0184] (3) Extract the features from the bounding boxes corresponding to the detected cells. These features include appearance features and motion features. The former facilitates feature comparison, while the latter facilitates Kalman filtering for prediction.

[0185] (4) Calculate the matching degree between targets in two consecutive frames using the Hungarian algorithm and cascaded matching, and assign an ID to each tracked target.

[0186] We tested the method on a public dataset and extracted two sets of images to demonstrate its tracking performance between consecutive frames, as shown below. Figure 8 , 9 As shown.

[0187] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A method for cell segmentation and tracking in medical images, characterized in that, Includes the following steps: Step 1: Data processing: including image normalization, image denoising, image segmentation, and image enhancement; Step 2: Feature extraction: Feature extraction is performed on the image after scaling to 640*640. The backbone of the model is used to extract features from the preprocessed image. YOLOv8 adopts the CSPDarknet structure, which is a Darknet network with a CSP structure. Step 3: FPN-PAN multi-scale feature fusion: FPN conveys semantic features from top to bottom, and PAN conveys strong localization features from bottom to top. PAN and FPN work together to aggregate parameters from different backbone layers to different detection layers. Step 4: Head predicts based on multi-scale features: In the detection head, the two sub-modules Proto and cv4 predict the discrimination criteria of the mask and the confidence feature matrix of the mask, respectively. Then, the output results of these two modules are multiplied by matrix to finally obtain the mask confidence matrix corresponding to each grid. After processing by the sigmoid function, this matrix is ​​the probability value of each pixel belonging to the cell. Step 4.1: Set up a separate Proto for each scale, then the corresponding Proto matrix prediction for each scale is represented as follows: Where x represents the feature set extracted by the model at different scales, and protos represents the set of Proto modules corresponding to different scales; Step 4.2: Apply the Simam attention mechanism to the features of the input Segment module; Step 4.3: Add a layer to the PAN-FPN module to bring in the output of the second layer in the backbone network, so that the model can better incorporate the low-level features; Step 4.4: In the segmentation part, the loss function of BCE+LovaszHingeLoss is adopted; Step 4.5: Combine YOLOv8 with the deepsort tracking algorithm to track the movement trajectory of cells.

2. The medical image cell segmentation and tracking method according to claim 1, characterized in that, Step 2 uses the CSPDarknet53 backbone network to extract target features. It contains 5 CSP modules. The downsampling of each CSP module is achieved by a 3×3 convolutional kernel. The YOLOv8 network model defines the size of the input image as 640×640. After feature extraction by the five CSP models in the backbone network, the feature map size changes five times, and finally changes from 640×640 to a 20×20 feature map, thus achieving rapid dimensionality reduction of the feature map.

3. The medical image cell segmentation and tracking method according to claim 1, characterized in that, In step 3, YOLOv8 adopts the design concept of a decoupled-head, that is, classification and localization use two branches that do not interfere with each other.

4. The medical image cell segmentation and tracking method according to claim 1, characterized in that, In the classification branch, the binary cross-entropy loss is used, and its calculation formula is as follows: Specifically, when the predicted category is the same as the true category, It is 1, otherwise it is 0.

5. The medical image cell segmentation and tracking method according to claim 1, characterized in that, In the regression branch, CIOU+DFL is used as the loss function. CIOU is an improvement on IOU, which stands for Intersection over Union, and its calculation formula is as follows: We propose GIOU, DIOU, and CIOU as loss functions. GIOU directly sets IOU as the regression loss, and its calculation formula is as follows: in, Let be the area of ​​the smallest closure region between the two boxes, that is, the area of ​​the smallest box that simultaneously contains both the predicted box and the ground truth box. U is the area of ​​the union region of the two boxes. This refers to the proportion of the closed region that does not belong to either of the two bounding boxes, and ultimately 1-GIOU is used as the loss function. The formula for calculating DIOU is: Where, b, These represent the center points of the predicted bounding box and the ground truth bounding box, respectively. This represents the Euclidean distance between two center points, and c represents the diagonal distance of the smallest closure region that can simultaneously contain both the predicted and ground truth boxes. The CIOU loss function, based on the GIOU and DIOU functions, also considers the aspect ratio of the three elements of bounding box regression in the calculation, adding a penalty term, as shown in the following formula: in, It is a weighting function, and The similarity used to measure aspect ratio is defined as follows: , The complete CIOU loss function is defined as follows: The gradient of CIOU loss is similar to that of DIOU loss, but the gradient of v must also be considered, especially when the width and height are [0,1]. The value of is usually very small, which can lead to gradient explosion, therefore in When implemented, it will be replaced with 1; DFL loss, short for Distance-based Focal Loss, is a loss function used for classification tasks. It's an improvement on Focal Loss and Euclidean distance, designed to better handle imbalanced classifications and uneven difficulty levels. Its formula is: in, It is the actual label vector. It is the predicted label vector, where N is the number of samples. It is a hyperparameter that adjusts the difficulty weights, and d is the Euclidean distance.

6. The medical image cell segmentation and tracking method according to claim 1, characterized in that, In the splitting branch, the binary cross-entropy loss function is used in conjunction with LovaszHingeLoss. LovaszHingeLoss is also a loss function used for classification tasks, designed to minimize the average ranking error in multi-label classification tasks. The formula is defined as follows: Where n is the number of labels, and y is the actual label vector. To predict the label vector, Given a permutation function, This indicates the position of the i-th label after sorting. This represents the hinge loss function at that location; The specific hinge loss function for each label is: in This represents the true value of the i-th label. This represents the predicted value for the i-th label.

7. The medical image cell segmentation and tracking method according to claim 1, characterized in that, Step 4 includes the following specific steps: In the Head section, i.e., the Segment module, the Simam attention mechanism is added. Simam infers 3D attention weights for the feature map without increasing the original network parameters, and calculates the minimum energy for each neuron. Because the lower the energy, the greater the difference between neuron t and surrounding neurons, and the higher its importance, the importance of a neuron is determined by… Based on the definition of the attention mechanism, the features are enhanced using the following formula: YOLOv8 has abandoned the anchor-based design approach used in previous models, adopting an anchor-free approach instead. During training, YOLOv8 abandons the static positive sample allocation strategy and adopts the dynamic positive sample allocation strategy of TaskAlignedAssigner. The matching strategy of TaskAlignedAssigner is as follows: positive samples are selected based on the weighted scores of classification and regression. For each label box, for all predicted boxes, an alignment score (alignment_metrics) is obtained by weighting the classification score corresponding to the label box category with the IOU between the predicted box and the label box. For each label box, the top K largest alignment scores are selected as positive samples based directly on the alignment_metrics.

8. The medical image cell segmentation and tracking method according to claim 1, characterized in that, Based on instance segmentation, and combined with the deepsort tracking method, the movement trajectory of cells in streaming media is tracked. Deepsort, short for DeepSimpleOnlineandRealtimeTracking, is used to track targets in video and identify them with consecutive IDs. The specific steps of cell tracking are as follows: (1) Data preprocessing: Input the streaming media containing cell motion trajectories frame by frame into YOLOv8; (2) Target detection and segmentation: The YOLOv8 algorithm is used to detect cells and obtain their location and size information; (3) Target assignment: The Deepsort algorithm is used to match each detected cell with the previously tracked cells and assign a unique ID number to it; (4) Continuous tracking: Deepsort uses Kalman filtering for tracking, fusing previous and current cell position information to more accurately estimate future motion; (5) Cell attribute extraction: Based on the tracking results, extract various cell attributes: movement speed, growth rate; (6) Data visualization: Visualize the tracking results to facilitate further analysis and research on cell behavior; To obtain the training dataset for the model, LabelMe software was used to label the cells in the images, resulting in segmentation mask data for each cell. Each cell in each image was uniquely identified by an integer. OpenCV was then used to obtain the bounding box corresponding to each cell. After filtering and segmentation of this data, each small image was cleaned to obtain the final cell segmentation dataset. To accurately evaluate the model's performance, a triple cross-validation method was used. The dataset was divided into three parts, with two parts used as the training set and the other as the validation set. Training and validation were performed sequentially, and the average test metric on the validation set was taken as the final test result.

9. The medical image cell segmentation and tracking method according to claim 1, characterized in that, The YOLOv8 inference and post-processing workflow includes the following steps: (1) Converting the bbox integral form to 4D bbox format: Convert the bbox branch output by Head, and use Softmax and Conv to calculate and convert the integral form to 4D bbox format; (2) Dimension transformation: YOLOv8 outputs three feature maps with scales of 80x80, 40x40 and 20x20. The Head part outputs feature maps with a total of 6 scales for classification and regression. The three different scales of the class prediction branch and the bounding box prediction branch are concatenated and the dimension transformation is performed. For the convenience of subsequent processing, the original channel dimension is replaced at the end. The shapes of the class prediction branch and the bounding box prediction branch are (b,80x80+40x40+20x20,80)=(b,8400,80) and (b,8400,4), respectively. (3) Decoding and restoring to the original image scale: The classification prediction branch performs Sigmoid calculation, while the bbox prediction branch needs to be decoded and restored to the original image in the xyxy format after decoding; (4) Threshold filtering: Traverse each image in the batch and use score_thr for threshold filtering. In this process, multi_label and nms_pre also need to be considered to ensure that the number of detected boxes after filtering is no more than nms_pre. (5) Restore to the original image scale and NMS: Based on the preprocessing, restore the remaining detection boxes to the original image scale before the network output, and then perform NMS. The final output detection boxes cannot be more than max_per_img.

Images