A Cell-SORT cell tracking method and device based on deep learning

Abstract

The present invention relates to a Cell‑SORT cell tracking method and device based on deep learning. In view of the difficulties in cell tracking, such as few target appearance features and low resolution, the optimal model YOLOv7‑X in the current mainstream target detection model YOLO series is selected as the cell detection model, and its feature fusion module is modified to propose a high-resolution YOLOv7‑X‑A cell detection model. In view of the challenges of obvious target deformation and complex and changeable background noise in cell tracking, based on the SORT data association architecture, an optimized Kalman filter (KF+) is adopted to better predict the position of the target, and a BYTE matching mechanism is introduced to match the target detection frame twice, and a trajectory interpolation post-processing method is introduced to fill the trajectory. Finally, the improved YOLOv7‑X‑A cell detection model is combined with the reconstructed SORT data association method to propose a Cell‑SORT cell tracking method based on deep learning to achieve cell tracking with strong robustness and higher accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of visual target tracking and relates to a Cell-SORT cell tracking method and device based on deep learning. Background Art

[0002] Studying cellular activity is of great significance to the biomedical field. Cells are the fundamental structural and functional units of organisms. Understanding cellular activity provides insights into the normal functions of organisms and the mechanisms of disease, thereby driving the development and progress of medicine. Studying cellular activity is crucial for disease diagnosis and treatment. Observing abnormal cellular changes can provide clues to disease diagnosis and treatment, such as the discovery of early markers for cancer. Furthermore, cells are important targets and models for drug development. Assessing the effects of drugs on cells provides a foundation for the development of new drugs. Understanding cellular activity is crucial for the formation and functional restoration of organs and tissues. Controlling cellular activity can advance the development of tissue engineering and artificial organs. Studying cellular gene regulatory networks and signaling pathways also provides a theoretical basis for gene therapy. By modifying or regulating cellular gene expression, abnormalities in disease progression can be corrected. Overall, studying cellular activity deepens our understanding of the structure and function of organisms, providing a scientific basis for the diagnosis, treatment, and prevention of diseases. This is of great significance in promoting the development of drug development, organ and tissue engineering, and gene therapy. To further analyze cellular activity throughout their lifecycle, researchers record and analyze images and videos of living cells, accurately recording the position and movement of cells over time, a task also known as cell tracking.

[0003] Cell tracking is a multi-target tracking task used in the biomedical field. However, compared with mainstream multi-target tracking tasks, cell tracking tasks are more challenging, specifically in three aspects.

[0004] (1) The cells have few external features.

[0005] (2) The cells are obviously deformed throughout the video.

[0006] (3) Complex background.

[0007] Therefore, there is an urgent need for a method to track cells that is both robust to noise, cell deformation, and cell events and has excellent tracking performance. Summary of the Invention

[0008] The present invention is aimed at the difficulties of cell tracking, such as few target appearance features and low resolution. The optimal model YOLOv7-X in the current mainstream target detection model YOLO series is selected as the cell detection model, and its feature fusion module is modified to propose a high-resolution YOLOv7-XA cell detection model. In view of the challenges of obvious target deformation and complex and changeable background noise in cell tracking, based on the SORT data association architecture, an optimized Kalman filter (KF+) is adopted to better predict the position of the target. At the same time, a BYTE matching mechanism is introduced to match the target detection frame twice, and a trajectory interpolation post-processing method is introduced to fill the trajectory. Finally, the improved YOLOv7-XA cell detection model is combined with the reconstructed SORT data association method to propose a Cell-SORT cell tracking method based on deep learning to achieve robustness and higher accuracy cell tracking.

[0009] The present invention provides a Cell-SORT cell tracking method based on deep learning, which specifically includes the following steps:

[0010] (1) Send the video frame to the trained target detection model and obtain the detection target box confidence score bb Pred ;

[0011] (2) Use the Kalman filter to predict the position of the previous frame trajectory in this frame; if it is the first frame, then according to bb pred Create a new trajectory. If it is not the first frame, use the Kalman filter to predict the position of the cell in this frame based on the trajectory established in the previous frame;

[0012] (3) Using BYTE mechanism to bb pred The trajectory uses IOU as the similarity metric and uses the Hungarian algorithm for minimum cost matching. First, according to the confidence threshold μ high and μ low , bb pred Divided into high-score detection boxes D high and low-score detection box D low , and then use the Hungarian algorithm to convert the high score detection box D high The first match is performed with the trajectory. After the first match is completed, the Hungarian algorithm is used to find the low-score detection box D low Perform a second match with the unmatched trajectory; after the two matches are completed, D high Create a new track for the target frame that is still not matched and update it to the track set;

[0013] (4) Return to step (2) and process the next frame until all video frames are processed;

[0014] (5) After processing all video frames, the trajectory set T is obtained, and then the trajectory interpolation post-processing method is used to fill the trajectory of T to obtain the final tracking result Tracks.

[0015] Furthermore, the target detection model adopts the original YOLOv7-X model, and the YOLOv7-X consists of three parts: Backbone, Neck and Head, wherein Backbone consists of multiple CBS, ELAN, and MPConv modules to extract image features, wherein CBS is a set of convolution operations with different scales, including a convolution layer, a batch data normalization layer and an activation function SiLU layer, ELAN is composed of multiple CBS and connection operations, and MPConv is composed of CBS convolution operations and MaxPooling layers, which enhances the model's ability to fuse and extract features; the Neck part is used to fuse features of different dimensions to obtain richer semantic information; the Head predicts the position and category of the target object based on the feature information.

[0016] Furthermore, the target detection model adopts YOLOv7-XA, which modifies the fusion path of the Neck part based on the original YOLOv7-X.

[0017] In the original YOLOv7-X model, the feature maps of Neck fusion come from layers 28, 43, and 59 in Backbone. The feature map of layer 28 is input to layer 76 and then to layer 77 for fusion. The feature map of layer 43 is input to layer 62 and then to layer 63 for fusion. The feature map of layer 59 is input to layer 107 for fusion.

[0018] In YOLOv7-XA, the feature maps of Neck fusion come from the 13th, 28th and 43rd layers in Backbone respectively, that is, the feature map of the 13th layer is input to the 76th layer and then to the 77th layer for fusion, the feature map of the 28th layer is input to the 62nd layer and then to the 63rd layer for fusion, and in the fusion path of the 43rd layer, a 3*3 convolution module is added, which is the 107th layer here. The original 107th layer becomes the 108th layer, and the subsequent layer numbers are increased by one, that is, the feature map of the 43rd layer is output to the 107th layer and then input to the 108th layer for fusion, and the 1*1 convolution modules of the 76th and 62th layers in the original model are replaced by 3*3 convolution modules. The rest of the model is the same as the original model.

[0019] Furthermore, the use of the Kalman filter KF+ in step (2) includes two steps: prediction and update. The prediction stage is to predict the state and covariance of the cell. The state of the cell is the center coordinates and the width and height. The update step is to use the cell state obtained by the target detection model and the state estimate obtained in the prediction stage to update the posterior estimate of the system and update the covariance matrix. After continuous prediction and update, the Kalman filter can accurately predict the position of the cell at the next moment.

[0020] The predicted target state vector is represented as an octet, as shown in formula (1), where (x c ,y c ) represents the center coordinates of a target on the image plane, (w,h) represents the width and height of the target frame, and the following four parameters represent the rate of change of the first four parameters. The observation vector is shown in formula (2), which represents the center coordinates (z) of a target obtained by the sensor or the detection model. xc ,z yc ) and width z w and Gaoz h According to the modified Kalman filter state vector and observation vector, namely formula (1) and formula (2), the process noise matrix Q k and the observation noise matrix R k It is also modified accordingly, as shown in formula (3)

[0021] As shown in (4),

[0022]

[0023]

[0024]

[0025]

[0026]

[0027] Here, σ p and σ v represents the noise factor, and Represent the width and height obtained in the k-1 time update phase, that is, in the following update phase The third and fourth parameters in;

[0028]

[0029] Here, σ m represents the noise factor, and Represents the width and height of the prediction stage at the kth moment, that is, the following prediction stage The third and fourth parameters in;

[0030] The specific prediction and update implementation steps are as follows:

[0031]

[0032] Here, represents the state quantity predicted at time k, as shown in formula (1), Q k Represents the process noise matrix, that is, formula (3), P k|k-1 represent The covariance matrix, F k Represents the prediction matrix, F k The specific form is:

[0033]

[0034]

[0035] Here, P k|k represents the updated covariance matrix, z k Represents the observed quantity, that is, formula (2), R k Represents the observation noise matrix, that is, formula (4), H k Represents the mapping matrix, H k The specific form is:

[0036]

[0037] Furthermore, the specific implementation of step (3) is as follows:

[0038] (31) Send the video frame into the target detection model and obtain the detection target frame bb Pred ;

[0039] (32) According to the predicted target box bb Pred The confidence score and detection confidence threshold μ high , μ low The detection target frame obtained in step (31) is divided into high-score detection frames D high and low-score detection box D low , that is, predict the target box bb pred The confidence score is greater than μ high , then add the target frame to the high-score detection frame set D high In the case of predicting the target box bb pred The confidence score is less than μ high and is greater than μ low , then add the target frame to the low-score detection frame set Dlow middle;

[0040] (33) Use the Kalman filter to predict the new position of each trajectory in the previous frame trajectory set to obtain the current trajectory set T;

[0041] (34) Using IOU as a similarity metric, the Hungarian algorithm is used to compare the trajectory set T with the high-score detection box D. high Perform the first matching, and add the unmatched trajectories to the set T remain , the unmatched target box is added to D remain , then use IOU as the similarity metric, and use the Hungarian algorithm again to match the unmatched tracks T remain Compared with the low-score detection box D low Perform a second match and delete the tracks that are still unassigned;

[0042] (35) is D remain Create a new trajectory for the target box in;

[0043] (36) Repeat steps (31)-(35) until all video frames are processed.

[0044] Furthermore, the IOU indicator used in the matching in step (34) refers to the intersection-over-union ratio, which measures the degree of overlap between two detection frames. The formula is as follows: Among them, B pred and B gt are the target bounding box pred and the real label bounding box gt estimated by the algorithm, respectively, and |·| represents the area of ​​the region.

[0045] Furthermore, the specific implementation of step (5) is as follows:

[0046] Assume that there is a cell trajectory T. Due to cell movement, the trajectory loses its target between frame t1 and frame t2. The position of trajectory T at frame t1 is P. t1 (x1, y1, x2, y2), which is the quadruple consisting of the coordinates of the upper left corner and the lower right corner, P t2 is the coordinate of trajectory T in frame t2. The hyperparameter μ is set as the maximum interval for performing trajectory interpolation. That is, when t2-t1≤μ, the trajectory interpolation method will be executed. The interpolation position of the trajectory in frame t is calculated as follows:

[0047]

[0048] Among them, t1 <t<t2。

[0049] Furthermore, the method further includes step (6), wherein the objective evaluation indicators MOTA and IDF1 are used to evaluate the cell tracking effect.

[0050] The present invention also provides a deep learning-based Cell-SORT cell tracking device, comprising the following modules:

[0051] The confidence score acquisition module is used to feed the video frame into the trained target detection model and obtain the confidence score bb of the detected target frame. Pred ;

[0052] The position prediction module is used to use the Kalman filter to predict the position of the previous frame trajectory in this frame; if it is the first frame, it is used according to bb pred Create a new trajectory. If it is not the first frame, use the Kalman filter to predict the position of the cell in this frame based on the trajectory established in the previous frame;

[0053] Trajectory update module, used to use BYTE mechanism to update bb pred The trajectory uses IOU as the similarity metric and uses the Hungarian algorithm for minimum cost matching. First, according to the confidence threshold μ high and μ low , bb pred Divided into high-score detection boxes D high and low-score detection box D low , and then use the Hungarian algorithm to convert the high score detection box D high The first match is performed with the trajectory. After the first match is completed, the Hungarian algorithm is used to find the low-score detection box D low Perform a second match with the unmatched trajectory; after the two matches are completed, D high Create a new track for the target frame that is still not matched and update it to the track set;

[0054] Return to the position prediction module and process the next frame until all video frames are processed. After processing all video frames, the trajectory set T is obtained, and then the trajectory interpolation post-processing method is used to fill T with trajectories to obtain the final tracking result Tracks.

[0055] Compared with existing technologies, this invention offers several advantages and benefits: it utilizes the new cell tracking dataset CTMC for training, uses coarser bounding boxes for cell detection rather than more precise cell segmentation methods, and optimizes the target detection model and association method based on cell characteristics. Compared with previous methods, this method is more robust and can achieve good tracking results even in situations with few visual features and complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is the original YOLOv7-X structure diagram.

[0057] Figure 2 This is a diagram of the improved YOLOv7-XA structure according to an embodiment of the present invention.

[0058] Figure 3 This is a framework diagram of the cell tracking method in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0060] like Figure 3 As shown, an embodiment of the present invention provides a Cell-SORT cell tracking method based on deep learning, which specifically includes the following steps:

[0061] (1) Each frame of the cell video sequence is sent to the trained target detection model YOLOv7-XA for cell detection. When sent to the detection model, the cell image size will be adjusted to 416×416; after passing through the target detection model, the predicted target box bb will be obtained pred ;

[0062] (2) If it is the first frame, then according to bb pred Create a new trajectory. If it is not the first frame, use the Kalman filter (KF+) to predict the cell position in this frame based on the trajectory established in the previous frame;

[0063] (3) Using BYTE mechanism to bb pred The trajectory uses IOU as the similarity metric and uses the Hungarian algorithm for minimum cost matching. First, according to the confidence threshold μ high and μ low bb pred Divided into high-score detection boxes D high and low-score detection box D low (If the predicted target box bb pred The confidence score is greater than μ high , then add the target frame to the high-score detection frame set D high In the case of predicting the target box bb pred The confidence score is less than μ high and is greater than μ low , then add the target frame to the low-score detection frame set D low Then use the Hungarian algorithm to convert the high score detection box D high The first match is performed with the trajectory. After the first match is completed, the Hungarian algorithm will be used to match the low-score detection box Dlow Perform a second match with the unmatched trajectory; after the two matches are completed, D high Create a new track for the target box that is still not matched and update it to the track set.

[0064] (4) Return to step (2) and process the next frame until all video frames are processed.

[0065] (5) After processing all video frames, a trajectory set T is obtained, and then the trajectory interpolation post-processing method is used to fill the trajectory of T to obtain the final tracking result Tracks.

[0066] Furthermore, in step (1), the target tracking model adopts YOLOv7-XA, which is proposed by the present invention. The detection model modifies the fusion path of the Neck part on the basis of YOLOv7-X. YOLOv7-XA mainly consists of three parts: Backbone, Neck and Head. The Backbone is composed of multiple modules such as CBS, ELAN, MPConv, etc., which are used to extract image features. CBS is a set of convolution operations with different scales, including a convolution layer, a batch data normalization layer and an activation function SiLU layer. ELAN is composed of multiple CBS layers and connection operations, which can extract rich features of different dimensions. MPConv is composed of a combination of CBS convolution operations and MaxPooling layers, which enhances the model's ability to extract features. The Neck part is used to fuse features of different dimensions to obtain richer semantic information. The Head predicts the position and category of the target object based on the feature information.

[0067] In the original YOLOv7-X model, the feature maps of Neck fusion come from layers 28, 43, and 59 in Backbone. The feature map of layer 28 is input to layer 76 and then to layer 77 for fusion. The feature map of layer 43 is input to layer 62 and then to layer 63 for fusion. The feature map of layer 59 is input to layer 107 for fusion.

[0068] In YOLOv7-XA, the feature maps of Neck fusion come from layers 13, 28, and 43 of Backbone. That is, the feature map of layer 13 is input to layer 76 and then to layer 77 for fusion, while the feature map of layer 28 is input to layer 62 and then to layer 63 for fusion. In the fusion path of layer 43, a 3*3 convolution module is added (here, layer 107, the original 107 layers become layer 108, and the sequence numbers of subsequent layers are incremented by one). That is, the feature map of layer 43 is output to layer 107 and then to layer 108 for fusion. The 1*1 convolution modules of layers 76 and 62 in the original model are replaced with 3*3 convolution modules. The rest of the model is the same as the original model.

[0069] Furthermore, during the data association phase, an improved Kalman filter (KF+) is employed. The KF+ algorithm is a recursive algorithm for estimating system states. It fuses multiple, inaccurate sensor measurements with the system model to arrive at an optimal estimate of the system's true state. Specifically, the KF+ algorithm consists of two main steps: prediction and update.

[0070] In the prediction step, the KF+ algorithm uses the state estimate from the previous moment to predict the current state and covariance matrix using the state transition equation. In the update step, the Kalman filter uses the sensor measurements and the state estimate obtained in the prediction step to calculate the posterior estimate of the system state using the measurement update equation. It also updates the covariance matrix. This step corrects and optimizes the system state. By recursively performing the prediction and update steps, the Kalman filter algorithm continuously optimizes the state estimate and reduces the uncertainty of the estimate.

[0071] In the multi-target tracking algorithm SORT, the state vector of the target is expressed as In this invention, the target state vector is represented as an octet, as shown in formula (1), where (x c ,y c ) represents the center coordinates of a target on the image plane, (w,h) represents the width and height of the target frame, and the following four parameters represent the rate of change of the first four parameters. The observation vector is shown in formula (2), which represents the center coordinates of a target obtained by the sensor (i.e., the detection model). and width w and Gaoz h According to the modified Kalman filter state vector and observation vector (i.e., formula (1) and formula (2)), the process noise matrix Q k and the observation noise matrix R k It is also modified accordingly, as shown in formulas (3) and (4), and the noise factor σ p , σ m and σ v Set them to 0.05, 0.05 and 0.00625 respectively (can be set by yourself).

[0072]

[0073]

[0074]

[0075]

[0076] Here, σ p and σ vrepresents the noise factor, and Represent the width and height obtained in the k-1 time update phase, that is, in the following update phase The third and fourth parameters in .

[0077]

[0078] Here, σ m represents the noise factor, and Represents the width and height of the prediction stage at the kth moment, that is, the following prediction stage The third and fourth parameters in .

[0079] In this invention, the prediction phase predicts the cell state (i.e., center coordinates, width, height) and covariance. The update phase uses the cell state obtained by the sensor (i.e., the detection model) and the state estimate obtained in the prediction phase to update the system's posterior estimate and covariance matrix. The specific prediction and update steps are as follows:

[0080]

[0081] Here, represents the state quantity predicted at time k (as shown in formula (1)), Q k represents the process noise matrix (i.e., formula (3)), P k|k-1 represent The covariance matrix, F k Represents the prediction matrix, F k The specific form is:

[0082]

[0083]

[0084] Here, P k|k represents the updated covariance matrix, z k represents the observed quantity (i.e., formula (2)), R k represents the observation noise matrix (Formula (4)), H k Represents the mapping matrix, H k The specific form is:

[0085]

[0086] Furthermore, in the data association stage, BYTE is used as the matching strategy. The specific process is as follows:

[0087] (1) Send the video frame to the detection model YOLOv7-XA and obtain the detection target box bb Pred ;

[0088] (2) According to the predicted target box bb Pred The confidence score and detection confidence threshold μ high , μ low The predicted target box obtained in step (1) is divided into high-score detection boxes D high and low-score detection box D low ; (where the predicted target box bb pred The confidence score is greater than μ high , then add the target frame to the high-score detection frame set D high In the case of predicting the target box bb pred The confidence score is less than μ high and is greater than μ low , then add the target frame to the low-score detection frame set D low middle);

[0089] (3) Use the Kalman filter to predict the new position of each trajectory in the trajectory set T of the previous frame to obtain the current trajectory set T.

[0090] (4) Use the Hungarian algorithm to compare the trajectory T with the high-score detection box D high Perform the first matching (using IOU as the similarity metric), and add the unmatched trajectories to the set T remain , the unmatched target box is added to D remain Then use Hungarian matching again to match the unmatched trajectory T remain Compared with the low-score detection box D low Perform a second matching (using IOU as similarity metric) and remove tracks that are still unassigned.

[0091] (5) is D remain Create a new trajectory for the target box in;

[0092] (6) Repeat steps (1)-(5) until all video frames are processed.

[0093] The IOU indicator used in the matching in step (4) here refers to the intersection-over-union ratio, which measures the degree of overlap between two detection boxes (for target detection). The formula is as follows: Among them, B pred and B gt are the target bounding box estimated by the algorithm and the true label bounding box, respectively, and |·| represents the area of ​​the region.

[0094] Finally, trajectory interpolation is used as a post - processing method to fill in the trajectory discontinuities caused by target loss or occlusion during the tracking process. Trajectory interpolation has a certain effect on the trajectory breaks caused by the target being occluded or losing sight. Suppose there is a cell trajectory T now, and due to the movement of the cell, the target is lost between frame t1 and frame t2. The position of the trajectory T at frame t1 is P t1 (x1, y1, x2, y2), that is, a quadruple composed of the upper - left and lower - right coordinates, and P t2 is the coordinate of the trajectory T at frame t2. We set the hyperparameter μ as the maximum interval for performing trajectory interpolation. That is, when t2 - t1 ≤ μ, the trajectory interpolation method will be executed. The calculation method of the interpolation position of the trajectory at the t - th frame (t1 < t < t2) is as follows:

[0095]

[0096] On the other hand, an embodiment of the present invention also provides a Cell - SORT cell tracking device based on deep learning, including the following modules:

[0097] A confidence - score acquisition module, which is used to send a video frame into a trained target - detection model and obtain the confidence score bb of the detected target box Pred ;

[0098] A position - prediction module, which is used to predict the position of the previous - frame trajectory in this frame using a Kalman filter; if it is the first frame, a new trajectory is created according to bb pred If it is not the first frame, the position of the cell in this frame is predicted using the Kalman filter according to the trajectory established in the previous frames;

[0099] A trajectory - update module, which is used to use the BYTE mechanism to use the IOU as a similarity metric for bb pred and the trajectory and perform minimum - cost matching using the Hungarian algorithm. First, according to the confidence - threshold μ high and μ low , bb pred is divided into high - score detection boxes D high and low - score detection boxes D low ]>, and then the Hungarian algorithm is used to perform the first matching of the high - score detection boxes D high with the trajectories. After the first matching is completed, the Hungarian algorithm is used to perform the second matching of the low - score detection boxes D low with the unmatched trajectories; after the two - matching ends, new trajectories are established for the target boxes that are still unmatched in D high and updated into the trajectory set;

[0100] Return to the position prediction module and process the next frame until all video frames are processed. After processing all video frames, the trajectory set T is obtained, and then the trajectory interpolation post-processing method is used to fill T with trajectories to obtain the final tracking result Tracks.

[0101] To illustrate the design of the present invention in terms of tracking accuracy and robustness, the present invention compares and analyzes the tracking performance of two stages of the multi-target tracking process, namely the target detection stage and the data association stage. The present invention is further explained below with reference to charts and specific embodiments.

[0102] Table 1: Comparison of target detection model accuracy

[0103]

[0104] Table 1 shows the YOLOv7-X (original model) and YOLOv7-XA (the present invention proposes, the specific structure is as follows Figure 1 To compare the detection performance of the two detection models, we used mAP, a commonly used metric in the field of object detection, as the evaluation metric for a fair comparison. The two detection models were trained on the CTMC dataset for 200 cycles. The COCO pre-trained model was used as the initial weights, the image input size was 416×416, the learning rate was 0.01, and the optimizer used SGD with a weight decay of 5×10. -4 , the momentum is set to 0.937. Table 1 shows that under the same training parameters, YOLOv7-XA improves the mAP index by 0.8 compared to the original model.

[0105] The explanation of the mAP indicator is as follows:

[0106] First, let’s introduce the IOU indicator. The intersection-over-union (IOU) is a measure of the degree of overlap between two detection boxes (for target detection). The formula is as follows: Among them, B pred and B gt are the target bounding box estimated by the algorithm and the real label bounding box, respectively, and |·| represents the area of ​​the region. For example, AP 50 An IOU greater than or equal to 0.5 is considered to be a detected object. mAP is the average value calculated after setting the IOU threshold to (0.5, 0.55, ..., 0.95). This metric can effectively evaluate the accuracy of the object detection model.

[0107] The accuracy of the target detection model plays a crucial role in the target tracking process. Generally speaking, the higher the detection model's accuracy, the higher the tracking accuracy. To verify the effectiveness of the YOLOv7-XA model in cell tracking, we applied the YOLOv7-X and YOLOv7-XA models to the SORT algorithm and measured the tracking performance using the MOTA and IDF1 metrics. The specific results are shown in Table 2. The SORT tracking algorithm, using YOLOv7-XA as the detection model, achieved significant improvements in both MOTA and IDF1, achieving 2.2 and 1.2, respectively.

[0108] Table 2: Tracking results generated on the test set using different detectors

[0109]

[0110] Here the MOTA and IDF1 indicators are explained as follows:

[0111]

[0112] Here FN t Represents the number of GT bounding boxes that are not matched by any detection box in the t-th frame, FP t represents the number of detection boxes in the tth frame that do not match any GT bounding box, IDSW t Represents the number of times each target ID is switched in the tth frame.

[0113]

[0114] Here, IDTP and IDFP represent the number of correctly assigned and incorrectly assigned IDs, respectively, and IDFN represents the number of IDs that were not assigned due to missed detection.

[0115] The present invention uses the improved Kalman filter (KF+) as a motion model and integrates it into the data association stage. Its target state vector is represented as an eight-dimensional vector, as shown in formula (1), where (x c ,y c ) represents the center coordinate of the target on the image plane, (w,h) represents the width and height of the target frame, and the following four represent the rate of change of each variable. The observation vector is shown in formula (2). According to the modified Kalman filter state vector, its process covariance matrix Q k and the observation noise covariance matrix R k It is also modified accordingly, as shown in formulas (3) and (4). Through continuous iteration, KF+ can accurately predict the target position and provide effective support for subsequent trajectory matching.

[0116]

[0117]

[0118]

[0119]

[0120] In the present invention, BYTE is used as the trajectory matching strategy, and the specific process is as follows.

[0121] (1) Send the video frame to the detection model YOLOv7-XA and obtain the detection target box bb Pred ;

[0122] (2) According to the predicted target box bb Pred The confidence score and detection confidence threshold μ high , μ low The predicted target box obtained in step (1) is divided into high-score detection boxes D high and low-score detection box D low ; (where the predicted target box bb pred The confidence score is greater than μ high , then add the target frame to the high-score detection frame set D high In the case of predicting the target box bb pred The confidence score is less than μ high and is greater than μ low , then add the target frame to the low-score detection frame set D low middle)

[0123] (3) Use the Kalman filter to predict the new position of each trajectory in the trajectory set T of the previous frame to obtain the current trajectory set T.

[0124] (4) Use the Hungarian algorithm to compare the trajectory T with the high-score detection box D high Perform the first matching, and add the unmatched trajectories to the set T remain , the unmatched target box is added to D remain Then use Hungarian matching again to match the unmatched trajectory T remain Compared with the low-score detection box D low Perform a second match;

[0125] (5) is D remain Create a new track for the target box in

[0126] (6) Repeat steps (1)-(5) until all video frames are processed.

[0127] In the present invention, trajectory interpolation is also introduced as a post-processing method. Trajectory interpolation has a certain effect on trajectory breaks caused by the occlusion of the target or the loss of the field of view. Suppose there is now a cell trajectory T, and due to the movement of the cell, the target is lost between frame t1 and frame t2. The coordinates of the trajectory T at frame t1 are P t1 (i.e., the upper left and lower right coordinates of the target), and P t2 are the coordinates of the trajectory T at frame t2. We set the hyperparameter μ as the maximum interval for performing trajectory interpolation. That is, when t2 - t1 ≤ μ, the trajectory interpolation method will be executed. The calculation method of the interpolation position of the trajectory at the t-th frame (t1 < t < t2) is as follows.

[0128]

[0129] Table 3 shows the tracking effects of the cell tracking algorithm after SORT fuses KF+, BYTE, and the trajectory interpolation method respectively. In the experiments in the table, YOLOv7-X-A is used as the detector. In the BYTE mechanism, μ high and μ low are set to 0.6 and 0.25 respectively, and other parameters are the same. It can be seen that after fusing the KF+ and BYTE strategies and performing trajectory interpolation processing, the tracking effect has been significantly improved.

[0130] Table 3: Comparison of tracking effects after fusing KF+ and BYTE

[0131]

[0132] The present invention uses the proposed high-resolution detection model YOLOv7-X-A as the object detection model, and at the same time introduces the KF+ and BYTE mechanisms, as well as trajectory interpolation to reconstruct the SORT data association method. Finally, a deep learning-based Cell-SORT cell tracking method is proposed, which can achieve robust and more accurate cell tracking in the case of few cell features, low resolution, obvious deformation, and complex and variable background noise.

[0133] To illustrate the high robustness and high accuracy of the present invention, mainstream multi-object tracking algorithms are selected for comparison on the same CTMC test set.

[0134] Table 5: Evaluation metrics of mainstream algorithms on the CTMC dataset

[0135]

[0136] As shown in Table 5, the deep learning-based Cell-SORT cell tracking method proposed by the present invention has achieved the best results in multi-object tracking evaluation metrics such as MOTA and IDF1, and has high robustness and accuracy.

[0137] The above content is a further detailed description of the present invention in conjunction with the preferred embodiment, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art should understand that various modifications can be made to the details without departing from the scope of the appended claims, and all should be considered to fall within the scope of protection of the present invention.

Claims

1. A Cell-SORT cell tracking method based on deep learning, characterized in that: The following steps are involved: (1) Send the video frame to the trained target detection model and obtain the detection target box confidence score bb Pred ; (2) Use the Kalman filter to predict the position of the previous frame trajectory in this frame; if it is the first frame, then according to bb pred Create a new trajectory. If it is not the first frame, use the Kalman filter to predict the position of the cell in this frame based on the trajectory established in the previous frame; (3) Using BYTE mechanism to bb pred The trajectory uses IOU as the similarity metric and uses the Hungarian algorithm for minimum cost matching. First, according to the confidence threshold μ high and μ low , bb pred Divided into high-score detection boxes D high and low-score detection box D low , and then use the Hungarian algorithm to convert the high score detection box D high The first match is performed with the trajectory. After the first match is completed, the Hungarian algorithm is used to find the low-score detection box D low Perform a second match with the unmatched trajectory; after the two matches are completed, D high Create a new track for the target frame that is still not matched and update it to the track set; (4) Return to step (2) and process the next frame until all video frames are processed; (5) After processing all video frames, the trajectory set T is obtained, and then the trajectory interpolation post-processing method is used to fill the trajectory of T to obtain the final tracking result Tracks.

2. The deep learning-based Cell-SORT cell tracking method according to claim 1, wherein: The target detection model adopts the original YOLOv7-X model. The YOLOv7-X consists of three parts: Backbone, Neck and Head. The Backbone consists of multiple CBS, ELAN and MPConv modules to extract image features. CBS is a set of convolution operations with different scales, including a convolution layer, a batch normalization layer and an activation function SiLU layer. ELAN is composed of multiple CBS and connection operations. MPConv is composed of CBS convolution operations and MaxPooling layers, which enhances the model's feature fusion extraction capabilities. The Neck part is used to fuse features of different dimensions to obtain richer semantic information. The Head predicts the position and category of the target object based on the feature information.

3. The deep learning-based Cell-SORT cell tracking method according to claim 1, wherein: The target detection model uses YOLOv7-XA. YOLOv7-XA modifies the fusion path of the Neck part based on the original YOLOv7-X. In the original YOLOv7-X model, the feature maps of Neck fusion come from layers 28, 43, and 59 in Backbone. The feature map of layer 28 is input to layer 76 and then to layer 77 for fusion. The feature map of layer 43 is input to layer 62 and then to layer 63 for fusion. The feature map of layer 59 is input to layer 107 for fusion. In YOLOv7-XA, the feature maps of Neck fusion come from the 13th, 28th and 43rd layers in Backbone respectively, that is, the feature map of the 13th layer is input to the 76th layer and then to the 77th layer for fusion, the feature map of the 28th layer is input to the 62nd layer and then to the 63rd layer for fusion, and in the fusion path of the 43rd layer, a 3*3 convolution module is added, which is the 107th layer here. The original 107th layer becomes the 108th layer, and the subsequent layer numbers are increased by one, that is, the feature map of the 43rd layer is output to the 107th layer and then input to the 108th layer for fusion, and the 1*1 convolution modules of the 76th and 62th layers in the original model are replaced by 3*3 convolution modules. The rest of the model is the same as the original model.

4. The deep learning-based Cell-SORT cell tracking method according to claim 1, wherein: The use of the Kalman filter KF+ in step (2) includes two steps: prediction and update. The prediction stage is to predict the state and covariance of the cell. The state of the cell is the center coordinates and the width and height. The update step is to use the cell state obtained by the target detection model and the state estimate obtained in the prediction stage to update the posterior estimate of the system and update the covariance matrix. After continuous prediction and update, the Kalman filter can accurately predict the position of the cell at the next moment. The predicted target state vector is represented as an octet, as shown in formula (1), where (x c ,y c ) represents the center coordinates of a target on the image plane, (w,h) represents the width and height of the target frame, and the following four parameters represent the rate of change of the first four parameters. The observation vector is shown in formula (2), which represents the center coordinates of a target obtained by the sensor or the detection model. and width w and Gaoz h According to the modified Kalman filter state vector and observation vector, namely formula (1) and formula (2), the process noise matrix Q k and the observation noise matrix R k It is also modified accordingly, as shown in formula (3) As shown in (4), Here, σ p and σ v represents the noise factor, and Represent the width and height obtained in the k-1 time update phase, that is, in the following update phase The third and fourth parameters in; Here, σ m represents the noise factor, and Represents the width and height of the prediction stage at the kth moment, that is, the following prediction stage The third and fourth parameters in; The specific prediction and update implementation steps are as follows: Prediction stage: Here, represents the state quantity predicted at time k, as shown in formula (1), Q k Represents the process noise matrix, that is, formula (3), P k|k-1 represent The covariance matrix, F k Represents the prediction matrix, F k The specific form is: Update phase: Here, P k|k represents the updated covariance matrix, z k Represents the observed quantity, that is, formula (2), R k Represents the observation noise matrix, that is, formula (4), H k Represents the mapping matrix, H k The specific form is:

5. The deep learning-based Cell-SORT cell tracking method according to claim 1, wherein: The specific implementation of step (3) is as follows: (31) Send the video frame into the target detection model and obtain the detection target frame bb Pred ; (32) According to the predicted target box bb Pred The confidence score and detection confidence threshold μ high , μ low The detection target frame obtained in step (31) is divided into high-score detection frames D high and low-score detection box D low , that is, predict the target box bb pred The confidence score is greater than μ high , then add the target frame to the high-score detection frame set D high In the case of predicting the target box bb pred The confidence score is less than μ high and is greater than μ low , then add the target frame to the low-score detection frame set D low middle; (33) Use the Kalman filter to predict the new position of each trajectory in the previous frame trajectory set to obtain the current trajectory set T; (34) Using IOU as a similarity metric, the Hungarian algorithm is used to compare the trajectory set T with the high-score detection box D. high Perform the first matching, and add the unmatched trajectories to the set T remain , the unmatched target box is added to D remain , then use IOU as the similarity metric, and use the Hungarian algorithm again to match the unmatched tracks T remain Compared with the low-score detection box D low Perform a second match and delete the tracks that are still unassigned; (35) is D remain Create a new trajectory for the target box in; (36) Repeat steps (31)-(35) until all video frames are processed.

6. The deep learning-based Cell-SORT cell tracking method according to claim 5, characterized in that: The IOU indicator used in the matching in step (34) refers to the intersection-over-union ratio, which measures the degree of overlap between two detection frames. The formula is as follows: Among them, B pred and B gt are the target bounding box pred and the real label bounding box gt estimated by the algorithm, respectively, and |·| represents the area of ​​the region.

7. The deep learning-based Cell-SORT cell tracking method according to claim 1, wherein: The specific implementation of step (5) is as follows: Assume that there is a cell trajectory T. Due to cell movement, the trajectory loses its target between frame t1 and frame t2. The position of trajectory T at frame t1 is P. t1 (x1, y1, x2, y2), which is the quadruple consisting of the coordinates of the upper left corner and the lower right corner, P t2 is the coordinate of trajectory T in frame t2. The hyperparameter μ is set as the maximum interval for performing trajectory interpolation. That is, when t2-t1≤μ, the trajectory interpolation method will be executed. The interpolation position of the trajectory in frame t is calculated as follows: Among them, t1 <t<t2。 8. The deep learning-based Cell-SORT cell tracking method according to claim 1, wherein: The method further includes step (6), wherein the objective evaluation indicators MOTA and IDF1 are used to evaluate the cell tracking effect.

9. A Cell-SORT cell tracking device based on deep learning, characterized in that: Includes the following modules: The confidence score acquisition module is used to feed the video frame into the trained target detection model and obtain the confidence score bb of the detected target frame. Pred ; The position prediction module is used to use the Kalman filter to predict the position of the previous frame trajectory in this frame; if it is the first frame, it is used according to bb pred Create a new trajectory. If it is not the first frame, use the Kalman filter to predict the position of the cell in this frame based on the trajectory established in the previous frame; Trajectory update module, used to use BYTE mechanism to update bb pred The trajectory uses IOU as the similarity metric and uses the Hungarian algorithm for minimum cost matching. First, according to the confidence threshold μ high and μ low , bb pred Divided into high-score detection boxes D high and low-score detection box D low , and then use the Hungarian algorithm to convert the high score detection box D high The first match is performed with the trajectory. After the first match is completed, the Hungarian algorithm is used to find the low-score detection box D low Perform a second match with the unmatched trajectory; after the two matches are completed, D high Create a new track for the target frame that is still not matched and update it to the track set; Return to the position prediction module and process the next frame until all video frames are processed. After processing all video frames, the trajectory set T is obtained, and then the trajectory interpolation post-processing method is used to fill T with trajectories to obtain the final tracking result Tracks.

Images