Cell tracking method based on intra-frame relation modeling and edge feature map attention network

Abstract

The invention belongs to the technical field of cell tracking, and particularly provides a cell tracking method based on intra-frame relation modeling and an edge feature map attention network, and the method comprises the steps: firstly carrying out the basic feature information extraction and identity distribution of a cell data set; further feature extraction is carried out through deep metric learning, the same cells are classified into one class, and different cells are distinguished; the information of the two is fused, and local tracking features of cell tracking are obtained by applying intra-frame relation modeling; an edge feature map attention network is introduced to update nodes and edge features, and wrong matching information is processed through an edge classifier; and finally, drawing a cell lineage tree according to the probability matrix and the connection matrix to perform lineage tracking on the cells. According to the method, accurate construction and tracking of the cell trajectory are successfully realized, the research process of cell lineage analysis is greatly promoted, and powerful technical support is provided for scientific research in related fields.

Description

Technical Field

[0001] The present invention relates to the technical field of cell tracking, and particularly to a cell tracking method based on intra-frame relationship modeling and edge feature map attention network. Background Art

[0002] Cells, as the basic units of living organisms, exhibit phenomena such as cell growth, cell movement, cell division, and cell apoptosis, and these phenomena are closely related to the development and lesions of living organisms. For example: in biology, analyzing how complex multi-organisms are gradually constructed from a single cell is crucial for studying cell fate determination; in cancer research, cell tracking helps to reveal the invasion and metastasis mechanisms of tumor cells and provides a basis for developing new treatment strategies; in drug evaluation, the effects and mechanisms of drugs are evaluated by observing specific cell behavior changes before and after drug treatment. However, tracking individual cells in consecutive frames, especially in co-culture, is confounded by problems such as cell overlap, morphological variability, and complex cell-cell interactions.

[0003] With the rapid development of computer science and bio-image informatics, cell tracking is no longer limited to manual observation of staining tracking. Researchers calculate the similarity matrix between cells and use strategies such as nearest neighbor and minimum distance matching to connect cells in adjacent frames. This method is relatively automated, but it is still limited by factors such as cell overlap and morphological changes. The optical flow method that infers the movement direction and speed of cells based on the change of pixel intensity in the image sequence is applicable to the situation where cell movement is relatively smooth and continuous. These traditional cell tracking methods often need to be combined with image preprocessing techniques, such as background subtraction, denoising, enhancement, etc., to improve the accuracy and robustness of tracking. Modern cell tracking more tends to use algorithms to improve the degree of automation and tracking accuracy.

[0004] Cell tracking faces more uncertainty challenges compared to vehicle and pedestrian tracking. First of all, the movement of cells is completely random. They do not follow a fixed path or direction like vehicles or pedestrians, nor do they have a unified speed norm. Secondly, in the same observation sequence, the appearance of some cells is extremely similar, increasing the difficulty of distinction; even more, some cells will change their appearance characteristics over time. Moreover, due to the frequent occurrence of cell division, the number of cells increases rapidly during the tracking process, and the cell lineage also changes rapidly. These uncertainty factors make it difficult for traditional manual feature selection methods to achieve precise tracking of cells.

[0005] Therefore, we propose a cell tracking technical method based on intra-frame relationship modeling and edge feature map attention network to solve the above problems. Summary of the Invention

[0006] (1) Technical problem to be solved

[0007] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a cell tracking method based on intra-frame relationship modeling and edge feature map attention network, which solves the problems of cell mis-matching and trajectory disconnection existing in the prior art.

[0008] (2) Technical solution

[0009] In order to achieve the above object, the present invention proposes a cell tracking method based on intra-frame relationship modeling and edge feature map attention network. By applying deep metric learning to extract features of cell images, and then using intra-frame relationship modeling and edge feature map attention network to train and update node and edge features, finally activating edge vectors through an edge classifier to display the lineage tree of cells. The present invention can automatically analyze and process a large amount of cell image data to efficiently and accurately track the dynamic changes of cells.

[0010] The technical idea for implementing the present invention is to combine deep metric learning with an edge feature map attention network to build an end-to-end cell tracking framework, which can identify and track cells in a complex background while ensuring the continuity and accuracy of tracking. Its specific implementation includes the following steps:

[0011] S1. Prepare the dataset: Use the 2D dataset in the cell tracking challenge as the research object of this experiment;

[0012] S2. Feature extraction: Apply deep metric learning to the 2D dataset to form a metric space, and optimize the similarity by learning the cosine distance metric in this space, so that the distance between cells of the same class is as small as possible, and the distance between cells of different classes is as large as possible;

[0013] S3. Intra-frame relationship modeling: Detect the two neighbors closest to each cell within the frame for the cell image sequence, and aggregate the feature information of the neighbors through directed edges;

[0014] S4. Message passing: Apply the message passing paradigm to process cell image data, and the correlation of cell instances will be affected by higher-order connections along the sequence until a farther distance;

[0015] S5. Update of node features: Use the node feature information of intra-frame relationship modeling as the initial feature vector of the node, and apply the edge feature map attention network to update the features of the node;

[0016] S6. Update of edge features: The edge feature update block of the edge feature map attention network receives a set of node features and a set of edge features, updates the edge features, and then returns the updated features of each edge through an edge update function;

[0017] S7. Edge classifier: The output edge feature vector is the input of the edge classifier, which consists of a multi-layer perceptron with three linear layers. The edge classifier is trained according to the edge activation vector of GT to represent the probability of actual cell association;

[0018] S8. Construction of trajectories: Finally, a directed graph with association probabilities is used to predict the movement path of cells, and the probability matrix for identifying active edges and the connection matrix are applied together to construct the movement trajectories of cells.

[0019] Furthermore, in the feature extraction of step S2, deep metric learning is mainly applied, which includes three major components: sampler, network model architecture, and loss function. In order to maintain the consistency of the time series during cell tracking, the original sampler is improved so that it samples in adjacent frames to avoid confusion.

[0020] Furthermore, in the intra-frame relationship modeling of step S3, after fusing the spatio-temporal features and the features learned by deep metric learning, intra-frame relationship modeling is applied. The weight value assigned to itself by the node is 1. When calculating the new intra-frame feature vector of the node, if the neighbor is actually the target node itself, then the weight of the edge is set to 1. This processing is to ensure that during the process of propagating information, the basic features of each node are retained while integrating the information of neighbors.

[0021] Furthermore, in the message passing of step S4, the GNN model and message passing are mainly used to track the entire cell orbit simultaneously, rather than associating cell instances frame by frame. For the feature update of a certain cell in a certain frame, it is not only affected by its direct neighbors in the same frame, but also affected by the cells in the previous frame or earlier frames.

[0022] Furthermore, in the update of edge features in step S6, when updating edge features, the nodes and edges need to be mutually converted, which requires creating a new graph based on the original graph, and its nodes and edges are the edges and nodes in the original graph respectively.

[0023] Furthermore, in the edge classifier of step S7, the edge feature vector completed after edge feature update is the input of the edge classifier. The Sigmoid function is applied to process the edge feature vector in the output layer. (\(= 1\)) indicates that the edge is in an active state, and (\(= 0\)) indicates that the edge is in an inactive state, that is, the association probability of the actual cell.

[0024] (III) Beneficial effects

[0025] Compared with the prior art, the present invention provides a cell tracking method based on intra-frame relationship modeling and edge feature map attention network, having the following beneficial effects:

[0026] Improved Sampler: In the feature extraction stage, by improving the sampling method, sampling is ensured between adjacent frames, thus maintaining the consistency of the time series and avoiding the confusion problems that may occur in traditional methods. This helps to improve the accuracy and coherence of cell tracking.

[0027] Intra-frame Relationship Modeling: After feature fusion, by assigning a weight value of 1 to the node itself and specifically considering the influence of neighbor nodes when calculating the new feature vector of the node, it is ensured that the basic features of each node are retained during the information transmission process, while effectively integrating the information of surrounding neighbors, improving the richness and accuracy of feature representation.

[0028] GNN-based Message Passing: In the message passing stage, a graph neural network (GNN) model is used to track the entire cell trajectory simultaneously, rather than the traditional method of associating cell instances frame by frame. This method not only considers the inter-cell relationships within the same frame but also the influence of cells across frames, and can capture the dynamic change process of cells more accurately.

[0029] Dynamic Update of Edge Features: In the edge feature update stage, a new mechanism is proposed, that is, converting nodes and edges mutually to create a new graph structure, whose nodes and edges are respectively the edges and nodes in the original graph. This innovation helps to better capture and express the dynamic relationships between cells.

[0030] Application of Edge Classifier: Finally, in the edge classifier stage, the Sigmoid function is used to process the edge feature vector to determine whether the edge is active, that is, the actual association probability between cells. This processing method provides an effective decision-making mechanism for cell tracking and can help accurately distinguish real cell associations from false associations. Description of the Drawings

[0031] Figure 1 Cell tracking framework for intra-frame relationship modeling and edge feature map attention network.

[0032] Figure 2 Network model architecture for deep metric learning.

[0033] Figure 3 Node and edge transformation matrix for edge feature map attention network.

[0034] Figure 4 Cosine distance similarity image of edge encoder.

[0035] Figure 5 Lineage graph of cells. Detailed Implementation Manner

[0036] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0037] Embodiment

[0038] As Figures 1-5 shown, a cell tracking method based on intra-frame relationship modeling and edge feature map attention network proposed in an embodiment of the present invention includes the following steps:

[0039] S1. Prepare the dataset: Use the 2D dataset in the cell tracking challenge as the research object of this experiment;

[0040] S11. First, apply the label function to the cell grayscale image to label the connected regions, and assign a unique label to each different connected region, where the background is labeled as 0;

[0041] S12. Apply the regionprops function to receive this labeled image as input and calculate attributes for each region: calculate the area by the number of pixels in the region, use the coordinates of the center point of the region as the centroid of the cell, and use the smallest rectangle containing the region as the bounding box of the cell.

[0042] S2. Feature extraction: Apply deep metric learning to the 2D dataset to form a metric space, and optimize the similarity by learning the cosine distance metric in this space, so that the distance between cells of the same class is as small as possible, and the distance between cells of different classes is as large as possible;

[0043] S21. Deep metric learning mainly consists of three parts: sampler, network model architecture, and loss function;

[0044] S22. The sampler of deep metric learning randomly samples from the dataset. However, in cell tracking, the consistency of the time series needs to be maintained. Therefore, the sampling is improved so that it samples from adjacent times. By randomly selecting a central frame and selecting symmetric and asymmetric frame sets before and after this central frame, the rationality of sampling cell images in the video sequence and time series data center is ensured;

[0045] S23. The network model architecture uses the ResNet18 network applied to large-scale datasets such as ImageNet. Before feature extraction, the cell images are first normalized and uniformly scaled to 224×224 pixels, and then passed through a series of residual blocks to gradually extract higher-level features while retaining lower-level feature information;

[0046] S24. To mine those samples that are difficult to be correctly classified, the present invention applies a multi-similarity loss function to measure hard positive examples and hard negative examples, and the formula is as follows:

[0047]

[0048] where α, β, and λ are hyperparameters, and S ik is the cosine similarity of the sample pair.

[0049] S3. Intra-frame relationship modeling: For the cell image sequence, select the two neighbors closest to each cell within the frame for detection, and aggregate the feature information of the neighbors through directed edges;

[0050] S31. Before performing intra-frame relationship modeling, it is necessary to fuse the spatio-temporal features V ST extracted by applying the regionprops function and the features V DML extracted by deep metric learning. The formula is as follows:

[0051] V V = V ST + V DML

[0052] S32. Select the two neighbors closest to each cell for detection, and aggregate the feature information of the neighbors through directed edges. The formula for calculating the weight of the edge is as follows:

[0053]

[0054] where (x i , y i ) is the coordinate position information of the centroid of node i, j, s, and t are all nodes in the cell image, s represents the s-th neighbor of node i, t represents the t-th neighbor of node j, and e i,s and e j,t are the weights of the edges connecting two nodes within the same frame;

[0055] S33. Aggregate the feature information of neighbor nodes according to the weight of the edge. The formula is as follows:

[0056]

[0057] where the node assigns a weight value of 1 to itself. When calculating the new intra-frame feature vector of node j, if neighbor t is actually the target node itself, then the weight of the edge is set to 1. This processing is to ensure that during the process of propagating information, the basic features of each node are retained while integrating the information of neighbors. Each node itself is the most important source of information, and the local features of the node within the frame are obtained

[0058] S4. Message Passing: Apply the message passing paradigm to process cell image data. The correlation of cell instances is affected by higher-order connections along the sequence up to greater distances.

[0059] S41. Use GNN and message passing to process graph-structured data, enabling the inclusion of information about the entire cell trajectory during the study, rather than performing local associations frame by frame as in traditional methods.

[0060] S42. Introduce an edge encoder to expand message passing. Use the node features modeling the intra-frame relationship as the initial features of the nodes, and apply edge feature map attention to update the nodes and edges, which is called the message passing neural network with intra-frame and edge encoding.

[0061] S43. In the last message passing block, each cell and the cell instances of its first-order connections are affected by the cell associations in the entire sequence of frames and the combined effect of higher-order connections at greater distances.

[0062] S5. Update of Node Features: Use the node feature information modeling the intra-frame relationship as the initial feature vector of the nodes, and apply the edge feature map attention network to update the features of the nodes.

[0063] S51. Use the node features modeling the intra-frame relationship as the initial features of the nodes, and apply edge feature map attention to update them. The edge feature map attention network mainly consists of two blocks: the node update block and the edge update block.

[0064] S52. When updating the nodes, the node update block receives a set of node features X and a set of edge features Z and generates a new set of node features.

[0065] S53. For each node, weights w i are calculated for each j ∈ N ij , where N i is the set including the first-order neighbors of node i and node i itself. During this process, the features will be concatenated and parameterized by the weight vector , and LeakyReLU is applied as the activation function. The softmax function is used to normalize these weights obtained from the selection of nodes j in all N i . The whole process is expressed by the following formula:

[0066]

[0067] S54. After obtaining the normalized attention weights for each neighborhood, perform a weighted sum of these neighborhood node features. The output of node update can be expressed as:

[0068]

[0069] S6. Update of edge features: The edge feature update block of the edge feature map attention network receives a set of node features and a set of edge features, updates the edge features, and then returns the updated features of each edge through an edge update function;

[0070] S61. The update of edge features is to update the features of each edge by aggregating the features of adjacent edges. To achieve aggregation, the nodes and edges in the graph are mutually transformed, and a new graph is created based on the original graph, whose nodes and edges are the edges and nodes of the original graph respectively;

[0071] S62. For each edge p, the normalized attention weight formula of edge q is as follows:

[0072]

[0073] where N p is the set of first-order neighbors of edge p (including p), is the weight vector;

[0074] S63. Similar to node features, the calculation formula of the new edge feature set is as follows:

[0075]

[0076] S64. Use D-S to represent a function that returns the distance and similarity vector metric of two feature vectors connecting nodes. The formula is as follows:

[0077]

[0078] S65. Return the updated features of each edge e i,j through an edge update function. The formula is as follows:

[0079]

[0080] S7. Edge classifier: The output edge feature vector is the input of the edge classifier. The edge classifier consists of a multi-layer perceptron with three linear layers. The edge classifier is trained according to the edge activation vector of GT to represent the probability of actual cell association;

[0081] S71. To identify the behavior and dynamics of cells, the outgoing / incoming edges of specific nodes are considered. The following situations may occur:

[0082] S72. All outgoing / incoming edges are inactive, which may indicate the end or start of a trajectory;

[0083] S73. Only one outgoing edge is active, which usually indicates the normal movement of cells in consecutive frames, that is, a cell instance continues to move to another position in the next frame;

[0084] S74. Two or more outgoing edges are active, which may indicate cell mitosis, that is, a cell divides into two or more cells;

[0085] S75. Multiple incoming edges are active, which means that different cell instances are associated with the same cell instance, possibly due to occlusion or other reasons for incorrect matching. To avoid multiple cell instances being incorrectly associated with the same cell instance, the incoming edge with the highest association probability (as long as it is higher than 0.5) is regarded as the active edge.

[0086] S8. Construction of the trajectory: Finally, a directed graph with association probabilities is used to predict the movement path of cells, and the probability matrix for identifying active edges and the connection matrix are applied together to construct the movement trajectory of cells;

[0087] S81. To create the longest relevant cell sequence belonging to the same biological cell in consecutive frames, the time points of the first and last appearances of the cell are marked, and a corresponding biological number is assigned to each cell. Even if the cell is marked as different instances at different time points, the longest continuous dynamic trajectory of each cell from its first appearance to its final disappearance can be correctly assembled.

[0088] The above makes an exemplary description of the present invention. It should be noted that the above is only the preferred embodiment of the present invention without departing from the core of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

[0089] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A cell tracking method based on intra-frame relationship modeling and edge feature map attention network, characterized in that: The following steps are involved: S1. Prepare the dataset: Use the 2D dataset in the cell tracking challenge as the research object of this experiment; S2. Feature extraction: Apply deep metric learning to the 2D dataset to form a metric space, and optimize the similarity in this space by learning the cosine distance metric, so that the distance between cells of the same type is as small as possible, and the distance between cells of different types is as large as possible; S3, intra-frame relationship modeling: for the cell image sequence, the two nearest neighbors to each cell are selected within the frame for detection, and the feature information of the neighbors is aggregated through directed edges; S4. Message passing: Applying the message passing paradigm to process cell image data, the correlation of cell instances is affected by higher-order connections along the sequence up to longer distances; S5, node feature update: the node feature information of the intra-frame relationship modeling is used as the initial feature vector of the node, and the edge feature graph attention network is applied to update the node features; S6. Update of edge features: The edge feature update block of the edge feature graph attention network receives a set of node features and a set of edge features, updates the edge features, and then returns the updated features of each edge through an edge update function; S7, edge classifier: The output edge feature vector is the input of the edge classifier, which consists of a multi-layer perceptron with three linear layers. The edge classifier is trained based on the edge activation vector of GT to represent the probability of actual cell association. S8. Trajectory construction: Finally, a directed graph with associated probabilities is used to predict the movement path of the cell, and the probability matrix of the identified active edges and the connection matrix are used to jointly construct the movement trajectory of the cell.

2. According to claim 1, a cell tracking method based on intra-frame relationship modeling and edge feature map attention network is characterized in that: Feature extraction in step S2: The samples that have passed through the sampler are subjected to a ResNet18 network to uniformly scale the cell images to 224×224 pixels, and higher-level features are extracted through a series of residual modules while retaining lower-level feature information, and a loss function is applied to mine samples that are difficult to be correctly classified.

3. The cell tracking method based on intra-frame relationship modeling and edge feature map attention network according to claim 1, characterized in that: In step S3, the intra-frame relationship modeling uses the features learned through deep metric learning as the input of the intra-frame relationship modeling, considers the interactive information between cells and neighboring cells, and extracts the local information of cells.

4. The cell tracking method based on intra-frame relationship modeling and edge feature map attention network according to claim 1, characterized in that: The message passing in step S4 is: by expanding the message passing module of the cell tracking framework - based on the intra-frame and edge message passing neural network, the node and edge features can cooperate with each other during the update process and be carried out simultaneously with the edge feature map attention network.

5. The cell tracking method based on intra-frame relationship modeling and edge feature map attention network according to claim 1, characterized in that: The edge classifier in step S7: the updated edge feature vector is the input of the edge classifier, and the Sigmoid function is applied to the edge feature vector in the output layer, (=1) indicates that the edge is in an active state, and (=0) indicates that the edge is in an inactive state, that is, the association probability of the actual cell.

6. The cell tracking method based on intra-frame relationship modeling and edge feature map attention network according to claim 1, characterized in that: The construction of the trajectory in step S8: In order to create the longest related cell sequence belonging to the same biological cell in consecutive frames, the time points of the first and last appearance of the cell are marked, and a corresponding biological number is assigned to each cell. Even if the cells are marked as different instances at different time points, the longest continuous dynamic trajectory of each cell from its first appearance to its final disappearance can be correctly assembled.

Images