Cell lineage tracking system and method

Through the combination of deep metric learning and graph neural network processing modules, stable tracking of high-density cell populations and morphologically variable cells is achieved, solving the problem of low cell tracking accuracy and reliability in the prior art.

CN118887220APending Publication Date: 2024-11-01INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410787312.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-18
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

The prior art reduces the accuracy and reliability of cell tracking when dealing with high-density cell populations, large changes in cell migration speed and variable morphology.

Method used

A cell lineage tracking system is adopted, including a cell segmentation module, a feature extraction module and a tracking module. Through deep metric learning and graph neural network processing module, cell image segmentation, feature extraction and association map construction are carried out to achieve stable cell tracking.

Benefits of technology

It improves the accuracy and reliability of cell tracking, and can track cells stably in continuous frames to adapt to complex cell movements and morphological changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118887220A_ABST
    Figure CN118887220A_ABST
Patent Text Reader

Abstract

The invention provides a cell lineage tracking system and method, and the system comprises a cell segmentation module which is used for carrying out the image segmentation of a plurality of cell instances in a microscopic image, and generating a cell mask; the feature extraction module is used for extracting potential features respectively corresponding to the plurality of cell instances from the microscopic image according to a deep metric learning method, and obtaining comprehensive feature description of the plurality of cell instances according to cell spatial features and the potential features; the tracking module is used for constructing an association graph among the plurality of cell instances according to the comprehensive feature description, and tracking the plurality of cell instances by iteratively updating node information in the association graph to obtain a cell lineage tracking trajectory; according to the method, the cells can be stably tracked in continuous frames, and the accuracy and reliability of cell tracking are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a cell lineage tracking system and method. Background Art

[0002] In the biomedical field, accurate monitoring of cell behavior is crucial for understanding complex life processes, and automatic cell detection, instance segmentation, and tracking techniques are key technologies for analyzing complex cellular activities.

[0003] In related technologies, nearest neighbor tracking is usually used to match cells in each frame with the nearest cells in the next frame, but tracking is prone to errors when cells move quickly or change direction; when using a Kalman filter to predict cell positions, it relies on model assumptions and has limited adaptability to nonlinear or complex motions; when using a particle filter to predict cell positions, it has good adaptability to nonlinear and non-Gaussian noise, but the computational cost is relatively high, that is, the current cell tracking equipment has reduced accuracy and reliability in cell tracking when dealing with high-density cell populations, large changes in cell migration speeds, and variable morphologies. Summary of the invention

[0004] The present invention provides a cell lineage tracking system and method, which are used to solve the defects in the prior art that cell tracking is prone to errors when dealing with high-density cell populations, cells with large changes in migration speed, and variable morphologies, resulting in low reliability of cell tracking results, thereby improving the accuracy and reliability of cell tracking.

[0005] The present invention provides a system, comprising the following modules: A cell segmentation module, wherein the cell segmentation module is used to perform image segmentation on multiple cell instances in a microscopic image and generate a cell mask; A feature extraction module, the feature extraction module is used to extract the potential features corresponding to the multiple cell instances from the microscopic image according to the deep metric learning method, and obtain a comprehensive feature description of the multiple cell instances according to the cell spatial features and the potential features; A tracking module is used to construct an association graph between the multiple cell instances according to the comprehensive feature description, and track the multiple cell instances by iteratively updating the node information in the association graph to obtain a cell lineage tracking trajectory.

[0006] According to a cell lineage tracking system provided by the present invention, the cell segmentation module comprises: A SAM large model based on LoRA fine-tuning; the SAM large model based on LoRA fine-tuning is used to perform prompt encoding according to the spatial positions of multiple cell instances to obtain prompt encoding results, encode the high-resolution image according to the pre-trained VisionTransformer to obtain the image encoding result, and perform image segmentation on the multiple cell instances according to the prompt encoding result and the embedded features corresponding to the image encoding result to obtain the cell mask.

[0007] According to a cell lineage tracing system provided by the present invention, the cell segmentation module is further used to perform roundness detection and central cavity detection on the cell mask to obtain a new cell mask.

[0008] According to a cell lineage tracing system provided by the present invention, the feature extraction module is trained based on multiple similarity loss functions.

[0009] According to a cell lineage tracking system provided by the present invention, the tracking module comprises: A graph neural network processing module calculates the attention weights between multiple cell instances through an edge attention mechanism and the comprehensive feature description, and determines the association graph according to the attention weights.

[0010] According to a cell lineage tracking system provided by the present invention, the system further comprises: A trajectory optimization module, wherein the trajectory optimization module is used to perform trajectory optimization and smoothing processing according to the cell lineage tracking trajectory to obtain an optimized cell lineage tracking trajectory; A visualization module is used to display at least one of the cell lineage tracing trajectory and the optimized cell lineage tracing trajectory.

[0011] According to a cell lineage tracking system provided by the present invention, the system further comprises: An analysis and statistics module is used to perform quantitative analysis and counting statistics on at least one of the cell lineage tracing trajectory and the optimized cell lineage tracing trajectory to obtain cell dynamic data.

[0012] The present invention also provides a cell lineage tracing method, comprising the following steps: Perform image segmentation on multiple cell instances in the microscopic image to generate cell masks; Extracting potential features corresponding to the multiple cell instances respectively from the microscopic image according to a deep metric learning method, and obtaining a comprehensive feature description of the multiple cell instances according to the cell spatial features and the potential features; A correlation graph between the multiple cell instances is constructed according to the comprehensive feature description, and the multiple cell instances are tracked by iteratively updating the node information in the correlation graph to obtain a cell lineage tracking trajectory.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the cell lineage tracing methods described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the cell lineage tracing methods described above.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the cell lineage tracing methods described above.

[0016] The cell lineage tracking system and method provided by the present invention respectively perform image segmentation on multiple cell instances in a microscopic image through a cell segmentation module to generate cell masks, extract potential features corresponding to the multiple cell instances from the microscopic image according to a deep metric learning method through a feature extraction module, and obtain comprehensive feature descriptions of the multiple cell instances based on the cell spatial features and the potential features; the tracking module is used to construct an association graph between the multiple cell instances according to the comprehensive feature description, and the multiple cell instances are tracked by iteratively updating the node information in the association graph, so that the cells can be stably tracked in continuous frames, thereby improving the accuracy and reliability of cell tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is one of the structural schematic diagrams of the cell lineage tracing device provided by the present invention.

[0019] Figure 2 This is the second structural schematic diagram of the cell lineage tracing device provided by the present invention.

[0020] Figure 3 It is a schematic diagram of the backbone network framework of the tracking module provided by the present invention.

[0021] Figure 4It is a branch network EA-MPNN framework diagram of the tracking module provided by the present invention.

[0022] Figure 5 This is the third structural schematic diagram of the cell lineage tracing device provided by the present invention.

[0023] Figure 6 It is a schematic diagram of the process of the cell lineage tracing method provided by the present invention.

[0024] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention.

[0025] Reference numerals: 110: cell segmentation module; 120: feature extraction module; 130: tracking module; 131: graph neural network processing module; 140: trajectory optimization module; 150: visualization module; 160: Analysis and statistics module. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Combine the following Figure 1-Figure 6 The cell lineage tracing systems and methods of the present invention are described.

[0028] Figure 1 is one of the structural schematic diagrams of the cell lineage tracing system provided by the present invention, such as Figure 1 As shown, the cell lineage tracking system includes: a cell segmentation module 110 , a feature extraction module 120 and a tracking module 130 .

[0029] The cell segmentation module 110 is used to perform image segmentation on multiple cell instances in the microscopic image to generate cell masks.

[0030] In this embodiment, the cell detection module can be used to automatically identify and locate cell instances in microscopic images based on advanced image processing algorithms and deep learning techniques. For example, the cell detection module is set to a fine-tuned YOLO-V8 model, which is trained for a specific type of cell and can accurately identify and locate the basic position information of the cell. This embodiment can effectively deal with the problems of dense arrangement and mutual occlusion of cells, and can accurately identify the boundaries of each cell so as to obtain its position and morphological information in detail.

[0031] In this embodiment, the cell segmentation module 110 can automatically generate effective prompt information according to the spatial position of the image, and use this information to achieve one-shot segmentation of the image, thereby greatly improving the flexibility and generalization ability of the system; for example, the cell segmentation module 110 passes the location information of the cell instance to the SAM segmentation network in the form of a prompt to optimize the instance segmentation recall rate of specific cells.

[0032] The feature extraction module 120 is used to extract the potential features corresponding to the multiple cell instances respectively from the microscopic image according to the deep metric learning method, and obtain a comprehensive feature description of the multiple cell instances according to the cell spatial features and the potential features.

[0033] In this embodiment, the cell instance includes a single cell instance; in the process of extracting features of the single cell instance, deep metric learning (DML) is used to extract cell potential features from the single cell segmentation mask to discriminate the features of the cell instance.

[0034] In this embodiment, the deep metric learning part chooses to use the neural network basic framework based on Res-Net. According to the cell type category, various network infrastructures such as Res-Net34 and Res-Net18 can be selected.

[0035] In this embodiment, the feature extraction module 120 is trained based on multiple similarity loss functions.

[0036] Specifically, instances of the same cell in different time series are taken as members of the same class, and N mutually exclusive classes are set (N represents the total number of biological cells in the entire sequence). Multiple similarity losses are used to distinguish different cell instances and cluster all instances of the same biological cell. Considering that cells change their appearance over time, this embodiment uses adjacent frame sampling in the training process, and creates a feature affinity matrix by extracting k-class instances of adjacent m frames. In addition, by training the feature extraction DML network, the embedded features of the cell image in the latent space are obtained, and the latent space features of the cell image and the corresponding spatiotemporal features (including cell center coordinates, frame number, intensity statistics, area, major and minor axes of the bounding ellipse, and bounding box coordinates) are encoded into a spatiotemporal feature matrix. ST ) and the spatiotemporal feature matrix (V DML ) for matrix connection (V N =V ST +V DML ) as the node features of the graph neural network, and create a feature matrix, each row of which represents a different cell instance and each column represents the features of the corresponding instance.

[0037] In this embodiment, the DML method is used to extract latent features of cell instances from microscopic images. Through training with multiple similarity losses, instances of the same cell are closely clustered in the feature space to form obvious clusters; while instances of different cells are far away from each other in the feature space, maintaining clear boundaries; the way this feature space is constructed provides strong support for subsequent cell recognition and tracking, allowing the cell lineage tracking system to more accurately identify and distinguish different cell instances.

[0038] In this embodiment, in addition to extracting latent features, basic spatial feature information of cells, such as morphology, size, intensity, etc., is also extracted. These basic spatial features intuitively reflect the appearance and properties of cells and are an important basis for cell recognition. By combining the basic spatial features with the latent spatial features extracted by the DML method, the cell system of this embodiment obtains a comprehensive feature description of the cell instance.

[0039] The tracking module 130 is used to construct an association graph between multiple cell instances based on the comprehensive feature description, and track multiple cell instances by iteratively updating the node information in the association graph to obtain a cell lineage tracking trajectory.

[0040] In this embodiment, an association graph between cell instances is constructed based on the extracted cell instance features. In the association graph, each node represents a cell instance, and the edge represents the relationship between cell instances, such as adjacent, similar, etc.; by constructing the association graph, the dynamic changing relationship between cells can be deeply understood, which provides strong support for subsequent tracking.

[0041] In this embodiment, in order to achieve continuous and stable tracking of cell instances, a classification network can be set to classify these edge features. The classification network can accurately distinguish which edges represent effective connections between cell instances, that is, real tracking trajectories; and which edges are noise or interference and should not be included in the tracking results.

[0042] In this embodiment, the feature vectors of each edge are first extracted during the classification process. The feature vectors of these edges contain rich cell dynamic information. Then, the classification network uses these feature vectors and combines the classification boundaries learned during the training process to efficiently classify the edges. Through the processing of the classification network, the system can accurately identify the effective tracking trajectories between cell instances and eliminate those irrelevant or misleading edges. This tracking method based on edge classification greatly improves the tracking accuracy and stability of the system, allowing cell instances to be effectively tracked and associated in a continuous time series.

[0043] The cell lineage tracking system provided by the embodiment of the present invention performs image segmentation on multiple cell instances in a microscopic image respectively through a cell segmentation module 110 to generate a cell mask, extracts potential features corresponding to the multiple cell instances from the microscopic image according to a deep metric learning method through a feature extraction module 120, and obtains a comprehensive feature description of the multiple cell instances based on the cell spatial features and the potential features; constructs an association graph between the multiple cell instances according to the comprehensive feature description through a tracking module 130, and tracks the multiple cell instances by iteratively updating the node information in the association graph, which can stably track cells in continuous frames and improve the accuracy and reliability of cell tracking.

[0044] Figure 2 This is the second structural schematic diagram of the cell lineage tracking device provided by the present invention. Figure 2 In the illustrated embodiment, in the data processing stage, the cell instance mask annotation, trajectory and lineage information are first obtained from the full spatiotemporal sequence multi-cell image, the cell instance image is edge enhanced or normalized through the image and processing module, and the image is divided to obtain the corresponding verification set, training set or test set; the training set or test set is respectively input into the time series feature module and the image encoder module to obtain the corresponding time series feature and encoding information, and then the mask information is obtained through the lightweight mask decoder module; the training set or test set is respectively input into the graph structure encoding module and the image feature extraction module 120 to obtain the graph structure encoding feature and the corresponding image feature, and the graph structure encoding feature and the corresponding image feature are input into The image is input into the graph neural network message passing module, and the output of the module is input into the edge classification module to obtain the corresponding edge features; in the model application stage, the original cell microscopic image collected in the laboratory is preprocessed by the image preprocessing module, and the preprocessed image and mask information are input into the image segmentation model to obtain the corresponding cell mask, and the cell tracking model is input into the cell tracking model according to the cell mask and edge features to obtain the cell lineage tracking trajectory; finally, the single cell spatiotemporal sequence image is determined according to the cell lineage tracking trajectory, or the cell lineage tracking trajectory is visualized through the visualization module 150; or the tracking result is evaluated through the evaluation model to obtain the corresponding cell dynamic data.

[0045] In some embodiments, the cell segmentation module 110 includes: a SAM large model fine-tuned based on LoRA; the SAM large model fine-tuned based on LoRA is used to perform prompt encoding according to the spatial positions of multiple cell instances to obtain prompt encoding results, encode the high-resolution image according to the pre-trained Vision Transformer to obtain the image encoding result, and perform image segmentation on multiple cell instances according to the embedded features corresponding to the prompt encoding results and the image encoding results to obtain cell masks.

[0046] In this embodiment, advanced deep learning technology is used in the cell image segmentation part. For example, the cell segmentation module 110 is set up using the Segment Anything model based on Vision Transformer (ViT), and combined with the AutoPrompt method to achieve accurate segmentation of cell instances in microscopic images.

[0047] In this embodiment, the ViT-based SAM segmentation network is used as the core segmentation model. The ViT model has a powerful feature extraction capability and can fully capture the detailed information in the image; while the SAM segmentation network uses the alternating update of prompt embedded features and image embedded features to achieve accurate segmentation of cell instances. This prompt-based segmentation method can not only improve the accuracy of segmentation, but also reduce the need for manual participation and improve work efficiency.

[0048] Specifically, image segmentation is achieved through the following steps: (1) First, the original cell microscopic image is cut into multiple patches of the same size and input into the image encoder. The high-resolution image is encoded using the Vision Transformer pre-trained with MAE to obtain image embedding features for the subsequent segmentation process.

[0049] (2) Prompt encoding combined with prior knowledge: The prompt information of the cell image obtained in the detection part is used as prior knowledge, and the embedded feature vector of the prompt is obtained through encoding by the prompt encoder to guide the subsequent segmentation process.

[0050] (3) Based on the segmentation and update of prompt, the prompt embedding features and image embedding features are input into the mask decoder. The prompt content is added to the mask decoding process through alternating and repeated calculation and update of the prompt embedding features and the image embedding features to obtain the prompt-based segmentation result; the prompt embedding features are continuously self-attentioned and cross-attentioned with the image embedding features to update the prompt embedding features. After two layers of MLP, the image embedding features are cross-attentioned again to update the image embedding features; after two alternating updates, the image embedding is upsampled by two layers of transposed convolution to generate a specific segmentation instance output, and three MLPs are performed on some tokens in the prompt encoding to generate the prediction score for the corresponding instance segmentation mask.

[0051] The cell lineage tracing system provided by the embodiment of the present invention uses a large SAM model fine-tuned based on LoRA to perform prompt encoding according to the spatial positions of multiple cell instances, encodes high-resolution images according to a pre-trained Vision Transformer, obtains embedded features according to the two encoding results, and performs image segmentation on multiple cell instances according to the embedded features to obtain cell masks, thereby improving the accuracy and efficiency of cell instance segmentation.

[0052] In some embodiments, the cell segmentation module 110 is further used to perform roundness detection and central cavity detection on the cell mask to obtain a new cell mask.

[0053] In this embodiment, after obtaining the above-mentioned cell mask, a suitable threshold can be set to screen the segmentation mask according to the characteristics of the cell microscopic image, and steps such as roundness detection and center hole detection can be used to screen out suitable instance segmentation masks. Finally, the system saves the screened and processed segmentation results, providing a reliable data basis for subsequent analysis and tracking.

[0054] Specifically, the instance segmentation masks are first screened and saved; a large number of instance segmentation masks are obtained according to the maximum segmentation mask score; according to the average size of cells in the microscopic image, a suitable threshold is set to perform threshold screening on the obtained instance segmentation masks; finally, the obtained segmentation masks are subjected to roundness detection and center hole detection, etc., to screen out suitable instance segmentation masks, and the final segmentation results are saved.

[0055] The cell lineage tracing system provided in the embodiment of the present invention performs roundness detection and central cavity detection on the cell mask through the cell segmentation module 110 to obtain a new cell mask, thereby further improving the quality of the cell mask and providing reliable data support for subsequent cell trajectory tracing.

[0056] In some embodiments, the tracking module 130 includes: a graph neural network processing module 131, which calculates the attention weights between multiple cell instances through an edge attention mechanism and a comprehensive feature description, and determines an association graph based on the attention weights.

[0057] In this embodiment, the nodes in the association graph represent cell instances, and the edges represent the association relationships between instances; the dynamic changes of cell instances in time series are learned through the information propagation and update mechanism of GNN, and the positions of these cell instances in subsequent frames are predicted.

[0058] In this embodiment, the constructed cell graph structure is deeply processed by a graph neural network (GNN), and the node information is propagated and updated in the graph in an iterative manner, thereby achieving deep learning and understanding of the characteristics of cell instances. This embodiment effectively captures the complex relationship between cell instances through alternating updates of edges and nodes, and creates accurate cell tracking trajectories based on this.

[0059] In this embodiment, the key relationship between cell instances is emphasized by introducing an edge attention mechanism to calculate the attention weight between the current edge and adjacent edges and nodes during the GNN processing.

[0060] Specifically, the correlation between each edge and its adjacent edges and nodes is calculated first, and then attention weights are generated based on these correlations; these weights reflect the importance of different edges in the cell tracking process, allowing the system to pay more attention to those edges that have an important impact on the dynamic changes of cells; by calculating attention weights and updating edge features, the system can more accurately capture the dynamic changes between cell instances.

[0061] For example, when cells divide or move, the system can adapt to these changes by adjusting the attention weights of the edges, thereby ensuring the continuity and accuracy of tracking; in addition, the edge attention mechanism can also improve the stability of the system, maintaining stable tracking performance even when cells are densely packed or morphologically complex, so that the system can more comprehensively understand the relationship between cell instances.

[0062] In this embodiment, after fine processing by the graph neural network (GNN), the association graph contains edge feature information of the cell tracking graph structure. These edge features not only contain the spatial relationship between cell instances, but also incorporate the dynamic change information between cells, providing a key basis for subsequent tracking.

[0063] In this embodiment, the graph neural network processing module 131 is set up through the following steps: first, the edge and node features are initialized, where the cell feature matrix is ​​used as the node feature of the graph neural network structure; the edge feature of the graph neural network structure is constructed by the spatial distance and similarity between two adjacent node features, and because V ST and V DML They belong to different scales and need to be homogenized, so a multi-layer perceptron (MLP) network is used to map them to the graph network structure in an end-to-end manner.

[0064] Figure 3 is a schematic diagram of the backbone network framework of the tracking module provided by the present invention. Figure 3In the illustrated embodiment, the GNN network is designed to capture graph dependencies through message passing between nodes, and to calculate node features by recursively aggregating and transforming feature vectors of adjacent nodes; the number of message passing neural network (MPNN) blocks determines the degree of remote interaction between nodes and edges in the separated frames; the traditional framework PDN-Conv uses the incoming edge features to weight the impact on the connected nodes and optimizes the way node features are updated. Based on the above-mentioned message passing network, this embodiment designs a message passing neural network (EA-MPNN) framework for edge features; this embodiment adds an edge encoding module to the MPNN block, uses a two-layer MLP network to encode edge features, and alternately updates node and edge features through a graph attention mechanism to achieve information transmission, thereby achieving mutual updating of edge and node features.

[0065] Figure 4 It is a branch network EA-MPNN framework diagram of the tracking module provided by the present invention. Figure 4 In the embodiment shown, the EA-MPNN network includes an attention layer, a graph convolution layer, and an edge attention layer; the node feature x corresponding to the l_1th node in the association graph is (l-1) and edge features e (l-1) Input to the attention layer, and output the node feature x corresponding to the lth node through the graph convolution calculation of the graph convolution layer (l) , the graph convolution layer uses e (l-1) Perform edge attention calculation to obtain the edge feature e corresponding to the lth node (l) .

[0066] In this embodiment, the GNN network first performs attention sampling on multiple cell instances. In graph data, node updates do not need to calculate the attention of the entire graph, but instead consider the most relevant nodes, thereby reducing the complexity of data processing and filtering the information of noise nodes.

[0067] In this embodiment, an edge attention mechanism is proposed for edge features, and the attention of edge features is calculated using line graph (the current edge features are updated with node features and surrounding edge features), and the node features are merged with the current edge features through the self-attention mechanism to update the node features.

[0068] Specifically, an edge attention block is proposed, each of which accepts a set of node features X (l-1) and edge feature E (l-1) , and generate a new set of edge features E (l) First, based on the edges and nodes of the original graph, a new graph is created, with the edges of the original graph as the nodes of the new graph and the nodes of the original graph as the edges of the new graph, to convert the acyclic directed graph. The calculation formula of the attention weight is as follows: Among them, β pq is the attention weight, N p is the first-order adjacency set of edge p (including p), is the weight vector, F E and F X Represent the number of edge features and node features respectively; is the edge feature of edge p, is the edge feature of edge q, is the connection node feature between edge pq, is the connection node feature between edge pk.

[0069] Among them, the above edge features Updated by the following formula: Among them, σ is the Sigmoid activation function; the node features are updated using the updated edge features, and the expression is: Among them, f Θ is a function implemented by a multilayer perceptron network, which is trained to output a scalar representing the edge weight, f Ω is a vector function implemented by a multi-layer perceptron network, which updates the current feature vector through training; is the node feature of node j, is the node feature of node i.

[0070] In this embodiment, after being processed by the graph neural network, the above-mentioned edge features are input into the edge classification network block (the network block adopts a two-layer MLP structure), and the output is converted into a probability distribution through the softmax activation function; these probabilities are used to determine whether the current link is a cell tracking trajectory.

[0071] In this embodiment, in order to ensure the integrity of tracking, a threshold is set, and only edges with an association probability higher than the threshold are considered to be valid tracking trajectories; the overall network is iteratively optimized using a cross entropy loss function to minimize the difference between the predicted trajectory and the true trajectory.

[0072] The cell lineage tracking system provided by the embodiment of the present invention, by setting a graph neural network processing module 131, utilizes the edge attention mechanism and comprehensive feature description to calculate the attention weights between multiple cell instances, and determines the association graph according to the attention weights, thereby achieving that the cell instances can be effectively tracked and associated in a continuous time series, can accurately identify the effective tracking trajectories between the cell instances, and eliminate those irrelevant or misleading edges, thereby improving the accuracy and stability of cell tracking.

[0073] Figure 5 This is the third structural schematic diagram of the cell lineage tracking device provided by the present invention. Figure 5 In the illustrated embodiment, the cell lineage tracking system further includes: a trajectory optimization module 140, which is used to perform trajectory optimization and smoothing according to the cell lineage tracking trajectory to obtain an optimized cell lineage tracking trajectory; and a visualization module 150, which is used to display at least one of the cell lineage tracking trajectory and the optimized cell lineage tracking trajectory.

[0074] In this embodiment, the broken or wrong connection of the tracking trajectory that may occur in the cell lineage tracking trajectory is repaired and adjusted through the trajectory optimization algorithm; at the same time, the movement continuity, speed change and morphological characteristics of the cell instance in the time series are considered to achieve the connection of the broken trajectory and the correction of the wrong connection, thereby ensuring the continuity and accuracy of the tracking trajectory; For example, mitosis detection is used to repair and adjust the cell lineage tracking trajectory. Since daughter cells usually have different visual features from their parent cells, in most mitotic events, the parent tracking trajectory will terminate and two new daughter tracking trajectories will start. In order to associate all daughter cells with their parent cells, this embodiment considers the detection trajectories of all cells and uses their spatial coordinate features to determine whether they exist in the same neighborhood, and finally generates a cell lineage tracking result, that is, the complete trajectory of each cell from division to extinction.

[0075] In this embodiment, the cell lineage tracking trajectory is smoothed to eliminate possible noise and jitter.

[0076] In this embodiment, the visualization module 150 includes a terminal with a display function, such as a monitoring device, a computer or a mobile phone.

[0077] In this embodiment, the movement trajectory and change process of the cell instance in the time series are displayed through the visualization module 150, thereby realizing the recognition, segmentation and trajectory tracking monitoring of the cell image.

[0078] exist Figure 5 In the illustrated embodiment, the cell lineage tracking system further includes: an analysis and statistics module 160, which is used to perform quantitative analysis and counting statistics on at least one of the cell lineage tracking trajectory and the optimized cell lineage tracking trajectory to obtain cell dynamic data.

[0079] In some embodiments, the cell lineage tracing system supports integration with other bioinformatics tools and databases to enable broader data sharing and analysis collaboration.

[0080] The cell lineage tracing method provided by the present invention is described below. The cell lineage tracing method described below and the cell tracking device described above can be referenced to each other.

[0081] Figure 6 is a schematic diagram of the process of the cell lineage tracing method provided by the present invention, such as Figure 6 As shown, the cell lineage tracing method comprises the following steps: Step 610: Perform image segmentation on multiple cell instances in the microscopic image to generate cell masks.

[0082] In this step, the cell detection module can be used to automatically identify and locate cell instances in the microscopic image based on advanced image processing algorithms and deep learning techniques. For example, the cell detection module is set to a fine-tuned convolutional neural network, for example, the convolutional neural network can be a YOLO-V8 model.

[0083] In this embodiment, the cell segmentation module can automatically generate effective prompt information according to the spatial position of the image, and use this information to achieve one-shot segmentation of the image, thereby greatly improving the flexibility and generalization ability of the system; for example, the cell segmentation module passes the location information of the cell instance to the SAM segmentation network in the form of a prompt to optimize the instance segmentation recall rate of specific cells.

[0084] Step 620: extracting potential features corresponding to the multiple cell instances respectively from the microscopic image according to the deep metric learning method, and obtaining a comprehensive feature description of the multiple cell instances according to the cell spatial features and the potential features.

[0085] In this step, the cell instance includes a single cell instance; in the single cell instance feature extraction process, the feature extraction module uses deep metric learning to extract cell potential features from the single cell segmentation mask and discriminate the features of the cell instance.

[0086] In this embodiment, the deep metric learning part chooses to use the neural network basic framework based on Res-Net. According to the cell type category, various network infrastructures such as Res-Net34 and Res-Net16 can be selected.

[0087] In this embodiment, the feature extraction module is trained based on multiple similarity loss functions.

[0088] In this embodiment, the DML method is used to extract latent features of cell instances from microscopic images. Through training with multiple similarity losses, instances of the same cell are closely clustered in the feature space to form obvious clusters; while instances of different cells are far away from each other in the feature space, maintaining clear boundaries; the way this feature space is constructed provides strong support for subsequent cell recognition and tracking, allowing the cell lineage tracking system to more accurately identify and distinguish different cell instances.

[0089] In this embodiment, in addition to extracting latent features, basic spatial feature information of cells, such as morphology, size, intensity, etc., is also extracted. These basic spatial features intuitively reflect the appearance and properties of cells and are an important basis for cell recognition. By combining the basic spatial features with the latent spatial features extracted by the DML method, the cell system of this embodiment obtains a comprehensive feature description of the cell instance.

[0090] Step 630: construct an association graph between multiple cell instances based on the comprehensive feature description, and track multiple cell instances by iteratively updating node information in the association graph to obtain a cell lineage tracking trajectory.

[0091] In this step, in the association graph, each node represents a cell instance, and the edge represents the relationship between cell instances, such as adjacent, similar, etc.; by constructing the association graph, we can deeply understand the dynamic changing relationship between cells, providing strong support for subsequent tracking.

[0092] In this embodiment, the tracking module extracts the feature vectors of each edge during the classification process, and the feature vectors of these edges contain rich cell dynamic information; then, the classification network uses these feature vectors and combines the classification boundaries learned during the training process to efficiently classify the edges; through the processing of the classification network, the system can accurately identify the effective tracking trajectories between cell instances and eliminate those irrelevant or misleading edges. This tracking method based on edge classification greatly improves the tracking accuracy and stability of the system, allowing cell instances to be effectively tracked and associated in a continuous time series.

[0093] The cell lineage tracing method provided by the embodiment of the present invention performs image segmentation on multiple cell instances in a microscopic image respectively to generate cell masks, extracts potential features corresponding to the multiple cell instances respectively from the microscopic image according to a deep metric learning method, and obtains a comprehensive feature description of the multiple cell instances according to the cell spatial features and the potential features; constructs an association graph between the multiple cell instances according to the comprehensive feature description, and tracks the multiple cell instances by iteratively updating the node information in the association graph, which can stably track cells in continuous frames and improve the accuracy and reliability of cell tracking.

[0094] Figure 7 The following is a schematic diagram of the physical structure of an electronic device, such as Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the cell lineage tracking method, which includes: performing image segmentation on multiple cell instances in the microscopic image to generate cell masks; extracting potential features corresponding to multiple cell instances from the microscopic image according to the deep metric learning method, and obtaining a comprehensive feature description of the multiple cell instances according to the cell spatial features and the potential features; constructing an association graph between multiple cell instances according to the comprehensive feature description, and tracking multiple cell instances by iteratively updating the node information in the association graph to obtain a cell lineage tracking trajectory.

[0095] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0096] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the cell lineage tracing method provided by the above-mentioned methods, which includes: performing image segmentation on multiple cell instances in the microscopic image respectively to generate cell masks; extracting potential features corresponding to multiple cell instances respectively from the microscopic image according to a deep metric learning method, and obtaining a comprehensive feature description of the multiple cell instances based on the cell spatial features and the potential features; constructing an association graph between multiple cell instances based on the comprehensive feature description, and tracking multiple cell instances by iteratively updating the node information in the association graph to obtain a cell lineage tracking trajectory.

[0097] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the cell lineage tracing method provided by the above-mentioned methods, the method comprising: performing image segmentation on multiple cell instances in a microscopic image respectively to generate cell masks; extracting potential features corresponding to the multiple cell instances respectively from the microscopic image according to a deep metric learning method, and obtaining a comprehensive feature description of the multiple cell instances according to the cell spatial features and the potential features; constructing an association graph between the multiple cell instances according to the comprehensive feature description, and tracking the multiple cell instances by iteratively updating the node information in the association graph to obtain a cell lineage tracking trajectory.

[0098] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0099] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cell lineage tracking system, characterized in that: include: A cell segmentation module, wherein the cell segmentation module is used to perform image segmentation on multiple cell instances in a microscopic image and generate a cell mask; A feature extraction module, the feature extraction module is used to extract the potential features corresponding to the multiple cell instances from the microscopic image according to the deep metric learning method, and obtain a comprehensive feature description of the multiple cell instances according to the cell spatial features and the potential features; A tracking module is used to construct an association graph between the multiple cell instances according to the comprehensive feature description, and track the multiple cell instances by iteratively updating the node information in the association graph to obtain a cell lineage tracking trajectory.

2. The cell lineage tracking system according to claim 1, characterized in that: The cell segmentation module comprises: A SAM large model based on LoRA fine-tuning; the SAM large model based on LoRA fine-tuning is used to perform prompt encoding according to the spatial positions of multiple cell instances to obtain prompt encoding results, encode the high-resolution image according to the pre-trained Vision Transformer to obtain the image encoding result, and perform image segmentation on the multiple cell instances according to the prompt encoding result and the embedded features corresponding to the image encoding result to obtain the cell mask.

3. The cell lineage tracking system according to claim 1, characterized in that: The cell segmentation module is also used to perform roundness detection and central cavity detection on the cell mask to obtain a new cell mask.

4. The cell lineage tracking system according to claim 1, characterized in that: The feature extraction module is trained based on multiple similarity loss functions.

5. The cell lineage tracing system according to claim 1, characterized in that: The tracking module includes: A graph neural network processing module calculates the attention weights between multiple cell instances through an edge attention mechanism and the comprehensive feature description, and determines the association graph according to the attention weights.

6. The cell lineage tracking system according to any one of claims 1 to 5, characterized in that: The system further comprises: A trajectory optimization module, wherein the trajectory optimization module is used to perform trajectory optimization and smoothing processing according to the cell lineage tracking trajectory to obtain an optimized cell lineage tracking trajectory; A visualization module is used to display at least one of the cell lineage tracing trajectory and the optimized cell lineage tracing trajectory.

7. The cell lineage tracking system according to any one of claims 1 to 5, characterized in that: The system further comprises: An analysis and statistics module is used to perform quantitative analysis and counting statistics on at least one of the cell lineage tracing trajectory and the optimized cell lineage tracing trajectory to obtain cell dynamic data.

8. A cell lineage tracing method corresponding to the cell lineage tracing system according to claim 1, characterized in that: include: Perform image segmentation on multiple cell instances in the microscopic image to generate cell masks; Extracting potential features corresponding to the multiple cell instances respectively from the microscopic image according to a deep metric learning method, and obtaining a comprehensive feature description of the multiple cell instances according to the cell spatial features and the potential features; A correlation graph between the multiple cell instances is constructed according to the comprehensive feature description, and the multiple cell instances are tracked by iteratively updating the node information in the correlation graph to obtain a cell lineage tracking trajectory.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the cell lineage tracing method as claimed in claim 8 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the cell lineage tracing method according to claim 8 is implemented.

Citation Information

Cited By

  • Edge-enhanced pavement crack segmentation network processing method and system

    CN119579629A

  • Cell tracking method based on intra-frame relation modeling and edge feature map attention network

    CN120070505A

  • Evaluation management method, system and server for cell analysis project

    CN120852313A