Follow-up CT lymph node tracking method, system, device and medium based on time and space

Abstract

The present invention discloses a method, system, device and medium for follow-up CT lymph node tracking based on time and space, which belongs to lymph node tracking in the field of artificial intelligence technology, and its purpose is to solve the technical problem of poor lymph node tracking effect in the prior art. The constructed CT lymph node tracking model introduces a self-supervised morphological enhancement module to improve the encoder's ability to capture lymph node features; an adaptive graph matching method is adopted to dynamically update the feature representation of lymph nodes using the powerful spatial feature extraction capability of the graph convolutional neural network; it also includes a time enhancement module and a space-time fusion module to further improve the feature learning of the time dimension, and improve the prediction accuracy of new or disappeared lymph nodes by fine-tuning and correcting the graph matching results. This framework is not only suitable for matching a variety of lesion organ tissues, but also has significant advantages for lymph nodes with small volume and blurred imaging in a complex pelvic environment, and is particularly outstanding in colorectal cancer lymph node matching and tracking.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and relates to the tracking of lymph nodes, and in particular to the tracking of lymph nodes in colorectal CT images, and more specifically to a time-space-based follow-up CT lymph node tracking method, system, device and medium. Background Art

[0002] With the development of medical technology, some cancers have gradually shown a trend of becoming chronic. Colorectal cancer is a common chronic disease worldwide, with high morbidity and mortality rates. For the image analysis of lymph nodes in colorectal cancer, doctors mostly analyze the cross-comparison of multiple scan images and slice sequences to evaluate the progression of the disease and guide treatment decisions.

[0003] With the development and application of artificial intelligence technology, its application in the field of disease diagnosis and treatment technology has become more and more extensive, especially in the intelligent recognition of medical images. The invention patent application with application number 202110626711.X discloses a method for diagnosing lymph node metastasis of rectal cancer based on deep learning multimodal CT, which includes: preprocessing the multimodal CT images of rectal cancer, and extracting image features from the cropped fixed-size 3D plain scan CT images and 3D enhanced CT images through the newly constructed Mlenet (multimodal Lenet convolutional neural network) convolutional neural network, splicing the feature maps to form new feature maps, and inputting the new feature maps into the new Mlenet convolutional neural network for binary classification prediction to obtain binary classification prediction results.

[0004] The invention patent application with application number 202110936944.X also discloses a method, system, intelligent terminal and computer-readable storage medium for predicting the prognosis of stage II colorectal cancer. It establishes a prognosis prediction model for stage II CRC patients by fusing the imaging genomics features and deep learning features of the primary lesion and surrounding lymph nodes in CT scans. Compared with a single imaging genomics model or a single deep learning model, the joint model of this application has the highest prediction efficiency, providing a preoperative, non-invasive and repeatable prognosis prediction method for stage II colorectal cancer patients. On the basis of the segmentation model of the deep convolutional neural network, it fixes the deep feature layer of the deep convolutional neural network, connects the fully connected layer with the downstream pooling layer, establishes an initial deep learning model, and uses the Adam optimizer with a learning rate of 0.0001 and cross entropy to train the initial deep learning model to obtain a trained deep learning model.

[0005] The invention patent application with application number 202310536861.0 also discloses a method for predicting lymph node metastasis of rectal cancer based on a neural network combined with traditional imaging, which predicts whether lymph node metastasis occurs in rectal cancer; in order to construct a feature extraction model, deep learning technology is used to construct a convolutional neural network (CNN) and a U-Net framework based on a fully convolutional neural network, and the Pyradiomics package is used to extract radiological features; finally, feature selection is performed on the extracted features and a binary classification model is constructed, and F1-Score is used for evaluation. In the prediction model of rectal cancer lymph node metastasis, it is found that the XGBoost ensemble learning algorithm is better than other traditional algorithms, and the prediction accuracy can reach 91%.

[0006] As in the above-mentioned invention patent application, most of the prior art uses neural networks to automatically identify and process CT images, which is more about identifying, segmenting and other processing of lymph nodes in CT images at a certain time point. However, in the management of chronic diseases, medical image follow-up examinations are an indispensable component. The comparison and analysis of follow-up images are of great significance for evaluating the condition, monitoring the treatment effect and selecting treatment plans. Therefore, it is necessary to obtain CT images of the same patient at different time points, and compare and track the lymph nodes in the CT images at different time points. The prior art basically does not pay attention to the time series data of the same patient, nor does it develop relevant models for tracking lymph nodes. In addition, due to the changes in the patient's body position when scanning at different time points, fluctuations in health conditions and discontinuity of scanning time intervals, the size, appearance and shape of the same lymph node in CT images at different time points will be significantly different. Therefore, how to track the follow-up CT lymph nodes, how to establish a front-to-back relationship between the lymph nodes in the abdominal CT images taken at different times, and how to identify the lymph nodes in the CT images of different periods as the same lymph nodes will be the key issues to be solved in the process of CT lymph node tracking. Summary of the invention

[0007] The purpose of the present invention is to provide a time-space-based follow-up CT lymph node tracking method, system, device and medium in order to solve the technical problem in the prior art that the size, appearance shape and the like of the target lymph nodes in CT images at different time points may change due to changes in body position, health status, scanning time interval, and lymph node growth and dissipation, which interferes with lymph node identification and tracking and results in poor tracking effect.

[0008] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0009] A time-space-based follow-up CT lymph node tracking method comprises the following steps:

[0010] Step 1: Obtain sample and label data;

[0011] Obtain abdominal CT three-dimensional images taken of multiple patients at different time points, and extract abdominal CT two-dimensional images from the abdominal CT three-dimensional images to obtain CT three-dimensional sample data and CT two-dimensional sample data; annotate the matching relationship between each lymph node in the CT three-dimensional sample data of the same patient at different time points to obtain label data;

[0012] Step 2, constructing a CT lymph node tracking model;

[0013] Construct a CT lymph node tracking model, which includes a two-dimensional lymph node extraction network, a three-dimensional lymph node extraction network, a three-dimensional time-enhanced feature extraction network, a graph neural network, and a lymph node addition and disappearance prediction module;

[0014] Step 3, training the CT lymph node tracking model;

[0015] The CT two-dimensional sample data and the CT three-dimensional sample data are respectively input into the two-dimensional lymph node extraction network and the three-dimensional lymph node extraction network, and are trained using the self-supervised contrast loss; the CT three-dimensional sample data and the label data are input into the three-dimensional time-enhanced feature extraction network, and are trained using the time contrast loss; the features output by the three-dimensional lymph node extraction network or the three-dimensional time-enhanced feature extraction network and the center coordinates of each lymph node in the abdominal CT three-dimensional image are input into the graph neural network, the graph neural network first calculates the coordinate difference between each lymph node and other lymph nodes, and then calculates the weight according to the coordinate difference, and then selects the features of the corresponding lymph nodes according to the weight for feature aggregation, and adds the aggregated features to the original features of the lymph node to complete the feature update of the lymph node, and calculates the similarity between the updated features and all the lymph nodes in the CT image of the same patient at another time point, and judges that the lymph node and a lymph node in the CT image at another time point belong to the same lymph node according to the calculated similarity; the abdominal CT three-dimensional image and its corresponding two abdominal CT two-dimensional images are input into the lymph node addition and disappearance prediction module, and the lymph node addition and disappearance prediction module outputs whether a certain lymph node is a new or disappeared lymph node;

[0016] Step 4, real-time tracking of lymph nodes;

[0017] The abdominal CT three-dimensional test images taken at different time points of the patient to be tested are obtained, and the abdominal CT two-dimensional test images are extracted from the abdominal CT three-dimensional test images. The abdominal CT three-dimensional test images and the abdominal CT two-dimensional test images are input into the trained CT lymph node tracking model, and the CT lymph node tracking model outputs the tracking results of each lymph node.

[0018] Furthermore, in step 2, the two-dimensional lymph node extraction network adopts a two-dimensional ResNet-34 network, and the three-dimensional lymph node extraction network adopts a three-dimensional ResNet-34 network, and the residual structures of the two-dimensional ResNet-34 network and the three-dimensional ResNet-34 network include a batch normalization layer, a linear rectification unit, a weight calculation unit, a batch normalization layer, a linear rectification unit and a weight calculation unit arranged in sequence, and the input passes through the batch normalization layer, the linear rectification unit, the weight calculation unit, the batch normalization layer, the linear rectification unit and the weight calculation unit in sequence, and the output of the weight calculation unit is superimposed on the input as the output of the entire residual structure.

[0019] Furthermore, in step 2, the three-dimensional time enhanced feature extraction network adopts a three-dimensional ResNet-34 network, and the residual structure of the three-dimensional ResNet-34 network includes a batch normalization layer, a linear rectification unit, a weight calculation unit, a batch normalization layer, a linear rectification unit and a weight calculation unit arranged in sequence, and the input passes through the batch normalization layer, the linear rectification unit, the weight calculation unit, the batch normalization layer, the linear rectification unit and the weight calculation unit in sequence, and the output of the weight calculation unit is superimposed on the input as the output of the entire residual structure.

[0020] Furthermore, in step 2, in the graph neural network, the specific processing method is:

[0021] First, according to the lymph nodes With lymph nodes The normalized center coordinates of , , count lymph nodes With lymph nodes The weight between :

[0022] ;

[0023] By aggregating all connected lymph nodes Features , count lymph nodes Aggregation features :

[0024] ;

[0025] Use the self-attention mechanism to calculate the importance weight of aggregated information :

[0026] ;

[0027] Add the aggregated features to the original features of the lymph node to obtain the updated lymph node features , and update the CT image; updated lymph node features The update method is:

[0028] ;

[0029] For updated CT images from two different CT scans, calculate the cosine similarity of each pair of lymph nodes , and based on the calculated similarity, it is determined that the lymph node belongs to the same lymph node as a lymph node in the CT image at another time point; cosine similarity The calculation method is:

[0030] ;

[0031] in, Indicates a click operation. Indicates calculating the modulus of the vector. Indicates lymph nodes The normalized center coordinates of Indicates lymph nodes The normalized center coordinates of Indicates lymph nodes The initial characteristics of represents transpose, represents the dimension of the key vector, Indicates lymph nodes The initial characteristics of Represents the lymph nodes in the CT image taken at time T1 Updated features, Represents the lymph nodes in the CT image taken at time T2 Updated features.

[0032] Furthermore, in step 2, in the lymph node new disappearance prediction module, the specific processing method is as follows:

[0033] Select two most representative abdominal CT two-dimensional images from the nine abdominal CT two-dimensional images of the abdominal CT three-dimensional image, specifically:

[0034] ;

[0035] For all lymph nodes in a pair of CT scans, the abdominal CT 3D images of each pair of lymph nodes and the two selected abdominal CT 2D images were fused pairwise; for each pair of lymph nodes from two CT scans ( , ), calculate the pairwise fusion features:

[0036] ;

[0037] The fused features are input into the fully connected layer, which outputs the similarity of each pair of lymph nodes and determines whether a lymph node is a newly added lymph node or a disappeared lymph node based on the similarity.

[0038] in, Indicates The first two-dimensional view in the orientation The adaptive weights of the features of the lymph nodes, Indicates The first two-dimensional view in the orientation The adaptive weights of the features of the lymph nodes, represents the value of the independent variable that makes the function reach the maximum value, Indicates In the view The characteristics of the lymph nodes, Indicates In the view The characteristics of the lymph nodes, Represents a fusion operation.

[0039] Furthermore, the fusion features are input into the fully connected layer and the similarity of each pair of lymph nodes is output, specifically:

[0040] First, each fusion feature is placed in the fully connected layer for mapping transformation to obtain the mapping value; the mapping transformation is specifically:

[0041] ;

[0042] Remap the mapped value to between 0 and 1 to obtain the mapping result; the remapping is as follows:

[0043] ;

[0044] Change the mapping results into binary classification results, specifically:

[0045] ;

[0046] Calculate the comprehensive similarity of each fusion feature, specifically:

[0047] ;

[0048] Prediction is performed based on the binary classification results and comprehensive similarity, and the prediction matching similarity of each fusion feature , specifically:

[0049]

[0050] For the nine predicted matching similarities generated, generate the final similarity ;

[0051] ;

[0052] According to the final similarity , the final similarity lower than 0.5 is defined as mismatch, and a lymph node is judged as a newly added or disappeared lymph node;

[0053] in, represents the weight matrix, represents the bias term, represents the fusion feature, Represents the Sigmoid activation function.

[0054] Furthermore, in step 3, the overall loss function of the CT lymph node tracking model is for:

[0055] ;

[0056] The 2D lymph node extraction network and the 3D lymph node extraction network are trained using self-supervised contrast loss. The 2D-3D self-supervised contrast loss function for:

[0057] ;

[0058] The 3D time-enhanced feature extraction network is trained using time contrast loss. The time contrast loss function for:

[0059] ;

[0060] Graph neural network uses binary cross entropy for supervised training, position information loss function for:

[0061] ;

[0062] When training the new lymph node disappearance prediction module, a new disappearance prediction loss function is added for:

[0063] ;

[0064] in, Represents the three-dimensional image The three-dimensional characteristics of lymph nodes, Indicates The first two-dimensional view in the orientation The two-dimensional features of lymph nodes, Indicates The first two-dimensional view in the orientation The two-dimensional features of lymph nodes, represents the temperature parameter, represents the cosine similarity function, Indicates lymph nodes at T1 The three-dimensional features of Indicates lymph nodes at T2 The three-dimensional features of Indicates the number of lymph node pairs that can be formed at T1 and T2. Represents the lymph nodes in the CT image at time T1 Updated lymph node characteristics, Represents the lymph nodes in the CT image at time T2 updated lymph node characterization; Represents label data, which is 0 or 1, 1 means the two lymph nodes match, 0 means the two lymph nodes do not match; and same, Indicates the final similarity.

[0065] A time-space-based follow-up CT lymph node tracking system includes the following steps:

[0066] The sample and label data acquisition module is used to obtain abdominal CT three-dimensional images taken by multiple patients at different time points, and extract abdominal CT two-dimensional images from the abdominal CT three-dimensional images to obtain CT three-dimensional sample data and CT two-dimensional sample data; annotate the matching relationship between each lymph node in the CT three-dimensional sample data of the same patient at different time points to obtain label data;

[0067] CT lymph node tracking model construction module, used to build a CT lymph node tracking model. The CT lymph node tracking model includes a two-dimensional lymph node extraction network, a three-dimensional lymph node extraction network, a three-dimensional time-enhanced feature extraction network, a graph neural network, and a lymph node addition and disappearance prediction module;

[0068] The CT lymph node tracking model training module is used to input CT two-dimensional sample data and CT three-dimensional sample data into the two-dimensional lymph node extraction network and the three-dimensional lymph node extraction network respectively, and use self-supervised contrast loss for training; input CT three-dimensional sample data and label data into the three-dimensional time-enhanced feature extraction network, and use time contrast loss for training; input the features output by the three-dimensional lymph node extraction network or the three-dimensional time-enhanced feature extraction network and the center coordinates of each lymph node in the abdominal CT three-dimensional image into the graph neural network, and the graph neural network first calculates the coordinate difference between each lymph node and other lymph nodes, and then uses the coordinate difference to train the three-dimensional lymph node extraction network. to calculate the weight, and then select the features of the corresponding lymph nodes according to the weight for feature aggregation, and add the aggregated features to the original features of the lymph node to complete the feature update of the lymph node, and calculate the similarity between the updated features and all the lymph nodes in the CT image of the same patient at another time point, and judge whether the lymph node and a lymph node in the CT image at another time point belong to the same lymph node according to the calculated similarity; input the abdominal CT three-dimensional image and its corresponding two abdominal CT two-dimensional images into the lymph node addition and disappearance prediction module, and the lymph node addition and disappearance prediction module outputs whether a lymph node is a new or disappeared lymph node;

[0069] The real-time lymph node tracking module is used to obtain the abdominal CT three-dimensional test images taken of the patient at different time points, and extract the abdominal CT two-dimensional test images from the abdominal CT three-dimensional test images, and input the abdominal CT three-dimensional test images and the abdominal CT two-dimensional test images into the trained CT lymph node tracking model, and the CT lymph node tracking model outputs the tracking results of each lymph node.

[0070] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0071] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the steps of the above method.

[0072] The beneficial effects of the present invention are as follows:

[0073] 1. In the present invention, the morphological, temporal and spatial characteristics of lymph nodes are integrated to optimize the matching process of lymph nodes in follow-up CT scans. This can solve the technical problem that the size and appearance of target lymph nodes in CT images at different time points may change due to changes in body position, health status, scanning time interval, and lymph node growth and disappearance, which interferes with lymph node identification and tracking and causes poor tracking effect. This can effectively cope with changes in the number of lymph nodes in CT scans and detect new or disappeared targets. To enhance feature expression, it innovatively introduces a self-supervised morphological enhancement module to improve the encoder's ability to capture lymph node features. It adopts an adaptive graph matching method and uses the powerful spatial feature extraction capability of a graph convolutional neural network to dynamically update the feature representation of lymph nodes. It also includes a temporal enhancement module and a spatiotemporal fusion module to further improve feature learning in the temporal dimension, and improves the prediction accuracy of new or disappeared lymph nodes by fine-tuning and correcting the graph matching results. This framework is not only suitable for matching a variety of lesion organ tissues, but also has significant advantages for small and blurred lymph nodes in complex pelvic environments, and is particularly outstanding in colorectal cancer lymph node matching and tracking.

[0074] 2. In the present invention, a self-supervised contrastive learning method is used to input different sequences into the network, and the feature extraction ability of the three-dimensional network is improved through the two-dimensional modality; this method is highly transferable and suitable for a variety of approximate three-dimensional medical image inputs, which significantly enhances the performance of the encoder, reduces the dependence on labeled data, reduces the cost of data labeling, and provides reliable support for a variety of future research.

[0075] 3. In this invention, based on the inter-lymph node vector generated by normalized coordinates, the distance and direction information is integrated into the initial weight of the edge in the graph; at the same time, combined with the self-attention mechanism, the features of each node are adaptively updated during the feature aggregation and node update process of the graph; this scheme provides a new idea for the modeling and application of location information in medical images. Experimental results show that this method effectively improves the accuracy of feature update and enhances the sensitivity of the model to surrounding information.

[0076] 4. In the present invention, two-dimensional and three-dimensional features from multiple perspectives are used to construct a robust feature fusion and prediction process through a multi-instance learning scheme, which successfully solves the problem that the model has poor learning effect on the essential features of lymph nodes in the time dimension in the traditional feature fusion process. Experiments have shown that this scheme significantly improves the performance of the model, enabling it to more accurately identify and match lymph nodes.

[0077] 5. In the present invention, multi-instance learning innovatively combines multiple fusion vectors with classification results and numerical regression results, while ensuring the reliability of the results, and enhancing the model's ability to learn data from different perspectives and multi-dimensional data. This scheme is applicable to input sequences of different modalities, providing strong support for future research. In addition, algorithm complexity analysis shows that this method has good feasibility and efficiency when processing multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a schematic diagram of the process of the present invention;

[0079] Figure 2 It is a schematic diagram of the structure of the CT lymph node tracking model in the present invention;

[0080] Figure 3 It is a schematic diagram of the structure of the residual structure of the ResNet-34 network in the present invention;

[0081] Figure 4 is a schematic diagram of updating lymph node characteristics by the neural network in the present invention;

[0082] Figure 5 It is a structural schematic diagram of the time dimension mixer in the present invention. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0084] Therefore, based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0085] Example 1

[0086] This embodiment provides a time-space-based follow-up CT lymph node tracking method for tracking each lymph node in CT images collected at different time points during the follow-up process, such as Figure 1 As shown, the following steps are included:

[0087] Step 1: Obtain sample and label data;

[0088] Obtain abdominal CT three-dimensional images of multiple patients taken at different time points, and extract abdominal CT two-dimensional images from the abdominal CT three-dimensional images to obtain CT three-dimensional sample data and CT two-dimensional sample data; annotate the matching relationship between each lymph node in the CT three-dimensional sample data of the same patient at different time points to obtain label data.

[0089] The abdominal CT 3D images come from various hospitals, such as the West China Hospital of Sichuan University. In many medical imaging tasks, the 2D method has been proven to be more effective in capturing details and processing efficiency; therefore, in order to better capture the non-rigid changes of lymph nodes and avoid losing too much information, this embodiment will extract nine 2D views in different directions from the 3D lymph node area of ​​the abdominal CT 3D image, and use both the 3D image and the 2D image for subsequent model training, further enhancing the learning ability of the 3D model, thereby improving the 3D model's ability to express the morphological characteristics of lymph nodes.

[0090] Step 2, constructing a CT lymph node tracking model;

[0091] A CT lymph node tracking model was constructed. The CT lymph node tracking model includes a two-dimensional lymph node extraction network, a three-dimensional lymph node extraction network, a three-dimensional time-enhanced feature extraction network, a graph neural network, and a lymph node addition and disappearance prediction module. Figure 2 shown.

[0092] In order to bridge the natural gap between extracting features from two-dimensional and three-dimensional images in contrastive learning, a two-dimensional lymph node extraction network and a three-dimensional lymph node extraction network were designed in the CT lymph node tracking model. The two-dimensional lymph node extraction network and the three-dimensional lymph node extraction network respectively use a two-dimensional ResNet-34 network and a three-dimensional ResNet-34 network, which extract 128-dimensional two-dimensional and three-dimensional features of lymph nodes respectively. Among them, the two-dimensional ResNet-34 network and the three-dimensional ResNet-34 network can use the existing ResNet-34 network without creative labor.

[0093] The residual structures of the two-dimensional ResNet-34 network and the three-dimensional ResNet-34 network are as follows Figure 3 As shown, they all include a batch normalization layer, a linear rectification unit, a weight calculation unit, a batch normalization layer, a linear rectification unit and a weight calculation unit, which are arranged in sequence. The input passes through the batch normalization layer, the linear rectification unit, the weight calculation unit, the batch normalization layer, the linear rectification unit and the weight calculation unit in sequence. The output of the weight calculation unit is superimposed on the input as the output of the entire residual structure.

[0094] The CT lymph node tracking model adopts a 3D time enhancement strategy, that is, a 3D time enhancement feature extraction network is added to the CT lymph node tracking model. The 3D time enhancement feature extraction network adopts a 3D ResNet-34 network, and the specific structure of the 3D time enhancement feature extraction network is the same as that of the 3D lymph node extraction network. The 3D time enhancement feature extraction network compares different lymph node features from the same pair of CT scans and combines real annotations (i.e., label data) for supervised learning to ensure that the model can learn core features that are not affected by non-rigid changes.

[0095] Since lymph nodes may undergo non-rigid changes and the images may differ greatly in subsequent CT scans, doctors usually rely on environmental information around the lymph nodes to match lymph nodes in different CT scans of the same patient. This environmental information includes not only adjacent organs, but also adjacent lymph nodes. Therefore, in order to effectively deal with this non-rigid change, a graph neural network is introduced into the CT lymph node tracking model. In the graph neural network of this embodiment, each lymph node in the same CT scan is regarded as a node, and the 128-dimensional features extracted by the three-dimensional lymph node extraction network are used as the initial node features. All nodes (lymph nodes) construct a graph structure in a fully connected manner, and the central coordinates of each lymph node in the CT scan are normalized. In the node update process, the coordinate difference between each node and other nodes is first calculated, and the difference is used to calculate the weight of the edge, which plays a key role in the node information aggregation process. Subsequently, the importance weight of the aggregated information is calculated by the self-attention mechanism to better determine the characteristics of the lymph nodes. Finally, the aggregated features are added to the original features to complete the node update.

[0096] In the graph neural network, the specific processing method is:

[0097] Lymph node (i.e., the nodes in the graph ) has the following initial characteristics: , lymph nodes The normalized coordinates of First, according to the lymph nodes With lymph nodes The normalized center coordinates of , , calculate the weight between the two :

[0098] ;

[0099] By aggregating all connected lymph nodes Features , count lymph nodes Aggregation features , specifically:

[0100] ;

[0101] Next, the importance weight of the aggregated information is calculated using the self-attention mechanism , the calculation formula is:

[0102] ;

[0103] Add the aggregated features to the original features of the lymph node to obtain the updated lymph node features , and update the CT image; updated lymph node features The update method is:

[0104] .

[0105] The updating process of the graph is as follows Figure 4 shown.

[0106] For updated CT images from two different CT scans, calculate the cosine similarity of each pair of lymph nodes , and based on the calculated similarity, it is determined that the lymph node belongs to the same lymph node as a lymph node in the CT image at another time point; cosine similarity The calculation method is:

[0107] ;

[0108] in, Indicates a click operation. Indicates calculating the modulus of the vector. Indicates lymph nodes The normalized center coordinates of Indicates lymph nodes The normalized center coordinates of Indicates lymph nodes The initial characteristics of represents transpose, represents the dimension of the key vector, Indicates lymph nodes The initial characteristics of Represents the lymph nodes in the CT image taken at time T1 Updated features, Represents the lymph nodes in the CT image taken at time T2 Updated features.

[0109] For the lymph node matching results after spatial enhancement, the Hungarian matching algorithm was finally used to achieve global matching of all lymph nodes in the two CT scans. However, this method has limitations in identifying newly appeared or disappeared lymph nodes. In addition, after applying the constraints of spatial and morphological dimensions, it is hoped that the model can further learn the temporal consistency of non-rigidly changing lymph nodes. Therefore, this prompted the introduction of more constraints in the model to explore whether matching can be performed only by the characteristics of the lymph nodes themselves. To this end, a lymph node addition and disappearance prediction module was introduced into the CT lymph node tracking model. Specifically, the lymph node addition and disappearance prediction module selects the two most representative features from the 128-dimensional features extracted from the two-dimensional views in nine different directions using adaptive weights, and combines them with the corresponding 128-dimensional three-dimensional features. The specific processing method is:

[0110] Set up The two-dimensional features of nine different directions of lymph nodes are ,in Represents 9 two-dimensional views in different orientations. The adaptive weights of these features are expressed as Based on these weights, the two most representative 2D features are selected , ,in:

[0111] ;

[0112] Therefore, each lymph node has three characteristics: , , .

[0113] For all lymph nodes in a pair of CT scans, the abdominal CT 3D images of each pair of lymph nodes and the two selected abdominal CT 2D images are fused pairwise to predict their similarities. This means that a total of 9 predicted matching similarities will be generated for each pair of lymph nodes. Redefine, let and Respectively represent the two selected 2D view features, let Represents the 3D view features of a lymph node. For each pair of lymph nodes from two CT scans ( , ), calculate the pairwise fusion features:

[0114] ;

[0115] The fusion diagram is as follows: Figure 5 shown.

[0116] The fused features are input into the fully connected layer, which outputs the similarity of each pair of lymph nodes and determines whether a lymph node is a newly added lymph node or a disappeared lymph node based on the similarity.

[0117] in, Indicates The first two-dimensional view in the orientation The adaptive weights of the features of the lymph nodes, Indicates The first two-dimensional view in the orientation The adaptive weights of the features of the lymph nodes, represents the value of the independent variable that makes the function reach the maximum value, Indicates In the view The characteristics of the lymph nodes, Indicates In the view The characteristics of the lymph nodes, Represents a fusion operation.

[0118] Then, for each mixed feature vector, each fused feature is placed in the fully connected layer for mapping transformation to obtain the mapping value; the mapping transformation is specifically:

[0119] ;

[0120] Remap the mapped value to between 0 and 1 to obtain the mapping result; the remapping is as follows:

[0121] ;

[0122] Change the mapping results into binary classification results, specifically:

[0123] ;

[0124] In addition, each mixed feature vector needs to obtain a similarity value and a distribution value to obtain a comprehensive similarity; specifically:

[0125] ;

[0126] Predict the predicted matching similarity of each fusion feature based on the binary classification results and comprehensive similarity , specifically:

[0127] ;

[0128] For the nine predicted matching similarities generated, the final similarity is generated by combining ;

[0129] ;

[0130] For the final similarity , the model defines a mismatch if the final similarity is less than 0.5. If a lymph node in one CT scan does not find any similar lymph nodes in another CT scan, it will be classified as a new or disappeared lymph node, and a lymph node will be judged as a new or disappeared lymph node (determined according to the time point of the CT scan. If it is present in the previous CT scan but not in the later CT scan, it will be judged as disappeared; if it is not present in the previous CT scan but in the later CT scan, it will be judged as a new lymph node;

[0131] in, represents the weight matrix, represents the bias term, represents the fusion feature, Represents the Sigmoid activation function.

[0132] Step 3, training the CT lymph node tracking model;

[0133] The CT two-dimensional sample data and the CT three-dimensional sample data are respectively input into the two-dimensional lymph node extraction network and the three-dimensional lymph node extraction network, and are trained using the self-supervised contrast loss; the CT three-dimensional sample data and the label data are input into the three-dimensional time-enhanced feature extraction network, and are trained using the time contrast loss; the features output by the three-dimensional lymph node extraction network or the three-dimensional time-enhanced feature extraction network and the center coordinates of each lymph node in the abdominal CT three-dimensional image are input into the graph neural network, the graph neural network first calculates the coordinate difference between each lymph node and other lymph nodes, and then calculates the weight according to the coordinate difference, and then selects the features of the corresponding lymph nodes according to the weight for feature aggregation, and adds the aggregated features to the original features of the lymph node to complete the feature update of the lymph node, and calculates the similarity between the updated features and all the lymph nodes in the CT image of the same patient at another time point, and judges that the lymph node belongs to the same lymph node as a lymph node in the CT image at another time point according to the calculated similarity; the abdominal CT three-dimensional image and its corresponding two abdominal CT two-dimensional images are input into the lymph node addition and disappearance prediction module, and the lymph node addition and disappearance prediction module outputs whether a lymph node is a new or disappeared lymph node.

[0134] Overall loss function of CT lymph node tracking model for:

[0135] ;

[0136] When performing 2D-3D self-supervised contrastive learning, the 2D features of the same lymph node are used as positive samples of its corresponding 3D features, while the 2D features of other lymph nodes in the same batch are used as negative samples. The two-dimensional features of nine different directions of lymph nodes are ,in, represents 9 two-dimensional views in different directions, and the three-dimensional features are ; Assume that the rest of the same batch The remaining nine two-dimensional features in different directions are expressed as ,in This strategy enhances the ability of the 3D lymph node extraction network to extract meaningful lymph node features. The 2D lymph node extraction network and the 3D lymph node extraction network are trained using self-supervised contrast loss. The 2D-3D self-supervised contrast loss function for:

[0137] .

[0138] In the supervised learning of the 3D time-enhanced feature extraction network, 256-dimensional 3D features are used and projected into different feature spaces, so that the two contrasting training strategies focus on different aspects of learning features. The characteristics of the lymph nodes at time point T1 are , No. The characteristics of the lymph nodes at time point T2 are , the 3D time-enhanced feature extraction network is trained using time contrast loss, which helps the model learn the final changes of lymph nodes over time. Time contrast loss function for:

[0139] .

[0140] Binary cross entropy is used as the loss function to supervise the training of the graph neural network, ensuring that it can effectively distinguish the same lymph nodes that have undergone non-rigid changes without over-homogenizing the features of the lymph nodes in the same CT scan. Position information loss function during graph neural network training for:

[0141] ;

[0142] If the lymph nodes at T1 and lymph nodes at T2 For the same node in the label, =0, otherwise =0.

[0143] When judging whether a lymph node is new or missing, the model will incorporate this information into the spatial matching results to form a final comprehensive judgment. for:

[0144] ;

[0145] in, Represents the three-dimensional image The three-dimensional characteristics of lymph nodes, Indicates The first two-dimensional view in the orientation The two-dimensional features of lymph nodes, Indicates The first two-dimensional view in the orientation The two-dimensional features of lymph nodes, represents the temperature parameter, represents the cosine similarity function, Indicates lymph nodes at T1 The three-dimensional features of Indicates lymph nodes at T2 The three-dimensional features of Indicates the number of lymph node pairs that can be formed at T1 and T2. Represents the lymph nodes in the CT image at time T1 Updated lymph node characteristics, Represents the lymph nodes in the CT image at time T2 updated lymph node characterization; Represents label data, which is 0 or 1, 1 means the two lymph nodes match, 0 means the two lymph nodes do not match; and same, Represents the final similarity.

[0146] Step 4, real-time tracking of lymph nodes;

[0147] The abdominal CT three-dimensional test images taken at different time points of the patient to be tested are obtained, and the abdominal CT two-dimensional test images are extracted from the abdominal CT three-dimensional test images. The abdominal CT three-dimensional test images and the abdominal CT two-dimensional test images are input into the trained CT lymph node tracking model, and the CT lymph node tracking model outputs the tracking results of each lymph node.

[0148] Example 2

[0149] This embodiment provides a time-space-based follow-up CT lymph node tracking system, including:

[0150] The sample and label data acquisition module is used to obtain abdominal CT three-dimensional images taken of multiple patients at different time points, and extract abdominal CT two-dimensional images from the abdominal CT three-dimensional images to obtain CT three-dimensional sample data and CT two-dimensional sample data; the matching relationship between each lymph node in the CT three-dimensional sample data of the same patient at different time points is annotated to obtain label data.

[0151] The abdominal CT 3D images come from various hospitals, such as the West China Hospital of Sichuan University. In many medical imaging tasks, the 2D method has been proven to be more effective in capturing details and processing efficiency; therefore, in order to better capture the non-rigid changes of lymph nodes and avoid losing too much information, this embodiment will extract nine 2D views in different directions from the 3D lymph node area of ​​the abdominal CT 3D image, and use both the 3D image and the 2D image for subsequent model training, further enhancing the learning ability of the 3D model, thereby improving the 3D model's ability to express the morphological characteristics of lymph nodes.

[0152] The CT lymph node tracking model construction module is used to construct the CT lymph node tracking model. The CT lymph node tracking model includes a two-dimensional lymph node extraction network, a three-dimensional lymph node extraction network, a three-dimensional time-enhanced feature extraction network, a graph neural network, and a lymph node addition and disappearance prediction module.

[0153] The two-dimensional lymph node extraction network and the three-dimensional lymph node extraction network use the two-dimensional ResNet-34 network and the three-dimensional ResNet-34 network respectively, which extract the 128-dimensional two-dimensional and three-dimensional features of the lymph nodes respectively. Among them, the two-dimensional ResNet-34 network and the three-dimensional ResNet-34 network can use the existing ResNet-34 network without creative labor.

[0154] The residual structures of the two-dimensional ResNet-34 network and the three-dimensional ResNet-34 network are as follows Figure 3 As shown, they all include a batch normalization layer, a linear rectification unit, a weight calculation unit, a batch normalization layer, a linear rectification unit and a weight calculation unit, which are arranged in sequence. The input passes through the batch normalization layer, the linear rectification unit, the weight calculation unit, the batch normalization layer, the linear rectification unit and the weight calculation unit in sequence. The output of the weight calculation unit is superimposed on the input as the output of the entire residual structure.

[0155] The three-dimensional time-enhanced feature extraction network adopts a three-dimensional ResNet-34 network, and the specific structure of the three-dimensional time-enhanced feature extraction network is the same as that of the three-dimensional lymph node extraction network.

[0156] In the graph neural network of this embodiment, each lymph node in the same CT scan is regarded as a node, and the 128-dimensional features extracted by the three-dimensional lymph node extraction network are used as the initial node features. All nodes (lymph nodes) construct a graph structure in a fully connected manner, and the center coordinates of each lymph node in the CT scan are normalized. In the node update process, the coordinate difference between each node and other nodes is first calculated, and the difference is used to calculate the weight of the edge, which plays a key role in the node information aggregation process. Subsequently, the importance weight of the aggregated information is calculated through the self-attention mechanism to better determine the characteristics of the lymph nodes. Finally, the aggregated features are added to the original features to complete the node update.

[0157] In the graph neural network, the specific processing method is:

[0158] Lymph node (i.e., the nodes in the graph ) has the following initial characteristics: , lymph nodes The normalized coordinates of First, according to the lymph nodes With lymph nodes The normalized center coordinates of , , calculate the weight between the two :

[0159] ;

[0160] By aggregating all connected lymph nodes Features , count lymph nodes Aggregation features , specifically:

[0161] ;

[0162] Next, the importance weight of the aggregated information is calculated using the self-attention mechanism , the calculation formula is:

[0163] ;

[0164] Add the aggregated features to the original features of the lymph node to obtain the updated lymph node features , and update the CT image; updated lymph node features The update method is:

[0165] .

[0166] The updating process of the graph is as follows Figure 4 shown.

[0167] For updated CT images from two different CT scans, calculate the cosine similarity of each pair of lymph nodes , and based on the calculated similarity, it is determined that the lymph node belongs to the same lymph node as a lymph node in the CT image at another time point; cosine similarity The calculation method is:

[0168] ;

[0169] in, Indicates a click operation. Indicates calculating the modulus of the vector. Indicates lymph nodes The normalized center coordinates of Indicates lymph nodes The normalized center coordinates of Indicates lymph nodes The initial characteristics of represents transpose, represents the dimension of the key vector, Indicates lymph nodes The initial characteristics of Represents the lymph nodes in the CT image taken at time T1 Updated features, Represents the lymph nodes in the CT image taken at time T2 Updated features.

[0170] The lymph node addition and disappearance prediction module selects the two most representative features from the 128-dimensional features extracted from the two-dimensional views in nine different directions using adaptive weights and combines them with the corresponding 128-dimensional three-dimensional features. The specific processing method is as follows:

[0171] Set up The two-dimensional features of nine different directions of lymph nodes are ,in Represents 9 two-dimensional views in different orientations. The adaptive weights of these features are expressed as Based on these weights, the two most representative 2D features are selected , ,in:

[0172] ;

[0173] Therefore, each lymph node has three characteristics: , , .

[0174] For all lymph nodes in a pair of CT scans, the abdominal CT 3D images of each pair of lymph nodes and the two selected abdominal CT 2D images are fused pairwise to predict their similarities. This means that a total of 9 predicted matching similarities will be generated for each pair of lymph nodes. Redefine, let and Respectively represent the two selected 2D view features, let Represents the 3D view features of a lymph node. For each pair of lymph nodes from two CT scans ( , ), calculate the pairwise fusion features:

[0175] ;

[0176] The fusion diagram is as follows: Figure 5 shown.

[0177] The fused features are input into the fully connected layer, which outputs the similarity of each pair of lymph nodes and determines whether a lymph node is a newly added lymph node or a disappeared lymph node based on the similarity.

[0178] in, Indicates The first two-dimensional view in the orientation The adaptive weights of the features of the lymph nodes, Indicates The first two-dimensional view in the orientation The adaptive weights of the features of the lymph nodes, represents the value of the independent variable that makes the function reach the maximum value, Indicates In the view The characteristics of the lymph nodes, Indicates In the view The characteristics of the lymph nodes, Represents a fusion operation.

[0179] Then, for each mixed feature vector, each fused feature is placed in the fully connected layer for mapping transformation to obtain the mapping value; the mapping transformation is specifically:

[0180] ;

[0181] Remap the mapped value to between 0 and 1 to obtain the mapping result; the remapping is as follows:

[0182] ;

[0183] Change the mapping results into binary classification results, specifically:

[0184] ;

[0185] In addition, each mixed feature vector needs to obtain a similarity value and a distribution value to obtain a comprehensive similarity; specifically:

[0186] ;

[0187] Predict the predicted matching similarity of each fusion feature based on the binary classification results and comprehensive similarity , specifically:

[0188] ;

[0189] For the nine predicted matching similarities generated, the final similarity is generated by combining ;

[0190] ;

[0191] For the final similarity , the model defines a mismatch if the final similarity is less than 0.5. If a lymph node in one CT scan does not find any similar lymph node in another CT scan, it is classified as a new or disappeared lymph node, and a lymph node is judged to be a new or disappeared lymph node (determined according to the time point of the CT scan. If it is present in the previous CT scan but not in the later CT scan, it is judged to be disappeared; if it is not present in the previous CT scan but in the later CT scan, it is judged to be a new lymph node).

[0192] in, represents the weight matrix, represents the bias term, represents the fusion feature, Represents the Sigmoid activation function.

[0193] The CT lymph node tracking model training module is used to input CT two-dimensional sample data and CT three-dimensional sample data into the two-dimensional lymph node extraction network and the three-dimensional lymph node extraction network respectively, and use self-supervised contrast loss for training; input CT three-dimensional sample data and label data into the three-dimensional time-enhanced feature extraction network, and use time contrast loss for training; input the features output by the three-dimensional lymph node extraction network or the three-dimensional time-enhanced feature extraction network and the center coordinates of each lymph node in the abdominal CT three-dimensional image into the graph neural network, and the graph neural network first calculates the coordinate difference between each lymph node and other lymph nodes, and then uses the coordinate difference to train the three-dimensional lymph node extraction network. to calculate the weight, and then select the features of the corresponding lymph nodes according to the weight for feature aggregation, and add the aggregated features to the original features of the lymph node to complete the feature update of the lymph node, and calculate the similarity between the updated features and all the lymph nodes in the CT image of the same patient at another time point, and judge whether the lymph node belongs to the same lymph node as a lymph node in the CT image at another time point according to the calculated similarity; input the abdominal CT three-dimensional image and its corresponding two abdominal CT two-dimensional images into the lymph node addition and disappearance prediction module, and the lymph node addition and disappearance prediction module outputs whether a lymph node is a new or disappeared lymph node.

[0194] Overall loss function of CT lymph node tracking model for:

[0195] ;

[0196] When performing 2D-3D self-supervised contrastive learning, the 2D features of the same lymph node are used as positive samples of its corresponding 3D features, while the 2D features of other lymph nodes in the same batch are used as negative samples. The two-dimensional features of nine different directions of lymph nodes are ,in, represents 9 two-dimensional views in different directions, and the three-dimensional features are ; Assume that the rest of the same batch The remaining nine two-dimensional features in different directions are expressed as ,in This strategy enhances the ability of the 3D lymph node extraction network to extract meaningful lymph node features. The 2D lymph node extraction network and the 3D lymph node extraction network are trained using self-supervised contrast loss. The 2D-3D self-supervised contrast loss function for:

[0197] .

[0198] In the supervised learning of the 3D time-enhanced feature extraction network, 256-dimensional 3D features are used and projected into different feature spaces, so that the two contrasting training strategies focus on different aspects of learning features. The characteristics of the lymph nodes at time point T1 are , No. The characteristics of the lymph nodes at time point T2 are , the 3D time-enhanced feature extraction network is trained using time contrast loss, which helps the model learn the final changes of lymph nodes over time. Time contrast loss function for:

[0199] .

[0200] Binary cross entropy is used as the loss function to supervise the training of the graph neural network, ensuring that it can effectively distinguish the same lymph nodes that have undergone non-rigid changes without over-homogenizing the features of the lymph nodes in the same CT scan. Position information loss function during graph neural network training for:

[0201] ;

[0202] If the lymph nodes at T1 and lymph nodes at T2 For the same node in the label, =0, otherwise =0.

[0203] When judging whether a lymph node is new or missing, the model will incorporate this information into the spatial matching results to form a final comprehensive judgment. for:

[0204] ;

[0205] in, Represents the three-dimensional image The three-dimensional characteristics of lymph nodes, Indicates The first two-dimensional view in the orientation The two-dimensional features of lymph nodes, Indicates The first two-dimensional view in the orientation The two-dimensional features of lymph nodes, represents the temperature parameter, represents the cosine similarity function, Indicates lymph nodes at T1 The three-dimensional features of Indicates lymph nodes at T2 The three-dimensional features of Indicates the number of lymph node pairs that can be formed at T1 and T2. Represents the lymph nodes in the CT image at time T1 Updated lymph node characteristics, Represents the lymph nodes in the CT image at time T2 updated lymph node characterization; Represents label data, which is 0 or 1, 1 means the two lymph nodes match, 0 means the two lymph nodes do not match; and same, Indicates the final similarity.

[0206] The real-time lymph node tracking module is used to obtain the abdominal CT three-dimensional test images taken of the patient at different time points, and extract the abdominal CT two-dimensional test images from the abdominal CT three-dimensional test images, and input the abdominal CT three-dimensional test images and the abdominal CT two-dimensional test images into the trained CT lymph node tracking model, and the CT lymph node tracking model outputs the tracking results of each lymph node.

[0207] Example 3

[0208] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of a time-space-based follow-up CT lymph node tracking method.

[0209] The computer device may be a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with the user through a keyboard, a mouse, a remote control, a touch pad, or a voice control device.

[0210] The memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of the computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is often used to store the operating system and various application software installed on the computer device, such as the program code of the time-space-based follow-up CT lymph node tracking method. In addition, the memory can also be used to temporarily store various types of data that have been output or are to be output.

[0211] The processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor is generally used to control the overall operation of the computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data, such as running the program code of the spatiotemporal-based follow-up CT lymph node tracking method.

[0212] Example 4

[0213] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor executes the steps of a time-space-based follow-up CT lymph node tracking method.

[0214] The computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned time-space-based follow-up CT lymph node tracking method.

[0215] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the time-space-based follow-up CT lymph node tracking method described in the embodiment of the present application.

Claims

1. A time-space-based follow-up CT lymph node tracking method, characterized in that: The following steps are involved: Step 1: Obtain sample and label data; Obtain abdominal CT three-dimensional images taken of multiple patients at different time points, and extract abdominal CT two-dimensional images from the abdominal CT three-dimensional images to obtain CT three-dimensional sample data and CT two-dimensional sample data; annotate the matching relationship between each lymph node in the CT three-dimensional sample data of the same patient at different time points to obtain label data; Step 2, constructing a CT lymph node tracking model; Construct a CT lymph node tracking model, which includes a two-dimensional lymph node extraction network, a three-dimensional lymph node extraction network, a three-dimensional time-enhanced feature extraction network, a graph neural network, and a lymph node addition and disappearance prediction module; Step 3, training the CT lymph node tracking model; The CT two-dimensional sample data and the CT three-dimensional sample data are respectively input into the two-dimensional lymph node extraction network and the three-dimensional lymph node extraction network, and are trained using the self-supervised contrast loss; the CT three-dimensional sample data and the label data are input into the three-dimensional time-enhanced feature extraction network, and are trained using the time contrast loss; the features output by the three-dimensional lymph node extraction network or the three-dimensional time-enhanced feature extraction network and the center coordinates of each lymph node in the abdominal CT three-dimensional image are input into the graph neural network, the graph neural network first calculates the coordinate difference between each lymph node and other lymph nodes, and then calculates the weight according to the coordinate difference, and then selects the features of the corresponding lymph nodes according to the weight for feature aggregation, and adds the aggregated features to the original features of the lymph node to complete the feature update of the lymph node, and calculates the similarity between the updated features and all the lymph nodes in the CT image of the same patient at another time point, and judges that the lymph node and a lymph node in the CT image at another time point belong to the same lymph node according to the calculated similarity; the abdominal CT three-dimensional image and its corresponding two abdominal CT two-dimensional images are input into the lymph node addition and disappearance prediction module, and the lymph node addition and disappearance prediction module outputs whether a certain lymph node is a new or disappeared lymph node; Among them, the overall loss function of the CT lymph node tracking model is for: ； The 2D lymph node extraction network and the 3D lymph node extraction network are trained using self-supervised contrast loss. The 2D-3D self-supervised contrast loss function for: ； The 3D time-enhanced feature extraction network is trained using time contrast loss. The time contrast loss function for: ； Graph neural network uses binary cross entropy for supervised training, position information loss function for: ； When training the new lymph node disappearance prediction module, a new disappearance prediction loss function is added for: ； in, Represents the three-dimensional image The three-dimensional characteristics of lymph nodes, Indicates The first two-dimensional view in the orientation The two-dimensional features of lymph nodes, Indicates The first two-dimensional view in the orientation The two-dimensional features of lymph nodes, represents the temperature parameter, represents the cosine similarity function, Indicates lymph nodes at T1 The three-dimensional features of Indicates lymph nodes at T2 The three-dimensional features of Indicates the number of lymph node pairs that can be formed at T1 and T2. Represents the lymph nodes in the CT image at time T1 Updated lymph node characteristics, Represents the lymph nodes in the CT image at time T2 updated lymph node characterization; Represents label data, which is 0 or 1, 1 means the two lymph nodes match, 0 means the two lymph nodes do not match; and , Indicates the final similarity; Step 4, real-time tracking of lymph nodes; The abdominal CT three-dimensional test images taken at different time points of the patient to be tested are obtained, and the abdominal CT two-dimensional test images are extracted from the abdominal CT three-dimensional test images. The abdominal CT three-dimensional test images and the abdominal CT two-dimensional test images are input into the trained CT lymph node tracking model, and the CT lymph node tracking model outputs the tracking results of each lymph node.

2. A time-space-based follow-up CT lymph node tracking method as claimed in claim 1, characterized in that: In step 2, the two-dimensional lymph node extraction network adopts a two-dimensional ResNet-34 network, and the three-dimensional lymph node extraction network adopts a three-dimensional ResNet-34 network, and the residual structures of the two-dimensional ResNet-34 network and the three-dimensional ResNet-34 network include a batch normalization layer, a linear rectification unit, a weight calculation unit, a batch normalization layer, a linear rectification unit and a weight calculation unit arranged in sequence, and the input passes through the batch normalization layer, the linear rectification unit, the weight calculation unit, the batch normalization layer, the linear rectification unit and the weight calculation unit in sequence, and the output of the weight calculation unit is superimposed on the input as the output of the entire residual structure.

3. The time-space-based follow-up CT lymph node tracking method according to claim 1, characterized in that: In step 2, the three-dimensional time enhanced feature extraction network adopts a three-dimensional ResNet-34 network, and the residual structure of the three-dimensional ResNet-34 network includes a batch normalization layer, a linear rectification unit, a weight calculation unit, a batch normalization layer, a linear rectification unit and a weight calculation unit arranged in sequence. The input passes through the batch normalization layer, the linear rectification unit, the weight calculation unit, the batch normalization layer, the linear rectification unit and the weight calculation unit in sequence, and the output of the weight calculation unit is superimposed on the input as the output of the entire residual structure.

4. The time-space-based follow-up CT lymph node tracking method according to claim 1, characterized in that: In step 2, in the graph neural network, the specific processing method is: First, according to the lymph nodes With lymph nodes The normalized center coordinates of , , count lymph nodes With lymph nodes The weight between : ； By aggregating all connected lymph nodes Features , count lymph nodes Aggregation features : ； Use the self-attention mechanism to calculate the importance weight of aggregated information : ； Add the aggregated features to the original features of the lymph node to obtain the updated lymph node features , and update the CT image; updated lymph node features The update method is: ； For updated CT images from two different CT scans, calculate the cosine similarity of each pair of lymph nodes , and based on the calculated similarity, it is determined that the lymph node belongs to the same lymph node as a lymph node in the CT image at another time point; cosine similarity The calculation method is: ； in, Indicates a click operation. It means to calculate the modulus of the vector. Indicates lymph nodes The normalized center coordinates of Indicates lymph nodes The normalized center coordinates of Indicates lymph nodes The initial characteristics of represents transpose, represents the dimension of the key vector, Indicates lymph nodes The initial characteristics of Represents the lymph nodes in the CT image taken at time T1 Updated features, Represents the lymph nodes in the CT image taken at time T2 Updated features.

5. The time-space-based follow-up CT lymph node tracking method according to claim 1, characterized in that: In step 2, in the newly added lymph node disappearance prediction module, the specific processing method is as follows: Select two most representative abdominal CT two-dimensional images from the nine abdominal CT two-dimensional images of the abdominal CT three-dimensional image, specifically: ； For all lymph nodes in a pair of CT scans, the abdominal CT 3D images of each pair of lymph nodes and the two selected abdominal CT 2D images were fused pairwise; for each pair of lymph nodes from two CT scans ( , ), calculate the pairwise fusion features: ； The fused features are input into the fully connected layer, which outputs the similarity of each pair of lymph nodes and determines whether a lymph node is a newly added lymph node or a disappeared lymph node based on the similarity. in, Indicates The first two-dimensional view in the orientation The adaptive weights of the features of the lymph nodes, Indicates The first two-dimensional view in the orientation The adaptive weights of the features of the lymph nodes, represents the value of the independent variable that makes the function reach the maximum value, Indicates In the view The characteristics of the lymph nodes, Indicates In the view The characteristics of the lymph nodes, Represents a fusion operation.

6. A time-space-based follow-up CT lymph node tracking method as claimed in claim 5, characterized in that: The fusion features are input into the fully connected layer and the similarity of each pair of lymph nodes is output, specifically: First, each fusion feature is placed in the fully connected layer for mapping transformation to obtain the mapping value; the mapping transformation is specifically: ； Remap the mapped value to between 0 and 1 to obtain the mapping result; the remapping is as follows: ； Change the mapping results into binary classification results, specifically: ； Calculate the comprehensive similarity of each fusion feature, specifically: ； Prediction is performed based on the binary classification results and comprehensive similarity, and the prediction matching similarity of each fusion feature , specifically: ； For the nine predicted matching similarities generated, generate the final similarity ; ； According to the final similarity , the final similarity lower than 0.5 is defined as mismatch, and a lymph node is judged as a newly added or disappeared lymph node; in, represents the weight matrix, represents the bias term, represents the fusion feature, Represents the Sigmoid activation function.

7. A time-space-based follow-up CT lymph node tracking system, characterized in that: The following steps are involved: The sample and label data acquisition module is used to obtain abdominal CT three-dimensional images taken by multiple patients at different time points, and extract abdominal CT two-dimensional images from the abdominal CT three-dimensional images to obtain CT three-dimensional sample data and CT two-dimensional sample data; annotate the matching relationship between each lymph node in the CT three-dimensional sample data of the same patient at different time points to obtain label data; CT lymph node tracking model construction module, used to build a CT lymph node tracking model. The CT lymph node tracking model includes a two-dimensional lymph node extraction network, a three-dimensional lymph node extraction network, a three-dimensional time-enhanced feature extraction network, a graph neural network, and a lymph node addition and disappearance prediction module; The CT lymph node tracking model training module is used to input CT two-dimensional sample data and CT three-dimensional sample data into the two-dimensional lymph node extraction network and the three-dimensional lymph node extraction network respectively, and use self-supervised contrast loss for training; input CT three-dimensional sample data and label data into the three-dimensional time-enhanced feature extraction network, and use time contrast loss for training; input the features output by the three-dimensional lymph node extraction network or the three-dimensional time-enhanced feature extraction network and the center coordinates of each lymph node in the abdominal CT three-dimensional image into the graph neural network, and the graph neural network first calculates the coordinate difference between each lymph node and other lymph nodes, and then uses the coordinate difference to train the three-dimensional lymph node extraction network. to calculate the weight, and then select the features of the corresponding lymph nodes according to the weight for feature aggregation, and add the aggregated features to the original features of the lymph node to complete the feature update of the lymph node, and calculate the similarity between the updated features and all the lymph nodes in the CT image of the same patient at another time point, and judge whether the lymph node and a lymph node in the CT image at another time point belong to the same lymph node according to the calculated similarity; input the abdominal CT three-dimensional image and its corresponding two abdominal CT two-dimensional images into the lymph node addition and disappearance prediction module, and the lymph node addition and disappearance prediction module outputs whether a lymph node is a new or disappeared lymph node; Overall loss function of CT lymph node tracking model for: ； The 2D lymph node extraction network and the 3D lymph node extraction network are trained using self-supervised contrast loss. The 2D-3D self-supervised contrast loss function for: ； The 3D time-enhanced feature extraction network is trained using time contrast loss. The time contrast loss function for: ； Graph neural network uses binary cross entropy for supervised training, position information loss function for: ； When training the new lymph node disappearance prediction module, a new disappearance prediction loss function is added for: ； in, Represents the three-dimensional image The three-dimensional characteristics of lymph nodes, Indicates The first two-dimensional view in the orientation The two-dimensional features of lymph nodes, Indicates The first two-dimensional view in the orientation The two-dimensional features of lymph nodes, represents the temperature parameter, represents the cosine similarity function, Indicates lymph nodes at T1 The three-dimensional features of Indicates lymph nodes at T2 The three-dimensional features of Indicates the number of lymph node pairs that can be formed at T1 and T2. Represents the lymph nodes in the CT image at time T1 Updated lymph node characteristics, Represents the lymph nodes in the CT image at time T2 updated lymph node characterization; Represents label data, which is 0 or 1, 1 means the two lymph nodes match, 0 means the two lymph nodes do not match; and , Indicates the final similarity; The real-time lymph node tracking module is used to obtain the abdominal CT three-dimensional test images taken of the patient at different time points, and extract the abdominal CT two-dimensional test images from the abdominal CT three-dimensional test images, and input the abdominal CT three-dimensional test images and the abdominal CT two-dimensional test images into the trained CT lymph node tracking model, and the CT lymph node tracking model outputs the tracking results of each lymph node.

8. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 4.

Images