A longitudinal image sequence evolution rule modeling method based on deep clustering
The end-to-end temporal modeling framework based on GRU encoder-deep embedding clustering solves the problems of fragmented temporal data and insufficient high-dimensional feature extraction in the modeling of multiple CT images of patients. It enables accurate modeling of dynamic correlations of lesions and identification of complex patterns, supporting personalized treatment strategies.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIANGSU OCEAN UNIV
- Filing Date
- 2026-06-09
- Publication Date
- 2026-07-31
AI Technical Summary
Existing technologies suffer from fragmented temporal data, lack of high-dimensional feature extraction capabilities, and ineffective temporal clustering mechanisms in patient modeling with multiple CT images. This leads to the loss of continuous information on lesion evolution and difficulty in recognizing complex patterns, making it difficult to support personalized treatment strategies.
An end-to-end temporal modeling framework based on GRU encoder-deep embedding clustering is adopted. Through self-supervised pre-training and unsupervised clustering, the temporal dependency features of lesion mask sequences are extracted and pattern recognition is performed. The deep embedding clustering method is combined to automatically learn and classify the nonlinear evolution trajectory of lesions.
It achieves accurate modeling of the dynamic correlation of lesions, improves the ability to identify complex evolution patterns, can automatically classify patients with similar evolution trajectories, and supports disease subtype identification and personalized treatment.
Smart Images
Figure CN122492679A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method for modeling the evolution of longitudinal image sequences based on deep clustering. Background Technology
[0002] In clinical medical practice, longitudinal analysis of multiple CT images of patients to track the evolution of lesions is an important means of disease diagnosis, efficacy evaluation, and prognosis. Traditional lesion evolution analysis mainly relies on manual comparison and experience judgment by radiologists, which has limitations such as strong subjectivity, low efficiency, and easy missed diagnoses. With the rapid development of deep learning technology, a variety of automated methods have emerged in the field of intelligent medical image analysis.
[0003] There are two main technical approaches to modeling the evolution of lesions from multiple CT images in patients: The first is deep learning models such as 2D / 3D CNN and U-Net, which are based on static single-image analysis. These methods treat each examination as an independent sample and, while performing well in lesion detection and segmentation tasks from a single image, cannot effectively capture the dynamic changes of lesions over time. The second approach is to use simple time-series statistical methods such as linear regression and difference analysis to analyze quantitative indicators from multiple images. Although this strategy can quantify changes in parameters such as lesion volume and density, it lacks the ability to deeply mine high-dimensional time-series features and is difficult to identify complex evolution patterns.
[0004] The existing methods for modeling the evolution of lesions using multiple CT images of patients have three main shortcomings:
[0005] First, the deep learning paradigm based on static single-image analysis fragments the temporal data, resulting in the loss of continuous information on lesion evolution and making it impossible to accurately model the dynamic correlation between lesions at different time points.
[0006] Second, simple time-series statistical methods rely too heavily on manually designed quantitative indicators and lack the ability to automatically extract high-dimensional features from the original images, making it difficult to discover potential evolution patterns and clustering patterns.
[0007] Third, existing methods generally lack effective temporal clustering mechanisms, which cannot automatically classify lesions with similar evolutionary trajectories, thus limiting their ability to support disease subtype identification and personalized treatment strategy development. Summary of the Invention
[0008] The purpose of this invention is to provide a method for modeling the evolution of longitudinal image sequences based on deep clustering. This method addresses issues such as loss of temporal continuity information, insufficient high-dimensional feature extraction capabilities, and a lack of effective temporal clustering mechanisms during lesion evolution modeling from multiple CT images of patients. An end-to-end temporal modeling framework based on a GRU encoder and deep embedding clustering is designed, featuring integrated self-supervised pre-training and unsupervised clustering. This framework extracts temporal dependency features of lesion mask sequences using an RNN encoder and combines this with deep embedding clustering for pattern recognition. It aims to achieve automatic learning and accurate classification of the nonlinear evolution trajectory of lesions, ultimately enabling refined identification of complex treatment response patterns. This provides quantifiable and interpretable clinical decision support for individualized efficacy assessment, drug resistance warning, and dynamic adjustment of treatment strategies.
[0009] The above-mentioned technical objective of the present invention is achieved through the following technical solution:
[0010] A method for modeling the evolution patterns of longitudinal image sequences based on deep clustering includes the following steps:
[0011] Step 1: Establish an image preprocessing and feature extraction module: extract corresponding deep learning features, 3D morphological features, and omics features based on the input image and annotation file;
[0012] Step 2: Establish a temporal modeling pre-training module: Use a GRU encoder to perform temporal modeling on multiple sets of deep learning features, multiple sets of 3D morphological features and multiple sets of omics features extracted in Step 1 to obtain their high-dimensional temporal features; use an MLP decoder to reconstruct the temporal features to improve the accuracy and effectiveness of the temporal features.
[0013] Step 3: Establish a clustering module: Based on high-dimensional temporal features, patients with similar evolution trajectories of lesions in temporal medical images are classified in an unsupervised manner.
[0014] Step 4: Establish a deep embedding clustering module: Combine the GRU encoder and the clustering module, denoted as the deep embedding clustering module. Based on the multiple sets of deep learning features, three-dimensional morphological features and omics features extracted in Step 1, patients with similar evolutionary trajectories of lesions in longitudinal medical images are classified in an unsupervised manner to identify clinically significant subtypes, providing support for disease subtype identification and personalized treatment.
[0015] Preferably, step one includes:
[0016] Step 1, Image Preprocessing: For the extraction of morphological and omics features, multiple sets of medical image sequences and labeled files are resampled to a fixed voxel size; for the extraction of deep learning features, multiple sets of medical image sequences and labeled files are resampled to a fixed input size.
[0017] Step 2, Omics Feature Extraction: Extract omics features from each preprocessed medical image and mask annotation file, denoted as... ;
[0018] Step 3, Morphological Feature Extraction: Extract the morphological feature count of all connected components in the mask annotation file of each preprocessed medical image, denoted as . ;
[0019] Step 4, Deep Learning Feature Extraction: Deep learning features are extracted for each preprocessed medical image. Specifically, the 3D SwinUnet model is pre-trained based on the preprocessed medical images and mask files, enabling it to accurately encode and decode the image's inherent features. First, the 2D model is extended to a 3D structure, and the parameters within the 3D structure are adjusted. Its encoder part is then used as a deep learning feature extractor to extract deep learning features, denoted as... .
[0020] Preferably, step two includes:
[0021] Step 1, Feature Assembly: Input features include deep learning features, radiomics features, and morphological features, i.e., patient... The eigenvector at time point t is:
[0022]
[0023] in, , , , , The number of deep learning features extracted for each medical image and mask. The number of omics features extracted for each medical image and mask. The number of morphological features extracted for each medical image and mask;
[0024] Step 2, GRU encoder extracts temporal features: The GRU encoder is used to extract the spliced features mentioned above. Extracting high-dimensional temporal features: The input data has a shape of [B,T,D], where B is the size of the input data batch, T is the time dimension, and D is the number of all features; the output is high-dimensional temporal features. The shape is [B,H], where B is the size of the input data batch and H is the set number of hidden states;
[0025] Step 3, MLP Decoder Feature Reconstruction: Utilize the MLP decoder to reconstruct the features concatenated above. Perform feature reconstruction, with the input being The output is That is, its size is consistent with the original input; mean square error is used as the loss function to pre-train the network structure so that the output of the MLP decoder is as close as possible to the original input of the GRU encoder, so as to ensure the effectiveness and accuracy of the high-dimensional temporal features extracted by the GRU encoder.
[0026] Preferably, step three includes:
[0027] Step 1, Parameter Initialization: Module initialization, the input parameters are the number of clusters n_clusters, the embedding dimension H, the Student-t distribution degree of freedom parameter α, the learnable cluster center matrix μ, which is sampled from a standard normal distribution and then multiplied by 0.1 to reduce the magnitude;
[0028] Step 2, Cluster Center Update Strategy: Update the cluster centers using the furthest-end strategy, with the input being the temporal high-dimensional feature vector of all samples. Its shape is [B, ];
[0029] Step 3, Soft Clustering Prediction: First, calculate the Euclidean distance between all samples and all cluster centers μ, obtaining the distance matrix dist2, with shape [B, n_clusters]. Then, use the Student-t distribution kernel function to calculate the unnormalized similarity, the formula of which is...
[0030]
[0031] in, , , For the degree of freedom parameters; secondly, by The soft clustering assignment matrix is constructed with shape [B, n_clusters] and normalized, denoted as . Its calculation formula is
[0032]
[0033] Finally, the soft clustering assignment matrix is returned. The column index corresponding to the largest element in each row is the predicted category.
[0034] As a preferred option, the implementation process of Step 2 in step three is as follows:
[0035] (i) Randomly select a sample index idx0 as the first cluster center;
[0036] (ii) Initialize the central index list centers = [idx0];
[0037] (iii) Calculate the Euclidean distance from all samples to the first cluster center to obtain the distance vector d_list;
[0038] (iv) Repeat the following process n_clusters-1 times: find the index idx of the sample that is farthest from the current cluster center and add idx to the centers list; for each sample, take the smaller value between its distance to the newly added center and its distance to the previous center;
[0039] (v) Using the temporal high-dimensional features of the corresponding samples in centers The value is directly assigned to the cluster center μ.
[0040] Preferably, step four includes:
[0041] Multiple sets of deep learning features Three-dimensional morphological features and omics characteristics The extraction module serves as the preprocessing module of the overall system. When connected in series with the GRU encoder and clustering module, it forms an end-to-end prediction system for the evolution trajectory of patient lesions.
[0042] A model for the evolution of longitudinal image sequences based on deep clustering includes:
[0043] The image preprocessing and feature extraction module takes as input multiple image sequences of the patient and the corresponding annotation file for each time step; and outputs multiple sets of extracted deep learning features. Multiple sets of three-dimensional morphological features and multi-omics characteristics ;
[0044] The time series modeling pre-training module takes as input multiple sets of deep learning features. Three-dimensional morphological features and omics characteristics The output is: high-dimensional temporal features, denoted as... ;
[0045] The clustering module takes high-dimensional time-series features as input. The output is: predicted category;
[0046] The deep embedding clustering module takes multiple sets of deep learning features as input. Three-dimensional morphological features and omics characteristics The output is the predicted category.
[0047] In summary, the present invention has the following beneficial effects:
[0048] The deep learning temporal clustering model proposed in this patent shows significant advantages in modeling the dynamic evolution of longitudinal images.
[0049] First, by using the GRU encoder-decoder pre-training architecture, we can effectively model the continuity of lesion time series data, avoiding the problem of time series information fragmentation caused by traditional static single image analysis methods, and accurately capturing the dynamic correlation of lesions at different time points.
[0050] Secondly, the model does not rely on manually designed quantitative indicators, but instead automatically extracts high-dimensional temporal features of lesion mask sequences through end-to-end learning, which significantly improves the ability to identify complex evolution patterns.
[0051] Third, the designed deep embedding clustering method can automatically classify patients with similar evolutionary trajectories and identify clinically significant subtypes such as "pseudo-progressive regression type", providing strong support for disease subtype identification and personalized treatment. Attached Figure Description
[0052] Figure 1 This is a flowchart illustrating the implementation of a deep learning-based temporal clustering model.
[0053] Figure 2 This is a schematic diagram of a feature extraction method for multiple image sequences;
[0054] Figure 3 This is a 2D SwinUnet network structure diagram;
[0055] Figure 4 This is the network structure diagram of the time series modeling pre-training module;
[0056] Figure 5 This is a diagram showing the main steps involved in implementing the clustering module;
[0057] Figure 6 This is a diagram showing the main steps involved in implementing a deeply embedded clustering module. Detailed Implementation
[0058] The present invention will be further described in detail below with reference to the accompanying drawings.
[0059] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.
[0060] This invention proposes a longitudinal image dynamics research modeling method based on a deep learning temporal clustering model, which mainly consists of four modules: image preprocessing and feature extraction module, temporal modeling pre-training module, clustering module, and deep embedding clustering module. The method flowchart is shown below. Figure 1 As shown.
[0061] The specific implementation of this invention uses three sets of time-series images as example data: before treatment, after the first treatment, and after the second treatment. The medical images used in the examples are 3D CT images, and the disease in the examples is pulmonary tuberculosis. Other modalities (such as MRI, ultrasound) or other diseases follow the same principle as this example. For clarity, the accompanying drawings and visualizations are based on these three stages of data. The specific implementation details of each module are as follows:
[0062] 1. Image preprocessing and feature extraction module
[0063] Objective: To extract corresponding deep learning features, 3D morphological features, and omics features based on input images and labeled files. The detailed implementation method is as follows: Figure 2 As shown.
[0064] Input: Multiple image sequences (Image1,...,ImageN) of the patient and the corresponding annotation file (Mask1,...,MaskN) at each time step;
[0065] Output: Extracted deep learning features from multiple sets Multiple sets of three-dimensional morphological features and multi-omics characteristics .
[0066] The specific implementation steps are as follows:
[0067] Step 1 (Image Preprocessing): For the extraction of morphological and omics features, multiple sets of 3D image sequences and annotation (Mask) files are resampled to a fixed voxel size, with each voxel being [1.0mm, 1.0mm, 1.0mm]. For the extraction of deep learning features, multiple sets of 3D image sequences and annotation files are resampled to a fixed input size, with the resampled 3D images and Mask files being [224, 224, 224].
[0068] Step 2 (Omics Feature Extraction): Using the open-source omics toolkit PyRadiomics (based on the Python platform), omics features were extracted from each preprocessed 3D CT and mask annotation file. These features included first-order statistical features, 3D-based shape features, gray-level co-occurrence matrix, gray-level run-length matrix, adjacency gray-level difference matrix, and gray-level dependency matrix, denoted as... .
[0069] The PyRadiomics parameters are set as follows:
[0070]
[0071] Step 3 (Morphological Feature Extraction): Using the regionprops function from the open-source morphological tool library Scikit-image, morphological features are extracted from all connected regions (each connected region can be understood as a lesion) in the mask annotation file of each preprocessed 3D CT case. These features include the maximum / average diameter, maximum / average eccentricity, maximum / average sphericity, maximum / average area, maximum / average extend value, and maximum / average convexity ratio of the lesion's minimum circumscribed ellipse. These features are denoted as follows: .
[0072] Step 4 (Deep Learning Feature Extraction): The 3D SwinUnet segmentation model is used to extract deep learning features from each preprocessed 3D CT image. The 3D SwinUnet model is pre-trained based on the preprocessed 3D CT and mask files, enabling it to accurately encode and decode the image's inherent features. Its original 2D network structure is as follows: Figure 3 As shown. In this invention, the 2D model is first expanded into a 3D structure, and the parameters in the 3D structure are adjusted according to the example data and the server's hardware conditions, using its encoder portion (see...). Figure 2 As a deep learning feature extractor, it extracts deep learning features, denoted as . .
[0073] The direct dilation method is used to extend the 2D model into a 3D structure, directly extending the 2D convolution kernel into a 3D convolution kernel;
[0074] The system consists of four downsampling layers and four upsampling layers, with the number of channels (96->192->384->768), the number of attention points (3,6,12,24), and the block window size (7,7,7).
[0075] The size of the input image can be appropriately increased if the video memory is large enough.
[0076] Pre-training is fine-tuning based on a publicly available dataset. It utilizes the BTCV dataset published in the paper "Self-supervised pre-training of swin transformers for 3D medical image analysis," and fine-tunes 300 sequence data points (including three spatial point CT scans plus labeled masks) provided by the hospital. Details are omitted here. The learning rate in this invention is set as follows:
[0077] lr_config=dict(policy='CosineAnnealing',warmup='linear',warmup_iters=50,warmup_ratio=1.0 / 3,min_lr=23e-6), batch_size=4, epoch is 10.
[0078] 2. Temporal Modeling Pre-training Module
[0079] Objective: To perform temporal modeling of multiple sets of deep learning features, multiple sets of 3D morphological features, and multiple sets of omics features extracted by the image preprocessing and feature extraction modules using a GRU encoder, thereby obtaining high-dimensional temporal features; and to reconstruct the temporal features using an MLP decoder, improving the accuracy and effectiveness of the temporal features. The structure is shown in [see attached diagram]. Figure 4 .
[0080] Input: Multiple sets of deep learning features Three-dimensional morphological features and omics characteristics ;
[0081] Output: High-dimensional time series features (HDTS features), denoted as .
[0082] The specific implementation steps are as follows:
[0083] Step 1 (Feature Assembly): Input features include deep learning features, radiomics features, and morphological features, i.e., patient features. The eigenvector at time point t is:
[0084]
[0085] in, , , , , The number of deep learning features extracted for each 3D CT and Mask case. The number of omics features extracted for each 3D CT and mask case. The number of morphological features extracted for each 3D CT and Mask case;
[0086] Step 2 (Extracting Temporal Features from the GRU Encoder): Utilize the GRU encoder to extract the spliced features described above. Extracting high-dimensional time-series features: The input data has a shape of [B, T, D], where B is the size of the input data batch, T is the time dimension (in the example data, there are three time points, i.e., T=3), and D is the number of all features; the output is a high-dimensional time-series feature. The shape is [B,H], where B is the size of the input data batch and H is the set number of hidden states, which is set to 128 in this specific embodiment;
[0087] Step 3 (MLP Decoder Feature Reconstruction): Utilizing the MLP decoder (see detailed structure) Figure 4 Regarding the above-mentioned splicing characteristics Perform feature reconstruction, with the input being The output is (Shape is [B,T,D]), meaning its size is consistent with the original input; mean squared error (nn.MSELoss) is used as the loss function. Figure 4 The network structure is pre-trained to make the output of the MLP decoder as close as possible to the original input of the GRU encoder, so as to ensure the effectiveness and accuracy of the high-dimensional temporal features extracted by the GRU encoder.
[0088] 3. Clustering Module
[0089] Objective: To perform unsupervised classification of patients with similar trajectories of lesions in temporal 3D CT scans based on high-dimensional temporal features. The detailed implementation process is as follows: Figure 5 As shown.
[0090] Input: High-dimensional temporal features ;
[0091] Output: Predicted categories (total number of categories is an empirical parameter).
[0092] The specific implementation steps are as follows:
[0093] Step 1 (Parameter Initialization): Module initialization. The input parameters are the number of clusters n_clusters. In this example, the data is tuberculosis data, and n_clusters=2 (manually set the number of disease subtypes to be obtained), the embedding dimension H (the number of hidden states set in the decoder GRU), and the Student-t distribution degrees of freedom parameter α, which defaults to 1.0 (if there is a lot of noisy data, i.e., the clustering effect does not reach the predetermined level, reduce α to 0.5~0.8 to enhance the tail effect and reduce the impact of outliers on cluster centers; conversely, if there is less noisy data, α can be increased to 2.0~5.0). The learnable cluster center matrix μ (shape [n_clusters,H]) is sampled from a standard normal distribution and then multiplied by 0.1 to reduce the magnitude (i.e., small random initialization).
[0094] Step 2 (Cluster Center Update Strategy): Update the cluster centers using the furthest-farthest strategy. The input is the temporal high-dimensional feature vector of all samples. Its shape is [B, The implementation process is as follows: (i) Randomly select a sample index idx0 as the first cluster center; (ii) Initialize the center index list centers=[idx0]; (iii) Calculate the Euclidean distance from all samples to the first cluster center to obtain the distance vector d_list (length B); (iv) Repeat the following process n_clusters-1 times: find the sample index idx that is farthest from the current cluster center (i.e., the position of the maximum value in d_list), and add idx to the centers list; for each sample, take the smaller value between its distance to the newly added center and the distance to the previous center (i.e., maintain the nearest distance to the selected center set); (v) Use the temporal high-dimensional features of the corresponding samples in centers. , directly assign the value to the cluster center μ (overwrite the initial random value);
[0095] Step 3 (Soft Clustering Prediction): First, calculate all samples (high-dimensional time-series features). The Euclidean distances between the cluster centers μ and all cluster centers μ are used to obtain the distance matrix dist2, which has the shape [B, n_clusters]. Then, the unnormalized similarity is calculated using the Student-t distribution kernel function, and the calculation formula is as follows:
[0096]
[0097] in, , , This is the degree of freedom parameter (default is 1.0); secondly, it is... The soft clustering assignment matrix is constructed with shape [B, n_clusters] and normalized, denoted as . Its calculation formula is
[0098]
[0099] Finally, the soft clustering assignment matrix is returned. The column index corresponding to the largest element in each row is the predicted category.
[0100] 4. Deeply embedded clustering module
[0101] Objective: To combine the GRU encoder and clustering module, denoted as the deep embedding clustering module, and based on multiple sets of deep learning features, 3D morphological features, and omics features extracted by the image preprocessing and feature extraction modules, to perform unsupervised classification of patients with similar evolutionary trajectories of lesions on longitudinal 3D CT, identifying clinically significant subtypes such as "pseudo-progressive regression type," thus providing support for disease subtype identification and personalized treatment. Its structure is shown in [link to diagram]. Figure 6 .
[0102] Input: Multiple sets of deep learning features Three-dimensional morphological features and omics characteristics ;
[0103] Output: Predicted category.
[0104] Multiple sets of deep learning features Three-dimensional morphological features and omics characteristics The extraction module can be used as a preprocessing module for the overall system. When connected in series with the GRU encoder and clustering module, it forms an end-to-end prediction system for the evolution trajectory of patient lesions.
[0105] In summary, this invention proposes a method for modeling the evolution of lesions in multiple CT images of patients based on a deep learning temporal clustering model. The core innovation of this technology lies in:
[0106] (1) Design a dedicated RNN encoder-decoder pre-training architecture: The GRU encoder is used to extract the main features of the time series data, and the MLP decoder (MLPDecoder) is used for pre-training. The loss function (usually the mean squared error loss function: nn.MSELoss) is used to ensure the consistency between the reconstruction result and the original input, and to ensure the accuracy and stability of the time series features extracted by the GRU encoder.
[0107] (2) Deep Embedding Clustering Method: After pre-training, the RNN encoder is combined with the deep embedding clustering method to form an end-to-end temporal clustering framework. This method can not only effectively extract temporal features, but also automatically identify and classify different lesion evolution patterns, providing strong support for clinical decision-making.
[0108] This specific implementation method uses the dynamic evolution modeling of lesions from three longitudinal CT follow-up images (before treatment - first treatment - second treatment) of pulmonary tuberculosis patients as an example. The model development set is a multi-center CT dataset containing follow-up CT images of 300 patients before treatment, after the first treatment, and after the second treatment. Each case provides paired thin-slice axial CT images and a lesion mask manually annotated by the physician, serving as input representation for temporal modeling.
[0109] Experimental results show that the clustering results of this method are significantly correlated with clinical prognosis (progression-free survival, PFS, overall survival, OS) (HR=2.41, p<0.001), providing clinicians with more accurate, objective, and efficient clinical decision support for lesion evolution analysis.
Claims
1. A method for modeling evolution rule of longitudinal image sequence based on deep clustering, characterized in that, Includes the following steps: Step 1: Establish an image preprocessing and feature extraction module: extract corresponding deep learning features, 3D morphological features, and omics features based on the input image and annotation file; Step 2: Establish a temporal modeling pre-training module: Use a GRU encoder to perform temporal modeling on multiple sets of deep learning features, multiple sets of 3D morphological features and multiple sets of omics features extracted in Step 1 to obtain their high-dimensional temporal features; use an MLP decoder to reconstruct the temporal features to improve the accuracy and effectiveness of the temporal features. Step 3: Establish a clustering module: Based on high-dimensional temporal features, patients with similar evolution trajectories of lesions in temporal medical images are classified in an unsupervised manner. Step 4: Establish a deep embedding clustering module: Combine the GRU encoder and the clustering module, denoted as the deep embedding clustering module. Based on the multiple sets of deep learning features, three-dimensional morphological features and omics features extracted in Step 1, patients with similar evolutionary trajectories of lesions in longitudinal medical images are classified in an unsupervised manner to identify clinically significant subtypes, providing support for disease subtype identification and personalized treatment.
2. The method of claim 1, wherein, Step one includes: Step 1, Image Preprocessing: For the extraction of morphological and omics features, multiple sets of medical image sequences and labeled files are resampled to a fixed voxel size; for the extraction of deep learning features, multiple sets of medical image sequences and labeled files are resampled to a fixed input size. Step 2, extraction of omics features: extract the omics features from each pre-processed medical image and mask annotation file, denoted as ; Step 3, morphological feature extraction: extract the morphological feature number of all connected domains in the mask annotation file of each preprocessed medical image, denoted as ; Step 4, deep learning feature extraction: extract deep learning features for each preprocessed medical image, wherein the 3D SwinUnet model is pre-trained based on the preprocessed medical image and the mask file, so that the 3D SwinUnet can accurately encode and decode the image itself features, first expand the 2D model into a 3D structure, adjust the parameters in the 3D structure, use the encoder part as a deep learning feature extractor, extract deep learning features, denoted as .
3. The method of claim 1, wherein, Step two includes: Step 1, feature concatenation: input features include deep learning features, radiomics features and morphological features, i.e., patient The feature vector at time point t is: in, , , , , The number of deep learning features extracted for each medical image and mask. The number of omics features extracted for each medical image and mask. The number of morphological features extracted for each medical image and mask; Step 2, GRU encoder extracts temporal features: The GRU encoder is used to extract the spliced features mentioned above. Extracting high-dimensional temporal features: The input data has a shape of [B,T,D], where B is the size of the input data batch, T is the time dimension, and D is the number of all features; the output is high-dimensional temporal features. The shape is [B,H], where B is the size of the input data batch and H is the set number of hidden states; Step 3, MLP Decoder Feature Reconstruction: Utilize the MLP decoder to reconstruct the features concatenated above. Perform feature reconstruction, with the input being The output is That is, its size is consistent with the original input; mean square error is used as the loss function to pre-train the network structure so that the output of the MLP decoder is as close as possible to the original input of the GRU encoder, so as to ensure the effectiveness and accuracy of the high-dimensional temporal features extracted by the GRU encoder.
4. The method for modeling the evolution pattern of longitudinal image sequences based on deep clustering according to claim 1, characterized in that, Step three includes: Step 1, Parameter Initialization: Module initialization, the input parameters are the number of clusters n_clusters, the embedding dimension H, the Student-t distribution degree of freedom parameter α, the learnable cluster center matrix μ, which is sampled from a standard normal distribution and then multiplied by 0.1 to reduce the magnitude; Step 2, Cluster Center Update Strategy: Update the cluster centers using the furthest-end strategy, with the input being the temporal high-dimensional feature vector of all samples. Its shape is [B, ]; Step 3, Soft Clustering Prediction: First, calculate the Euclidean distance between all samples and all cluster centers μ, obtaining the distance matrix dist2, with shape [B, n_clusters]. Then, use the Student-t distribution kernel function to calculate the unnormalized similarity, the formula of which is... in, , , For the degree of freedom parameters; secondly, by The soft clustering assignment matrix is constructed with shape [B, n_clusters] and normalized, denoted as . Its calculation formula is Finally, the soft clustering assignment matrix is returned. The column index corresponding to the largest element in each row is the predicted category.
5. The method for modeling the evolution pattern of longitudinal image sequences based on deep clustering according to claim 4, characterized in that, The implementation process of Step 2 in Step 3 is as follows: (i) Randomly select a sample index idx0 as the first cluster center; (ii) Initialize the central index list centers = [idx0]; (iii) Calculate the Euclidean distance from all samples to the first cluster center to obtain the distance vector d_list; (iv) Repeat the following process n_clusters-1 times: find the index idx of the sample that is farthest from the current cluster center and add idx to the centers list; for each sample, take the smaller value between its distance to the newly added center and its distance to the previous center; (v) Using the temporal high-dimensional features of the corresponding samples in centers The value is directly assigned to the cluster center μ.
6. The method for modeling the evolution pattern of longitudinal image sequences based on deep clustering according to claim 1, characterized in that, Step four includes: Multiple sets of deep learning features Three-dimensional morphological features and omics characteristics The extraction module serves as the preprocessing module of the overall system. When connected in series with the GRU encoder and clustering module, it forms an end-to-end prediction system for the evolution trajectory of patient lesions.
7. A model for the evolution of longitudinal image sequences based on deep clustering, wherein the method described in any one of claims 1-6 is characterized in that, include: The image preprocessing and feature extraction module takes as input: multiple sets of image sequences of the patient and the annotation file corresponding to each time step; The output is: multiple sets of extracted deep learning features. Multiple sets of three-dimensional morphological features and multi-omics characteristics ; The time series modeling pre-training module takes as input multiple sets of deep learning features. Three-dimensional morphological features and omics characteristics ; The output is: high-dimensional temporal features, denoted as ; The clustering module takes high-dimensional time-series features as input. The output is: predicted category; The deep embedding clustering module takes multiple sets of deep learning features as input. Three-dimensional morphological features and omics characteristics The output is: predicted category.