A pancreatitis time sequence image dynamic evaluation method based on a self-attention mechanism

Abstract

The application discloses a kind of pancreatitis time sequence image dynamic evaluation methods based on self-attention mechanism, it is related to medical image intelligent analysis technical field, including, to the multi-time point pancreas enhancement CT image after pre-processing is carried out spatial registration, and using Swin-UNet medical image segmentation network extracts the multiscale self-attention feature of pancreas and lesion, generates pancreatitis time sequence image feature sequence;Pancreatitis time sequence image feature sequence is carried out lesion identification and cross-time association, establishes lesion trajectory index table, and calculates trajectory level attention weight in the self-attention layer of Swin-UNet medical image segmentation network, aggregation obtains lesion trajectory weighted representation matrix.The application is dynamically adjusted trajectory level attention weight by using time sequence weighted fusion method, realizes the adaptive fusion of image feature and clinical index in time dimension, improves the consistency and clinical explainability of severity evaluation time sequence.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical image analysis technology, and in particular to a method for dynamic evaluation of time-series images of pancreatitis based on a self-attention mechanism. Background Technology

[0002] Pancreatitis is a common digestive system disease, and dynamic monitoring of its course is crucial for treatment decisions and prognostic assessment. In clinical practice, enhanced computed tomography (CT) imaging is the primary means of assessing pancreatic morphology and lesion changes. Conventional assessment methods typically rely on radiologists visually comparing and manually measuring multi-time-point CT images, or employing traditional image processing algorithms (such as thresholding and region growing) for lesion extraction and quantitative analysis. For time-series imaging data, images at each time point are usually processed independently, and disease progression is inferred by calculating changes in the static characteristics of lesion size and density.

[0003] However, existing methods still have certain limitations. They typically treat images at different time points as independent entities and estimate the trend of change through linear interpolation, failing to effectively capture the nonlinear evolution of lesions in the spatiotemporal dimension. In terms of feature representation, conventional methods often rely on hand-designed features, which are often limited to local texture or shape information and cannot fully capture the multi-scale structural details and global contextual relationships of lesions, resulting in insufficient feature discriminative power. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a method for dynamic evaluation of pancreatitis temporal images based on a self-attention mechanism to address the problems of insufficient spatiotemporal dynamic modeling of lesions and limited feature discrimination power.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a method for dynamic evaluation of temporal images of pancreatitis based on a self-attention mechanism. The method includes: acquiring and preprocessing multi-timepoint enhanced CT images of the pancreas and clinical temporal auxiliary data; spatially registering the preprocessed multi-timepoint enhanced CT images of the pancreas; extracting multi-scale self-attention features of the pancreas and lesions using the Swin-UNet medical image segmentation network to generate a temporal image feature sequence of pancreatitis; identifying and temporally associating lesions within the pancreatitis temporal image feature sequence; establishing a lesion trajectory index table; and calculating the self-attention feature sequence within the self-attention layer of the Swin-UNet medical image segmentation network. The trajectory-level attention weights are calculated, and the lesion trajectory weighted representation matrix is ​​aggregated. Based on the lesion trajectory weighted representation matrix, dynamic weight allocation based on image features is performed on the preprocessed clinical time-series auxiliary data. The trajectory-level attention weights are dynamically adjusted in the time dimension using a time-series weighted fusion method to generate a disease course time-series feature matrix. Time-weighted regression and probability threshold analysis are performed on the disease course time-series feature matrix to generate a pancreatitis severity change curve. The trajectory-level attention weights are mapped to multi-time-point enhanced CT images of the pancreas to form an interpretable heat map of the lesion area, and a dynamic assessment report of the pancreatitis course is output.

[0007] As a preferred embodiment of the pancreatitis time-series imaging dynamic assessment method based on self-attention mechanism described in this invention, wherein: the clinical time-series auxiliary data includes laboratory test indicators, treatment intervention records, and complication occurrence information; The preprocessing includes window width and window level adjustment, intensity normalization, and missing value filling.

[0008] As a preferred embodiment of the time-series image dynamic evaluation method for pancreatitis based on the self-attention mechanism described in this invention, the steps for spatial registration of the preprocessed multi-timepoint enhanced CT images of the pancreas are as follows: The time point with the clearest pancreatic anatomy was selected from the preprocessed multi-time point enhanced CT images of the pancreas as a fixed reference time point;

[0009] The enhanced CT images of the pancreas at each time point are registered with the enhanced CT images of the pancreas at a fixed reference time point to generate the spatial transformation parameter matrix for each time point; By using the spatial transformation parameter matrix at each time point, the enhanced CT images of the pancreas at the corresponding time point are resampled and transformed to generate a multi-time point enhanced CT image registration sequence.

[0010] As a preferred embodiment of the pancreatitis time-series image dynamic evaluation method based on self-attention mechanism described in this invention, the steps for generating the pancreatitis time-series image feature sequence are as follows:

[0011] The registration sequence of enhanced CT images of the pancreas at multiple time points is input into the Swin-UNet medical image segmentation network for forward propagation calculation, and the pancreas and lesion segmentation atlas set and self-attention feature map set set at each time point are output.

[0012] Based on the pancreas and lesion segmentation atlas, the lesion region is identified and spatially located to obtain the lesion spatial coordinate set;

[0013] Based on the spatial coordinate set of lesions, multi-scale self-attention features of the lesion region are extracted directionally from the self-attention feature map sets at each level; Upsampling and channel stitching operations were performed on the multi-scale self-attention features at each time point, and the features were arranged in chronological order to generate a time-series image feature sequence of pancreatitis.

[0014] As a preferred embodiment of the time-series image dynamic evaluation method for pancreatitis based on the self-attention mechanism described in this invention, the steps for establishing the lesion trajectory index table are as follows: The lesion regions corresponding to the multi-scale self-attention features at each time point in the time-series image feature sequence of pancreatitis are instantiated and distinguished to generate a set of lesion instances;

[0015] Based on the set of lesion instances, the similarity between the multi-scale self-attention features corresponding to lesion instances at adjacent time points is calculated to generate a lesion instance similarity matrix; The lesion instance pairs with similarity exceeding the preset similarity threshold are selected from the lesion instance similarity matrix for association matching, and a globally unique trajectory ID is assigned to establish a lesion trajectory index table.

[0016] As a preferred embodiment of the time-series image dynamic evaluation method for pancreatitis based on the self-attention mechanism described in this invention, the steps for aggregating and obtaining the weighted representation matrix of lesion trajectories are as follows:

[0017] Based on the lesion trajectory index table, multi-scale self-attention features corresponding to each trajectory ID are extracted from the pancreatitis time-series image feature sequence to form a lesion trajectory feature sequence;

[0018] Based on the self-attention mechanism of the Swin-UNet medical image segmentation network, the importance scores of multi-scale self-attention features at different time points in the lesion trajectory feature sequence are calculated to obtain the trajectory-level attention weight vector. By using trajectory-level attention weight vectors, multi-scale self-attention features at all time points in the lesion trajectory feature sequence are weighted and aggregated to generate a weighted representation matrix of the lesion trajectory.

[0019] As a preferred embodiment of the self-attention mechanism-based dynamic assessment method for time-series images of pancreatitis described in this invention, the steps of dynamically assigning weights to the preprocessed clinical time-series auxiliary data based on image features according to the lesion trajectory weighted representation matrix are as follows: The lesion trajectory weighted representation matrix is ​​strictly aligned with the preprocessed clinical time-series auxiliary data, and the imaging features representing the severity of the lesions are extracted from the aligned lesion trajectory weighted representation matrix to generate an image-guided feature vector.

[0020] Based on the image-guided feature vector, a fixed importance weight is matched to each indicator in the aligned clinical time-series auxiliary data from a preset clinical weight lookup table to generate a clinical time-series weight vector. The indicators include C-reactive protein, white blood cell count, creatinine, amylase, and lipase.

[0021] As a preferred embodiment of the time-series image dynamic assessment method for pancreatitis based on the self-attention mechanism described in this invention, the steps for generating the time-series feature matrix of the disease course are as follows:

[0022] The clinical time-series weight vector is multiplied element-by-element by the trajectory-level attention weight vector to generate dynamic trajectory-clinical joint weights. By using dynamic trajectory-clinical joint weights, the aligned lesion trajectory weighted representation matrix is ​​reweighted to generate a disease course temporal feature matrix.

[0023] As a preferred embodiment of the time-series image dynamic assessment method for pancreatitis based on the self-attention mechanism described in this invention, the steps for generating the pancreatitis severity change curve are as follows: A time-weighted regression analysis was performed on the disease course time sequence feature matrix along the time dimension to fit the continuous change trend of pancreatitis severity and generate a preliminary severity change curve. The system calls the preset clinical severity grading threshold to convert the initial severity change curve into discrete severity levels, generating a pancreatitis severity change curve and severity level sequence.

[0024] As a preferred embodiment of the time-series image dynamic assessment method for pancreatitis based on the self-attention mechanism described in this invention, the steps for outputting the dynamic assessment report of pancreatitis disease progression are as follows:

[0025] The trajectory-level attention weights are upsampled to the original image resolution using a bilinear interpolation algorithm and then superimposed onto the corresponding lesion areas of multi-time point enhanced CT images of the pancreas to generate an interpretable heatmap sequence of lesion areas. By integrating the severity change curve of pancreatitis, the severity grade sequence, and the interpretable heat map sequence of lesion areas, a dynamic assessment report of the course of pancreatitis is generated.

[0026] The beneficial effects of this invention are as follows: by obtaining the lesion trajectory weighted representation matrix, continuous tracking and feature fusion of the same lesion at different time points are realized, thereby capturing the dynamic evolution law of lesion morphology and range, and accurately depicting the individualized evolution trajectory of the lesion; by adopting the temporal weighted fusion method to dynamically adjust the trajectory-level attention weight, adaptive fusion of imaging features and clinical indicators in the time dimension is realized, improving the consistency of severity assessment time sequence and clinical interpretability. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of a time-series image dynamic assessment method for pancreatitis based on a self-attention mechanism.

[0029] Figure 2 This is a flowchart for spatial registration and feature extraction.

[0030] Figure 3 A flowchart for lesion identification and trajectory indexing.

[0031] Figure 4 This is a flowchart for dynamic weight adjustment and generation of the disease course feature matrix. Detailed Implementation

[0032] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0033] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0034] Secondly, the term "one embodiment" or "example" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the invention. The appearance of an embodiment in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that mutually excludes other embodiments.

[0035] Reference Figures 1-4This is one embodiment of the present invention, which provides a method for dynamic evaluation of pancreatitis time-series images based on a self-attention mechanism, comprising the following steps: S1. Acquire multi-timepoint enhanced CT images of the pancreas and clinical time-series auxiliary data and perform preprocessing; Clinical time-series auxiliary data include laboratory test indicators, treatment intervention records, and information on the occurrence of complications; It should be noted that laboratory test indicators are data obtained directly from the testing of patients' blood and body fluid samples using automated analytical instruments; treatment intervention records include information on fluid resuscitation volume, antibiotic use, and analgesia regimens, which are recorded by medical staff in the electronic medical record on the hospital information platform during the diagnosis and treatment process; information on complications, including acute kidney injury, respiratory failure, and pancreatic necrosis infection, are recorded in the medical record after being determined by the physician according to the corresponding clinical diagnostic criteria.

[0036] Preprocessing includes window width and window level adjustment, intensity normalization, and missing value filling; It should be noted that window width and level adjustment refers to mapping the original CT value range of multi-timepoint enhanced pancreatic CT images to the grayscale range of the display device by setting specific window widths and levels (e.g., window width of 250 HU and window level of 40 HU), in order to optimize the contrast of specific tissues (such as the pancreas and lesions) and suppress interference from irrelevant tissues; intensity normalization refers to unifying the CT values ​​of multi-timepoint enhanced pancreatic CT images after window width and level adjustment to a specific range through Z-score standardization, eliminating intensity differences caused by different scanning devices and scanning parameters; missing value imputation refers to filling in missing values ​​in clinical time-series auxiliary data (such as laboratory test indicators) caused by undetected or unrecorded values ​​using the mean imputation method, ensuring the integrity of clinical time-series auxiliary data.

[0037] S2. Spatial registration is performed on the preprocessed multi-time-point enhanced CT images of the pancreas, and the Swin-UNet medical image segmentation network is used to extract multi-scale self-attention features of the pancreas and lesions to generate a temporal image feature sequence of pancreatitis. The time point with the clearest pancreatic anatomy was selected from the preprocessed multi-time point enhanced CT images of the pancreas as a fixed reference time point; Furthermore, at each time point of the preprocessed multi-time-point enhanced CT images of the pancreas, the Sobel operator is used to calculate the gradient magnitude map at each time point, and it is overlapped with the boundary region of the preset pancreatic region of interest template. The average gradient magnitude of all pixels in the overlap region is calculated to quantify the clarity of the pancreatic contour; the average CT value difference between the pancreatic region of interest template region and the outer ring region is calculated to quantify the contrast between the pancreas and the surrounding tissue structures; the standard deviation of CT values ​​within the pancreatic region of interest template region is calculated to quantify the uniform enhancement of pancreatic parenchyma; the average gradient magnitude, average CT value difference, and the reciprocal of the standard deviation calculated at each time point are normalized and weighted and summed to obtain a comprehensive score, and the time point with the highest comprehensive score is selected as the fixed reference time point.

[0038] It should be noted that the Sobel operator is an edge detection operator used in image processing. It uses two 3x3 convolution kernels (one for the horizontal direction and one for the vertical direction) to perform convolution operations with the image to approximate the gradient magnitude and direction of each pixel in the image, thereby highlighting the edge regions in the image.

[0039] The enhanced CT images of the pancreas at each time point are registered with the enhanced CT images of the pancreas at a fixed reference time point to generate the spatial transformation parameter matrix for each time point; Furthermore, using enhanced CT images of the pancreas at a fixed reference time point as fixed images and enhanced CT images of the pancreas at the time point requiring registration as moving images, a similarity measure based on mutual information is used to quantify the alignment between the fixed and moving images. The spatial transformation parameters of the moving images are iteratively adjusted using a regularized gradient descent method, and the spatial transformation parameters of the moving images together constitute the spatial transformation parameter matrix. The optimization process is continued until the mutual information value between the moving and fixed images reaches its maximum, and the final spatial transformation parameter matrix is ​​recorded. The above process is repeated for enhanced CT images of the pancreas at each time point requiring registration to generate spatial transformation parameter matrices for each time point.

[0040] It should be noted that mutual information similarity measure is based on information theory. It quantifies the amount of information contained in one image by calculating the difference between the joint probability distribution of corresponding pixel gray values ​​of two images and the product of their respective independent probability distributions (measured by Kullback-Leibler divergence). The larger the mutual information value, the higher the consistency of the two images in spatial structure, that is, the better the alignment. Regularized gradient descent is an optimization algorithm that incorporates regularization techniques into the standard gradient descent process. The principle is to linearly combine the negative gradient direction with the parameter vector itself in a certain proportion (determined by the learning rate and regularization coefficient) in the parameter update formula of standard gradient descent. In this way, while pursuing the minimization of the loss function, the size of the parameters is constrained to obtain a more stable solution with stronger generalization ability.

[0041] The pancreatic enhanced CT images at corresponding time points are resampled and transformed using the spatial transformation parameter matrix at each time point to generate a multi-time point pancreatic enhanced CT image registration sequence. Furthermore, the coordinates of each voxel in the moving image are multiplied with the corresponding spatial transformation parameter matrix to obtain the new coordinates of each voxel in the fixed image space. In the fixed image space, the eight nearest voxel vertices surrounding the new coordinates are located, and two one-dimensional linear interpolations are performed on the four adjacent XY plane slices to obtain four intermediate values. Then, a third linear interpolation is performed on the four intermediate values ​​along the Z direction to obtain the gray value estimate at the new coordinates. The gray value estimate at the new coordinates is then assigned to the voxel coordinates corresponding to the transformed moving image. After performing the above coordinate transformation and gray value interpolation operations on the enhanced pancreatic CT images at each time point, the enhanced pancreatic CT images at all time points are transformed to the anatomical space consistent with the fixed reference time point, thereby generating a multi-time point enhanced pancreatic CT image registration sequence.

[0042] The registration sequence of enhanced CT images of the pancreas at multiple time points is input into the Swin-UNet medical image segmentation network for forward propagation calculation, and the pancreas and lesion segmentation atlas set and self-attention feature map set set at each time point are output. Furthermore, the enhanced CT images of the pancreas at each time point in the multi-time point pancreatic enhanced CT image registration sequence are sequentially input into the Swin-UNet medical image segmentation network. The encoder part of the Swin-UNet medical image segmentation network extracts features from the enhanced CT images of the pancreas at each time point through a hierarchical structure that includes sliding window self-attention calculation, and generates and retains the self-attention feature map of the corresponding resolution at each level. The features extracted by the encoder are fused and reconstructed by upsampling and skip connections in the decoder part, and pixel-level classification pancreatic and lesion segmentation atlases are generated at the output of the Swin-UNet medical image segmentation network for each time point. After all time points of the multi-time point pancreatic enhanced CT image registration sequence have been processed, the pancreatic and lesion segmentation atlases of each time point are obtained, as well as the set of self-attention feature maps of each level composed of the self-attention feature maps of all time points and the encoder level.

[0043] It should be noted that training the Swin-UNet medical image segmentation network involves collecting historical time-series pancreatitis images and dividing the single-time-point enhanced CT images of the pancreas and their corresponding pancreatic and lesion-annotated segmentation maps into training and validation sets based on the patient's unique identifier (such as medical record number or anonymized ID). (This means all single-time-point enhanced CT images of the pancreas and their corresponding segmentation maps of the same patient are treated as an indivisible whole and completely assigned to one set in the training or validation set.) A fixed number of samples are randomly selected from the training set to form a training batch. The single-time-point enhanced CT images of the pancreas from the training batch are then input into the Swin-UNet medical image segmentation network for processing. Forward propagation is performed to obtain the predicted segmentation map. The error between the predicted segmentation map and the pancreas and lesion labeled segmentation map is calculated using a loss function. The weight parameters of the Swin-UNet medical image segmentation network are updated using the backpropagation algorithm and stochastic gradient descent optimizer. The above process is iterated until all samples in the training set have been traversed, which is considered a training cycle. After each training cycle, the Swin-UNet medical image segmentation network is validated using a validation set, and the segmentation performance index is calculated. When the segmentation performance index does not reach a new optimal value in multiple consecutive training cycles, training is stopped, the historical optimal weight parameters are saved, and the trained Swin-UNet medical image segmentation network is output.

[0044] Based on the pancreas and lesion segmentation atlas, the lesion region is identified and spatially located to obtain the lesion spatial coordinate set; Furthermore, for the pancreas and lesion segmentation maps at each time point in the pancreas and lesion segmentation map set, a connected component analysis algorithm is applied to traverse all pixels marked as lesions in the pancreas and lesion segmentation map. Based on the eight-neighbor connectivity of pixels, spatially adjacent lesion pixels are merged into the same connected region, and each independent connected region is identified as a lesion region. All voxels contained in each identified lesion region are traversed, and the minimum and maximum values ​​of all voxels in the X, Y, and Z coordinate axes are found. The cube region defined by the extreme points (the minimum and maximum values ​​in the X, Y, and Z coordinate axes) is the minimum bounding box of the independent lesion region. The vertex coordinates of the minimum bounding boxes corresponding to all time points and all lesion regions are summarized to form the lesion spatial coordinate set.

[0045] It should be noted that the connected component analysis algorithm is an image processing algorithm used to identify independent connected regions in a binary image. It is used to cluster pixels belonging to the lesion category in the pancreas and lesion segmentation map in order to identify and segment spatially connected lesion pixel regions into independent lesion regions.

[0046] Based on the spatial coordinate set of lesions, multi-scale self-attention features of the lesion region are extracted directionally from the self-attention feature map sets at each level; Furthermore, based on the time point to which each lesion region belongs in the lesion spatial coordinate set, the self-attention feature maps from different levels of the Swin-UNet medical image segmentation network encoder at the corresponding time point are located from the self-attention feature map sets at each level. The vertex coordinates of the minimum bounding box of the lesion region are scaled according to the downsampling ratio of the self-attention feature maps at each level relative to the enhanced CT images of the pancreas at each time point to obtain scaled coordinate values. The region determined by the scaled coordinate values ​​on the self-attention feature maps at all levels is taken as the region of interest of the lesion. From the deep, middle and shallow layers of the encoder, all feature vectors of the region of interest of the lesion on the corresponding self-attention feature maps are extracted as multi-scale self-attention features of the lesion region.

[0047] Upsampling and channel stitching operations were performed on the multi-scale self-attention features at each time point, and the features were arranged in chronological order to generate a time-series image feature sequence of pancreatitis. Furthermore, for the multi-scale self-attention features extracted at each time point, bilinear upsampling is performed on the self-attention features from the shallow and middle layers of the encoder to ensure that the spatial size is consistent with the spatial size of the self-attention features from the deep layers of the encoder. In the channel dimension, the self-attention features from the deep, middle, and shallow layers after size unification are directly connected to form a comprehensive multi-scale self-attention feature map. The comprehensive multi-scale self-attention feature maps generated at all time points are arranged in the order of their corresponding acquisition times and assembled into a pancreatitis time-series image feature sequence.

[0048] It should be noted that bilinear upsampling is an image scaling technique. The principle is to estimate the pixel value at the new coordinates of the target by performing two linear interpolations on the values ​​of four adjacent known pixels in a two-dimensional plane (first interpolating in the horizontal direction, and then interpolating in the vertical direction), thereby achieving image magnification.

[0049] S3. Identify and correlate lesions across time-series images of pancreatitis, establish a lesion trajectory index table, and calculate trajectory-level attention weights in the self-attention layer of the Swin-UNet medical image segmentation network to aggregate and obtain the weighted representation matrix of lesion trajectories. The lesion regions corresponding to the multi-scale self-attention features at each time point in the time-series image feature sequence of pancreatitis are instantiated and distinguished to generate a set of lesion instances; Furthermore, the time-series image feature sequence of pancreatitis is read. Based on the upsampling factor performed when generating the comprehensive multi-scale self-attention feature map at each time point, each spatial location in the comprehensive multi-scale self-attention feature map is subjected to inverse coordinate transformation to map back to the original three-dimensional spatial coordinates of the enhanced CT images of the pancreas at each time point. Under the original three-dimensional spatial coordinates, the L2 norm of the feature vector at each spatial location on the comprehensive multi-scale self-attention feature map is calculated as the activation intensity to obtain a three-dimensional activation intensity map. Using a three-dimensional connected component analysis algorithm, voxels with activation intensities exceeding a preset activation intensity threshold and adjacent layers in three-dimensional space are merged into the same connected region. Each connected region is treated as an independent lesion instance and assigned a unique intra-time point identifier. The identifiers and spatial range information of all independent lesion instances within all time points are summarized to generate a lesion instance set.

[0050] It should be noted that the 3D connected component analysis algorithm is an image processing algorithm used to identify and label interconnected voxel regions in 3D volume data. By traversing each voxel in the 3D activation intensity map and according to the 26-neighborhood connectivity rule (i.e., a voxel is considered connected to its neighboring voxels in the front, back, left, right, top, bottom, and all diagonal directions), it recursively searches and labels neighboring voxels whose activation intensity exceeds a preset activation intensity threshold, thereby merging the set of spatially connected high-activation voxels into the same independent connected region. The activation intensity threshold is set based on the statistical quantile (such as the 95th percentile) of the feature norm distribution of lesion regions and non-lesion regions on the validation set. It is a threshold value used to distinguish salient activation regions. An exemplary value range is [0, +). Values ​​below 0 will misclassify noise as lesions.

[0051] Based on the set of lesion instances, the similarity between the multi-scale self-attention features corresponding to lesion instances at adjacent time points is calculated to generate a lesion instance similarity matrix; Furthermore, for each pair of adjacent time points, multi-scale self-attention features of all lesion instances at the previous and subsequent time points are extracted from the lesion instance set. For each lesion instance at the previous time point, the cosine similarity formula is used to calculate the similarity between the multi-scale self-attention features and those of each lesion instance at the subsequent time point. The similarity matrix of lesion instances is generated by arranging them in the order of the lesion instances at the previous time point as rows and the order of the lesion instances at the subsequent time point as columns.

[0052] From the lesion instance similarity matrix, lesion instance pairs with similarity exceeding a preset similarity threshold are selected for association matching, and globally unique trajectory IDs are assigned to establish a lesion trajectory index table. Furthermore, for each pair of lesion instance similarity matrices between adjacent time points, all elements in the lesion instance similarity matrix are scanned to identify all lesion instance pairs with similarity exceeding a preset similarity threshold. For each lesion instance at a later time point, only connections with the highest similarity to the lesion instance at the previous time point that exceed the similarity threshold are retained to ensure the uniqueness of the match. Successfully matched lesion instance pairs are determined to be the manifestation of the same lesion at different time points, and a globally unique trajectory ID is assigned to each lesion instance pair. The trajectory ID of each lesion trajectory, as well as all time points associated with the trajectory ID and the lesion instance identifiers within the corresponding time points, are recorded to generate a lesion trajectory index table.

[0053] It should be noted that the similarity threshold is a threshold value set by statistical methods (such as the mean plus twice the standard deviation) to determine the probability of association based on the feature similarity distribution of known matching lesion instance pairs in historical data. For example, the value range is 0 to 1. A value higher than 1 will result in no lesion instance pairs being matched, while a value lower than 0 will result in all lesion instance pairs being matched, leading to incorrect association.

[0054] Based on the lesion trajectory index table, multi-scale self-attention features corresponding to each trajectory ID are extracted from the pancreatitis time-series image feature sequence to form a lesion trajectory feature sequence; Furthermore, for each lesion trajectory recorded in the lesion trajectory index table, all associated time points and lesion instance identifiers within the corresponding time points are queried based on the trajectory ID. Based on the queried time points and lesion instance identifiers, the comprehensive multi-scale self-attention feature map of the corresponding time point is located from the pancreatitis time-series image feature sequence, and the multi-scale self-attention features of the spatial region corresponding to the lesion instance identifier are extracted from the comprehensive multi-scale self-attention feature map. The multi-scale self-attention features extracted from all associated time points of the lesion trajectory are arranged in chronological order to form the lesion trajectory feature sequence corresponding to the trajectory ID.

[0055] Based on the self-attention mechanism of the Swin-UNet medical image segmentation network, the importance scores of multi-scale self-attention features at different time points in the lesion trajectory feature sequence are calculated to obtain the trajectory-level attention weight vector. Furthermore, the multi-scale self-attention feature at each time point in the lesion trajectory feature sequence is regarded as a feature block. In the initialization stage of the Swin-UNet medical image segmentation network, three globally trainable parameter matrices are defined as query weight matrix, key weight matrix, and value weight matrix, respectively. Query weight matrix, key weight matrix, and value weight matrix are assigned to each feature block in the lesion trajectory feature sequence, and the multi-scale self-attention feature vector at each time point is multiplied by the three weight matrices and linearly projected onto the new feature space, thereby generating the corresponding query vector, key vector, and value vector for each time point. The dot product of the query vector at each time point and the key vectors at all time points in the lesion trajectory feature sequence is calculated and normalized to obtain the attention weight. The attention weight at each time point is extracted as an importance score and aggregated to generate a trajectory-level attention weight vector.

[0056] Using trajectory-level attention weight vectors, multi-scale self-attention features at all time points in the lesion trajectory feature sequence are weighted and aggregated to generate a weighted representation matrix of the lesion trajectory. Furthermore, the trajectory-level attention weight vector defines the relative importance of features at different time points in the lesion trajectory feature sequence. Based on the attention weight of the trajectory-level attention weight vector, the multi-scale self-attention features at all time points in the lesion trajectory feature sequence are weighted and fused to generate a fused feature vector to represent the complete temporal dynamics of the lesion trajectory. The fused feature vectors corresponding to all lesion trajectories are collected to form a weighted representation matrix of the lesion trajectory.

[0057] S4. Based on the lesion trajectory weighted representation matrix, dynamic weight allocation based on image features is performed on the preprocessed clinical time-series auxiliary data, and the time-series weighted fusion method is used to dynamically adjust the trajectory-level attention weight in the time dimension to generate the disease course time-series feature matrix. The lesion trajectory weighted representation matrix is ​​strictly aligned with the preprocessed clinical time-series auxiliary data, and the imaging features representing the severity of the lesions are extracted from the aligned lesion trajectory weighted representation matrix to generate an image-guided feature vector. Furthermore, based on the timestamp, each row of the lesion trajectory weighted representation matrix is ​​precisely matched with the corresponding time point record in the preprocessed clinical time-series auxiliary data to ensure that the trajectory features at each time point correspond one-to-one with the clinical data. Each row of the lesion trajectory weighted representation matrix represents the lesion trajectory fusion feature at each time point. For the feature row at each time point, the arithmetic mean of all feature dimensions is calculated as the overall feature intensity of the lesion, and the standard deviation of all feature dimensions is used as the intrinsic heterogeneity of the lesion features. The Euclidean distance between the feature rows of consecutive time points is calculated using the Euclidean distance calculation formula to represent the dynamic change amplitude of the lesion features. The feature intensity, feature heterogeneity, and dynamic change amplitude of the lesion at each time point are used as image guidance features to quantify the severity of the lesion at each time point, and are aggregated to generate an image guidance feature vector.

[0058] Based on the image-guided feature vector, a fixed importance weight is matched to each indicator in the aligned clinical time-series auxiliary data from a preset clinical weight lookup table to generate a clinical time-series weight vector. Furthermore, for each time point in the aligned clinical time-series auxiliary data, the image guidance feature vector corresponding to the time point is used as input, and a precise search and nearest neighbor matching operation is performed in the clinical weight lookup table to directly obtain the fixed importance weights set for each indicator associated with the image guidance feature vector; the fixed importance weights of each indicator at all time points are collected to form the clinical time-series weight vector.

[0059] It should be noted that the clinical weight lookup table is based on the correlation analysis results of image guidance features and clinical indicators in historical data. The image guidance feature vectors are sorted according to their numerical values ​​and binned at equal frequencies. For all clinical indicators falling within the same bin interval, the absolute value of the Spearman rank correlation coefficient with the severity of pancreatitis is calculated, and after scaling, a fixed importance weight is obtained. This results in a mapping table containing feature vector intervals, a list of clinical indicators, and corresponding weight values.

[0060] The indicators include C-reactive protein, white blood cell count, creatinine, amylase, and lipase; It should be noted that C-reactive protein is obtained by immunoturbidimetric assay of patient serum samples and is used to characterize the intensity of acute inflammatory response in the body; white blood cell count is obtained by fully automated hematology analyzer of patient anticoagulated whole blood samples and is used to characterize the degree of immune system activation and infection; creatinine is obtained by picric acid assay of patient serum or plasma samples and is used to characterize renal function status and possible acute kidney injury; amylase is obtained by enzymatic reaction assay of patient serum or urine samples and is used to characterize the degree of pancreatic acinar cell damage; lipase is obtained by colorimetric assay of patient serum samples and is used to specifically characterize the impairment of pancreatic exocrine function.

[0061] The clinical time-series weight vector is multiplied element-by-element by the trajectory-level attention weight vector to generate dynamic trajectory-clinical joint weights. Furthermore, the clinical time-series weight vector and the trajectory-level attention weight vector have the same dimension, both representing the weight values ​​of the same set of time points. The weight value of each time point in the clinical time-series weight vector is multiplied by the corresponding weight value of the time point in the trajectory-level attention weight vector to obtain a new weight value for the time point that integrates clinical information and image trajectory information. The new weight values ​​of all time points are arranged in the row order of the lesion trajectory weighted representation matrix to form a dynamic trajectory-clinical joint weight vector.

[0062] By using dynamic trajectory-clinical joint weights, the aligned lesion trajectory weighted representation matrix is ​​reweighted to generate a disease course temporal feature matrix; Furthermore, the lesion trajectory feature vector at each time point in the aligned lesion trajectory weighted representation matrix is ​​multiplied by the new weight value of the corresponding time point in the dynamic trajectory-clinical joint weight vector to obtain the reweighted lesion trajectory feature vector at each time point; the reweighted lesion trajectory feature vectors at all time points are arranged in chronological order to form the disease course temporal feature matrix.

[0063] S5. Perform time-weighted regression and probability threshold analysis on the time-series feature matrix of the disease course to generate a curve of pancreatitis severity change, and map the trajectory-level attention weight to multi-time point enhanced CT images of the pancreas to form an interpretable heat map of the lesion area, and output a dynamic assessment report of the pancreatitis course. A time-weighted regression analysis was performed on the disease course time sequence feature matrix along the time dimension to fit the continuous change trend of pancreatitis severity and generate a preliminary severity change curve. Furthermore, each row of the disease progression time-series feature matrix is ​​considered as an observation sample at each time point, with the row index representing the time sequence. A weight value is assigned to each time point, with the initial weight value set to 1. The weight value increases linearly with the row index according to a preset linear coefficient. Using the time point as the independent variable and the lesion trajectory feature vector of each time point in the disease progression time-series feature matrix as the dependent variable, the regression coefficient is solved using the least squares method. Using the regression coefficient, the predicted value of pancreatitis severity corresponding to each time point is calculated using the linear regression prediction formula. The predicted value of pancreatitis severity is used as the vertical axis, and the corresponding time point is used as the horizontal axis to plot the data points on a two-dimensional plane. The cubic spline interpolation algorithm is used to connect the data points into a smooth continuous curve to generate a preliminary severity change curve.

[0064] The system calls the preset clinical severity grading threshold to convert the initial severity change curve into discrete severity levels, generating a pancreatitis severity change curve and severity level sequence. Furthermore, the predicted severity value of pancreatitis at each time point is extracted from the preliminary severity change curve and compared with the preset clinical severity grading threshold: if the predicted severity value of pancreatitis is lower than the clinical severity grading threshold, the severity is marked as mild; if the predicted severity value of pancreatitis is higher than the clinical severity grading threshold, the severity is marked as severe; if the predicted severity value of pancreatitis is within the clinical severity grading threshold, the severity is marked as moderate. The severity label is assigned to the severity level at each time point, and the severity levels at all time points are collected and arranged in chronological order to generate a severity level sequence. The preliminary severity change curve with severity level labels is used as the pancreatitis severity change curve.

[0066] It should be noted that the clinical severity grading threshold is a specific numerical range formed by quantifying the objective diagnostic criteria for pancreatitis severity (such as duration of organ failure and local complications) in authoritative clinical guidelines (such as the Atlanta classification) through statistical methods. For example, the range is [0,1]. If the predicted value of pancreatitis severity is higher than 1, it will lead to severe cases being underestimated as mild cases, and lower than 0, it will lead to mild cases being overestimated as severe cases, thus distorting the assessment results.

[0067] The trajectory-level attention weights are upsampled to the original image resolution using a bilinear interpolation algorithm and then superimposed onto the corresponding lesion areas of multi-time point enhanced CT images of the pancreas to generate an interpretable heatmap sequence of lesion areas. Furthermore, the trajectory-level attention weights are low-resolution three-dimensional arrays. The spatial scaling ratio between the trajectory-level attention weights and the corresponding multi-time-point enhanced pancreatic CT images is calculated. Based on the spatial scaling ratio, each voxel coordinate in the multi-time-point enhanced pancreatic CT images is represented by the three-dimensional grid of the trajectory-level attention weights. By performing cubic linear interpolation calculations at eight adjacent low-resolution grid points for each voxel coordinate, a high-resolution trajectory-level attention weight at each voxel coordinate is obtained, completing the upsampling. According to a preset weight value-color lookup table, the high-resolution trajectory-level attention weights are directly mapped to the corresponding pseudo-color values. The converted pseudo-color values ​​are then fused with the grayscale values ​​of the corresponding multi-time-point enhanced pancreatic CT images using the alpha channel, so that lesion areas with higher trajectory-level attention weights are superimposed and displayed with more prominent colors. The upsampling pseudo-color mapping and image fusion operations are repeated for each time point to generate an interpretable heatmap sequence of lesion areas arranged in chronological order.

[0068] It should be noted that the weight-color lookup table is a static mapping table containing 256 entries, constructed by linearly interpolating the RGB color channels from blue (corresponding to the minimum value 0) to red (corresponding to the maximum value 1) based on a continuous range of weight values ​​from 0 to 1. Alpha channel fusion is an image processing technique that semi-transparently overlays the foreground and background images. It is used to fuse the pseudo-color heatmap (as the foreground) representing the trajectory-level attention weight with the grayscale image of the multi-time point enhanced CT pancreas (as the background). By normalizing the trajectory-level attention weight and using it as the Alpha value, areas with higher trajectory-level attention weights are displayed more prominently (the foreground color is more prominent), while areas with lower trajectory-level attention weights reveal more background image details, thereby generating an intuitive heatmap that can be interpreted for lesion areas.

[0069] By integrating the severity change curve of pancreatitis, the severity grade sequence, and the interpretable heat map sequence of lesion area, a dynamic assessment report of the course of pancreatitis is generated. Furthermore, the severity change curve of pancreatitis is presented in graphical form to show the continuous trend of pancreatitis severity over time. The severity grade sequence is listed in tabular form to show the specific severity grade of pancreatitis at each time point. The explanatory heat map sequence of lesion areas is presented in image sequence form to show the attention weight distribution of lesion areas at different time points. The severity change curve of pancreatitis, the severity grade sequence, and the explanatory heat map sequence of lesion areas are aligned and correlated along the time axis and arranged together in a structured document to form a comprehensive dynamic assessment report of pancreatitis course that includes quantitative trend analysis, qualitative grade assessment, and image visualization evidence.

[0070] In summary, this invention achieves continuous tracking and feature fusion of the same lesion at different time points by obtaining a lesion trajectory weighted representation matrix, thereby capturing the dynamic evolution law of lesion morphology and range and accurately depicting the individualized evolution trajectory of the lesion; by adopting a time-series weighted fusion method to dynamically adjust the trajectory-level attention weight, it achieves adaptive fusion of imaging features and clinical indicators in the time dimension, improving the consistency of severity assessment time sequence and clinical interpretability.

[0071] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for dynamic evaluation of temporal images of pancreatitis based on a self-attention mechanism, characterized in that: include, Acquired and preprocessed multi-timepoint enhanced CT images of the pancreas and clinical time-series auxiliary data; Spatial registration was performed on the preprocessed multi-time-point enhanced CT images of the pancreas, and the Swin-UNet medical image segmentation network was used to extract multi-scale self-attention features of the pancreas and lesions to generate a temporal image feature sequence of pancreatitis. Lesion identification and cross-temporal correlation were performed on the time-series image feature sequences of pancreatitis, a lesion trajectory index table was established, and trajectory-level attention weights were calculated in the self-attention layer of the Swin-UNet medical image segmentation network to obtain the weighted representation matrix of lesion trajectories. Based on the lesion trajectory weighted representation matrix, dynamic weight allocation based on image features is performed on the preprocessed clinical time-series auxiliary data, and the time-series weighted fusion method is used to dynamically adjust the trajectory-level attention weight in the time dimension to generate a disease course time-series feature matrix. Time-weighted regression and probability threshold analysis were performed on the temporal feature matrix of the disease course to generate a curve of pancreatitis severity change. The trajectory-level attention weights were then mapped to multi-time-point enhanced CT images of the pancreas to form an interpretable heat map of the lesion area, and a dynamic assessment report of the pancreatitis course was output.

2. The method for dynamic evaluation of pancreatitis temporal images based on self-attention mechanism as described in claim 1, characterized in that: The clinical time-series auxiliary data includes laboratory test indicators, treatment intervention records, and information on the occurrence of complications; The preprocessing includes window width and window level adjustment, intensity normalization, and missing value filling.

3. The method for dynamic evaluation of pancreatitis time-series images based on self-attention mechanism as described in claim 1, characterized in that: The steps for spatial registration of the preprocessed multi-timepoint enhanced CT images of the pancreas are as follows: The time point with the clearest pancreatic anatomy was selected from the preprocessed multi-time point enhanced CT images of the pancreas as a fixed reference time point; The enhanced CT images of the pancreas at each time point are registered with the enhanced CT images of the pancreas at a fixed reference time point to generate the spatial transformation parameter matrix for each time point; By using the spatial transformation parameter matrix at each time point, the enhanced CT images of the pancreas at the corresponding time point are resampled and transformed to generate a multi-time point enhanced CT image registration sequence.

4. The method for dynamic evaluation of pancreatitis temporal images based on self-attention mechanism as described in claim 1, characterized in that: The steps for generating the temporal image feature sequence of pancreatitis are as follows: The registration sequence of enhanced CT images of the pancreas at multiple time points is input into the Swin-UNet medical image segmentation network for forward propagation calculation, and the pancreas and lesion segmentation atlas set and self-attention feature map set set at each time point are output. Based on the pancreas and lesion segmentation atlas, the lesion region is identified and spatially located to obtain the lesion spatial coordinate set; Based on the spatial coordinate set of lesions, multi-scale self-attention features of the lesion region are extracted directionally from the self-attention feature map sets at each level; Upsampling and channel stitching operations were performed on the multi-scale self-attention features at each time point, and the features were arranged in chronological order to generate a time-series image feature sequence of pancreatitis.

5. The method for dynamic evaluation of pancreatitis temporal images based on self-attention mechanism as described in claim 1, characterized in that: The steps for establishing the lesion trajectory index table are as follows: The lesion regions corresponding to the multi-scale self-attention features at each time point in the time-series image feature sequence of pancreatitis are instantiated and distinguished to generate a set of lesion instances; Based on the set of lesion instances, the similarity between the multi-scale self-attention features corresponding to lesion instances at adjacent time points is calculated to generate a lesion instance similarity matrix; The lesion instance pairs with similarity exceeding the preset similarity threshold are selected from the lesion instance similarity matrix for association matching, and a globally unique trajectory ID is assigned to establish a lesion trajectory index table.

6. The method for dynamic evaluation of pancreatitis temporal images based on self-attention mechanism as described in claim 1, characterized in that: The steps for obtaining the weighted representation matrix of lesion trajectories through aggregation are as follows: Based on the lesion trajectory index table, multi-scale self-attention features corresponding to each trajectory ID are extracted from the pancreatitis time-series image feature sequence to form a lesion trajectory feature sequence; Based on the self-attention mechanism of the Swin-UNet medical image segmentation network, the importance scores of multi-scale self-attention features at different time points in the lesion trajectory feature sequence are calculated to obtain the trajectory-level attention weight vector. By using trajectory-level attention weight vectors, multi-scale self-attention features at all time points in the lesion trajectory feature sequence are weighted and aggregated to generate a weighted representation matrix of the lesion trajectory.

7. The method for dynamic evaluation of pancreatitis temporal images based on self-attention mechanism as described in claim 1, characterized in that: The steps for dynamically assigning weights based on imaging features to the preprocessed clinical time-series auxiliary data according to the lesion trajectory weighted representation matrix are as follows: The lesion trajectory weighted representation matrix is ​​strictly aligned with the preprocessed clinical time-series auxiliary data, and the imaging features representing the severity of the lesions are extracted from the aligned lesion trajectory weighted representation matrix to generate an image-guided feature vector. Based on the image-guided feature vector, a fixed importance weight is matched to each indicator in the aligned clinical time-series auxiliary data from a preset clinical weight lookup table to generate a clinical time-series weight vector. The indicators include C-reactive protein, white blood cell count, creatinine, amylase, and lipase.

8. The method for dynamic evaluation of pancreatitis temporal images based on self-attention mechanism as described in claim 1, characterized in that: The steps for generating the disease progression time-series feature matrix are as follows: The clinical time-series weight vector is multiplied element-by-element by the trajectory-level attention weight vector to generate dynamic trajectory-clinical joint weights. By using dynamic trajectory-clinical joint weights, the aligned lesion trajectory weighted representation matrix is ​​reweighted to generate a disease course temporal feature matrix.

9. The method for dynamic evaluation of pancreatitis time-series images based on self-attention mechanism as described in claim 1, characterized in that: The steps for generating the pancreatitis severity change curve are as follows: A time-weighted regression analysis was performed on the disease course time sequence feature matrix along the time dimension to fit the continuous change trend of pancreatitis severity and generate a preliminary severity change curve. The system calls the preset clinical severity grading threshold to convert the initial severity change curve into discrete severity levels, generating a pancreatitis severity change curve and severity level sequence.

10. The method for dynamic evaluation of pancreatitis temporal images based on self-attention mechanism as described in claim 1, characterized in that: The steps for generating a dynamic assessment report on the course of pancreatitis are as follows: The trajectory-level attention weights are upsampled to the original image resolution using a bilinear interpolation algorithm and then superimposed onto the corresponding lesion areas of multi-time point enhanced CT images of the pancreas to generate an interpretable heatmap sequence of lesion areas. By integrating the severity change curve of pancreatitis, the severity grade sequence, and the interpretable heat map sequence of lesion areas, a dynamic assessment report of the course of pancreatitis is generated.

Images