Liver and gall postoperative lesion quantitative evaluation method and system based on DLPE algorithm

Abstract

The invention relates to the technical field of biomedical engineering and artificial intelligence crossing, in particular to a quantitative evaluation method and system for postoperative pathological changes of liver and gall based on a DLPE algorithm, and the method comprises the steps: obtaining a multi-time-point multi-modal medical image, and generating a four-dimensional data set of spatial alignment through deep learning registration; performing multi-modal spatial-temporal feature fusion by using a parallel 3D-CNN and a cross-modal attention mechanism; inputting the fusion feature sequence into a GRU network modeling lesion dynamic evolution process, and outputting a global evolution feature vector; a voxel-level pathological evolution probability graph is generated through a three-dimensional decoder, and quantitative indexes such as the volume, the form and the signal intensity of each pathological state are extracted. By means of the technical scheme, accurate distinguishing and dynamic quantification of tumor recurrence, inflammatory edema and scar tissue can be achieved, and objectivity, accuracy and clinical decision support capacity of liver and gall postoperative curative effect evaluation are improved.

Description

Technical Field

[0001] This invention relates to the field of biomedical engineering and artificial intelligence, and in particular to a method and system for quantitative assessment of postoperative hepatobiliary lesions based on the DLPE algorithm. Background Technology

[0002] With the rapid development of medical imaging technology and artificial intelligence algorithms, the application of lesion quantitative assessment methods in the diagnosis and treatment of hepatobiliary surgery has gradually become a research hotspot. Precise quantitative analysis of postoperative lesion areas can provide doctors with more accurate diagnostic information, thereby optimizing treatment plans and improving patient prognosis. However, existing lesion quantitative assessment methods still have many shortcomings when dealing with complex postoperative lesions, especially in terms of multimodal data fusion, lesion area segmentation accuracy, and dynamic quantification capabilities, which still need improvement. A search revealed a method for quantitative analysis of pneumonia fibrosis based on the DLPE algorithm, published in CN114820571B. This patent collects lung CT data sets and performs normalization processing, designs a segmentation model to segment the lungs, respiratory tract, blood vessels, and visible lesions, and combines a 2.5D segmentation algorithm with a human-in-the-loop procedure to establish a stronger segmentation and quantification model. However, this technical solution mainly targets the quantitative analysis of pneumonia fibrosis, and its application scenario differs significantly from that of postoperative hepatobiliary lesions. Specifically, this method does not fully consider the unique characteristics of postoperative hepatobiliary lesions, such as complex situations like postoperative tissue adhesions, scar formation, and local inflammatory responses, which may lead to insufficient adaptability and accuracy of the segmentation model in handling postoperative hepatobiliary lesions. Furthermore, this approach lacks the ability to model the dynamic changes in the lesion area, making it difficult to meet the need for precise quantification of lesion progression during long-term postoperative follow-up.

[0003] On the other hand, a liver biopsy data analysis method and system, disclosed in patent CN119919312B, extracts the geometric center points of cell nuclei, blood vessels, and fibrotic regions from liver biopsy slice images, calculates spatial distances and connection angles, and generates a set of tissue topology network association parameters, thereby enhancing the dynamic quantification capability of pathological features. However, this technical solution mainly relies on biopsy slice images, and its data source is relatively singular, failing to fully utilize the rich information provided by multimodal imaging data (such as CT, MRI, etc.), thus limiting the comprehensiveness and accuracy of lesion quantification analysis. Furthermore, while this solution quantifies the spatial distribution characteristics of the lesion area, it does not fully consider the temporal evolution characteristics of the lesion after surgery, potentially making it difficult to capture the dynamic changes of the lesion area at different postoperative stages, affecting the clinical applicability of the evaluation results.

[0004] The aforementioned problems indicate that existing methods for quantitative assessment of lesions after hepatobiliary surgery still have certain shortcomings in areas such as multimodal data fusion, accuracy in segmenting complex lesion regions, and dynamic quantification capabilities. Therefore, this invention provides a method and system for quantitative assessment of postoperative hepatobiliary lesions based on the DLPE algorithm. The aim is to achieve accurate quantitative assessment of postoperative hepatobiliary lesions by integrating multimodal imaging data, optimizing the lesion region segmentation model, and introducing time evolution characteristic modeling, thereby meeting the clinical demand for efficient and intelligent assessment methods. Summary of the Invention

[0005] The technical problem to be solved by this invention is that existing postoperative hepatobiliary lesion assessment technologies have limitations in their analytical paradigms when facing complex challenges such as postoperative anatomical changes, multiple pathological factors, and dynamic evolution of disease course. Specifically, they mainly rely on a single image modality and lack the ability to explicitly model the temporal evolution characteristics of lesions, resulting in deficiencies in segmentation accuracy, multi-source information fusion efficiency, and dynamic quantitative tracking capabilities in complex scenarios.

[0006] To achieve the above-mentioned objectives, this invention provides a method for quantitative assessment of postoperative hepatobiliary lesions based on the DLPE algorithm. The method is executed using a computing device and includes the following steps: Step A: Data acquisition and temporal alignment processing steps. Acquire a patient's postoperative medical image sequence at multiple discrete time points, containing at least two modalities. The modalities include at least computed tomography (CT) images and magnetic resonance imaging (MRI) images. The MRI images further include T1-weighted imaging (T1WI), T2-weighted imaging (T2WI), and diffusion-weighted imaging (DWI) sequences. Then, execute a deep learning-based anatomical structure-aware deformable registration process to uniformly register all modal images at the multiple time points to a patient-specific reference anatomical coordinate system, generating a spatially aligned multimodal four-dimensional image dataset.

[0007] Step B: Multimodal spatiotemporal feature deep fusion step. Using a parallel 3D convolutional neural network (3D-CNN) encoder group, independent deep feature extraction is performed on the image data of each modality at each time point in the spatially aligned multimodal four-dimensional image dataset to generate feature maps corresponding to each modality. Subsequently, through a cross-modal attention fusion module, the intrinsic correlation and complementarity weights between different modal feature maps are calculated, and the feature maps are weighted and fused to generate a fused spatiotemporal feature sequence that can represent comprehensive pathological information at each time point.

[0008] Step C: The dynamic evolution trajectory modeling step of the lesion involves inputting the fused spatiotemporal feature sequence into a gated recurrent unit (GRU) network in chronological order. The GRU network processes the fused features of each time node in the sequence through its internal update and reset gate structures, and updates its hidden state, thereby capturing and encoding the dynamic changes in the lesion region in terms of morphology, density, signal intensity, etc. Finally, at the last time node, it outputs a global evolution feature vector containing complete information on the evolution history of the lesion.

[0009] Step D: Pathological Evolution Probability Map Generation and Quantitative Analysis. The global evolution feature vector is input into a three-dimensional convolutional decoder network. The decoder network upsamples and decodes the global evolution feature vector to generate a three-dimensional voxel-level pathological evolution probability map. Each voxel in the pathological evolution probability map contains probability values ​​for multiple channels, each channel corresponding to a preset postoperative pathological state. The pathological states include at least tumor recurrence, active inflammation or edema, and stable scar tissue. Finally, based on the pathological evolution probability map, a series of quantitative evaluation indicators are calculated and output. These indicators include the volume, volume change rate, morphological parameters, and time evolution curves of the mean and standard deviation of signal intensity for each pathological state.

[0010] Accordingly, the present invention also provides a quantitative assessment system for postoperative hepatobiliary lesions based on the DLPE algorithm, the system comprising: A data interface and preprocessing module is configured to receive and parse multimodal medical imaging data from multiple follow-up time points after a patient's surgery, conforming to the DICOM standard, and perform data format conversion, intensity normalization, and data structuring to form a raw dataset organized by patient and time series.

[0011] A time-series image registration module is electrically connected to the data interface and the preprocessing module. It contains an anatomically aware three-dimensional deformable registration network model. The model is trained to identify key stable anatomical anchor points in the postoperative hepatobiliary region and calculates a non-rigid three-dimensional deformation field based on this. This field is used to accurately align the modal images of all time points to a unified anatomical coordinate space, thereby eliminating spatial displacement caused by respiratory movements, changes in body position, and natural tissue evolution.

[0012] A multimodal spatiotemporal feature fusion module is electrically connected to the temporal image registration module. Its core is a neural network structure containing multiple parallel feature extraction branches and a central fusion unit. Each feature extraction branch is a three-dimensional residual network (3D-ResNet), which is responsible for extracting deep spatial features from aligned CT, T1WI, T2WI and DWI images respectively. The central fusion unit is a multi-head self-attention mechanism based on the Transformer architecture, which is configured to calculate and apply attention weights between cross-modal feature maps to achieve effective aggregation of information and generate fused features.

[0013] A lesion dynamic evolution modeling module is electrically connected to the multimodal spatiotemporal feature fusion module. Its core is a multi-layer stacked gated recurrent unit (GRU) network, which receives fused features arranged in chronological order as input and encodes the continuous change pattern of the lesion in the time dimension into its final hidden state vector through cyclic processing.

[0014] A pathological evolution probability map generation and quantitative analysis module is electrically connected to the lesion dynamic evolution modeling module. It includes a three-dimensional U-Net decoder for decoding the hidden state vectors output by the GRU network into a voxel-level multi-channel pathological probability map. The module also integrates a set of analysis algorithms for automatically segmenting and extracting three-dimensional regions of different pathological components from the probability map, and calculating a series of quantitative indicators such as volume, sphericity, surface area to volume ratio, and average signal intensity on each original modality image, and generating a dynamic trend map.

[0015] A results visualization and report generation module, electrically connected to the pathological evolution probability map generation and quantitative analysis module, is configured to overlay the generated multi-channel pathological probability map on the original CT or MRI images in pseudo-color, providing an intuitive three-dimensional visualization interface, and automatically integrating all quantitative indicators, trend charts and key image snapshots into a structured electronic evaluation report.

[0016] The present invention will now be described in detail with reference to specific embodiments.

[0017] In a preferred embodiment of the present invention, the DLPE-based postoperative hepatobiliary lesion quantitative assessment system is deployed on a high-performance computing server. The server's hardware configuration is as follows: the central processing unit (CPU) is an Intel Xeon Platinum 8368 processor with 32 physical cores and a clock speed of at least 2.8 GHz; the system memory (RAM) is configured as 512 GB of DDR4 ECREG memory with a working frequency of 3200 MHz; the graphics processing unit (GPU) is configured as two parallel computing cards, each with at least 10,752 CUDA cores, 336 third-generation Tensor cores, and equipped with 80 gigabytes of high-bandwidth video memory (HBM2e), with a video memory bandwidth of at least 1,935 GB / s; data storage uses a 4 TB NVMe solid-state drive array (RAID 0) to ensure high-speed read and write of image data; the network interface is a dual-port 10 Gigabit Ethernet card. The entire system runs on a Linux kernel-based operating system (e.g., Ubuntu Server 22.04 LTS) and is equipped with CUDA Toolkit 11.8, cuDNN 8.6, and a deep learning environment that integrates PyTorch 1.13 and TensorFlow 2.10.

[0018] Specifically, in the data acquisition and time-series alignment process of step A, the data interface and preprocessing module first access the hospital's Picture Archiving and Communication System (PACS) via Network File System (NFS) or direct mounting. This module has a built-in DICOM parser that can recursively traverse the image folder of a specified patient. Based on fields such as "StudyDate" and "SeriesDescription" in the DICOM header file, it automatically identifies and retrieves abdominal CT plain and enhanced scan sequences, as well as MRI T1WI, T2WI fat-suppressed sequences, and DWI sequences (b-values ​​set to 0, 400, and 800 s / mm²) at different postoperative time points (e.g., 1 month, 3 months, 6 months, and 12 months postoperatively). After acquiring the data, a preprocessing procedure is performed: First, a three-dimensional median filter (kernel size of 3x3x3 voxels) is applied to all image sequences to suppress random noise; second, intensity normalization is performed. For CT images, their intensity values ​​are linearly mapped from Hounsfield Units (HU) to the [0,1] interval, and the window width and window level are adaptively set according to liver tissue (e.g., window level 40HU, window width 400HU); for MRI images, the Z-score normalization method is used, that is, for each three-dimensional image volume, the mean and standard deviation of all voxels within it are calculated, and then the intensity value of each voxel is subtracted from the mean and then divided by the standard deviation.

[0019] Furthermore, the anatomically aware deformable registration process executed by the temporal image registration module is based on an unsupervised deep learning network using the VoxelMorph architecture. This network takes a pair of 3D images (a floating image to be registered and a fixed image as the target) as input and outputs a dense 3D deformation field. The network's loss function is designed to include two parts: one is an image similarity loss, using a normalized cross-correlation (NCC) coefficient to measure the similarity between the registered floating and fixed images; the other is a deformation field smoothness regularization term, which penalizes the L2 norm of the deformation field gradient to ensure the topological continuity of the deformation and prevent tissue tearing or folding. To achieve "anatomical structure awareness," during network training, in addition to the original input images, a mask of key anatomical structures generated by a pre-trained segmentation network (e.g., nnU-Net) is also input. These structures include the abdominal aorta, inferior vena cava, main portal vein and its left and right branches, and the outline of the remnant liver. A Dice loss term based on these structural masks is added to the loss function, forcing the network to assign higher weights when aligning these key, relatively stable structures. In practical applications, the system selects the CT image from the first follow-up (1 month post-surgery) as the fixed image in the reference coordinate system, and then uses images from all other time points and all other modalities as floating images, registering them with the fixed image to generate a fully aligned four-dimensional image dataset with dimensions (time point, modality, height, width, number of slices).

[0020] In step B, the multimodal spatiotemporal feature deep fusion step, the multimodal spatiotemporal feature fusion module receives the aligned four-dimensional image dataset. Its internal parallel three-dimensional convolutional neural network encoder consists of four structurally identical 3D-ResNet-50 networks, processing CT, T1WI, T2WI, and DWI data respectively. Each 3D-ResNet-50 outputs multi-scale feature maps at different depth levels. To achieve fusion, this invention employs a hierarchical, progressive cross-modal attention fusion mechanism. Specifically, at each feature level (e.g., the output of ResNet's conv2_x, conv3_x, conv4_x, and conv5_x layers), feature maps from the four modalities are input into a multi-head attention module. This module uses a feature map from one modality (e.g., CT) as a "query" and concatenates the feature maps from all four modalities as "keys" and "values." Attention weights are obtained by calculating the dot product similarity between the query and the key and then normalizing it using Softmax. These weights reflect the importance of features from other modalities at the current spatial location for enhancing CT feature representation. Applying these weights to the values ​​and performing a weighted sum yields the fused and enhanced CT features. This process is performed once for each modality, resulting in four fused feature maps at each level. Finally, the four fused feature maps from the highest level are concatenated along the channel dimension to form the final fused feature representation for that time point. This process is repeated for each time point in the four-dimensional image dataset, outputting a fused spatiotemporal feature sequence.

[0021] In step C, the dynamic evolution trajectory modeling step of the lesion, the core of the dynamic evolution modeling module is a GRU network consisting of three stacked layers. The hidden unit dimension of this network is set to 1024. The input fused spatiotemporal feature sequence, where each element is a fused feature representation (a high-dimensional tensor) at the corresponding time node, is compressed into a fixed-length feature vector (e.g., 2048 dimensions) by a global average pooling layer and a fully connected layer before being input into the GRU network. This feature vector sequence is then fed into the GRU network. The mathematical expression of GRU is as follows: First, the reset gate is calculated. and Update Gate : in It is the input feature vector at the current time step. It is the hidden state of the previous time step. and It is a learnable weight matrix. It uses the Sigmoid activation function. Then, the candidate hidden states are calculated. : in This represents the Hadamard product. Finally, the final hidden state at the current time step is calculated. : Through this gating mechanism, the network can adaptively decide how much past information to retain (via...). (control) and how much new information is absorbed (through) (Control). The hidden state of the last time step after the entire time series has been processed. It is considered the final summary of the entire disease evolution process, namely the global evolutionary feature vector, and is passed to the next module.

[0022] In step D, the pathological evolution probability map generation and quantification analysis module receives a global evolution feature vector. This vector is first fed into a fully connected layer and reshaped into a three-dimensional tensor matching the size of the highest-level feature map in 3D-ResNet-50. This tensor serves as the input to the decoder. The decoder employs a symmetric structure similar to standard U-Net, progressively restoring spatial resolution through a series of upsampling (using trilinear interpolation or deconvolution) and convolutional layers. After each upsampling step, its output is skip-connected to the corresponding level of the fused feature map from the multimodal spatiotemporal feature fusion module. This design allows the decoder to utilize rich low-level feature information when reconstructing spatial details. The final layer of the decoder is a 1x1x1 convolutional layer with 3 output channels (corresponding to tumor recurrence, inflammation / edema, and scar tissue), and a Softmax activation function is applied to ensure that the sum of the probabilities of the three channels is 1 at each voxel location. This generates a three-dimensional voxel-level pathological evolution probability map.

[0023] Furthermore, the quantization analysis algorithm operates on this probability map. First, each pathological channel is binarized by applying a probability threshold (e.g., 0.5), resulting in three non-overlapping 3D masks. Then, for each mask, independent lesion regions are identified and labeled using a connected component analysis algorithm. For each labeled region, its volume (number of voxels multiplied by the physical volume of a single voxel), maximum 3D diameter, surface area, and sphericity (defined as...) are calculated. (Where V is volume and A is surface area). Furthermore, these masks are applied back to the original, aligned multimodal images to calculate the mean, standard deviation, median, and percentile signal intensity of each lesion region on CT, T1WI, T2WI, and the Apparent Diffusion Coefficient (ADC) map calculated from DWI. Since the system processes time-series data, all these quantifications are calculated at each time point, generating time-evolution curves. For example, the system might output a "Tumor Recurrence Area Volume Change over Time" or a "Trench of Average ADC Value Change in Inflammatory Area".

[0024] Finally, the results visualization and report generation module integrates the above analysis results. Its 3D visualization engine employs volume rendering technology, enabling the overlay of pathological evolution probability maps onto CT or MRI images of any cross-section with different colors and transparency. For example, tumor recurrence areas are highlighted in red, inflammatory areas in yellow, and scars in blue. Users can interactively rotate, zoom, and section the 3D model with the mouse to intuitively observe the spatial distribution of the lesion and its relationship with adjacent structures. The report generation unit uses a preset template to automatically fill in basic patient information, assessment time points, values ​​and charts of various quantitative indicators, and automatically generates textual descriptions of key findings (e.g., "The volume of the suspected tumor recurrence area increased by 35.7% between 3 and 6 months post-surgery, and the average ADC value decreased by 12.4%, suggesting high proliferative activity of the lesion"). The final generated PDF report can be directly archived into the electronic medical record system or used for clinical consultations.

[0025] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: 1. By constructing a unified analysis framework that integrates multimodal images (CT, multi-sequence MRI), the ability to identify pathological tissues with mixed components in the postoperative region is significantly enhanced, effectively distinguishing between tumor recurrence, inflammatory edema and scar tissue, and overcoming the deficiency of insufficient information from single-modal images. 2. It innovatively introduces a time series modeling mechanism based on GRU network, realizing explicit modeling and quantification of the dynamic evolution process of lesions, and upgrading the assessment from static "snapshot" analysis to dynamic "trajectory" analysis, which can reveal the disease progression pattern that cannot be discovered by a single examination. 3. The deep learning-based anatomical structure-aware deformable registration technology effectively solves the problem of image registration caused by drastic changes in postoperative anatomical structure, providing a reliable spatial benchmark for accurate longitudinal comparative analysis. 4. The pathological evolution probability map and multi-dimensional quantitative indicators output by the system provide clinicians with unprecedented objective, accurate and repeatable decision-making basis, which greatly improves the scientificity and accuracy of the efficacy assessment of hepatobiliary surgery and has significant clinical application value for realizing individualized treatment and prognosis prediction. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the method of the present invention; Figure 2 This is a structural block diagram of the system of the present invention; Figure 3 This is a schematic diagram of the internal structure of the multimodal spatiotemporal feature fusion module in this invention; Figure 4 This is a schematic diagram illustrating the process of generating a pathological evolution probability map from a global evolution feature vector in this invention; Figure 5 This is a schematic diagram illustrating the visualization effect of the pathological evolution probability map generated by this invention superimposed on the original image; Figure 6 This is a schematic diagram of the time evolution curve of the quantitative evaluation index generated by this invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0028] Reference Figures 1 to 6 This invention discloses a method and system for quantitative assessment of postoperative hepatobiliary lesions based on the DLPE algorithm. To facilitate understanding and implementation by those skilled in the art, the following will first provide a detailed description of the typical hardware and software environment supporting this method and system. Then, following the logical steps of the method and in conjunction with the functional modules of the system, an engineering analysis of the implementation details of the entire technical solution will be conducted.

[0029] In a preferred embodiment of the present invention, the DLPE-based postoperative hepatobiliary lesion quantitative assessment system is deployed in a specially optimized high-performance computing server cluster environment dedicated to medical image analysis. The core computing node hardware configuration of this environment is as follows: the central processing unit (CPU) uses two Intel Xeon Platinum 8368 processors, each with 38 physical cores and 76 threads, a base clock speed of 2.4GHz, and a turbo boost up to 3.4GHz, providing powerful general-purpose computing and task scheduling capabilities. The system's main memory (RAM) is configured as 1024 gigabytes (GB) of DDR4 error-correcting code (ECC) registered memory, operating in eight-channel mode at a frequency of 3200MHz, ensuring sufficient cache space and high-bandwidth data paths when processing large-scale three-dimensional image data. The graphics processing unit (GPU) is crucial for the training and inference of the deep learning model in this system; therefore, it is configured with four NVIDIA A100 Tensor Core computing cards, interconnected at high speed via NVLink technology. Each A100 compute card integrates 6912 CUDA cores and 432 third-generation Tensor cores, and is equipped with 80 gigabytes of high-bandwidth memory (HBM2e), with a memory bandwidth of up to 1935 GB / s, providing ultimate computing power support for parallel processing of 3D convolution operations. The data storage subsystem adopts a hybrid architecture. The operating system and applications are deployed on a RAID1 array consisting of two 2TB NVMe solid-state drives to ensure high system availability; while the workspace for storing image data and intermediate model results consists of a RAID0 array of eight 4TB NVMe solid-state drives, providing sustained read and write speeds of over 25 GB / s, greatly reducing data I / O bottlenecks. For network communication, each compute node is equipped with a dual-port 100 Gigabit Ethernet card, connecting to a high-speed switch supporting Remote Direct Memory Access (RDMA), providing low-latency network assurance for distributed training and data access.

[0030] At the software level, the entire system runs on a customized Linux distribution, such as Ubuntu Server 22.04 Long Term Support Edition, whose kernel has been optimized for low latency and high throughput. The basic environment deploys NVIDIA's official CUDA Toolkit 11.8, cuDNN 8.6.0, and NCCL 2.12.12, providing underlying GPU acceleration support for the upper-layer deep learning framework. The core algorithm development and runtime environment is based on Python 3.9, utilizing Conda for environment isolation and package management. The deep learning framework primarily uses PyTorch 1.13.1, combined with PyTorch Lightning 1.9.0 for structured management of the model training process and code decoupling. The processing and manipulation of medical image data relies on a series of specialized libraries, including Pydicom 2.3.1 for DICOM file parsing, SimpleITK 2.2.1 for image reading, writing, transformation, and registration, and the MONAI 1.1.0 framework, specifically designed for deep learning in medical imaging. This framework provides numerous pre-built, validated data augmentations, network structures, and loss functions. The entire system is packaged and deployed using Docker containerization technology, ensuring environmental consistency, portability, and reproducibility.

[0031] Specifically, step A of the method of this invention, namely the data acquisition and temporal alignment processing step, is completed collaboratively by the system's data interface, preprocessing module, and temporal image registration module. The data interface and preprocessing module run continuously as a background service. It passively receives image data pushed from the hospital's Picture Archiving and Communication System (PACS) through a secure DICMC-STORESCP server, or actively queries and retrieves image sequences for a specified patient through a configured DICMC-MOVESCU client. When a new image sequence is received, the DICOM parser within the module reads its metadata in detail. This parser not only extracts basic information such as patient ID, examination date, and sequence description, but also analyzes key physical parameters such as pixel pitch, slice thickness, window width and level, echo time (TE), repetition time (TR), flip angle, and the b-value for diffusion-weighted imaging (DWI). Based on this information, the module automatically organizes image data belonging to the same patient at different time points in a structured manner, creating a hierarchical folder structure on local storage (e.g., / data / patient_ID / YYYY-MM-DD_Study / Series_Description / ). After acquiring the raw data, a standardized preprocessing workflow was initiated. First, to unify the spatial resolution of different images, all 3D images were resampled to an isotropic voxel size of 1.0 mm x 1.0 mm x 1.0 mm using a cubic B-spline interpolation algorithm. Next, a 3D Gaussian smoothing filter (standard deviation σ = 0.5) was applied to all images to mildly suppress high-frequency noise. Intensity normalization was then performed: for CT images, the Henlein unit (HU) values ​​were truncated to the range of [-200, 600] HU to highlight the contrast of abdominal soft tissue, and then linearly mapped to the [0, 1] floating-point interval; for various MRI sequences, after initial mask extraction of the liver region, Z-score normalization based on the voxel intensity within the mask was used. This method can effectively alleviate the signal strength drift problem caused by differences in scanning equipment and parameters.

[0032] Furthermore, the temporal image registration module receives the preprocessed data and executes a deformable registration process based on anatomical structure awareness. The core of this process is an unsupervised registration model based on a convolutional neural network, whose architecture is deeply inspired by VoxelMorph and optimized for the postoperative hepatobiliary region. This network consists of an encoder-decoder structure, taking a pair of 3D images as input: a fixed image serving as the spatial reference and a moving image to be aligned; the output is a 3D dense displacement field. To achieve anatomical structure awareness, additional supervision information is introduced during the model's training process. Specifically, a high-performance nnU-Net segmentation model pre-trained on public datasets (such as LiTS) and internal datasets is used to automatically segment key and relatively stable anatomical structures from all images, including the main portal vein and its first-order branches, hepatic veins, abdominal aorta, and the overall outline of the remaining liver. During the training of the registration network, its composite loss function is designed as follows: .in, It is an image similarity loss, using local normalized cross-correlation (NCC) as the metric, with a window size of 9x9x9 voxels to enhance the robustness of matching local textures. It is a smoothness regularization term for the displacement field, which ensures the physical rationality of the deformation by penalizing the L2 norm of the displacement field gradient. The structural consistency loss is calculated as the Dice loss between the anatomical structure mask of the floating image after deformation and the anatomical structure mask of the fixed image. By adjusting the weighting factors α, β, and γ (e.g., set to 1.0, 0.5, and 2.0 respectively), the network optimizes overall image similarity while focusing on ensuring the accurate alignment of key vascular structures. In the actual evaluation process, the system automatically selects the arterial phase enhanced CT image from the patient's first postoperative follow-up (usually 1 month postoperatively) as the reference coordinate system for the entire time series. Subsequently, all other time points and all modalities of images (including plain CT scans, venous phase, and MRI T1WI, T2WI, DWI, etc.) are used as floating images and registered with the reference CT image. The generated deformation field is saved and applied to the corresponding floating images, ultimately generating a set of spatially fully aligned, voxel-to-voxel-corresponding multimodal four-dimensional image datasets, whose data structure can be represented as [Time, Modality, Depth, Height, Width].

[0033] In step B, the multimodal spatiotemporal feature deep fusion step, the multimodal spatiotemporal feature fusion module receives the aligned four-dimensional image dataset as input. Its core is a parallel multi-encoder and hierarchical attention fusion architecture, as shown in the attached diagram. Figure 3As shown. Specifically, for the four modalities of CT, T1WI, T2WI, and DWI, a separate 3D residual network (3D-ResNet-50) is instantiated as a feature extractor. To adapt to the characteristics of medical images, the standard ResNet-50 structure is modified to a fully 3D convolutional version, that is, all 2D convolutional and pooling layers are replaced with their corresponding 3D versions. The image data input to the network is sampled as 3D patches using an overlapping sliding window method, for example, with a size of 128x128x96 voxels. Feature fusion is not completed all at the end of the network, but adopts a layered and progressive strategy. At the output of each of the four main stages (res2, res3, res4, res5) of 3D-ResNet, a cross-modal attention fusion unit is set. The structure of this unit is based on the multi-head self-attention mechanism in Transformer. Taking the fusion in the res3 stage as an example, the feature maps from the CT modality are... As a "query", the feature map of four modalities in the same stage After concatenation along the channel dimension, "keys" and "values" are generated through different linear projection layers. An attention weight map is obtained by calculating the dot product similarity between the query and the key and applying a softmax function. The value at each spatial location of this weight map reflects the contribution of other modal features to enhancing the CT feature representation at that location. Applying this weight map to the value and performing a weighted sum yields a new CT feature map enhanced with other modal information. This process is repeated for T1WI, T2WI, and DWI, resulting in four deeply interactively fused feature maps output in the res3 stage. This hierarchical fusion mechanism allows the network to capture and utilize complementary information between modalities at different levels of abstraction and spatial scales. For example, at lower levels, CT anatomical information can help correct MRI geometric distortions; at higher levels, DWI functional information can provide crucial clues for determining the nature of high-signal regions on T2WI. Finally, the four fused feature maps output in the res5 stage are concatenated along the channel dimension and dimensionally reduced through a 1x1x1 convolutional layer, forming a comprehensive spatiotemporal feature representation at that time point. This process is performed independently for each time point in the sequence (e.g., 1, 3, 6, and 12 months post-surgery), and finally outputs a fused spatiotemporal feature sequence sorted by time.

[0034] Next, in step C, the dynamic evolution trajectory modeling step, the dynamic evolution modeling module processes the fused spatiotemporal feature sequence generated in the previous stage. The core of this module is a deep recurrent neural network, specifically a three-layer stacked bidirectional gated recurrent unit (BidirectionalGRU) network. The purpose of choosing a bidirectional GRU is to allow the network to utilize not only past historical information (forward GRU) but also future evolutionary trends (backward GRU) when encoding information at the current time point, thus providing a more comprehensive understanding of the entire disease course. Before inputting the feature sequence into the GRU network, the three-dimensional fused feature map at each time point (e.g., dimension [1024, 8, 8, 6]) passes through a three-dimensional adaptive average pooling layer, compressing its spatial dimension to 1x1x1, and then flattening it into a 1024-dimensional feature vector. This vector sequence (the sequence length being the number of follow-up time points) is then fed into the bidirectional GRU network. Within each GRU unit of the network, the input information flow is dynamically controlled through its ingenious reset and update gate structures. The reset gate determines the extent to which the hidden state from the previous time step is forgotten, while the update gate controls the fusion ratio of new input information with the old hidden state. Mathematically, its update rule strictly follows the definition of the standard GRU. After three stacked layers of loop processing, the network can learn higher-level, more abstract temporal dependencies. For example, the first layer may capture short-term fluctuations in lesion volume, while the third layer may learn the complex pattern of persistence, exponential growth, and continuous decline in ADC value unique to tumor recurrence. Finally, for each time point, the hidden states of the forward GRU and backward GRU are concatenated to form a comprehensive state vector containing the contextual evolution information of that time point. After the entire sequence is processed, the comprehensive state vector of the last time point is selected as the global evolutionary feature vector, which encapsulates all the dynamic information of lesion evolution throughout the entire observation period.

[0035] Proceed to step D, namely the generation and quantitative analysis of the pathological evolution probability map, as shown in the attached document. Figure 4As shown, the pathological evolution probability map generation and quantification analysis module receives the global evolution feature vector from the GRU network. This vector first passes through a fully connected layer, whose output is reshaped into a three-dimensional tensor with the same spatial size as the feature map atlas of the res5 stage of the multimodal fusion module, but with more channels. This tensor serves as the starting point for the three-dimensional decoder network. The decoder adopts the classic U-Net architecture, whose structure is symmetrical to the encoder (3D-ResNet) part. The decoder path consists of a series of upsampling modules, each containing a 3D transposed convolutional layer (used to double the spatial resolution), a skip connection that fuses the feature map at the corresponding level of the encoder, and two cascaded 3D convolutional layers (with batch normalization and ReLU activation functions). The design of the skip connection is crucial, as it directly passes the high-resolution local detail information preserved in the encoder to the decoder, greatly helping the latter to reconstruct the accuracy of lesion boundaries. After four consecutive upsampling and feature fusion operations, the network finally recovers to the same original resolution as the input image patch. The final layer of the decoder is a 1x1x1 convolutional layer with 3 output channels, corresponding to three preset pathological states: tumor recurrence, active inflammation or edema, and stable scar tissue. Finally, a voxel-level softmax activation function is applied to ensure that at each voxel location, the output values ​​of the three channels form a probability distribution whose sum is 1. The resulting final output is a three-dimensional pathological evolution probability map, where each voxel stores the probability that the spatial location belongs to one of the three aforementioned pathological states.

[0036] Next, the integrated quantitative analysis algorithm automates the processing of the probability map. First, by setting a probability threshold (e.g., 0.7 for tumor recurrence pathways and 0.6 for inflammation), the probability map is converted into three independent binary masks. Then, a three-dimensional connected component analysis algorithm is applied to each mask to identify and separate spatially disconnected lesions. For each identified lesion, the system automatically calculates a series of detailed quantitative indicators. Morphological indicators include volume (in cubic centimeters), surface area, maximum three-dimensional diameter, sphericity, and fractal dimension to describe morphological irregularities. Signal intensity indicators are calculated by applying the lesion mask back to the original aligned multimodal images, including the mean, standard deviation, median, skewness, and kurtosis on CT (HU value), T1WI, T2WI, and apparent diffusion coefficient (ADC) maps calculated from DWI sequences. Since the entire method is based on time-series data, this entire set of quantitative calculations is performed at each time point. The system automatically processes this data, generating the results shown in the attached figure. Figure 6The time evolution curves shown, such as the "recurrence lesion volume-time curve" and the "mean ADC value of the inflamed area-time curve", intuitively reveal the dynamic changes of the pathological process.

[0037] Finally, the results visualization and report generation module is responsible for presenting all analysis results to users in a clinically friendly and information-rich manner. Its 3D visualization interface is shown in the attached image. Figure 5 As shown, GPU-based volume rendering technology can fuse multi-channel pathological evolution probability maps with any modality of original image (such as enhanced CT) in pseudo-color. Users can specify the color and transparency of different pathological states; for example, rendering tumor recurrence areas as opaque red, inflammatory edema as translucent yellow, and scar tissue as blue. Through interactive translation, rotation, zooming, and sectioning plane settings, clinicians can examine the three-dimensional morphology, spatial adjacency, and internal composition of lesions from any angle. The report generation unit automatically integrates all information based on a structured template. The report content includes basic patient information, examination time series, screenshots of key images (automatically located at the lesion center level), a table of quantitative indicators for each pathological component, and a time evolution trend graph of all indicators. In addition, this module integrates a rule-based natural language generation algorithm, which can generate preliminary descriptive conclusions based on the changing trends of quantitative results. For example: "Compared with 3 months post-surgery, a new lesion was observed in liver segment V at 6 months post-surgery. The lesion volume increased by 120%, and the average ADC value decreased by 25%, which strongly suggests tumor recurrence on the probability plot." Finally, the system outputs a standardized PDF evaluation report with illustrations, which can be directly used for clinical decision-making, efficacy evaluation, and multidisciplinary discussions.

[0038] The training process of this invention was completed on a large-scale postoperative follow-up image dataset collected from multiple centers and jointly annotated by senior radiologists and hepatobiliary surgeons. The annotation information included precise 3D contour delineations of different pathological regions at different time points. The loss function of the entire end-to-end model is a composite loss, consisting of two weighted parts: one part is the voxel-level weighted cross-entropy loss between the pathological evolution probability map output by the decoder and the ground truth annotations, the formula of which is... ,in It is the probability predicted by the model. It's a real label. One part is the weighting factor used to balance positive and negative samples; the other part is the segmentation loss based on the Dice coefficient, used to improve the sharpness of the segmentation boundaries. Through thousands of iterations of training using the Adam optimizer, all neural network weights in the system described in this invention are optimized, enabling them to accurately perform the aforementioned evaluation task.

[0039] It should be noted that the modules described in this invention can be independent software program units or different functional code segments integrated into a single application in actual implementation. Their "electrical connections" are manifested at the software level as data exchange methods such as inter-process communication, function calls, API interfaces, or shared memory. This invention is not limited to the specific hardware configurations and software environments listed; any platform that can provide equivalent or higher computing power and support for corresponding algorithm libraries falls within the protection scope of this invention.

Claims

1. A method for quantitative assessment of postoperative hepatobiliary lesions based on the DLPE algorithm, wherein the method is executed on a computing device, characterized in that, Includes the following steps: Step A: Data acquisition and temporal alignment processing steps, acquiring a patient's postoperative medical image sequence at multiple discrete time points, containing at least two modalities, wherein the modalities include at least computed tomography (CT) images and magnetic resonance imaging (MRI) images; and executing a deep learning-based anatomical structure-aware deformable registration process to uniformly register all modal images at the multiple time points to a patient-specific reference anatomical coordinate system, generating a spatially aligned multimodal four-dimensional image dataset; Step B: Multimodal spatiotemporal feature deep fusion step. Using a parallel 3D convolutional neural network (3D-CNN) encoder group, independent deep feature extraction is performed on the image data of each modality at each time point in the spatially aligned multimodal four-dimensional image dataset to generate feature maps corresponding to each modality. Subsequently, the feature maps are weighted and fused through a cross-modal attention fusion module to generate a fused spatiotemporal feature sequence that can represent comprehensive pathological information at each time point. Step C: Modeling the dynamic evolution trajectory of the lesion. The fused spatiotemporal feature sequence is input into a gated recurrent unit (GRU) network in chronological order. The GRU network captures and encodes the dynamic change pattern of the lesion region in the time dimension, and outputs a global evolution feature vector containing complete lesion evolution history information at the last time node. Step D: Pathological Evolution Probability Map Generation and Quantitative Analysis. The global evolution feature vector is input into a three-dimensional convolutional decoder network. The global evolution feature vector is upsampled and decoded to generate a three-dimensional voxel-level pathological evolution probability map. Each voxel in the pathological evolution probability map contains probability values ​​for multiple channels. Each channel corresponds to a preset postoperative pathological state, which includes at least tumor recurrence, active inflammation or edema, and stable scar tissue. Based on the pathological evolution probability map, quantitative evaluation indicators are calculated and output.

2. The method for quantitative assessment of postoperative hepatobiliary lesions based on the DLPE algorithm according to claim 1, characterized in that, The anatomical structure-aware deformable registration process executed in step A specifically includes: An unsupervised deep learning network based on the VoxelMorph architecture is used as the registration network. The registration network takes a pair of 3D images, namely a floating image to be registered and a fixed image as the target, as input and outputs a dense 3D deformation field. The training process of the registration network is optimized using a composite loss function, which includes: an image similarity loss term, used to measure the similarity between the floating image and the fixed image after transformation by the 3D deformation field; a deformation field smoothness regularization term, used to penalize the gradient of the 3D deformation field to ensure the topological continuity of the deformation; and an anatomical structure consistency loss term, which calculates Dice loss on one or more preset key anatomical structure masks to force the registration network to assign higher weights when aligning the key anatomical structures, including the abdominal aorta, inferior vena cava, portal vein trunk and branches, and the outline of the remnant liver.

3. The method for quantitative assessment of postoperative hepatobiliary lesions based on the DLPE algorithm according to claim 1, characterized in that, The multimodal spatiotemporal feature deep fusion step in step B specifically employs a hierarchical, progressive cross-modal attention fusion mechanism, including: The parallel 3D convolutional neural network encoder group consists of multiple 3D residual networks (3D-ResNet) with identical structures. The 3D-ResNet outputs multi-scale feature maps at multiple feature levels at different depths. At least one of the multiple feature levels, a cross-modal attention fusion unit is set. The fusion unit takes the feature map from a certain modality as a query and the combination of feature maps from all modalities at the same level as a key and a value. Attention weights are generated by calculating the dot product similarity between the query and the key. The attention weights are applied to the values ​​and weighted summation is performed to generate a new feature map corresponding to the certain modality, which is enhanced by information from other modalities. The weighted summation process is performed once for each modality to obtain a set of feature maps that have undergone deep interactive fusion at the current level, and these are used as the input to the next network layer.

4. The method for quantitative assessment of postoperative hepatobiliary lesions based on the DLPE algorithm according to claim 1, characterized in that, The lesion dynamic evolution trajectory modeling step in step C specifically includes: Before inputting the fused spatiotemporal feature sequence into the GRU network, a global average pooling layer and a fully connected layer are first applied to the fused feature map of each time point in the sequence to compress it into a fixed-length feature vector, thus forming a feature vector sequence. The feature vector sequence is processed by a multi-layer stacked bidirectional gated recurrent unit (BidirectionalGRU) network. The forward GRU in the Bidirectional GRU network is used to capture historical evolution information from the past to the present, and the backward GRU is used to capture contextual information based on future trends. The forward hidden state and the backward hidden state of the Bidirectional GRU network at the last time point are concatenated to form the global evolution feature vector.

5. The method for quantitative assessment of postoperative hepatobiliary lesions based on the DLPE algorithm according to claim 1, characterized in that, In step D, the three-dimensional convolutional decoder network adopts a three-dimensional U-Net structure decoder. Furthermore, during the progressive restoration of spatial resolution, the decoder receives and fuses the fused feature map output by the encoder level corresponding to the current decoding level in the multimodal spatiotemporal feature deep fusion step of step B via a skip connection. The quantitative evaluation index calculated and output based on the pathological evolution probability map further includes the time evolution curve of at least one of the following indices calculated at each time point: morphological indices, including the volume, volume change rate, maximum three-dimensional diameter, surface area, and sphericity of each pathological state region; and signal intensity indices, including the mean, standard deviation, median, and percentile of the signal intensity calculated on CT images, T1-weighted imaging, T2-weighted imaging, and apparent diffusion coefficient (ADC) maps after mapping the masks of each pathological state region back to the original multimodal images.

6. A quantitative assessment system for postoperative hepatobiliary lesions based on the DLPE algorithm, characterized in that, The system includes: A data interface and temporal image registration unit are configured to acquire a multimodal medical image sequence of a patient at multiple discrete time points after surgery, including at least computed tomography (CT) images and magnetic resonance imaging (MRI) images, and to perform a deep learning-based anatomical structure-aware deformable registration process to generate a set of spatially aligned multimodal four-dimensional image datasets. A multimodal spatiotemporal feature fusion module, electrically connected to the data interface and the temporal image registration unit, is configured to extract deep features from the four-dimensional image dataset using a parallel three-dimensional convolutional neural network (3D-CNN) encoder group, and perform feature weighted fusion through a cross-modal attention fusion mechanism to generate a fused spatiotemporal feature sequence. A lesion dynamic evolution modeling module is electrically connected to the multimodal spatiotemporal feature fusion module. It integrates a gated recurrent unit (GRU) network, which is configured to receive and process the fused spatiotemporal feature sequence to output a global evolution feature vector containing complete lesion evolution history information. A pathological evolution probability map generation and quantitative analysis module is electrically connected to the lesion dynamic evolution modeling module. It includes a three-dimensional convolutional decoder network, which is configured to decode the global evolution feature vector into a three-dimensional voxel-level pathological evolution probability map containing multiple preset pathological state channels, and calculate quantitative evaluation indicators based on the probability map.

7. The quantitative assessment system for postoperative hepatobiliary lesions based on the DLPE algorithm according to claim 6, characterized in that, The data interface and the temporal image registration unit internally contain an unsupervised deep learning registration network model based on the VoxelMorph architecture. The model is trained to output a three-dimensional deformation field that drives the floating image to align with the fixed image. The optimization objective of the model is defined by a composite loss function, which includes an image similarity loss term based on normalized cross-correlation (NCC) coefficients, a smoothness regularization term to ensure the physical rationality of the deformation, and a Dice loss term based on key anatomical structure masks to prioritize the accurate alignment of the key anatomical structures during the registration process.

8. The quantitative assessment system for postoperative hepatobiliary lesions based on the DLPE algorithm according to claim 6, characterized in that, The core of the multimodal spatiotemporal feature fusion module is a neural network structure containing multiple parallel feature extraction branches and a hierarchical fusion unit. Each feature extraction branch is a three-dimensional residual network (3D-ResNet), which is responsible for extracting deep spatial feature maps of different levels from the aligned images of different modalities. The hierarchical fusion unit consists of multiple cross-modal attention modules set at different feature levels of the 3D-ResNet. Each attention module is configured to: take the feature map of a certain modality as a query, combine the feature maps of all modalities as keys and values, calculate and apply cross-modal attention weights to achieve information enhancement and effective aggregation between feature maps.

9. The quantitative assessment system for postoperative hepatobiliary lesions based on the DLPE algorithm according to claim 6, characterized in that, The core of the lesion dynamic evolution modeling module is a multi-layer stacked bidirectional gated recurrent unit (BidirectionalGRU) network, which is used to learn the evolution pattern of the lesion from both forward and reverse time series. Furthermore, the three-dimensional convolutional decoder network in the pathological evolution probability map generation and quantification analysis module has a three-dimensional U-Net decoder structure, which contains multiple skip connection paths. These paths directly transmit the fused feature maps of each encoder level in the multimodal spatiotemporal feature fusion module to the corresponding upsampling level in the decoder network to assist in the accurate reconstruction of high-resolution spatial details.

10. The quantitative assessment system for postoperative hepatobiliary lesions based on the DLPE algorithm according to claim 6, characterized in that, The system also includes: A results visualization and report generation module, electrically connected to the pathological evolution probability map generation and quantitative analysis module, is configured to perform at least one of the following functions: superimposing the pathological evolution probability map in pseudo-color onto the original CT or MRI images, and providing an interactive three-dimensional visualization interface through volume rendering technology, allowing users to observe the three-dimensional spatial distribution and internal composition of the lesion from any angle; automatically integrating the quantitative evaluation indicators, the time evolution trend map generated based on the indicators, and key image snapshots into a structured electronic evaluation report, and generating a text description of the lesion evolution trend according to preset rules.

Images