A Multimodal Artificial Intelligence-Based Evaluation System for the Transformation Process of Hepatocellular Carcinoma

Abstract

This application relates to a multimodal artificial intelligence-based system for assessing the transformation process of hepatocellular carcinoma. The system includes: a data acquisition module for acquiring multi-phase image sequences, clinical indicator sequences, and treatment record sequences; an image registration module for performing non-rigid registration of multi-phase image data to obtain spatially registered image sequences; a lesion feature extraction module for extracting a dynamic lesion feature set; a multimodal feature fusion module for fusing the dynamic lesion feature set with clinical indicators to obtain an observation variable vector; an encoding module for encoding the treatment record sequence into a treatment mode vector; a trajectory prediction module for obtaining a continuous transformation state trajectory based on a neuronormal differential equation model; and an evaluation decoding module for obtaining the transformation process assessment results. This system can handle irregular follow-up intervals and dynamic changes in lesions, accurately characterizing the continuous evolution of tumor biological behavior.

Description

Technical Field

[0001] This invention belongs to the field of medical artificial intelligence, and in particular relates to an assessment system for the transformation process of hepatocellular carcinoma based on multimodal artificial intelligence. Background Technology

[0002] With the rapid development of medical image analysis technology and artificial intelligence algorithms, hepatocellular carcinoma auxiliary diagnosis and treatment technology based on multimodal data has become a research hotspot. This technology can use computers to automatically analyze massive amounts of medical data, leading to current methods for tumor status monitoring and treatment response evaluation based on deep learning models.

[0003] Traditional techniques typically involve extracting spatial features from independent image data acquired at each follow-up visit, or analyzing time-series data using sequence models with fixed time steps (such as recurrent neural networks). When processing multi-source information, image features are often simply linearly concatenated with clinical indicators to predict or stratify clinical indicators at specific time points in hepatocellular carcinoma patients.

[0004] However, current assessment methods or traditional approaches have significant limitations in practical clinical applications, primarily in their inability to handle irregular follow-up intervals and the complex dynamic changes in the number of lesions during treatment. Existing discretization models lack the ability to capture continuous dynamic changes and struggle to process variable lesion sets and multimodal clinical data within a unified framework. This results in the inability to construct an accurate representation reflecting the continuous evolution of tumor biological behavior over time, thus affecting the accuracy of translational assessment results. Summary of the Invention

[0005] Therefore, it is necessary to provide a multimodal artificial intelligence-based assessment system for hepatocellular carcinoma transformation process that can cope with irregular follow-up time intervals and dynamic changes in the number of lesions, and accurately reflect the continuous evolution of tumor biological behavior, in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a multimodal artificial intelligence-based system for evaluating the transformation process of hepatocellular carcinoma, comprising: The data acquisition module is used to acquire multi-phase imaging sequences, clinical indicator sequences, and treatment record sequences; The image registration module is used to perform non-rigid registration of multi-period image data in a multi-period image sequence to obtain a spatially registered image sequence. The lesion feature extraction module is used to extract lesion features from spatially registered image sequences to obtain a dynamic lesion feature set. The multimodal feature fusion module is used to perform multimodal feature fusion on the dynamic lesion feature set and the clinical indicators in the clinical indicator sequence to obtain the observation variable vector; the observation variable vector is used to characterize the current hepatocellular carcinoma transformation status of liver cancer patients; The encoding module is used to encode the treatment record sequence into a treatment vector to obtain a treatment method vector. The trajectory prediction module is used to input the observed variable vector and treatment method vector into the neuronormal differential equation model to predict the continuous transformation state trajectory and obtain the continuous transformation state trajectory; the continuous transformation state trajectory is used to characterize the expected transformation state of hepatocellular carcinoma in liver cancer patients; The evaluation decoding module is used to evaluate and decode the continuous transformation state trajectory to obtain the transformation process evaluation results; the transformation process evaluation results include the probability of liver cancer resectableness and the predicted clinical occurrence probability.

[0007] In one embodiment, the multimodal feature fusion module includes: The ensemble attention encoding unit is used to perform ensemble attention encoding on the dynamic lesion feature set to obtain the global tumor representation vector. The clinical indicator temporal coding unit is used to extract features from the clinical indicator sequence to obtain a multimodal clinical representation vector; The vector splicing unit is used to splice the global tumor representation vector and the multimodal clinical representation vector to obtain the observation variable vector.

[0008] In one embodiment, the ensemble attention encoding unit includes: The feature compression subunit is used to compress the dynamic lesion feature set based on the preset induced point feature set to obtain the compressed lesion feature set. The multi-head self-attention computation subunit is used to perform multi-head self-attention computation on the compressed lesion feature set to obtain the lesion context feature matrix. The pooling multi-head attention aggregation subunit is used to perform pooling multi-head attention aggregation on the lesion context feature matrix to obtain the lesion aggregation features. The feedforward neural network mapping subunit is used to input the lesion aggregation features into the global tumor representation feedforward neural network to generate a global tumor representation vector.

[0009] In one embodiment, the trajectory prediction module includes: The neural ordinary differential equation solving unit is used to perform continuous state evolution calculations based on the observation variable vector and the treatment method vector through the neural ordinary differential equation model to obtain the predicted hidden state at the future target time. The continuous trajectory generation unit is used to connect the predicted latent states of the future target time in a time sequence to generate a continuous transition state trajectory.

[0010] In one embodiment, the solution unit for the ordinary differential equation includes: The hidden state derivative calculation subunit is used to calculate the derivative of the hidden state with respect to time based on the current hidden state, the observed variable vector, and the treatment vector, using the hidden state derivative function model. The formula for calculating the derivative of the hidden state with respect to time is: ; in, Let be the time derivative of the hidden state. Let be the hidden state at time t. Let be the observed variable at time t. Let be the treatment vector at time t. The parameters of the derivative function are denoted as . The numerical integration sub-unit is used to perform numerical integration based on the derivative of the hidden state with respect to time, so as to obtain the predicted hidden state at the future target time.

[0011] In one embodiment, the lesion feature extraction module includes: The three-dimensional convolutional feature extraction unit is used to input the spatially registered image sequence into the three-dimensional convolutional feature extraction model, and to perform three-dimensional convolutional feature extraction on the spatially registered image sequence to obtain multi-phase lesion features; The lesion region segmentation unit is used to segment lesion regions based on the lesion features of multiple phases and generate lesion masks for each phase of the image. The mask information extraction unit is used to extract the lesion space and feature information corresponding to each mask to obtain the initial lesion set; The lesion topology tracking unit is used to perform lesion topology tracking on the initial lesion set to obtain a dynamic lesion feature set.

[0012] In one embodiment, the lesion topology tracking unit includes: The spatial similarity calculation subunit is used to calculate the spatial overlap and centroid distance of lesion region information at adjacent time points in the initial lesion set, and generate a lesion spatial metric dataset. The lesion pairing subunit is used to compare the spatial overlap and centroid distance in the lesion spatial measurement dataset with the preset overlap threshold and preset distance threshold, respectively, to obtain the comparison results, and to determine the correspondence between lesion regions at adjacent time points based on the comparison results, thereby generating lesion pairing information. The lesion region marking subunit is used to merge lesion region information with corresponding relationships based on lesion pairing information, and mark newly added or disappeared lesion regions to construct a dynamic lesion set. The temporal feature fusion subunit is used to fuse the temporal feature sequences of each lesion in the dynamic lesion set to generate a dynamic lesion feature set.

[0013] Secondly, this application also provides a method for evaluating the transformation process of hepatocellular carcinoma based on multimodal artificial intelligence, including: Acquire multi-phase imaging sequences, clinical indicator sequences, and treatment record sequences; Non-rigid registration is performed on multi-phase image data in a multi-phase image sequence to obtain a spatially registered image sequence. Lesion features are extracted from spatially registered image sequences to obtain a dynamic lesion feature set; Multimodal feature fusion was performed on the dynamic lesion feature set and the clinical indicators in the clinical indicator sequence to obtain the observation variable vector; the observation variable vector was used to characterize the current hepatocellular carcinoma transformation status of liver cancer patients; The treatment record sequence is encoded into a treatment vector to obtain a treatment method vector; The observed variable vector and treatment method vector are input into the neuronormal differential equation model to predict the continuous transformation state trajectory, thus obtaining the continuous transformation state trajectory; the continuous transformation state trajectory is used to characterize the expected transformation state of hepatocellular carcinoma in liver cancer patients; The continuous transformation trajectory is evaluated and decoded to obtain the transformation process evaluation results; the transformation process evaluation results include the probability of liver cancer resectableness and the predicted clinical occurrence probability.

[0014] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a multimodal artificial intelligence-based hepatocellular carcinoma transformation process assessment system as described in the first aspect.

[0015] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a multimodal artificial intelligence-based hepatocellular carcinoma transformation process assessment system as described in the first aspect.

[0016] The aforementioned multimodal artificial intelligence-based hepatocellular carcinoma (HCC) transformation process assessment system acquires multi-phase image sequences, clinical indicator sequences, and treatment record sequences. It eliminates the influence of liver deformation on image analysis through non-rigid registration, extracting a feature set reflecting dynamic changes in lesions. This feature set is then fused with clinical indicators to generate an observation variable vector representing the patient's current HCC transformation state. Simultaneously, treatment records are encoded as treatment mode vectors. By inputting the observation variable vector and treatment mode vector into a neural network constant differential equation model, the system utilizes its continuous time-dimensional modeling capabilities to predict continuous transformation trajectory, effectively capturing the continuous evolution of tumor biological behavior over time and the dynamic changes under treatment intervention. Finally, the generated trajectory is evaluated and decoded, outputting the probability of HCC resectability and the predicted clinical occurrence probability, achieving accurate and dynamic assessment of the HCC transformation process. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic diagram of the structure of a hepatocellular carcinoma transformation process assessment system based on multimodal artificial intelligence provided by the present invention; Figure 2 This is a schematic diagram of the structure of the lesion feature extraction module in an optional embodiment of the present invention; Figure 3 This is a flowchart illustrating a method for evaluating the transformation process of hepatocellular carcinoma based on multimodal artificial intelligence, provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] In one embodiment, such as Figure 1 As shown, a multimodal artificial intelligence-based system for evaluating the transformation process of hepatocellular carcinoma is provided. This embodiment illustrates the application of the system to a terminal. It is understood that the system can also be applied to a server, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the system includes the following modules: The data acquisition module is used to acquire multi-phase imaging sequences, clinical indicator sequences, and treatment record sequences.

[0021] Optionally, the multi-phase imaging sequence, clinical indicator sequence, and treatment record sequence are all time-series medical data for the same liver cancer patient retrieved from a medical data storage terminal. The multi-phase imaging sequence includes liver computed tomography (CT) and magnetic resonance imaging (MRI) images acquired at different time points. The clinical indicator sequence covers time-series records of clinical test data such as liver function indicators, alpha-fetoprotein (AFP) concentration, and coagulation function indicators. The treatment record sequence includes records of the patient's surgical, interventional, and targeted therapies, along with their corresponding implementation times and durations of action. Specifically, the retrieved raw medical data is first parsed to support mainstream medical imaging formats and clinical data table formats. Then, the parsed data undergoes validity screening, removing invalid data that is missing, corrupted, or irrelevant to the diagnosis and treatment of hepatocellular carcinoma. Each valid data point is then labeled with its corresponding acquisition or implementation timestamp. Finally, the three types of data are sorted and organized according to their timestamps from earliest to latest, while simultaneously verifying the consistency of the time dimension among the three types of data. The data that meet the requirements are then concatenated sequentially to obtain the structured, time-series multi-phase imaging sequence, clinical indicator sequence, and treatment record sequence.

[0022] The image registration module is used to perform non-rigid registration of image data from multiple periods in a multi-period image sequence to obtain a spatially registered image sequence.

[0023] Optionally, the image data from each phase of the multi-phase image sequence is preprocessed, performing grayscale normalization and Gaussian smoothing denoising operations sequentially. Grayscale normalization maps the pixel values ​​of each phase image to the same numerical range, eliminating grayscale differences caused by different acquisition devices and parameters, while Gaussian smoothing denoising reduces the impact of random noise in the images on registration accuracy. Further, a non-rigid registration algorithm based on B-splines is used to register the preprocessed multi-phase image data. The image from the phase with the middle acquisition time in the multi-phase image sequence is selected as the fixed image, and the remaining phase images are used as moving images. A nonlinear spatial transformation function between the moving and fixed images is constructed. By iteratively optimizing the parameters of this transformation function, the liver anatomical structure in the moving image and the corresponding structure in the fixed image achieve accurate spatial matching. The iteration terminates when the registration error between the moving and fixed images tends to stabilize. During the registration process, liver deformation caused by respiratory movements, changes in body position, and changes in liver morphology after treatment will be specifically corrected. Finally, all registered moving images and fixed images will be integrated to obtain a spatially registered image sequence. All phase images in this sequence are based on a unified spatial coordinate system.

[0024] The lesion feature extraction module is used to extract lesion features from spatially registered image sequences to obtain a dynamic lesion feature set.

[0025] Optionally, the spatially registered image sequences are first stacked into a three-dimensional tensor in chronological order and input into a three-dimensional convolutional neural network for feature extraction. This network, through multiple three-dimensional convolutional and pooling layers, simultaneously captures the morphological and textural features of lesions in the spatial dimension and the dynamic changes in the temporal dimension, outputting multi-phase lesion features that integrate multi-phase information. Subsequently, a lesion mask is generated based on the multi-phase lesion features. First, an adaptive threshold segmentation algorithm is used to distinguish lesion areas from normal liver tissue areas. Then, the preliminary binary mask obtained from segmentation undergoes dilation and erosion morphological processing to fill the voids inside the lesion area and remove edge noise. After eliminating false lesion areas, an accurate lesion mask is obtained. Based on this mask, lesion features and spatial location information at each time point are extracted and integrated to form an initial lesion set. Preferably, the initial lesion set is subjected to lesion topology tracing, the three-dimensional intersection-union ratio and centroid distance between lesions at adjacent time points are calculated, the one-to-one correspondence of lesions is determined by a preset threshold, new, disappearing and continuing lesions are distinguished, and the temporal features of the same lesion are fused to finally obtain a dynamic lesion feature set containing the morphological, texture and positional features of all lesions that change over time, which fully reflects the dynamic evolution law of lesions.

[0026] The multimodal feature fusion module is used to perform multimodal feature fusion on the dynamic lesion feature set and the clinical indicators in the clinical indicator sequence to obtain the observation variable vector; the observation variable vector is used to characterize the current hepatocellular carcinoma transformation status of liver cancer patients.

[0027] Optionally, ensemble attention encoding is first performed on the dynamic lesion feature set, introducing a preset set of induction points to help capture the global and temporal correlations of lesion features. A lesion context feature matrix is ​​obtained through multi-head self-attention computation, and then generated via pooling multi-head attention aggregation and feedforward neural network mapping to produce a global tumor representation vector that condenses the overall dynamic features of the tumor. Simultaneously, temporal encoding is performed on the clinical indicator sequence. First, categorical indicators in the clinical indicator sequence are one-hot encoded, and continuous indicators are Z-score standardized. Then, the preprocessed clinical indicator sequence is input into a gated recurrent unit. This unit captures the temporal dependencies of clinical indicators through update and reset gates, outputting a multimodal clinical representation vector that integrates the temporal changes of clinical indicators. Further, the global tumor representation vector and the multimodal clinical representation vector are dimensionally validated. If dimensional differences exist, zero-padding is used to achieve dimension matching. Then, the vectors are concatenated using feature dimension concatenation, preserving the feature information of both types of vectors during the concatenation process. Finally, an observation variable vector that comprehensively represents the current hepatocellular carcinoma transformation state of liver cancer patients is obtained. This vector integrates the dynamic features of tumor lesions and the temporal features of clinical indicators.

[0028] The encoding module is used to encode the treatment record sequence into a treatment vector to obtain the treatment method vector.

[0029] Optionally, the treatment record sequence is preprocessed, and the implementation time node and effective period of each treatment method in the treatment record sequence are marked one by one. Treatment methods within the effective period at each time node are selected to form a set of effective treatment methods corresponding to each time node. The treatment method embedding function is a linear embedding function. Specifically, each individual treatment method is first one-hot encoded to convert the discrete treatment method into a binary feature vector. Then, a lightweight fully connected layer is used to linearly map the binary feature vector to obtain a fixed-dimensional treatment method embedding vector. This embedding vector can accurately represent the core features of the corresponding individual treatment method. Specifically, according to the preset treatment method embedding function, the embedding calculation is performed on each individual treatment method in the set of effective treatment methods at each time node to obtain the corresponding treatment method embedding vector. Then, all treatment method embedding vectors at the same time node are summed according to the feature dimension to obtain the treatment vector for that time node. Finally, the treatment vectors of all time nodes are concatenated and integrated in chronological order to obtain a treatment method vector that can represent the treatment status at each time node. This vector fully reflects the treatment intervention characteristics received by the patient during the transformation of hepatocellular carcinoma. The trajectory prediction module is used to input the observed variable vector and treatment method vector into the neural network differential equation model to predict the continuous transformation state trajectory and obtain the continuous transformation state trajectory. The continuous transformation state trajectory is used to characterize the expected transformation state of hepatocellular carcinoma in liver cancer patients.

[0030] Optionally, the neural network constant differential equation model is a pre-constructed nonlinear differential equation model used to characterize the continuous evolution of the state over time. It uses the observed variable vector as the hidden state input and the treatment method vector as the control variable. The core of the model is the hidden state derivative function, which describes the rate of change of the hidden state over time, specifically implemented by a fully connected neural network. First, the observed variable vector and the treatment method vector are aligned in the time dimension to ensure a one-to-one correspondence between the observed variable vector and the treatment method vector at each time point. Then, the aligned vectors are input into the neural network constant differential equation model. The derivative of the hidden state with respect to time is calculated using the hidden state derivative function in the model. This derivative reflects the rate of change of the hepatocellular carcinoma transformation state. Furthermore, the fourth-order Runge-Kutta method is used to numerically integrate the neural network constant differential equation. The observed variable vector at the initial time node is selected as the initial condition for integration. The solution step size and target time range are set according to the actual diagnosis and treatment needs. The predicted hidden states at all target times are obtained through step-by-step iterative calculation. Finally, the predicted hidden states at all target times are integrated in chronological order to obtain a continuous transformation state trajectory that can characterize the expected state of hepatocellular carcinoma transformation in liver cancer patients. This trajectory realizes continuous time prediction of the hepatocellular carcinoma transformation process.

[0031] The evaluation decoding module is used to evaluate and decode the continuous transformation state trajectory to obtain the transformation process evaluation results; the transformation process evaluation results include the probability of liver cancer resectableness and the predicted clinical occurrence probability.

[0032] Optionally, the continuous transformation trajectory is first subjected to feature extraction and compression. A temporal pooling layer is used to process the continuous transformation trajectory, extracting trend features and key inflection point features. The trend features reflect the overall evolution direction of the hepatocellular carcinoma transformation state, while the key inflection point features correspond to the time points and magnitudes of significant changes in the transformation state. These two types of features are then fused to obtain a trajectory feature vector that condenses the core information of the trajectory. Subsequently, the trajectory feature vector is input into a fully connected neural network for evaluation and decoding. This network employs a dual-branch output structure. One branch's output layer uses a softmax activation function to map the trajectory feature vector to a probability interval, outputting the resectability probability of hepatocellular carcinoma. This probability value characterizes the feasibility of surgical resection treatment at the corresponding time point. The other branch outputs the predicted clinical occurrence probability, which includes the probability of tumor progression, treatment response, and other clinical events related to hepatocellular carcinoma transformation. Finally, the resectability probability and the predicted clinical occurrence probability are integrated to obtain a complete evaluation result of the transformation process, providing quantitative data support for clinical diagnosis and treatment decisions.

[0033] The aforementioned multimodal artificial intelligence-based hepatocellular carcinoma (HCC) transformation assessment system acquires and organizes multi-phase imaging sequences, clinical indicator sequences, and treatment record sequences of HCC patients. It then performs non-rigid registration on the multi-phase imaging data to obtain spatially registered image sequences. Dynamic features of the lesions are extracted and fused with clinical indicators using multimodal features to obtain an observation variable vector. Simultaneously, the treatment record sequences are encoded to obtain a treatment mode vector. These two types of vectors are input into a neural network constant differential equation model to predict the continuous transformation trajectory. Finally, the trajectory is evaluated and decoded to obtain the transformation process assessment result. This system effectively solves the problem that traditional HCC assessment methods struggle to handle irregular follow-up time intervals and dynamic changes in the number of lesions. It accurately captures the continuous temporal evolution of the HCC transformation process, achieving deep integration of tumor lesion characteristics, clinical indicator characteristics, and treatment intervention characteristics. This significantly improves the accuracy and comprehensiveness of the HCC transformation process assessment results, providing precise quantitative evidence for the development of personalized clinical treatment plans.

[0034] In one embodiment, the multimodal feature fusion module includes: The ensemble attention encoding unit is used to perform ensemble attention encoding on the dynamic lesion feature set to obtain the global tumor representation vector.

[0035] Optionally, the dynamic lesion feature set is a temporal feature set containing the morphological, texture, and location features of all lesions changing over time. Each element corresponds to the lesion feature vector at a single time node. The preset induction point set is a set of fixed feature points that helps capture the global and temporal correlations of lesion features. It is generated by using the K-means clustering algorithm on a large amount of historical lesion feature data to ensure that it covers the main change patterns of lesion features and that the dimensions are consistent with the lesion feature vector at a single time node. Specifically, the dynamic lesion feature set and the preset induction point set are first transformed by independent linear mapping layers to obtain query vectors, key vectors, and value vectors. The weights and biases of the linear mapping layers are obtained through iterative optimization using historical lesion time-series data to ensure that the key information of the original features is preserved. Furthermore, the three types of vectors are divided into multiple independent sub-vectors to achieve multi-head self-attention computation. The dot product of query and key is calculated separately for each sub-vector to obtain the attention score. After being normalized by the softmax function, the score is weighted and summed with the corresponding value vector. Then, all sub-vector outputs are concatenated according to the feature dimension to obtain the lesion context feature matrix. Subsequently, the core features are aggregated by global averaging and max pooling, and then input into a lightweight feedforward neural network to complete the nonlinear mapping. Finally, a global tumor representation vector that can condense the overall dynamic features of the tumor is obtained.

[0036] The clinical indicator temporal coding unit is used to extract features from the clinical indicator sequence to obtain a multimodal clinical representation vector.

[0037] Optionally, the clinical indicator sequence is a time-series collection of clinical test data at various time points during the diagnosis and treatment of liver cancer patients, including indicators such as liver function, alpha-fetoprotein, and coagulation function. These indicators are divided into categorical and continuous indicators, and both types require separate preprocessing to eliminate dimensional differences and data redundancy. Specifically, the clinical indicator sequence is first screened, removing invalid or missing data and labeling it with corresponding timestamps. Categorical indicators are converted into binary vectors using one-hot encoding to ensure that the categorical information can be recognized by the encoding model. Continuous indicators are mapped to the same numerical range using Z-score normalization to avoid interference from different indicator magnitudes. Preferably, a gated recurrent unit is used to implement time-series encoding. This unit contains two core structures: an update gate and a reset gate. The update gate adjusts the fusion ratio between the hidden state of the previous time step and the current input features, determining the weights of retaining historical and current clinical indicator information. The reset gate controls whether to ignore the hidden state of the previous time step, reducing interference from invalid historical information. The preprocessed clinical indicator sequence is input into the gated recurrent unit step by step in chronological order, and the hidden state is iteratively updated to capture the time dependence of the indicators. Finally, a multimodal clinical representation vector integrating the temporal change pattern of clinical indicators is output.

[0038] The vector splicing unit is used to splice the global tumor representation vector and the multimodal clinical representation vector to obtain the observation variable vector.

[0039] Optionally, the global tumor representation vector is the overall dynamic feature vector of the tumor output by the ensemble attention encoding unit, and the multimodal clinical representation vector is the temporal feature vector of clinical indicators output by the clinical indicator temporal encoding unit. Both need to undergo dimensionality verification to ensure concatenation feasibility and avoid feature loss due to dimensional mismatch. Specifically, the feature dimension parameters of the two vectors are first iterated to compare their dimensions. If a dimensional difference exists, zero-padding is used to adjust them to a matching state. Zero-padding only adds zero vectors to the end of the feature dimensions, without changing the feature values ​​and feature distribution of the original vectors. Further, the vectors are concatenated using a feature dimension concatenation method, with the concatenation axis set as the feature dimension axis. All feature dimensions of the global tumor representation vector and all feature dimensions of the multimodal clinical representation vector are sequentially connected. During the concatenation process, the core feature information of both types of vectors is fully preserved, ensuring that neither tumor dynamic evolution-related features nor temporal change features of clinical indicators are lost. After concatenation, an observation variable vector is obtained, which simultaneously integrates the key features of tumor lesions and clinical indicators, comprehensively representing the current hepatocellular carcinoma transformation status of liver cancer patients.

[0040] In the above embodiment, the dynamic lesion feature set is first encoded using a ensemble attention encoding unit to mine global and temporally correlated features of the lesions, resulting in a global tumor representation vector. Then, the clinical indicator sequence is preprocessed and temporally encoded using a gated recurrent unit to obtain a multimodal clinical representation vector. Finally, after dimensionality verification and zero-padding, the two types of vectors are concatenated to obtain the observation variable vector. This embodiment achieves accurate fusion of dynamic tumor lesion features and temporal features of clinical indicators, solving the problems of incomplete representation by a single feature and difficulty in synergistic utilization of multimodal features, effectively improving the completeness and accuracy of the observation variable vector in representing the patient's current state.

[0041] In one embodiment, the ensemble attention encoding unit includes: The feature compression subunit is used to compress the dynamic lesion feature set based on the preset induction point feature set to obtain the compressed lesion feature set.

[0042] Optionally, the preset induction point feature set is a fixed feature set used to assist in compressing dynamic lesion features, preserving core information, and reducing computational load. It is generated by using a K-means clustering algorithm on a large amount of historical lesion feature data. The clustering process focuses on the core feature dimensions of the lesions, ensuring that the induction point feature set can cover the main change patterns of dynamic lesion features, and that the dimension of the induction point features is completely consistent with the dimension of the lesion feature vector at a single time point in the dynamic lesion feature set. Specifically, the dynamic lesion feature set and the preset induction point feature set are first subjected to linear mapping processing. Two independent linear mapping functions are used to convert the two types of features into feature embedding vectors of the same dimension. The weights and biases of the linear mapping functions are obtained through iterative optimization using historical lesion time-series data, ensuring that the key information of the original features is completely preserved after mapping. Furthermore, the similarity between each feature vector in the dynamic lesion feature set and each induced point feature in the preset induced point feature set is calculated. The similarity is obtained by vector dot product operation. Then, the similarity is normalized by softmax to obtain the weight corresponding to each induced point feature. Finally, all induced point features and their corresponding weights are weighted and summed to obtain the compressed lesion features at each time point. After the compressed lesion features at all time points are integrated, a compressed lesion feature set is formed.

[0043] The multi-head self-attention computation subunit is used to perform multi-head self-attention computation on the compressed lesion feature set to obtain the lesion context feature matrix.

[0044] Optionally, the compressed lesion feature set is a temporal feature set output by the feature compression subunit, which retains the core dynamic features of the lesion and eliminates redundancy. Each element corresponds to a compressed lesion feature vector at a single time node. The core of multi-head self-attention computation is to simultaneously capture the global, local, and temporal dependencies between compressed lesion features through multiple independent attention heads. Specifically, the compressed lesion feature set is first linearly mapped, and then converted into query vectors, key vectors, and value vectors through three independent fully connected layers. The parameters of the three fully connected layers are obtained through iterative optimization using historical lesion feature data to ensure that the three types of vectors can accurately represent the different feature dimensions of the compressed lesion features. Furthermore, the query vector, key vector, and value vector are all divided into multiple independent sub-vectors according to the feature dimension. Each sub-vector corresponds to an attention head, and each attention head performs self-attention calculation independently: first, the dot product of the query vector and key vector corresponding to the attention head is calculated to obtain the attention score. The attention score represents the correlation strength between compressed lesion features at different time points. Then, the attention score is subjected to softmax normalization so that the sum of the normalized attention weights is 1. Subsequently, the attention weights are weighted and summed with the value vector of the corresponding attention head to obtain the output result of the attention head. The output results of all attention heads are concatenated and stitched together according to the feature dimension to obtain the lesion context feature matrix. This matrix fully integrates various correlation features captured by multiple attention heads and accurately reflects the temporal evolution and correlation relationship of compressed lesion features.

[0045] The pooled multi-head attention aggregation subunit is used to perform pooled multi-head attention aggregation on the lesion context feature matrix to obtain the lesion aggregation features.

[0046] Optionally, the lesion context feature matrix is ​​a feature matrix output by the multi-head self-attention computation subunit, containing various related information of compressed lesion features. It has high dimensionality and some redundant features. The core of pooling multi-head attention aggregation is to aggregate features from this matrix, extract core related features, and reduce feature dimensionality. Specifically, the lesion context feature matrix is ​​first split according to attention heads to obtain multiple independent single-head feature matrices. Each single-head feature matrix corresponds to the output of one attention head. Then, a hybrid pooling operation is performed on each single-head feature matrix. The hybrid pooling uses a combination of global average pooling and global max pooling. Global average pooling is used to calculate the average eigenvalue of each feature channel to capture the overall distribution pattern of lesion features, while global max pooling is used to extract the maximum eigenvalue of each feature channel to capture key prominent information in the lesion features. The average eigenvalue and maximum eigenvalue obtained from the hybrid pooling of each single-head feature matrix are concatenated to form a single-head aggregated feature. All single-head aggregated features are then concatenated and integrated according to feature dimensions to obtain the preliminary aggregated feature. Furthermore, a lightweight linear layer is used to adjust the dimensions of the initial aggregated features. The parameters of the linear layer are obtained through iterative optimization using historical lesion feature data. This layer is used to fuse aggregated features from different attention heads, eliminate dimensional differences and redundant information, and finally obtain lesion aggregated features. These features condense the core correlation information of the lesion context feature matrix, taking into account both the overall pattern and key details of the lesion features.

[0047] The feedforward neural network mapping subunit is used to input the lesion aggregation features into the global tumor representation feedforward neural network to generate a global tumor representation vector.

[0048] Optionally, the global tumor representation feedforward neural network is a pre-constructed nonlinear mapping model used to map lesion aggregation features to global tumor representation vectors. This network consists of an input layer, at least two hidden layers, and an output layer, with fully connected layers. The hidden layers all use the ReLU activation function to achieve nonlinear transformation of features, while the output layer uses a linear activation function to map the aggregation features to a global tumor representation vector of a preset dimension. Specifically, the lesion aggregation features output by the pooling multi-head attention aggregation subunit are first input to the input layer of the global tumor representation feedforward neural network. The input layer performs dimension adaptation on the lesion aggregation features to ensure that the dimension of the lesion aggregation features matches the input dimension of the hidden layer. If there is a dimensional difference, it is adjusted to the corresponding dimension through linear mapping. Preferably, the hidden layers perform nonlinear transformation of the input features using the ReLU activation function to suppress the propagation of invalid features and enhance the model's ability to capture key lesion features. The output of each hidden layer serves as the input of the next hidden layer. After multiple layers of nonlinear transformation, deep correlation information in the lesion aggregation features is mined, and irrelevant redundant features are eliminated. Furthermore, the output of the last hidden layer is input to the output layer. The output layer maps the deep features to a fixed dimension through a linear activation function to obtain the global tumor representation vector. All layer parameters of the global tumor representation feedforward neural network are obtained through iterative optimization using historical lesion feature data, ensuring that the mapped global tumor representation vector can accurately condense the core dynamic features of the lesion and comprehensively reflect the overall state of the tumor.

[0049] In the above embodiment, a feature compression subunit compresses the dynamic lesion feature set by combining it with a preset induction point feature set, eliminating redundancy and retaining core features to obtain a compressed lesion feature set. Then, a multi-head self-attention computation subunit captures various associations among the compressed lesion features, generating a lesion context feature matrix. Subsequently, a pooling multi-head attention aggregation subunit aggregates the core features to obtain aggregated lesion features. Finally, the aggregated lesion features are input into a global tumor representation feedforward neural network to generate a global tumor representation vector. This embodiment gradually mines the core information and association patterns of lesion features, solving the problems of lesion feature redundancy, incomplete association capture, and excessively high dimensionality making them difficult to utilize. The generated global tumor representation vector is accurate and comprehensive.

[0050] In one embodiment, the frequent differential equation modeling module includes: The neural ordinary differential equation solving unit is used to perform continuous state evolution calculations based on the observation variable vector and the treatment method vector through the neural ordinary differential equation model, so as to obtain the predicted hidden state at the future target time.

[0051] Optionally, the observed variable vector is a feature vector that comprehensively represents the current hepatocellular carcinoma transformation state of liver cancer patients after multimodal feature fusion, and the treatment method vector is a feature vector that reflects the treatment intervention state at each time point after encoding the treatment record sequence. The two need to be aligned in the time dimension to ensure that the observed variable vector and the treatment method vector at each time point correspond one-to-one, avoiding solution errors caused by temporal misalignment. The neural frequent differential equation model is a pre-constructed nonlinear evolution model, the core of which is the hidden state derivative function. This function is implemented by fitting a fully connected neural network and is used to describe the rate of change of the observed variable vector (hidden state) over time. When building the model, a large number of historical liver cancer patients' observed variable samples, treatment method samples, and corresponding transformation state samples are selected and divided into training set and validation set according to a preset ratio. The parameters of the fully connected neural network are iteratively optimized using the training set, and the model accuracy is verified to be stable using the validation set, thus completing the model construction. Specifically, the aligned vector of observed variables and the vector of treatment methods are first input into the model. The derivative of the hidden state with respect to time is calculated using the hidden state derivative function. This derivative quantifies the rate of change of the hepatocellular carcinoma transformation state. Furthermore, the fourth-order Runge-Kutta method is used to numerically integrate the neural ordinary differential equation. The vector of observed variables at the initial time node is selected as the initial condition for integration. The solution step size and the range of future target times are set according to the clinical follow-up needs. Through iterative calculation, the predicted hidden state corresponding to each future target time is obtained. The predicted hidden state can accurately represent the expected state of hepatocellular carcinoma transformation of the patient at the corresponding time.

[0052] The continuous trajectory generation unit is used to connect the predicted latent states of the future target time in a time sequence to generate a continuous transition state trajectory.

[0053] Optionally, the predicted hidden states at future target times are hidden state vectors output by the neural network constant differential equation solving unit, corresponding to multiple future target times respectively. Each predicted hidden state can only reflect the expected state of hepatocellular carcinoma transformation at a single time, and cannot reflect the continuous evolution of the state over time. Therefore, a continuous trajectory needs to be generated through temporal concatenation processing. Specifically, the predicted hidden states at all future target times are first sorted along the time axis, strictly following the order of future target times from early to late. Each predicted hidden state is then arranged sequentially according to its feature dimension to form an initial discrete hidden state sequence. Further, linear interpolation is performed on the initial discrete hidden state sequence to fill the hidden state gaps between adjacent future target times. The interpolation process is based on the feature values ​​of two adjacent predicted hidden states to calculate the estimated hidden state value at any time point between them, ensuring that the interpolated hidden state sequence is smooth and continuous, without trajectory breaks or abnormal fluctuations. Preferably, the interpolated hidden state sequence is slightly smoothed to eliminate minor noise that may be generated during the interpolation process, while preserving the core evolutionary trend of hepatocellular carcinoma transformation state. Finally, the smoothed hidden state sequence is integrated in chronological order to generate a continuous transformation state trajectory. This trajectory can intuitively and accurately characterize the continuous evolution of hepatocellular carcinoma transformation state over time in liver cancer patients.

[0054] In the above embodiment, the aligned observation variable vector and treatment method vector are first input into a pre-trained and validated neural frequent differential equation model through a neural frequent differential equation solving unit. The predicted hidden state at the future target time is obtained through derivative calculation and fourth-order Runge-Kutta integration. Then, a continuous trajectory generation unit sorts, interpolates, and smooths the predicted hidden state temporally to generate a continuous transition state trajectory. This embodiment solves the problem that traditional prediction methods can only obtain discrete states and cannot capture the continuous evolution of hepatocellular carcinoma. It accurately achieves continuous temporal prediction of the transition state, and the generated continuous transition state trajectory is complete and reliable, effectively improving the scientific validity and clinical reference value of the evaluation results.

[0055] In one embodiment, the solution unit for the ordinary differential equation includes: The hidden state derivative calculation subunit is used to calculate the derivative of the hidden state with respect to time based on the current hidden state, the observed variable vector, and the treatment vector, using the hidden state derivative function model. The formula for calculating the derivative of the hidden state with respect to time is: ; in, Let be the time derivative of the hidden state. Let be the hidden state at time t. Let be the observed variable at time t. Let be the treatment vector at time t. are the parameters of the derivative function.

[0056] Optionally, the hidden state at time t It is a feature vector reflecting the potential transformation state of hepatocellular carcinoma at time t in liver cancer patients. It cannot be directly observed and needs to be derived from the observed variable vector at the initial time. Its dimension is the same as that of the observed variable vector. Treatment vector Maintain consistency. Observation variable vector It is the patient's overall state vector at time t obtained after multimodal feature fusion, and the treatment vector. This is the comprehensive treatment state vector at time t obtained after encoding the treatment record. The three vectors (time, feature, and time) need to be aligned in both the time and feature dimensions to ensure a one-to-one correspondence and dimension matching between the three vectors at the same time. (Hidden state derivative function model) It is a pre-constructed nonlinear function, fitted by a fully connected neural network, used to accurately describe the rate of change of the hidden state over time. The parameters... The parameters to be optimized for the model are obtained through training with a large number of latent state samples, observed variable samples, and treatment vector samples from historical liver cancer patients. During training, the samples are divided into training and validation sets according to a preset ratio. The weights and biases of the fully connected neural network are iteratively optimized using the training set, and the model accuracy is verified and stabilized using the validation set. Finally, the parameters are determined. The optimal value. Specifically, first align the... , , Concatenate the features along their dimensions and substitute them into the hidden state derivative function model. Nonlinear computation is performed through fully connected layers within the model, and the ReLU activation function is used to capture the complex relationships among the three factors, ultimately outputting the derivative of the hidden state at time t with respect to time. This derivative quantifies the instantaneous rate of change in the transformation state of hepatocellular carcinoma.

[0057] The numerical integration sub-unit is used to perform numerical integration based on the derivative of the hidden state with respect to time, so as to obtain the predicted hidden state at the future target time.

[0058] Optionally, the time derivative of the hidden state This is the instantaneous rate of change of the hepatocellular carcinoma transformation state at time t, output by the hidden state derivative calculation subunit. The core of the numerical integration solution is to deduce the predicted hidden state at a future target time through integration based on this rate of change. The future target time can be preset according to clinical follow-up needs, and can be a single target time or multiple consecutive target times. Specifically, the fourth-order Runge-Kutta method is used to implement numerical integration. This method combines solution accuracy and efficiency and is a commonly used method for fitting nonlinear rates of change and solving numerical solutions to differential equations. Specifically, the initial conditions for integration are first determined, and the hidden state corresponding to the initial time node t0 is selected. As the starting point for points. The values ​​are directly assigned from the observed variable vector at the initial moment. Then, the integration step size is adaptively set according to the time interval between the future target moment and the initial moment to ensure a balance between solution accuracy and efficiency. Subsequently, the derivative of the hidden state with respect to time is used... The four intermediate variables required for the fourth-order Runge-Kutta method are calculated sequentially. ,in, , , , Where h is the integration step size, and finally, the formula is used: Iterative calculations are performed to gradually deduce the predicted hidden state corresponding to each future target time, ensuring that the predicted hidden state accurately matches the actual evolutionary pattern of hepatocellular carcinoma transformation.

[0059] In the above embodiment, the latent state derivative calculation subunit first substitutes the aligned current latent state, observation variable vector, and treatment vector into a pre-trained and validated latent state derivative function model to calculate the latent state derivative with respect to time. Then, the numerical integration subunit uses the fourth-order Runge-Kutta method to integrate this derivative and deduce the predicted latent state at the future target time. This embodiment solves the problems of inaccurate calculation of the latent state change rate and difficulty in continuously predicting the future state, accurately quantifies the instantaneous changes in the transformation state of hepatocellular carcinoma, and achieves accurate prediction of the transformation state at the future target time.

[0060] In an optional embodiment, such as Figure 2 As shown, the lesion feature extraction module 13 includes: The three-dimensional convolutional feature extraction unit 131 is used to input the spatially registered image sequence into the three-dimensional convolutional feature extraction model, perform three-dimensional convolutional feature extraction on the spatially registered image sequence, and obtain multi-phase lesion features.

[0061] Optionally, the spatially registered image sequence is a collection of multi-phase liver images based on a unified spatial coordinate system, after non-rigid registration to eliminate liver deformation. It includes images from different time points of the same patient and can be directly used for cross-phase extraction and comparison of lesion features. The 3D convolutional feature extraction model is a feature extraction model built on a 3D convolutional neural network. This model consists of multiple 3D convolutional layers, 3D pooling layers, and batch normalization layers. Its parameters are obtained through training with a large number of spatially registered image samples and corresponding lesion annotation samples from historical liver cancer patients. During training, the samples are divided into training and validation sets according to a preset ratio. The training set is used to iteratively optimize the network's convolutional kernel weights, biases, and other parameters, while the validation set verifies the model's feature extraction accuracy until it stabilizes, thus completing model construction. Specifically, the spatially registered image sequence is first preprocessed, performing grayscale normalization to eliminate grayscale differences caused by different acquisition parameters, and then Gaussian smoothing is performed to reduce image noise interference. Subsequently, the preprocessed images are stacked in chronological order into a 3D input tensor and input into the 3D convolutional feature extraction model. The model captures the morphological and textural features of lesions in the spatial dimension and the dynamic changes in the temporal dimension through sliding convolution operations of three-dimensional convolutional layers. Three-dimensional pooling layers reduce the dimensionality of convolutional features, retain key features and reduce computational load. Batch normalization layers accelerate model inference and improve stability. Finally, the output is a multi-phase lesion feature that integrates lesion information from multiple phases of imaging.

[0062] The lesion region segmentation unit 132 is used to segment the lesion region based on the lesion features of multiple phases and generate lesion masks for each phase of the image.

[0063] Optionally, the multi-phase lesion features are a set of feature vectors output by the 3D convolutional feature extraction unit, containing spatial and temporal features of lesions in multi-phase images. Each feature vector corresponds to feature information at a certain location in the image. The lesion mask is a binary mask used to accurately distinguish lesion areas from normal liver tissue areas in the image; areas with a mask value of 1 correspond to lesion areas, and areas with a mask value of 0 correspond to normal tissue areas. Specifically, the multi-phase lesion features are first enhanced by using a lightweight fully connected layer to amplify the differences between lesion features and normal tissue features, improving feature discrimination and avoiding missed or misdiagnosed cases during segmentation. Furthermore, an adaptive threshold segmentation algorithm is used to segment the enhanced feature map. The threshold is automatically determined by calculating the peak value of the gray-level histogram of the feature map, without manual intervention, ensuring that the segmentation threshold adapts to the differences in image features of different patients. After segmentation, a preliminary binary mask is obtained. Then, morphological processing is performed on the preliminary mask. First, the tiny voids inside the lesion area are filled by dilation operation. Then, the tiny noise areas at the edge of the mask are removed by erosion operation. False lesion areas that are too small or have indistinct features are eliminated. Finally, accurate lesion masks corresponding to each phase of the image are generated to ensure that the lesion area at each time point can be accurately delineated.

[0064] The mask information extraction unit 133 is used to extract the lesion space and feature information corresponding to each mask to obtain the initial lesion set.

[0065] Optionally, the lesion masks for each phase of the image accurately identify the location of the lesion region in the corresponding phase image. The lesion spatial information refers to the lesion's three-dimensional spatial coordinate range, center coordinates, and other positional parameters in the image coordinate system. The lesion feature information refers to the lesion features corresponding to the lesion region across multiple phases. Together, they constitute the complete lesion information for a single time node. Specifically, the lesion masks for each phase of the image are first read. The lesion region is located using the binary mask value of the mask. The three-dimensional coordinates of all pixels with a mask value of 1 are extracted. The maximum and minimum values ​​of these coordinates are calculated to determine the spatial coordinate range of the lesion. The mean value of the coordinates is calculated to obtain the three-dimensional center coordinates of the lesion, thus completing the extraction of the lesion spatial information. Simultaneously, based on the lesion region located by the lesion mask, the feature vector of that region is extracted from the lesion features of multiple phases, preserving the morphology, texture, and temporal dynamic features of the lesion to ensure that the feature information accurately corresponds to the lesion region. Furthermore, the spatial information of lesions in each phase of the images is associated and integrated with the corresponding lesion feature information, grouped by time node, with the lesion information of each time node as an independent element. After integrating the lesion information of all time nodes, an initial lesion set is obtained.

[0066] The lesion topology tracking unit 134 is used to perform lesion topology tracking based on the initial lesion set to obtain a dynamic lesion feature set.

[0067] Optionally, the initial lesion set contains spatial and feature information of independent lesions at each time point, but does not associate the same lesion at different time points, thus failing to reflect the dynamic evolution of lesions over time. The core of lesion topology tracking is to capture the topological feature associations of lesions to achieve the tracking of the same lesion across time points. Specifically, the initial lesion set is first sorted chronologically to determine the lesion information of two adjacent time points. The three-dimensional intersection-union ratio (IU / U) and centroid distance between lesions at adjacent time points are calculated. The IU / U quantifies the spatial overlap of lesions, and the centroid distance quantifies the spatial positional deviation of lesions. Subsequently, a comparison is made using preset IU / U thresholds and preset distance thresholds. Both thresholds are generated using statistical analysis methods based on a large amount of historical lesion tracking data. When the three-dimensional IU / U quotient of two lesions is greater than or equal to the preset IU / U quotient threshold and the centroid distance is less than or equal to the preset distance threshold, the two are determined to be the same lesion, and a topological association is established. Furthermore, new lesions, disappearing lesions, and persistent lesions are distinguished. Newly appearing lesions without a correlation are identified as new lesions, existing lesions without a correlation are identified as disappearing lesions, and lesions with a correlation are identified as persistent lesions. Then, the temporal characteristics of the same persistent lesion are fused to integrate its spatial and characteristic change information across time nodes, and finally a dynamic lesion feature set is obtained. The dynamic lesion feature set fully reflects the evolution of all lesions over time.

[0068] In the above embodiment, the spatially registered image sequence is first input into a three-dimensional convolutional feature extraction model to extract multi-phase lesion features. Then, the multi-phase lesion features are segmented and morphologically processed to generate lesion masks for each phase of the image. Subsequently, the spatial and feature information of the lesions corresponding to each mask is extracted to form an initial lesion set. Finally, topological tracing is performed on the initial lesion set to associate the same lesion across time nodes, resulting in a dynamic lesion feature set. This embodiment gradually achieves accurate lesion feature extraction, region localization, information integration, and dynamic tracking, solving the problems of inaccurate feature extraction, noisy lesion region segmentation, and inability to track the dynamic evolution of lesions in traditional lesion processing. The generated dynamic lesion feature set is complete and reliable.

[0069] In one embodiment, the lesion topology tracking unit includes: The spatial similarity calculation subunit is used to calculate the spatial overlap and centroid distance of lesion regions at adjacent time points in the initial lesion set, and generate a lesion spatial metric dataset.

[0070] Optionally, the initial lesion set includes spatial and feature information of independent lesions at each time point. Lesion region information specifically refers to the 3D spatial coordinate range and pixel distribution information of the lesions in the image coordinate system. Spatial overlap is used to quantify the degree of overlap between lesion regions at adjacent time points, and centroid distance is used to quantify the spatial offset between the centers of lesions at adjacent time points. The lesion spatial metric dataset is a collection integrating spatial similarity indices for lesions at all adjacent time points. Specifically, the initial lesion set is first sorted chronologically, and lesions at adjacent time points are selected one by one. Spatial similarity is calculated for each lesion at the previous time point and each lesion at the current time point. Spatial overlap is calculated using the 3D intersection-union ratio (IUU), with the following formula: ,in The volume of the intersection of the two lesion regions in three-dimensional space. The volume represents the union of the three-dimensional spatial regions of the two lesions. The centroid distance is obtained by calculating the Euclidean distance between the three-dimensional centroid coordinates of the two lesions. First, the three-dimensional centroid coordinates of each lesion are determined by the mean of the coordinates of all pixels within the lesion region, and then the Euclidean distance formula is substituted to complete the calculation. After each calculation is completed, the spatial overlap and centroid distance of each lesion pair are associated with the corresponding lesion identifier and integrated to form a lesion spatial metric dataset.

[0071] The lesion pairing subunit is used to compare the spatial overlap and centroid distance in the lesion spatial measurement dataset with the preset overlap threshold and preset distance threshold, respectively, to obtain the comparison results, and to determine the correspondence between lesion regions at adjacent time points based on the comparison results, thereby generating lesion pairing information.

[0072] Optionally, the preset overlap threshold and preset distance threshold are the core criteria for determining whether lesions at adjacent time points are the same lesion. Both are generated through statistical analysis of a large amount of historical lesion topology tracking data. The preset overlap threshold is used to define the minimum standard for lesion overlap, and the preset distance threshold is used to define the maximum spatial offset range of the same lesion at adjacent time points, ensuring that the pairing results conform to the clinical lesion evolution pattern. Specifically, the spatial lesion measurement dataset output by the spatial similarity calculation subunit is first called to extract the spatial overlap and centroid distance corresponding to each pair of lesions at adjacent time points, and then threshold comparisons are performed separately. Only when the spatial overlap of a lesion pair is greater than or equal to the preset overlap threshold and the centroid distance is less than or equal to the preset distance threshold is the lesion pair determined to be the same lesion, and a one-to-one correspondence is established; if only one of them is met or neither of them is met, the two are determined to be different lesions, and no correspondence is established. Furthermore, for scenarios with multiple coexisting lesions, a greedy matching algorithm is employed to handle the pairing logic. This algorithm prioritizes matching lesion pairs with the highest spatial overlap and the smallest centroid distance, avoiding many-to-one or one-to-many pairing anomalies and ensuring the uniqueness and rationality of the pairing results. After comparing all lesion pairs, lesion pairing information is compiled, clearly identifying the correspondence between lesions at adjacent time points, while also marking lesions for which no corresponding relationship was found.

[0073] The lesion region marking subunit is used to merge lesion region information with corresponding relationships based on lesion pairing information, and mark newly added or disappeared lesion regions to construct a dynamic lesion set.

[0074] Optionally, lesion pairing information clarifies the correspondence between lesions at adjacent time points. Merging lesion region information with corresponding relationships is to integrate the spatial and feature information of the same lesion across time nodes. Marking newly added or disappeared lesions is to completely record the dynamic evolution trajectory of lesions. The dynamic lesion set is a collection containing the temporal evolution information of all lesions. Specifically, the lesion pairing information output by the lesion pairing subunit is first read, and processed according to the pairing results in three scenarios. For lesions with one-to-one correspondence, the spatial region information and feature information of the lesion at the current time are merged with the information of the corresponding lesion at the previous time, retaining all temporal information of the same lesion across time nodes, and completely recording the spatial and feature changes of the lesion; for cases where there is a lesion at the current time but no corresponding lesion was found at the previous time, it is determined to be a newly added lesion, assigned a unique identifier, its initial spatial region information and feature information are recorded, and the newly added time node is marked; for cases where there is a lesion at the previous time but no corresponding lesion was found at the current time, it is determined to be a disappeared lesion, and the disappearance time node is marked on its original information, retaining all its historical temporal information. After processing all scenarios, the identification, temporal information, spatial and feature information of all lesions are integrated to construct a dynamic lesion set, which fully preserves the evolutionary trajectory of all lesions.

[0075] The temporal feature fusion subunit is used to fuse the temporal feature sequences of each lesion in the dynamic lesion set to generate a dynamic lesion feature set.

[0076] Optionally, the dynamic lesion set contains spatial region information and feature information of all lesions across time nodes. The temporal feature sequence of each lesion is a sequence composed of feature information of the lesion at different time nodes in chronological order. These discrete features do not form a unified feature that can reflect the dynamic evolution of the lesions. The core of temporal feature fusion is to integrate the discrete temporal features into a unified dynamic feature of the lesions. Specifically, the temporal feature sequence of each lesion is extracted one by one from the dynamic lesion set. The feature information in the sequence is preprocessed, and the Z-score normalization method is used to map the features of different time nodes to the same numerical range, eliminating the difference in feature dimension and magnitude interference between time nodes. Preferably, a gated loop unit is used to realize temporal feature fusion. This unit captures the temporal dependency of lesion features through update gate and reset gate. The update gate is used to determine the proportion of lesion features retained from the previous time node and the lesion features of the current time node, and the reset gate is used to determine whether to ignore invalid feature information from the previous time node. The standardized temporal feature sequence of each lesion is input into the gated recurrent unit step by step in chronological order. By iteratively updating the hidden state, the features and dynamic change information of each time node are integrated. Finally, the hidden state of the last time step of the gated recurrent unit is taken as the dynamic feature of the lesion. The dynamic features of all lesions are integrated to obtain the dynamic lesion feature set.

[0077] In the above embodiment, the spatial correlation between lesions at adjacent time points is first quantified by a spatial similarity calculation subunit to generate a lesion spatial metric dataset. Then, a lesion pairing subunit generates explicit lesion pairing information through threshold comparison and greedy matching. Subsequently, a lesion region marking subunit merges corresponding lesion information and marks newly added and disappeared lesions to construct a dynamic lesion set. Finally, a temporal feature fusion subunit integrates the discrete temporal features of the lesions to generate a dynamic lesion feature set. This embodiment solves the problems of inaccurate pairing, scattered temporal information, and lack of dynamic feature integration in traditional lesion topology tracking, accurately achieving full-cycle dynamic tracking and temporal feature fusion of lesions, and generating a complete and reliable dynamic lesion feature set.

[0078] In the aforementioned multimodal artificial intelligence-based hepatocellular carcinoma (HCC) transformation process assessment system, multi-phase imaging sequences, clinical indicator sequences, and treatment record sequences of HCC patients are first acquired. Non-rigid registration is performed on the multi-phase imaging data to eliminate liver deformation. Then, through three-dimensional convolutional feature extraction, lesion region segmentation, mask information extraction, and lesion topology tracking, including spatial similarity calculation, lesion pairing, lesion region labeling, and temporal feature fusion subunits, a dynamic lesion feature set is obtained. Subsequently, multimodal feature fusion is achieved through ensemble attention encoding, clinical indicator temporal encoding, and vector concatenation to obtain the observed variable vector. Simultaneously, the treatment record sequence is encoded to obtain the treatment method vector. These two types of vectors are input into a neural network ordinary differential equation model. The future predicted hidden states are obtained through hidden state derivative calculation and numerical integration. These are then connected temporally to generate a continuous transformation state trajectory. Finally, the assessment results, including the probability of HCC resectability and the predicted clinical occurrence probability, are output through evaluation decoding. Traditional hepatocellular carcinoma (HCC) transformation assessment suffers from technical problems such as inaccurate lesion tracking, insufficient fusion of multimodal features, discrete transformation trajectory, and insufficient assessment accuracy, failing to comprehensively reflect the tumor evolution pattern. This technical solution, through the collaborative design of various modules and sub-units, accurately achieves dynamic tracking of lesions throughout the entire cycle, deep fusion of multimodal features, and continuous prediction of transformation status, effectively solving the above-mentioned technical problems and significantly improving the accuracy, comprehensiveness, and scientific nature of HCC transformation assessment, providing reliable quantitative evidence for clinical diagnosis and treatment decisions.

[0079] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0080] Based on the same inventive concept, this application also provides a method for implementing the aforementioned multimodal artificial intelligence-based hepatocellular carcinoma transformation process assessment system. The solution provided by this method is similar to the implementation scheme described in the above system. Therefore, the specific limitations in one or more embodiments of the multimodal artificial intelligence-based hepatocellular carcinoma transformation process assessment method provided below can be found in the above-described limitations of the multimodal artificial intelligence-based hepatocellular carcinoma transformation process assessment system, and will not be repeated here.

[0081] In one exemplary embodiment, such as Figure 3 As shown, a method for assessing the transformation process of hepatocellular carcinoma based on multimodal artificial intelligence is provided, including: S1 is used to acquire multi-phase imaging sequences, clinical indicator sequences, and treatment record sequences; S2. Perform non-rigid registration on the multi-phase image data in the multi-phase image sequence to obtain a spatially registered image sequence; S3. Extract lesion features from the spatially registered image sequence to obtain a dynamic lesion feature set; S4. Multimodal feature fusion is performed on the dynamic lesion feature set and the clinical indicators in the clinical indicator sequence to obtain the observation variable vector; the observation variable vector is used to characterize the current hepatocellular carcinoma transformation status of liver cancer patients; S5 is used to encode the treatment record sequence into a treatment vector to obtain a treatment method vector. S6. Input the observed variable vector and treatment method vector into the neural network differential equation model to predict the continuous transformation state trajectory and obtain the continuous transformation state trajectory; the continuous transformation state trajectory is used to characterize the expected transformation state of hepatocellular carcinoma in liver cancer patients. S7. Evaluate and decode the continuous transformation trajectory to obtain the transformation process evaluation results; the transformation process evaluation results include the probability of liver cancer resectableness and the predicted clinical occurrence probability.

[0082] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a multimodal artificial intelligence-based hepatocellular carcinoma transformation process assessment system as described above.

[0083] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of a multimodal artificial intelligence-based hepatocellular carcinoma transformation process assessment system as described above.

[0084] For the device embodiments, since they basically correspond to the system embodiments, the relevant parts can be referred to in the description of the system embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0085] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A system for evaluating the transformation process of hepatocellular carcinoma based on multimodal artificial intelligence, characterized in that, The system includes: The data acquisition module is used to acquire multi-phase imaging sequences, clinical indicator sequences, and treatment record sequences; The image registration module is used to perform non-rigid registration of multi-period image data in the multi-period image sequence to obtain a spatially registered image sequence. The lesion feature extraction module is used to extract lesion features from the spatially registered image sequence to obtain a dynamic lesion feature set. A multimodal feature fusion module is used to perform multimodal feature fusion on the dynamic lesion feature set and the clinical indicators in the clinical indicator sequence to obtain an observation variable vector; the observation variable vector is used to characterize the current hepatocellular carcinoma transformation status of the liver cancer patient; The encoding module is used to encode the treatment record sequence into a treatment vector to obtain a treatment method vector; The trajectory prediction module is used to input the observed variable vector and the treatment method vector into the neuronormal differential equation model to perform continuous transformation state trajectory prediction and obtain the continuous transformation state trajectory; the continuous transformation state trajectory is used to characterize the expected transformation state of hepatocellular carcinoma in the liver cancer patient; An evaluation decoding module is used to evaluate and decode the continuous transformation state trajectory to obtain the transformation process evaluation result; the transformation process evaluation result includes the probability of liver cancer resectableness and the predicted clinical occurrence probability.

2. The system according to claim 1, characterized in that, The multimodal feature fusion module includes: A ensemble attention encoding unit is used to perform ensemble attention encoding on the dynamic lesion feature set to obtain a global tumor representation vector. A clinical indicator temporal coding unit is used to extract features from the clinical indicator sequence to obtain a multimodal clinical representation vector. The vector concatenation unit is used to concatenate the global tumor representation vector and the multimodal clinical representation vector to obtain the observation variable vector.

3. The system according to claim 2, characterized in that, The ensemble attention encoding unit includes: The feature compression subunit is used to compress the dynamic lesion feature set based on a preset set of induction point features to obtain a compressed lesion feature set. A multi-head self-attention calculation subunit is used to perform multi-head self-attention calculation on the compressed lesion feature set to obtain the lesion context feature matrix; A pooled multi-head attention aggregation subunit is used to perform pooled multi-head attention aggregation on the lesion context feature matrix to obtain lesion aggregation features; The feedforward neural network mapping subunit is used to input the lesion aggregation features into the global tumor representation feedforward neural network to generate the global tumor representation vector.

4. The system according to claim 1, characterized in that, The trajectory prediction module includes: The neural ordinary differential equation solving unit is used to perform continuous state evolution calculation based on the observed variable vector and the treatment method vector through the neural ordinary differential equation model to obtain the predicted hidden state at the future target time. The continuous trajectory generation unit is used to connect the predicted latent states of the future target time in a time sequence to generate the continuous transition state trajectory.

5. The system according to claim 4, characterized in that, The solution unit for the divine ordinary differential equation includes: The hidden state derivative calculation subunit is used to calculate the derivative of the hidden state with respect to time based on the current hidden state, the observed variable vector, and the treatment vector, using the hidden state derivative function model; the formula for calculating the derivative of the hidden state with respect to time is: ； in, Let be the time derivative of the hidden state. Let be the hidden state at time t. Let be the observed variable at time t. Let be the treatment vector at time t. The parameters of the derivative function are denoted as . The numerical integration subunit is used to perform numerical integration based on the derivative of the hidden state with respect to time to obtain the predicted hidden state at the future target time.

6. The system according to claim 1, characterized in that, The lesion feature extraction module includes: The three-dimensional convolutional feature extraction unit is used to input the spatially registered image sequence into the three-dimensional convolutional feature extraction model, and to perform three-dimensional convolutional feature extraction on the spatially registered image sequence to obtain multi-phase lesion features; The lesion region segmentation unit is used to segment the lesion region based on the multi-phase lesion features and generate lesion masks for each phase image. The mask information extraction unit is used to extract the lesion space and feature information corresponding to each mask to obtain the initial lesion set; The lesion topology tracking unit is used to perform lesion topology tracking on the initial lesion set to obtain the dynamic lesion feature set.

7. The system according to claim 6, characterized in that, The lesion topology tracking unit includes: The spatial similarity calculation subunit is used to calculate the spatial overlap and centroid distance of the lesion region information at adjacent time points in the initial lesion set, and generate a lesion spatial measurement dataset. The lesion pairing subunit is used to compare the spatial overlap and centroid distance in the lesion spatial measurement dataset with a preset overlap threshold and a preset distance threshold, respectively, to obtain the comparison result, and to determine the correspondence between lesion regions at adjacent time points based on the comparison result, thereby generating lesion pairing information. The lesion region marking subunit is used to merge lesion region information with corresponding relationships according to the lesion pairing information, and mark newly added or disappeared lesion regions to construct a dynamic lesion set. The temporal feature fusion subunit is used to fuse the temporal feature sequences of each lesion in the dynamic lesion set to generate the dynamic lesion feature set.

8. A method for evaluating the transformation process of hepatocellular carcinoma based on multimodal artificial intelligence, characterized in that, The method includes: Acquire multi-phase imaging sequences, clinical indicator sequences, and treatment record sequences; Non-rigid registration is performed on the multi-phase image data in the multi-phase image sequence to obtain a spatially registered image sequence. Lesion features are extracted from the spatially registered image sequence to obtain a dynamic lesion feature set; Multimodal feature fusion is performed on the dynamic lesion feature set and the clinical indicators in the clinical indicator sequence to obtain an observation variable vector; the observation variable vector is used to characterize the current hepatocellular carcinoma transformation status of liver cancer patients; The treatment record sequence is encoded using treatment vectors to obtain a treatment method vector; The observed variable vector and the treatment method vector are input into a neuronormal differential equation model to predict the continuous transformation state trajectory, thereby obtaining the continuous transformation state trajectory; the continuous transformation state trajectory is used to characterize the expected transformation state of hepatocellular carcinoma in the liver cancer patient; The continuous transformation trajectory is evaluated and decoded to obtain the transformation process evaluation results; the transformation process evaluation results include the probability of liver cancer resectableness and the predicted clinical occurrence probability.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the system according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the system according to any one of claims 1 to 7.

Images