An intelligent prediction system for pathological indicators of liver cancer patients

Abstract

The present invention provides an intelligent prediction system for pathological indicators of liver cancer patients. The system includes a sequence acquisition module, which acquires three-phase global CT sequences and tumor sequences; a multi-phase spatial feature extraction module, which acquires tumor spatial features of each phase and global spatial features of each phase; a multi-phase temporal feature extraction module, which uses three-phase tumor spatial features to obtain multi-phase temporal features of tumor images, and uses three-phase global spatial features to obtain multi-phase temporal features of global images; a feature splicing module, which splices multi-phase temporal features of tumor images, multi-phase temporal features of global images, laboratory data vector representation and / or clinical data vector representation to obtain multimodal feature data; a pathology multi-indicator prediction module, which performs multi-task processing on multimodal feature data to obtain MTM HCC subtype prediction results and MVI classification prediction results. Based on the deep fusion of CT images, laboratory data and clinical data, accurate pathology multi-indicator prediction results are simultaneously obtained.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent prediction system for pathological indicators of liver cancer patients. Background Art

[0002] Major clinical guidelines such as the American Association for the Study of Liver Diseases (AASLD) and the European Association for the Study of the Liver (EASL) point out that hepatocellular carcinoma can be directly diagnosed based on enhanced CT or MR without biopsy. However, many patients with liver cancer are already in the advanced stage when they seek medical treatment and cannot undergo liver resection. If preoperative multimodal information such as imaging information, laboratory test information, and clinical information can be used to intelligently judge the preoperative pathological indicators of the patient's pathology, it will be of great significance for the formulation of personalized treatment plans. Macrotrabecular-massive hepatocellular carcinoma (MTM HCC) is a highly invasive hepatocellular carcinoma with a poor prognosis, which can only be diagnosed by postoperative histopathological examination. The use of non-invasive methods to preoperatively diagnose the subtypes of hepatocellular carcinoma before treatment is helpful for the development of individual treatment plans for MTM patients. Microvascular invasion (MVI) of hepatocellular cells is highly correlated with tumor recurrence and is the most important pathological feature guiding the prevention and treatment of liver cancer recurrence. The diagnosis of MVI relies on histopathological examination of postoperative resection specimens, but the results of postoperative pathological examination are lagging, which is not conducive to using MVI information to guide preoperative and intraoperative treatment decisions. Therefore, the necessity and importance of preoperative prediction of MVI classification are highlighted.

[0003] However, the existing preoperative liver cancer pathology prediction methods have the following problems:

[0004] First, the existing liver cancer pathology prediction method only judges a single indicator in the pathology of hepatocellular carcinoma patients, such as MTM subtype or MVI classification, and cannot comprehensively predict the two pathological indicators of MTM subtype and MVI classification at the same time. There is a problem that the prediction indicators are not perfect. Comprehensive and comprehensive pathological prediction is the most beneficial for patient condition evaluation and preoperative judgment. Only by making a comprehensive judgment on the pathology can the patient's comprehensive state be grasped as comprehensively as possible. The inventors of this application found that the proportion of high-grade MVI (M2 grade) in hepatocellular carcinoma patients diagnosed with MTM HCC has been significantly improved, and the proportion of low-grade MVI (M0 grade) in NonMTM HCC patients is relatively high. The occurrence of MTM HCC and MVI is highly correlated. Multi-task learning can promote the mutual use of information between two tasks and improve the prediction accuracy by utilizing the correlation information between different tasks. The inventors found that the correlation between MTM HCC and MVI meets the correlation requirements of multi-task learning. Therefore, this application will construct a multi-task learning deep learning network based on model-task bidirectional drive to realize the prediction of pathological dual indicators while improving the detection accuracy of both.

[0005] Second, existing technical solutions mostly use visual or radiomics methods to perform intelligent diagnosis and prediction of liver cancer pathology information. The imaging feature evaluation method based on visual analysis is greatly affected by individual differences and subjective influences of doctors, and it is difficult to avoid potential selection bias; radiomic features reflect the image expression at the pixel level, and there is no consensus on the radiomics features that characterize pathology (such as MTM HCC, MVI, etc.). Deep learning technology can automatically learn high-dimensional features that are difficult for the human eye to perceive but contain rich predictive information. It has demonstrated strong diagnostic capabilities and potential in the field of disease identification. It extracts high-dimensional features in images through multi-layer convolutional neural networks, which are more stable and have stronger diagnostic capabilities than radiomics.

[0006] Third, liver contrast-enhanced CT images are divided into three development stages: arterial phase, venous phase, and equilibrium phase. Compared with liver tissue, tumor contrast metabolism is a "fast in and fast out" process, that is, liver tissue and tumor have different metabolism of contrast agents, showing different imaging characteristics in different phases. Current pathological prediction is often based on deep learning in the single venous phase, and it is impossible to track the dynamic changes of contrast. The inventors of this application found that the contrast metabolism time of MTM HCC subtypes is different from that of NonMTM HCC. However, the prior art does not capture this difference to improve the accuracy of pathological prediction. For example, the Chinese patent with publication number CN113657503A discloses a cancer survival analysis system based on multi-task and multi-modality. The patent uses an RNN network to directly characterize the time series features and extract the overall change pattern of the three phases of arterial phase, venous phase, and equilibrium phase of liver contrast-enhanced CT images, ignoring the "fast in and fast out" phenomenon of contrast agent metabolism changes in tumors relative to the metabolic changes of liver tissue. The patent does not capture more relative changes between tumors and liver tissues after contrast. Moreover, although the gated recurrent neural network used in this patent has a gating mechanism to control the flow of information, its modeling method is fixed, and the information processing method for different positions in the sequence is the same. It does not have the ability to dynamically adapt to extract dynamically changing information during the metabolic process. Summary of the invention

[0007] The present invention aims to solve the technical problems existing in the prior art and provide an intelligent prediction system for pathological indicators of liver cancer patients.

[0008] In order to achieve the above-mentioned purpose of the present invention, the present invention provides an intelligent prediction system for pathological indicators of liver cancer patients, which system includes: a sequence acquisition module, which acquires global CT sequences of three phases, and tumor sequences of three phases obtained by segmentation from the global CT sequences of the three phases, wherein the three phases include arterial phase, venous phase and equilibrium phase; a multi-phase spatial feature extraction module, which acquires the tumor spatial features of each phase based on the tumor sequence of each phase, and acquires the global spatial features of each phase based on the global CT sequence of each phase; a multi-phase temporal feature extraction module, which uses the tumor spatial features of the three phases to acquire multi-phase temporal features of tumor images, and uses the global spatial features of the three phases to acquire multi-phase temporal features of global images; a feature splicing module, which splices the multi-phase temporal features of tumor images, the multi-phase temporal features of global images, and the vector representation of laboratory data and / or the vector representation of clinical data to obtain multimodal feature data; and a pathological multi-indicator prediction module, which performs multi-task classification processing on the multimodal feature data to obtain MTM HCC subtype prediction results and MVI classification prediction results.

[0009] Further preferably, the multi-period spatial feature extraction module includes a tumor spatial feature extraction network and a global spatial feature extraction network; the tumor spatial feature extraction network extracts the tumor spatial features of each period from the tumor sequence of each period respectively; the global spatial feature extraction network extracts the global spatial features of each period from the global CT sequence of each period respectively.

[0010] Further preferably, the tumor spatial feature extraction network includes: a tumor high-order feature extraction network, which is used to extract high-order features of the tumor sequence of each period, recorded as tumor high-order features; and a tumor pooling layer, which compresses the high-order features of the tumor of each period into the tumor spatial features of each period.

[0011] Further preferably, the global spatial feature extraction network includes: a global high-order feature extraction network, used to extract high-order features of the global CT sequence of each period, recorded as global high-order features; a global pooling layer, compressing the global high-order features of each period into the global spatial features of each period.

[0012] Further preferably, the multi-period temporal feature extraction module includes a tumor deformable attention network and a global deformable attention network; the tumor deformable attention network obtains the tumor temporal features of each period based on the tumor spatial features of the three periods, and performs dimensionality reduction processing on the tumor temporal features of the three periods to obtain multi-period temporal features of tumor images; the global deformable attention network obtains the global temporal features of each period based on the global spatial features of the three periods, and performs dimensionality reduction processing on the global temporal features of the three periods to obtain global CT image sequence features.

[0013] Further preferably, the tumor deformable attention network includes a first tumor module, a second tumor module, a third tumor module and a tumor pooling layer; the first tumor module is used to map the tumor spatial features of each period to the value space of that period; the second tumor module performs position encoding on each feature point in the tumor spatial features of each period, and then uses the first linear mapping to learn multiple offsets for the feature points of the tumor spatial features of each period, and uses the second linear mapping to learn the attention weights of the multiple offsets in the value space of each period; the third tumor module is used to obtain the tumor image time series features of each period, and the feature value of each feature point of the tumor image time series features of each period is obtained by multiplying the multiple attention weights of the feature point with the mapping value of the feature point in the corresponding value space; the tumor pooling layer performs dimensionality reduction processing on the tumor image time series features of the three periods to obtain multi-period time series features of the tumor images.

[0014] Further preferably, the global deformable attention network includes a global first module, a global second module, a global third module and a global pooling layer; the global first module is used to map the global spatial features of each period to the value space of that period; the global second module performs position encoding on each feature point in the global spatial features of each period, and then uses the first linear mapping to learn multiple offsets for the feature points of the global spatial features of each period, and uses the second linear mapping to learn the attention weights of multiple offsets in the value space of each period; the global third module is used to obtain the global image time series features of each period, and the feature value of each feature point of the global image time series features of each period is obtained by multiplying the multiple attention weights of the feature point with the mapping value of the feature point in the corresponding value space; the global pooling layer performs dimensionality reduction processing on the global image time series features of the three periods to obtain the global image multi-period time series features. Further preferably, it also includes a segmentation module, which is used to segment the tumor area in the global CT sequence of the three periods, and the segmented tumor area constitutes the tumor sequence of the three periods.

[0015] Further preferably, it also includes a data processing module, which executes: acquiring laboratory data; selecting at least one significantly relevant experimental indicator from the laboratory data, and converting the significantly relevant experimental indicator value into a laboratory data vector representation using a value embedding method.

[0016] Further preferably, the data processing module also performs: acquiring clinical data; selecting at least one significantly relevant clinical indicator from the clinical data, and converting the significantly relevant clinical indicator value into a clinical data vector representation using a value embedding method.

[0017] Further preferably, the pathological multi-indicator prediction module performs multi-task prediction through a multi-layer perceptron, and the multi-task prediction includes MTM HCC subtype prediction and MVI classification prediction, the MTM HCC subtype prediction results include MTM HCC subtypes and NonMTM HCC subtypes, and the MVI classification prediction results include M0 level, M1 level and M2 level; during the training process of the intelligent prediction system for pathological indicators of liver cancer patients, the total loss includes the cross-entropy loss of MTM HCC subtype prediction and the cross-entropy loss of MVI classification prediction.

[0018] Beneficial technical effects of the present invention: the multi-phase spatial feature extraction module of the prediction system adopts deep learning technology to extract high-order high-dimensional spatial features of three phases of the global CT sequence and the tumor sequence respectively, and the obtained tumor spatial features have tumor detail features; the multi-phase temporal features of the tumor image are obtained by the multi-phase temporal features extraction module using the tumor spatial features of the three phases, and the global spatial features of the three phases are used to obtain the global image multi-phase temporal features, so as to track the dynamic change process of the contrast in the global CT sequence and the tumor sequence. Due to the difference in contrast metabolism between the tumor area and the liver area, the multi-phase temporal features of the tumor image and the multi-phase temporal features of the global image can capture more relative changes in the tumor area after contrast, and the relative changes are closely related to the MTM HCC subtype and MVI classification of hepatocellular carcinoma. Then the multimodal feature data is input into the pathology multi-index prediction module, which can simultaneously predict multiple pathology indicators, and can more accurately predict the MTM HCC subtype and MVI classification;

[0019] The pathological multi-index prediction module of the present invention is a multi-task learning module. It uses the correlation information between the two tasks of MTM HCC subtype prediction and MVI classification prediction to promote the two tasks to use the information in each other, realize the pathological dual-index prediction, and improve the detection accuracy of both and improve the prediction accuracy.

[0020] The present invention obtains multimodal feature data by splicing laboratory data and / or clinical data, as well as high-dimensional tumor image multi-phase time series features and global image multi-phase time series features, and performs multimodal data hybrid learning, which can more accurately and objectively reflect the patient's condition and further improve the accuracy of intelligent prediction of liver cancer pathology;

[0021] The tumor deformable attention network and global deformable attention network used in the present invention calculate the deformable attention of the surrounding features and other phase features in the phase for each spatial vector in the spatial feature to capture the changes in different phases, and can adaptively adjust the focus area according to the shape changes of the angiography, so as to more accurately capture the image feature information that changes dynamically during the metabolic process;

[0022] The present invention simultaneously obtains accurate pathology multi-index prediction results based on the deep fusion of CT images, laboratory data and clinical data, which is of great significance for preoperative judgment of patient pathology and making individualized treatment decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a structural schematic diagram of an intelligent prediction system for pathological indicators of liver cancer patients in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0024] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0025] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0026] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0027] The present invention discloses an intelligent prediction system for pathological indicators of liver cancer patients, referring to Figure 1 , which includes:

[0028] The sequence acquisition module acquires three-phase global CT sequences and three-phase tumor sequences segmented from the three-phase global CT sequences, wherein the three phases include an arterial phase, a venous phase, and a balanced phase.

[0029] In this embodiment, the global CT sequence is preferably a preoperative CT sequence image, and the tumor sequence is a sequence composed of tumor regions segmented from the global CT sequence. The three-phase global CT sequence includes global CT sequences of arterial phase, venous phase and equilibrium phase.

[0030] The three-phase tumor sequence includes the arterial phase, venous phase and equilibrium phase tumor sequence.

[0031] The multi-phase spatial feature extraction module obtains the tumor spatial features of each phase based on the tumor sequence of that phase, and obtains the global spatial features of that phase based on the global CT sequence of each phase.

[0032] In one example, global CT sequences and tumor sequences of arterial phase, venous phase and equilibrium phase are input. The multi-phase spatial feature extraction module extracts global spatial features of the arterial phase based on the global CT sequence of the arterial phase, extracts global spatial features of the venous phase based on the global CT sequence of the venous phase, extracts global spatial features of the equilibrium phase based on the global CT sequence of the equilibrium phase, extracts tumor spatial features of the arterial phase based on the tumor sequence of the arterial phase, extracts tumor spatial features of the venous phase based on the tumor sequence of the venous phase, and extracts tumor spatial features of the equilibrium phase based on the tumor sequence of the equilibrium phase.

[0033] The multi-phase time series feature extraction module uses the three-phase tumor spatial features to obtain the multi-phase time series features of tumor images, and uses the three-phase global spatial features to obtain the multi-phase time series features of global images.

[0034] In the above example, the multi-phase temporal feature extraction module uses the tumor spatial features of the arterial phase, the tumor spatial features of the venous phase, and the tumor spatial features of the equilibrium phase to obtain the multi-phase temporal features of the tumor image; the multi-phase temporal feature extraction module uses the global spatial features of the arterial phase, the global spatial features of the venous phase, and the global spatial features of the equilibrium phase to obtain the multi-phase temporal features of the global image.

[0035] The feature splicing module splices the multi-phase time series features of tumor images, the multi-phase time series features of global images, and the vector representation of laboratory data and / or the vector representation of clinical data to obtain multimodal feature data.

[0036] In this embodiment, the laboratory data vector representation and the clinical data vector representation are preferably vectors converted from laboratory data and clinical information collected before surgery. The laboratory data preferably include, but are not limited to, serum aspartate aminotransferase, alanine aminotransferase, albumin, platelet level, total bilirubin level, and alpha-fetoprotein level, and the clinical information preferably includes, but is not limited to, the patient's age, gender, complications, and medical history. To improve the accuracy of the prediction, it is further preferred that when there are multiple sets of preoperative laboratory data, the set of laboratory data with the closest time interval to the patient's CT scan is selected as the standard.

[0037] The pathological multi-index prediction module performs multi-task classification processing on multimodal feature data to obtain MTM HCC subtype prediction results and MVI classification prediction results.

[0038] Preferably, the pathological multi-indicator prediction module performs multi-task prediction through a multi-layer perceptron, and the multi-task prediction includes MTM HCC subtype prediction and MVI classification prediction. The HCC subtype prediction results include MTM HCC subtypes and NonMTM HCC subtypes, and the MVI classification prediction results include M0 level, M1 level and M2 level. During the training process of the intelligent prediction system for pathological indicators of liver cancer patients, the total loss loss includes the cross entropy loss loss1 of the MTM HCC subtype prediction and the cross entropy loss loss2 of the MVI classification prediction.

[0039] loss=γ1*loss1+γ2*loss2. Wherein, γ1 is the first weight parameter, and γ2 is the second weight parameter. In a preferred embodiment, in order to fully exploit the global and local features (i.e., tumor region features) of hepatocyte CT images, different branch networks are used to extract high-order features from CT image sequences. Figure 1 The multi-period spatial feature extraction module includes a tumor spatial feature extraction network and a global spatial feature extraction network. The tumor spatial feature extraction network extracts the tumor spatial features of each period from the tumor sequence of each period. The global spatial feature extraction network extracts the global spatial features of each period from the global CT sequence of each period.

[0040] In this embodiment, the rich detail information contained in the internal area of ​​the tumor is extracted.

[0041] More preferably, referring to Figure 1 , the tumor spatial feature extraction network includes:

[0042] The tumor high-order feature extraction network is used to extract the high-order features of each stage of tumor sequence, which are recorded as tumor high-order features. Each stage of tumor sequence obtains a tumor high-order feature. Figure 1 The tumor spatial feature extraction network is preferably but not limited to resnet+attention mechanism, which can learn the prominent details and high-order features inside the tumor and obtain the spatial features of tumors at different stages.

[0043] The tumor space extraction pooling layer compresses the high-order features of the tumor in each stage into the tumor space features of that stage.

[0044] In this embodiment, in order to better extract the characteristic information of the global CT sequence, refer to Figure 1 , the global spatial feature extraction network includes:

[0045] The global high-order feature extraction network is used to extract high-order features of the global CT sequence of each period, which are recorded as global high-order features; the backbone of the global high-order feature extraction network is preferably but not limited to a resnet model network, which can extract global information of the liver tissue area of ​​CT images from different angles.

[0046] The global space extraction pooling layer compresses the global high-order features of each period into the global space features of that period.

[0047] In a preferred embodiment, in order to extract the deformation features of the contrast imaging in multiple phases, a dual-branch method is used to extract the temporal features of the contrast imaging of the tumor region and the global region of the liver tissue respectively. Figure 1 ,The multi-period temporal feature extraction module includes a tumor deformable attention network and a global deformable attention network.

[0048] The tumor deformable attention network obtains the tumor time series features of each phase based on the tumor spatial features of the three phases, and performs dimensionality reduction processing on the tumor time series features of the three phases to obtain the multi-phase time series features of the tumor image. In the above example, the tumor time series features of each phase need to be extracted from the tumor spatial features of the arterial phase, venous phase, and equilibrium phase, and then the obtained tumor time series features of the arterial phase, venous phase, and equilibrium phase are reduced in dimension to obtain the multi-phase time series features of the tumor image.

[0049] The global deformable attention network obtains the global temporal features of each phase based on the global spatial features of the three phases, and performs dimensionality reduction processing on the global temporal features of the three phases to obtain the global CT image sequence features. In the above example, the global temporal features of each phase need to be extracted from the global spatial features of the arterial phase, venous phase, and equilibrium phase, and then the global temporal features of the three phases are reduced in dimension to obtain the global CT image sequence features.

[0050] In this embodiment, preferably, the tumor deformable attention network includes a first tumor module, a second tumor module, a third tumor module and a tumor pooling layer; the first tumor module is used to map the tumor spatial features of each period to the value space of that period; the second tumor module performs position encoding on each feature point in the tumor spatial features of each period, and then uses the first linear mapping to learn multiple offsets for the feature points of the tumor spatial features of each period, and uses the second linear mapping to learn the attention weights of the multiple offsets in the value space of each period; the third tumor module is used to obtain the tumor image time series features of each period, and the feature value of each feature point of the tumor image time series features of each period is obtained by multiplying the multiple attention weights of the feature point with the mapping value of the feature point in the corresponding value space; the tumor pooling layer performs dimensionality reduction processing on the tumor image time series features of the three periods to obtain multi-period time series features of the tumor images.

[0051] In this embodiment, the first tumor module, the second tumor module, and the third tumor module are preferably, but not limited to, linear layers.

[0052] In this embodiment, preferably, each phase has N offsets. In the above example, the second tumor module uses the first linear mapping to learn 3*N offsets for the feature points of the tumor spatial features of each phase. The temporal features of the tumor images of each phase are a feature map. The feature value of each feature point in the feature map is obtained by multiplying the 3*N attention weights of the feature point with the mapping value of the feature point in the corresponding value space. N is a positive integer, preferably but not limited to 3, 4, 5 or 6.

[0053] In this embodiment, further preferably, the global deformable attention network includes a global first module, a global second module, a global third module and a global pooling layer; the global first module is used to map the global spatial features of each period to the value space of that period; the global second module performs position encoding on each feature point in the global spatial features of each period, and then uses the first linear mapping to learn multiple offsets for the feature points of the global spatial features of each period, and uses the second linear mapping to learn the attention weights of the multiple offsets in the value space of each period; the global third module is used to obtain the global image time series features of each period, and the feature value of each feature point of the global image time series features of each period is obtained by multiplying the multiple attention weights of the feature point with the mapping value of the feature point in the corresponding value space; the global pooling layer performs dimensionality reduction processing on the global image time series features of three periods to obtain the global image multi-period time series features.

[0054] In this embodiment, the global first module, the global second module, and the global third module are preferably, but not limited to, linear layers.

[0055] In this embodiment, preferably, each period has 3*N offsets. In the above example, the global second module uses the first linear mapping to perform 3*N offset learning on the feature points of the global spatial features of each period. The global image time series features of each period are a feature map, and the feature value of each feature point in the feature map is obtained by multiplying the 3*N attention weights of the feature point with the mapping value of the feature point in the corresponding value space.

[0056] In this embodiment, the tumor deformable attention network and the global deformable attention network used in the multi-period temporal feature extraction module have the following beneficial technical effects:

[0057] Adaptability: Deformable attention can adaptively adjust the focus region according to the shape changes of the angiography, thereby more accurately capturing the relative changes of key liver tissue and tumor angiography.

[0058] Efficiency: By introducing an offset module to calculate the offset, deformable attention can avoid redundant calculations of irrelevant information and improve computational efficiency.

[0059] In a preferred embodiment, in order to automatically and accurately segment the tumor region and obtain the tumor sequence, a segmentation module is also included, and the segmentation module is used to segment the tumor region in the global CT sequence of the three phases, and the segmented tumor region constitutes the tumor sequence of the three phases. The segmentation module is preferably, but not limited to, the existing SAM segmentation model.

[0060] In a preferred embodiment, it further comprises a data processing module, the data processing module performs: acquiring laboratory data;

[0061] At least one significantly relevant experimental indicator is selected from the laboratory data, and the value of the significantly relevant experimental indicator is converted into a laboratory data vector representation using a value embedding method.

[0062] In this embodiment, the laboratory data include multiple experimental index values ​​such as serum aspartate aminotransferase, alanine aminotransferase, albumin, platelet level, total bilirubin level and alpha-fetoprotein level, and the correlation between each experimental index and MTM HCC subtype and MVI is analyzed. If the correlation is greater than the correlation threshold, the experimental index is considered to be a significantly correlated experimental index for MTM HCC subtype and MVI classification.

[0063] In a preferred embodiment, the data processing module further performs:

[0064] Obtain clinical data, including demographic data, etiology of chronic liver disease and other clinical indicators.

[0065] At least one significantly relevant clinical indicator is selected from the clinical data, and the value of the significantly relevant clinical indicator is converted into a clinical data vector representation using a value embedding method.

[0066] In this embodiment, the correlation between each clinical indicator and MTM HCC subtype and MVI is analyzed. If the correlation is greater than a correlation threshold, the clinical indicator is considered to be a significantly correlated clinical indicator for MTM HCC subtype and MVI classification.

[0067] In this embodiment, the use of data splicing method to perform multimodal fusion of laboratory data, clinical data and imaging data will cause the problem of misalignment. The value embedding method can solve the above problem and realize multimodal data fusion. The obtained laboratory index values, clinical index values, etc. with significant correlation are transformed to obtain vector representation of discrete data, and multimodal fusion is performed with high-dimensional image features to capture the potential relationship of the data to improve model performance. The value embedding expression is as follows, where data represents the laboratory index value or clinical index value, l represents the vector representation to be transformed, which is unified as 512 here, and j represents from 0, 1, 2,..., 255. Discrete data can be converted into vector representation of uniform length by calculation using the following formulas (1) and (2).

[0068]

[0069] The intelligent prediction system for pathological indicators of liver cancer patients provided by the present invention needs to be trained in advance, and the training method includes:

[0070] Step 1, refer to Figure 1 Build the system network structure.

[0071] Construct a tumor spatial feature extraction network and a global spatial feature extraction network in the multi-period spatial feature extraction module. The tumor spatial feature extraction network includes a tumor high-order feature extraction network and a tumor spatial extraction pooling layer connected in sequence, and the global spatial feature extraction network includes a global high-order feature extraction network and a global spatial extraction pooling layer connected in sequence. The tumor high-order feature extraction network can adopt the existing resnet+attention mechanism network, such as the attention basic network based on Resnet as the backbone in the Chinese patent with publication number CN202311005108A, such as the final feature map acquisition network structure disclosed in the Chinese patent with publication number CN202211569881A, and the basic feature extraction network and multi-head attention module disclosed in the Chinese patent with publication number CN202211402478A, which will not be repeated here. The global high-order feature extraction network adopts the existing resnet series model, which will not be repeated here.

[0072] A multi-period temporal feature extraction module is constructed, which includes a tumor deformable attention network and a global deformable attention network.

[0073] Construct a pathological multi-index prediction module. The pathological multi-index prediction module is a multi-task classifier, which is used to simultaneously obtain the MTM HCC subtype prediction results and the MVI classification prediction results. Preferably, the pathological multi-index prediction module processes the multimodal feature data through MLP (multi-layer perceptron) and flattens the processing results to obtain a probability vector, and then obtains the total cross entropy loss loss, loss = γ1*loss1+γ2*loss2. Loss1 represents the cross entropy loss loss1 of MTM HCC subtype prediction, and loss2 represents the cross entropy loss of MVI classification prediction. Of course, the pathological multi-index prediction module can also adopt the multi-task classification model of the convolutional neural network MobileNetV2 disclosed in the Chinese patent with publication number CN202110728236A, which will not be repeated here.

[0074] Step 2: Data collection and dataset construction

[0075] (1) Collection of preoperative patient information. Retrospective collection of HCC patient information: preoperative CT sequence images, clinical information, preoperative laboratory data, and pathological diagnostic indicators. Pathological diagnostic indicators include two indicators: MTM HCC subtype and MVI classification; clinical information includes patient demographic data, etiology of chronic liver disease, etc.; preoperative laboratory data include serum aspartate aminotransferase, alanine aminotransferase, albumin, platelet level, total bilirubin level, and alpha-fetoprotein level. If there are multiple sets of preoperative laboratory data, the set of records with the closest time interval to the patient's CT scan shall prevail.

[0076] (2) Postoperative pathological diagnosis. The pathological diagnostic criteria for MTM HCC are: the thickness of the thick beam structure is greater than or equal to 10 cells, and it involves more than 50% of the tumor; the rest are Non-MTM HCC. The pathological grading criteria for MVI are: M0: no MVI is found; M1: ≤5 MVIs, and they occur in the area of ​​liver tissue near the cancer (≤1 cm); M2: >5 MVIs, or MVIs occur in the area of ​​liver tissue far from the cancer (>1 cm) (see "Guidelines for Pathological Diagnosis of Primary Liver Cancer").

[0077] (3) Inclusion and exclusion criteria

[0078] Inclusion criteria:

[0079] ① Patients who have undergone surgical resection or liver transplantation for hepatocellular carcinoma;

[0080] ② Enhanced CT and other imaging examinations were performed within 2 months before surgery;

[0081] ③ Preoperative laboratory tests performed at the same time as imaging examinations;

[0082] ④ Patients with clinical information and pathological diagnosis results.

[0083] Exclusion criteria:

[0084] ① The patient has received local regional treatment before surgery (including transarterial chemoembolization or radioembolization, radiofrequency ablation, etc.);

[0085] ② Patients with other malignant tumors;

[0086] ③ Patients with poor imaging or pathological images.

[0087] Step 3: Multimodal data preprocessing

[0088] First, preoperative CT sequence image preprocessing.

[0089] (1) Data enhancement. In the deep learning network stage, in order to enhance the generalization ability of the model and expand the data set, the preoperative CT sequence images are enhanced using random rotation, horizontal flipping, random brightness and contrast transformation, scaling and cropping, etc. The preoperative CT sequence images after data enhancement are recorded as global CT sequences.

[0090] (2) Lesion localization

[0091] For each group of patient imaging data, three groups of CT image sequences, namely the arterial phase, venous phase and equilibrium phase, were selected for lesion localization.

[0092] Lesion localization includes precise segmentation of tumor boundaries. An interactive labeling method of human-computer collaboration is used to accurately segment tumor boundaries. The SAM model is currently an advanced medical image segmentation model that can quickly achieve precise lesion localization within a preset bounding box. The preoperative CT sequence images are grouped according to the arterial phase, venous phase, and equilibrium phase. For each group of global CT image sequences, a 3D bounding box of the tumor is manually outlined on the 3D CT sequence images. The powerful data processing and precise segmentation of the SAM model are used to achieve tumor segmentation in 3D sequences. Clinicians then perform fine corrections and reviews on each CT image containing a tumor, ultimately achieving precise tumor localization in each CT image and obtaining a tumor data set, i.e., a tumor sequence.

[0093] Second, clinical data and laboratory data vector conversion

[0094] The laboratory data of each patient is converted into a laboratory data vector representation, and the clinical data of each patient is converted into a clinical data vector representation.

[0095] Third, sample set division

[0096] The global CT sequence, tumor sequence, laboratory data vector representation and clinical data vector representation of each patient are taken together as the data of a sample, and the patient's pathological diagnosis index is used as the sample label of the sample to obtain the sample set. In order to ensure the relative independence between data samples, the sample set is divided into training set, test set and validation set at the patient level in a ratio of 8:1:1.

[0097] Fourth, iterative training and verification

[0098] The cross entropy loss function and stochastic gradient descent algorithm are used to train the deep learning model of the intelligent prediction system of pathological indicators of liver cancer patients provided by the present invention, and the model with the lowest verification loss is selected for testing after continuous iteration. The area under the receiver operating characteristic curve (AUC) is used to quantify the non-invasive diagnostic efficiency of the model for MTM HCC subtypes and MVI classification. Specific performance indicators include sensitivity, specificity, accuracy, positive predictive value (PPV) and negative predictive value (NPV) to evaluate the performance of the model.

[0099] It should be noted that the intelligent prediction system for liver cancer pathology provided by the present invention is not only effective for non-invasive intelligent diagnosis of multiple pathology indicators based on multimodal information of liver cancer, but also for multi-indicator intelligent diagnosis of pathology for diseases such as lung cancer, colon cancer, brain tumors, etc., in combination with multimodal information such as imaging, laboratory data and clinical information.

[0100] The present invention provides an intelligent prediction system for pathological indicators of liver cancer patients. Based on preoperative multimodal information, such as high-dimensional image features, laboratory data, and clinical information, the system uses multi-task learning to simultaneously perform non-invasive intelligent predictions on multiple pathological indicators such as MTM HCC subtypes and MVI. Multi-task learning can better promote the detection accuracy of MTM HCC subtypes and MVI; selecting multiple phases of CT sequences for spatial feature extraction and temporal feature extraction can more comprehensively utilize the three-phase features of CT sequences and fully explore the "fast in and fast out" process of liver metabolism; in addition to the global image, the present invention extracts the detailed features of the tumor sequence separately in order to extract and evaluate the image information more comprehensively.

[0101] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "an implementation", "a preferred implementation" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0102] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. An intelligent prediction system for pathological indicators of liver cancer patients, characterized in that: include: A sequence acquisition module, which acquires three-phase global CT sequences and three-phase tumor sequences segmented from the three-phase global CT sequences, wherein the three phases include an arterial phase, a venous phase, and a balanced phase; The multi-phase spatial feature extraction module obtains the tumor spatial features of each phase based on the tumor sequence of each phase, and obtains the global spatial features of each phase based on the global CT sequence of each phase; The multi-phase temporal feature extraction module uses the tumor spatial features of the three phases to obtain the multi-phase temporal features of the tumor image, and uses the global spatial features of the three phases to obtain the multi-phase temporal features of the global image; A feature splicing module, splicing multi-phase time series features of tumor images, multi-phase time series features of global images, and laboratory data vector representation and / or clinical data vector representation to obtain multimodal feature data; A pathological multi-index prediction module performs multi-task classification processing on the multimodal feature data to obtain MTM HCC subtype prediction results and MVI classification prediction results; The multi-period temporal feature extraction module includes a tumor deformable attention network and a global deformable attention network; The tumor deformable attention network obtains the tumor image time series features of each period based on the tumor spatial features of the three periods, and performs dimensionality reduction processing on the tumor image time series features of the three periods to obtain the tumor image multi-period time series features; The global deformable attention network obtains the global image time series features of each period based on the global spatial features of the three periods, and performs dimensionality reduction processing on the global image time series features of the three periods to obtain the global image multi-period time series features; The tumor deformable attention network includes the tumor first module, the tumor second module, the tumor third module and the tumor pooling layer; The first tumor module is used to map the tumor spatial features of each stage to the value space of that stage; After the second tumor module performs position encoding on each feature point in the tumor spatial feature of each period, it uses the first linear mapping to learn multiple offsets for the feature points of the tumor spatial feature of each period, and uses the second linear mapping to learn the attention weights of the multiple offsets in the value space of each period; The third tumor module is used to obtain the temporal features of tumor images of each period. The feature value of each feature point of the temporal features of tumor images of each period is obtained by multiplying the multiple attention weights of the feature point with the mapping value of the feature point in the corresponding value space; The tumor pooling layer performs dimensionality reduction processing on the tumor image time series features of the three phases to obtain the multi-phase time series features of the tumor images; The global deformable attention network includes a global first module, a global second module, a global third module and a global pooling layer; The global first module is used to map the global spatial features of each period to the value space of that period; After the global second module performs position encoding on each feature point in the global spatial feature of each period, it uses the first linear mapping to learn multiple offsets for the feature points of the global spatial feature of each period, and uses the second linear mapping to learn the attention weights of the multiple offsets in the value space of each period; The global third module is used to obtain the global image time series features of each period. The feature value of each feature point of the global image time series features of each period is obtained by multiplying the multiple attention weights of the feature point with the mapping value of the feature point in the corresponding value space; The global pooling layer performs dimensionality reduction processing on the global image time series features of the three periods to obtain the global image multi-period time series features.

2. The intelligent prediction system for pathological indicators of liver cancer patients according to claim 1, characterized in that: The multi-phase spatial feature extraction module includes a tumor spatial feature extraction network and a global spatial feature extraction network; The tumor spatial feature extraction network extracts the tumor spatial features of each stage from the tumor sequence of each stage respectively; The global spatial feature extraction network extracts the global spatial features of each period from the global CT sequence of each period.

3. The intelligent prediction system for pathological indicators of liver cancer patients according to claim 2, characterized in that: The tumor spatial feature extraction network comprises: Tumor high-order feature extraction network, used to extract high-order features of tumor sequences in each phase, recorded as tumor high-order features; A tumor space extraction pooling layer compresses the high-order features of the tumor in each stage into the tumor space features of each stage; and / or, The global spatial feature extraction network comprises: The global high-order feature extraction network is used to extract the high-order features of the global CT sequence of each period, which are recorded as global high-order features; The global space extraction pooling layer compresses the global high-order features of each period into the global space features of each period.

4. The intelligent prediction system for pathological indicators of liver cancer patients according to claim 1, characterized in that: It also includes a segmentation module, which is used to segment the tumor area in the global CT sequence of the three phases, and the segmented tumor area constitutes the tumor sequence of the three phases.

5. The intelligent prediction system for pathological indicators of liver cancer patients according to claim 4, characterized in that: It also includes a data processing module, which performs: Obtain laboratory data; At least one significantly relevant experimental indicator is selected from the laboratory data, and the value of the significantly relevant experimental indicator is converted into a laboratory data vector representation using a value embedding method.

6. The intelligent prediction system for pathological indicators of liver cancer patients according to claim 5, characterized in that: The data processing module also performs: Obtain clinical data; At least one significantly relevant clinical indicator is selected from the clinical data, and the value of the significantly relevant clinical indicator is converted into a clinical data vector representation using a value embedding method.

7. The intelligent prediction system for pathological indicators of liver cancer patients according to claim 5, characterized in that: The pathological multi-index prediction module performs multi-task prediction by a multi-layer perceptron, wherein the multi-task prediction includes MTM HCC subtype prediction and MVI classification prediction, wherein the MTM HCC subtype prediction results include MTM HCC subtype and NonMTM HCC subtype, and the MVI classification prediction results include M0, M1 and M2 grades; During the training process of the intelligent prediction system for pathological indicators of liver cancer patients, the total loss includes the cross-entropy loss of MTM HCC subtype prediction and the cross-entropy loss of MVI classification prediction.

Images