Pancreatic disease non-invasive evaluation method and system based on multi-modal data

Abstract

The invention provides a multi-modal data-based pancreatic disease non-invasive evaluation method and system. The method comprises the following steps: acquiring an ultrasonic endoscope image and enhanced CT image data; lesion type identification is carried out on the ultrasonic endoscope image; determining an image acquisition state corresponding to the CT image data; the method comprises the following steps: extracting dicom image sequences of an arterial phase, a portal phase and a delayed phase from CT image data, converting the dicom image sequences into image volume data, identifying pancreatic focus area volume data from each volume data, traversing focus area coordinates of the pancreatic focus area volume data, extracting ROI from each dicom image, and extracting statistical characteristics, morphological characteristics, textural characteristics and multiphase enhancement characteristics of each ROI; constructing a feature vector according to the lesion type, each feature of the lesion ROI and the image acquisition state; and classifying the feature vectors by adopting a classification network model to output multi-modal data as the prediction probability of autoimmune pancreatitis and pancreatic cancer, thereby realizing non-invasive auxiliary diagnosis of pancreatic diseases.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a non-invasive assessment method and system for pancreatic diseases based on multimodal data. Background Technology

[0002] Due to the anatomical proximity and shared embryonic origin of the biliary system and pancreas, they share many similarities in their disease spectrum. Beginners face significant challenges in differentiating the biliary tract and pancreas using endoscopic ultrasound, and the complex anatomy and diverse disease types of the biliary system and pancreas make identification of biliary and pancreatic diseases difficult. Furthermore, autoimmune pancreatitis can present on endoscopic ultrasound as focal hypoechoic masses and pancreatic duct stenosis, similar to pancreatic cancer, and these features can "masquerade" each other on ultrasound imaging, making it difficult to accurately distinguish between pancreatic cancer and autoimmune pancreatitis using endoscopic ultrasound images alone.

[0003] Therefore, achieving non-invasive assessment and auxiliary diagnosis of pancreatic diseases has significant clinical implications and application value. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide a non-invasive assessment method and system for pancreatic diseases based on multimodal data to overcome the above problems.

[0005] This invention provides a non-invasive assessment method for pancreatic diseases based on multimodal data, the method comprising: Acquire the multimodal data to be analyzed, including endoscopic ultrasound images and enhanced CT images; The lesion type is identified from the endoscopic ultrasound images; The gallbladder region is identified in the enhanced CT image data, and the image acquisition status corresponding to the current enhanced CT image data is determined based on the volume of the gallbladder region. The image acquisition status includes fasting status and non-fasting status. DICOM image sequences of the arterial phase, portal venous phase, and delayed phase are sequentially extracted from enhanced CT image data and converted into image volume data respectively. The contours of pancreatic lesion regions are identified from the volume data corresponding to the arterial phase, portal venous phase, and delayed phase, and pancreatic lesion region volume data are obtained. Based on the coordinate matching relationship between pancreatic lesion region volume data and image volume data, the coordinates of lesion regions in pancreatic lesion region volume data are traversed to extract lesion connected regions from the DICOM images of each DICOM image sequence and take the lesion connected regions as the region of interest (ROI) of the lesion. Statistical features, morphological features, texture features, and multiphase enhancement features of the lesion ROI are extracted. Based on the lesion type, the statistical characteristics, morphological characteristics, texture characteristics, and multiphase enhancement characteristics of the lesion ROI, and the image acquisition status, a feature vector of multimodal data is constructed. A pre-trained classification network model is used to classify the feature vectors, and the output multimodal data is used as the predicted probability of autoimmune pancreatitis and pancreatic cancer, so as to realize non-invasive auxiliary diagnosis of pancreatic diseases.

[0006] Another aspect of the present invention provides a non-invasive assessment system for pancreatic diseases based on multimodal data, the system comprising: The data acquisition module is used to acquire the multimodal data to be analyzed, including endoscopic ultrasound images and enhanced CT images. An ultrasound lesion identification module is used to identify the type of lesion in the ultrasound endoscopic images; The acquisition status detection module is used to identify the gallbladder region in the enhanced CT image data and determine the image acquisition status corresponding to the current enhanced CT image data based on the volume of the gallbladder region. The image acquisition status includes fasting status and non-fasting status. The feature extraction module is used to sequentially extract DICOM image sequences of the arterial phase, portal venous phase, and delayed phase from enhanced CT image data and convert them into image volume data respectively. The contours of pancreatic lesion regions are identified from the volume data corresponding to the arterial phase, portal venous phase, and delayed phase to obtain pancreatic lesion region volume data. Based on the coordinate matching relationship between the pancreatic lesion region volume data and the image volume data, the coordinates of the lesion region in the pancreatic lesion region volume data are traversed to extract the lesion connected regions from the DICOM images of each DICOM image sequence and take the lesion connected regions as the lesion region of interest (ROI). The PyRadiomics library is used to extract the statistical features, morphological features, texture features, and multiphase enhancement features of the lesion ROI. The feature vector construction module is used to construct feature vectors for multimodal data based on the lesion type, statistical characteristics, morphological characteristics, texture characteristics, and multiphase enhancement characteristics of the lesion ROI, as well as the image acquisition status. The classification and prediction module is used to classify the feature vector using a pre-trained classification network model, so as to output multimodal data as the predicted probability of autoimmune pancreatitis and pancreatic cancer, thereby realizing non-invasive auxiliary diagnosis of pancreatic diseases.

[0007] This invention provides a non-invasive assessment method and system for pancreatic diseases based on multimodal data. By fusing EUS image features and CT image features, a non-invasive auxiliary diagnostic method for pancreatic diseases based on multimodal data is constructed. This method can quickly and accurately obtain assessment results for pancreatic diseases, realize non-invasive assessment and auxiliary diagnosis of pancreatic diseases, and has important clinical application value.

[0008] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0009] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings: Figure 1 This is a flowchart of a non-invasive assessment method for pancreatic diseases based on multimodal data, according to an embodiment of the present invention. Figure 2 This is a structural block diagram of a non-invasive pancreatic disease assessment system based on multimodal data, according to an embodiment of the present invention. Detailed Implementation

[0010] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0011] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art and should not be interpreted in an idealized or overly formal sense unless specifically defined.

[0012] This invention provides a non-invasive assessment method for pancreatic diseases based on multimodal data, such as... Figure 1 As shown, the non-invasive assessment method for pancreatic diseases based on multimodal data proposed in this invention includes the following steps: S11. Acquire the multimodal data to be analyzed, including endoscopic ultrasound images and enhanced CT images. The endoscopic ultrasound images and enhanced CT images are examination data collected from the same patient at the same examination stage.

[0013] S12. Identify the lesion type in the endoscopic ultrasound image.

[0014] S13. Identify the gallbladder region in the enhanced CT image data and determine the image acquisition status corresponding to the current enhanced CT image data based on the volume of the gallbladder region. The image acquisition status includes fasting status and non-fasting status. S14. Extract the DICOM image sequences of the arterial phase, portal venous phase, and delayed phase from the enhanced CT image data and convert them into image volume data respectively. Identify the pancreatic lesion region contour from the volume data corresponding to the arterial phase, portal venous phase, and delayed phase to obtain pancreatic lesion region volume data. Based on the coordinate matching relationship between the pancreatic lesion region volume data and the image volume data, traverse the lesion region coordinates in the pancreatic lesion region volume data to extract the lesion connected regions from the DICOM images of each DICOM image sequence and take the lesion connected regions as the lesion region of interest (ROI). Extract the statistical features, morphological features, texture features, and multiphase enhancement features of the lesion ROI. S15. Construct a feature vector for multimodal data based on the lesion type, statistical characteristics, morphological characteristics, texture characteristics, and multiphase enhancement characteristics of the lesion ROI, as well as the image acquisition status. S16. The feature vector is classified using a pre-trained classification network model to output multimodal data as the predicted probability of autoimmune pancreatitis and pancreatic cancer, thereby achieving non-invasive auxiliary diagnosis of pancreatic diseases.

[0015] This invention provides a non-invasive assessment method for pancreatic diseases based on multimodal data. By fusing EUS image features and CT image features, a non-invasive auxiliary diagnostic method for pancreatic diseases based on multimodal data is constructed. This method can quickly and accurately obtain assessment results for pancreatic diseases, realize non-invasive assessment and auxiliary diagnosis of pancreatic diseases, and has important clinical application value.

[0016] In this embodiment of the invention, the acquisition method of the endoscopic ultrasound images in the multimodal data to be analyzed includes: acquiring digestive endoscopy images and identifying digestive tract sites in the digestive endoscopy images; guiding the endoscopic ultrasound to acquire endoscopic ultrasound images when a specified digestive tract site is identified; identifying gallbladder and pancreas sites in the acquired endoscopic ultrasound images to select endoscopic ultrasound images containing complete gallbladder and pancreas sites.

[0017] In this embodiment of the invention, a pre-trained digestive tract region recognition model can be used to identify digestive tract regions. The training process of the digestive tract region recognition model is as follows: The system acquires image data from historical gastroscopy examinations, manually classifies the image regions in the gastroscopy images, including: esophagus, cardia, gastric angle, gastric body, gastric fundus, gastric antrum, pylorus, duodenal bulb, and descending duodenum, and trains a RestNet classification network model to achieve classification and recognition of gastroscopy images.

[0018] In this embodiment of the invention, a pre-trained endoscopic ultrasound image gallbladder and pancreas region recognition model can be used to achieve gallbladder and pancreas region recognition. The training process of the endoscopic ultrasound image gallbladder and pancreas region recognition model is as follows: Acquire historical endoscopic ultrasound (EUS) images from gastroscopy examinations. Artificial contour annotations are performed on the gallbladder and pancreas regions within the EUS images. The segmentation regions are categorized as: pancreatic head, pancreatic body, pancreatic tail, pancreatic uncinate process, common bile duct, and gallbladder body. A segmentation network model is trained by adding the annotated gallbladder and pancreatic region data to the training set, enabling the model to segment and recognize the gallbladder and pancreas regions.

[0019] In this embodiment, the white light image of the digestive endoscope is first identified, and the internal location of the stomach is determined. After entering the designated digestive tract location within the stomach, the EUS (Endoscopic Ultrasound) endoscope is activated to observe pancreatic lesions. The ultrasound image gallbladder and pancreatic region recognition model is invoked to identify the ultrasound image in real time. By recognizing the corresponding internal location of the stomach, incorrectly identified areas can be filtered out. For example, in the duodenal bulb, only the pancreatic head and common bile duct can be observed; in the descending duodenum, only the pancreatic head, uncinate process, and common bile duct can be observed; and in the antrum, the pancreatic neck and common bile duct can be seen. In a specific example, the "internal location of the stomach" in the digestive endoscope image can determine which part of the stomach the current endoscope image belongs to. For example, if it is the duodenal bulb, the ultrasound image on the left side can only identify the pancreatic head and common bile duct based on medical experience. If the "gallbladder and pancreatic region recognition" model identifies other regions, it is judged as a misidentification and filtered out. Then, steps S12-S16 are called sequentially to obtain multimodal data for non-invasive auxiliary diagnosis of autoimmune pancreatitis and pancreatic cancer.

[0020] In this embodiment of the invention, step S12, which involves identifying the lesion type of the endoscopic ultrasound image, includes using a pre-trained pancreatic lesion identification model based on endoscopic ultrasound images to identify the lesion type.

[0021] Specifically, the training process of the pancreatic lesion recognition model in endoscopic ultrasound images is as follows: Acquire historical endoscopic ultrasound (EUS) images from gastroscopy examinations, including data on pancreatic cancer, autoimmune pancreatitis, other pancreatic diseases, and normal images without disease. Manually annotate the lesion regions in the EUS images, categorizing lesions into pancreatic cancer, autoimmune pancreatitis, and other pancreatic diseases. Train a segmentation network model using the annotated data and the disease-free data as background images. The trained segmentation network model will segment lesion regions and identify lesion types.

[0022] In this embodiment of the invention, step S13, which involves identifying the gallbladder region in the enhanced CT image data and determining the image acquisition status corresponding to the current enhanced CT image data based on the volume of the gallbladder region, includes the following steps (not shown in the accompanying drawings): S131. A pre-trained gallbladder location recognition model is used to identify the enhanced CT image data to identify the gallbladder region. S132. Estimate the gallbladder volume based on the area of ​​the gallbladder region. When the gallbladder volume is greater than a preset volume threshold, determine that the image acquisition state corresponding to the current enhanced CT image data is a fasting state; otherwise, the image acquisition state corresponding to the current enhanced CT image data is a non-fasting state. The training steps for the gallbladder location recognition model include: annotating the gallbladder region in CT image data from a preset abdominal enhanced CT image dataset to obtain a training dataset; and using the training dataset to train a preset nnU-Net 3D convolutional network to obtain the gallbladder location recognition model. Specifically, abdominal enhanced CT data is collected, the gallbladder region is delineated and annotated, the nnU-Net 3D convolutional network is used to train the gallbladder location recognition model, and the gallbladder volume is calculated for the identified gallbladder region. The gallbladder's shape is clearly visible when the body is fasting. Based on historical data, the gallbladder volume in adults is approximately 54 mL when fasting and approximately 21 mL after a meal. The difference in gallbladder volume determines whether the enhanced CT scan was taken in a fasting state. In an optional embodiment, the volume threshold can be set to 40 ml. When the gallbladder volume is greater than the preset volume threshold of 40 ml, the current abdominal enhanced CT image acquisition is determined to be in a fasting state.

[0023] Fasting is required before a contrast-enhanced CT scan of the pancreas to avoid irritating the gastrointestinal tract. However, some patients do not follow this fasting principle, leading to increased gastric motility or gas production, which affects the distribution and imaging effect of the contrast agent in the pancreatic region, thus impacting the accuracy of pancreatic disease diagnosis. Therefore, this embodiment of the invention also considers the influence of the image acquisition status when extracting features from the contrast-enhanced CT image data, pre-determining the image acquisition status corresponding to the current contrast-enhanced CT image data based on the volume of the gallbladder area.

[0024] The arterial, portal venous, and delayed phase DICOM image sequences were sequentially extracted from enhanced CT image data and converted into image volume data. The pancreatic lesion region contours were identified from the volume data corresponding to the arterial, portal venous, and delayed phases to obtain pancreatic lesion region volume data. Based on the coordinate matching relationship between the pancreatic lesion region volume data and the image volume data, the lesion region coordinates in the pancreatic lesion region volume data were traversed to extract the lesion connected regions from the DICOM images of each DICOM image sequence and take the lesion connected regions as the lesion region of interest (ROI). Based on the preset radiomics feature extraction library PyRadiomics, the statistical features, morphological features, texture features, and multiphase enhancement features of the lesion ROI were extracted. Unlike the lungs and liver, the pancreas has several unique characteristics, including: 1) Complex anatomical shape: it is elongated, asymmetrical, and has blurred boundaries, adjacent to the duodenum, stomach, and blood vessels. 2) Significant enhancement characteristics: dynamic contrast enhancement differences (arterial phase, portal venous phase, delayed phase) can reflect the difference between inflammation and cancer. 3) Heterogeneous texture: pancreatic tissue is dense; autoimmune pancreatitis often shows diffuse and homogeneous enhancement, while pancreatic cancer is mostly focal and hypo-enhanced with blurred boundaries. 4) Susceptibility to adjacent structures: pancreatic duct dilation, bile duct stenosis, surrounding fat infiltration, and vascular invasion can all affect radiomics characteristics. Based on these characteristics, this invention employs a segmentation model and image analysis to extract features from pancreatic lesions on CT images, and combines this with a subsequent classification model to classify lesions of autoimmune pancreatitis and pancreatic cancer.

[0025] Specifically as follows: In the segmentation model stage: historical enhanced CT scan data of autoimmune pancreatitis and pancreatic cancer were collected first. Dicom images of three enhanced sequences (arterial phase, portal venous phase, and delayed phase) were extracted and converted into .nii.gz volume data format. The pancreatic lesion region was labeled using the 3DSlicer annotation tool, with only one category: pancreatic lesion. The open-source nnU-Net 3D convolutional network was used to train the pancreatic lesion segmentation model.

[0026] In the feature extraction stage: arterial phase, portal venous phase, and delayed phase sequence DICOM images are sequentially extracted from an enhanced CT scan and converted into volumetric data format .nii.gz. Then, the pancreatic lesion segmentation model is called to infer and identify the lesions, resulting in volumetric data mask.nii.gz of the pancreatic lesion segmentation contour identified in each enhanced sequence. In this data, the pixel value of the pancreatic lesion is 1, and the pixel value of the non-lesion is 0.

[0027] Since the volume data of mask.nii.gz matches the volume data coordinates of image.nii.gz, by traversing the coordinates of lesions with a pixel value of 1 in the volume data of mask.nii.gz, each connected region of the lesion is extracted from the image data as a ROI. Statistical, morphological and textural features are extracted from the ROI using the standard radiomics feature extraction library PyRadiomics.

[0028] In this embodiment of the invention, the extraction of statistical features, morphological features, texture features, and multiphasic enhancement features of each lesion ROI specifically includes: The average CT value for each Region of Interest (ROI) is calculated to obtain the statistical characteristics of the CT grayscale distribution in the lesion region. Specifically, the average pixel CT value of each ROI is extracted to obtain first-order statistical features that reflect the CT grayscale distribution characteristics.

[0029] The three-dimensional geometric attributes of each Region of Interest (ROI) are extracted, and its volume, surface area, compactness, elongation, sphericity, major axis dimension, and minor axis dimension are calculated to obtain the morphological characteristics of the lesion region. Autoimmune pancreatitis lesions are diffuse, with high elongation and low sphericity; pancreatic cancer lesions are localized, nodular, and have relatively high compactness. Therefore, the morphological features that can be extracted include the volume, surface area, compactness, elongation, sphericity, major axis, and minor axis information of the lesion ROI.

[0030] Indicators representing local texture homogeneity were extracted, including contrast (Contrast), inverse variance (Homogeneity), energy (Energy), and entropy (Contrast). In pancreatic lesions, GLCM second-order texture features can reflect the spatial correlation and heterogeneity of tissue gray levels. In autoimmune pancreatitis, due to the relatively uniform distribution of inflammation and fibrosis, texture homogeneity is increased (Homogeneity and Energy are elevated), while texture complexity is decreased (Entropy and Contrast are reduced). Conversely, pancreatic cancer, due to significant tumor necrosis, uneven fibrosis, and substantial tissue structural destruction, exhibits marked texture heterogeneity (Entropy and Contrast are elevated), while homogeneity is decreased (Homogeneity and Energy are reduced).

[0031] The average CT value of each ROI is extracted, and the average CT value in the arterial phase and the average CT value in the delayed phase are calculated. The multiphasic enhancement feature ΔHU_delay-art is then calculated based on the average CT value in the arterial phase and the average CT value in the delayed phase, as shown in the following formula: ΔHU_delay-art=HUart-HUdelay, where represents the enhancement change from the arterial phase to the delayed phase.

[0032] In this embodiment, multi-phase features reflect the enhancement patterns and hemodynamic characteristics of the lesion and surrounding tissues during the dynamic distribution of contrast agent, and can be used to differentiate between benign and malignant lesions, tumor grades, or assess treatment response. The average CT value of the lesion's ROI is extracted. HUart represents the average CT value in the arterial phase, HUdelay represents the average CT value in the delayed phase, and ΔHU_delay-art represents the multi-phase enhancement feature, indicating the enhancement change from the arterial phase to the delayed phase. Pancreatic cancer typically shows low enhancement in the arterial phase and mild enhancement in the delayed phase, while autoimmune pancreatitis exhibits a "delayed enhancement" characteristic.

[0033] In this embodiment of the invention, the training steps of the classification network model include: A multimodal data sample dataset was constructed, which includes endoscopic ultrasound image samples, enhanced CT image data samples, and corresponding pancreatic disease labels.

[0034] The lesion type characteristics of the sample are obtained based on the endoscopic ultrasound image sample.

[0035] Acquire the sample image acquisition status of enhanced CT image data samples.

[0036] Statistical features, morphological features, texture features, and multiphase enhancement features of the lesion ROIs in the enhanced CT image data samples are obtained.

[0037] The sample feature vector is constructed based on the characteristics of the sample lesion type, the statistical characteristics, morphological characteristics, texture characteristics, and multiphase enhancement characteristics of the sample lesion ROI, as well as the sample image acquisition status.

[0038] A classification model training set is constructed based on the feature vectors of each multimodal data sample and the corresponding pancreatic disease label. A pre-defined machine learning model is then trained on this training set to predict the probability of the multimodal data sample being autoimmune pancreatitis or pancreatic cancer based on the input feature vectors. In this embodiment, the pre-defined machine learning model is implemented using the XGBoost algorithm.

[0039] Furthermore, after constructing the sample feature vector based on the characteristics of the sample lesion type, the statistical characteristics, morphological characteristics, texture characteristics, and multiphase enhancement characteristics of the sample lesion ROI, as well as the sample image acquisition status, the method further includes: correcting the sample feature vector by using the sample features in the fasting state in the sample image acquisition status as the standard, and correcting the sample features in the non-fasting state to the feature distribution state in the fasting state.

[0040] In this embodiment of the invention, the feature parameters of each sample are compiled into a single CSV (data.csv) to form a sample feature vector. Fields include the lesion type obtained from the endoscopic ultrasound image sample, represented by 0 or 1 to indicate autoimmune pancreatitis or pancreatic cancer. The feature parameters also include numerical features from CT, including the average CT value per pixel for each ROI, the volume, surface area, compactness, elongation, sphericity, major axis, and minor axis of the lesion ROI, and indices representing local texture uniformity: Contrast, Homogeneity, Energy, Entropy, and ΔHU_delay-art. Furthermore, the image acquisition status of the enhanced CT image data samples is added as a fasting variable (fasting) as a regular feature in XGBoost training, represented by 1 (fasting is represented by 0), allowing the model to automatically learn the impact of this status on the features.

[0041] Before training the XGBoost model, ComBat batch effect correction was performed on the feature matrix. "Hungry" features were used as the standard, and the feature distribution of "non-hungry samples" was corrected to match that of "hungry samples" to reduce the impact of feeding status on the extracted ROI features (CT value, morphology, texture). ComBat batch effect correction was implemented using the neuroCombat library in Python. It was assumed that each "batch" (hungry / non-hungry) would introduce a systematic bias to the mean and variance of the features. This bias was estimated and removed using an empirical Bayesian method, aligning the data from different batches to obtain "corrected non-hungry features" from the hungry data.

[0042] The specific implementation is as follows: Construct feature vectors for each sample. The feature vector obtained by the segmentation model for each CT image sample includes: the average CT value per pixel for each ROI, the volume, surface area, compactness, elongation, sphericity, major axis, minor axis, and indices representing local texture uniformity: Contrast, Homogeneity, Energy, Entropy, and ΔHU_delay-art (average CT value in the arterial phase minus the average CT value in the delayed phase). Assuming that each batch (fasting and non-fasting) introduces systematic bias into the feature mean and variance, the model assumptions are as follows: Xij=αbatch[i]+βbatch[i]×δij+εij; Xij represents the observed value of the i-th sample on the j-th feature; δij represents the "actual value" of the i-th sample on the j-th feature, which is the "true distribution" that the model attempts to recover (the true biological signal after batch effect correction, removing batch bias); αbatch represents the batch mean shift; βbatch represents the batch variance scaling; and εij represents the in-sample random error. ComBat models and adjusts the batch effect parameters using Bayesian estimation, aligning all sample features to the target batch (empty sample) distribution.

[0043] The non-fasting samples were corrected using the above method to obtain the corrected sample feature vectors, which were then used to train the XGBoost machine learning model to train the classification network and output the predicted probabilities of two pancreatic diseases.

[0044] Furthermore, the method also includes: The classification network model is validated based on the constructed validation sample set, and the validation set results are collected. The validation set results include the prediction scores of the validation samples for autoimmune pancreatitis and pancreatic cancer, as well as the true labels of the validation samples. A temperature model with a single parameter T is established to scale and calibrate the predicted score of each validation sample. A negative log-likelihood loss function is constructed to measure the difference between the calibrated probability and the true label. The gradient descent algorithm is used to optimize and adjust the value of T to minimize the loss of the validation sample set, thus obtaining an optimal temperature value T. When calling the classification network model to perform classification prediction on the multimodal data to be analyzed, a temperature model is used to scale the prediction score, and the normalized score is then used to obtain the calibrated prediction probability.

[0045] In this embodiment, a post-calibrated temperature scaling model is introduced after the trained classification network model to address the "overconfidence" problem. Temperature scaling makes the model's output probability more consistent with the actual risk probability distribution. Temperature scaling is a single-parameter model that does not affect the classification results of the XGBoost machine learning model (the class argmax remains unchanged). After the classification model outputs a score, this single-parameter temperature scaling module (T) is added. By minimizing the loss function NLL on the validation set, the optimal T is automatically learned, improving the confidence consistency of the classification model's output probability and making the model's output probability more consistent with the actual confidence distribution. In deep learning or machine learning classification models, the final step is usually softmax or sigmoid to convert the "score" (logits) into a probability. The temperature T appears in this step: this invention divides the model's logits by T before performing softmax or sigmoid calculation. Before calibration: when the model predicts p = 0.9, the proportion of patients who are actually positive is only 0.75; after calibration: after adjusting T, the true probability when predicting p = 0.75 is closer to 75%. The specific implementation is as follows: 1) Collect validation set results: the validation set's logits = log(p / (1 - p)), and the validation set's true labels. 2) Define the calibration model: establish a temperature model containing only a single parameter T, which scales (divides by T) the logits of each sample. 3) Define the optimization objective: set the loss function as negative log-likelihood (NLL) to measure the difference between the calibrated probability and the true label. 4) Optimize T: repeatedly adjust the value of T using the gradient descent algorithm to minimize the NLL of the validation set, i.e., the calibrated probability distribution best matches the true label, obtaining an optimal temperature value T and saving it as a temperature.pt file. 5) Apply T calibration: when calling XGBoost inference, divide the logits of the test set or new sample by T, and then apply softmax or sigmoid to obtain the calibrated probability. The calibrated probability is closer to the true risk level, and the confidence level is more reliable.

[0046] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0047] This invention provides a non-invasive assessment system for pancreatic diseases based on multimodal data. The system includes functional modules for implementing the non-invasive assessment method for pancreatic diseases based on multimodal data as described in any of the preceding embodiments. Figure 2 The schematic diagram illustrates the structure of a non-invasive pancreatic disease assessment system based on multimodal data provided in an embodiment of the present invention. (Refer to...) Figure 2 The system described in this embodiment of the invention includes: The data acquisition module 201 is used to acquire the multimodal data to be analyzed, including endoscopic ultrasound images and enhanced CT image data. Ultrasound lesion identification module 202 is used to identify the lesion type in the ultrasound endoscopic image; The acquisition status detection module 203 is used to identify the gallbladder location in the enhanced CT image data and determine the image acquisition status corresponding to the current enhanced CT image data based on the volume of the gallbladder location. The image acquisition status includes fasting status and non-fasting status. The feature extraction module 204 is used to sequentially extract the DICOM image sequences of the arterial phase, portal venous phase, and delayed phase from the enhanced CT image data and convert them into image volume data respectively. The contours of pancreatic lesion regions are identified from the volume data corresponding to the arterial phase, portal venous phase, and delayed phase to obtain the volume data of pancreatic lesion regions. Based on the coordinate matching relationship between the volume data of pancreatic lesion regions and the image volume data, the coordinates of the lesion regions in the volume data of pancreatic lesion regions are traversed to extract the lesion connected regions from the DICOM images of each DICOM image sequence and take the lesion connected regions as the regions of interest (ROIs) of the lesion. The statistical features, morphological features, texture features, and multiphase enhancement features of the lesion ROI are extracted from the PyRadiomics library. The feature vector construction module 205 is used to construct feature vectors for multimodal data based on the lesion type, statistical features, morphological features, texture features, and multiphase enhancement features of the lesion ROI, as well as the image acquisition status. The classification prediction module 206 is used to classify the feature vector using a pre-trained classification network model, so as to output multimodal data as the predicted probability of autoimmune pancreatitis and pancreatic cancer, thereby realizing non-invasive auxiliary diagnosis of pancreatic diseases.

[0048] In this embodiment of the invention, the system further includes a training set construction module and a model training module (not shown in the accompanying drawings), wherein: The training set construction module is used to construct a multimodal data sample dataset, which includes multimodal data samples such as endoscopic ultrasound images, enhanced CT images, and corresponding pancreatic disease labels. The ultrasound lesion identification module is also used to obtain lesion type characteristics of the sample based on the ultrasound endoscopic image sample; The acquisition status detection module is also used to acquire the sample image acquisition status of enhanced CT image data samples. The feature extraction module is also used to obtain the statistical features, morphological features, texture features and multiphase enhancement features of the lesion ROI from the enhanced CT image data sample; The feature vector construction module is also used to construct sample feature vectors based on the sample lesion type characteristics, the statistical characteristics, morphological characteristics, texture characteristics, and multiphase enhancement characteristics of the sample lesion ROI, as well as the sample image acquisition status. The model training module is used to construct a classification model training set based on the sample feature vectors of each multimodal data sample and the corresponding pancreatic disease labels. Based on the classification model training set, a classification network model is trained on a preset machine learning model so that the classification network model outputs the predicted probability of the multimodal data sample as autoimmune pancreatitis and pancreatic cancer based on the input sample feature vector.

[0049] Furthermore, the system also includes a feature correction module (not shown in the accompanying drawings). The feature correction module is used to correct the sample feature vector after constructing the sample feature vector based on the sample lesion type characteristics, the statistical characteristics, morphological characteristics, texture characteristics, and multiphase enhancement characteristics of the sample lesion ROI, as well as the sample image acquisition state. The feature correction module uses the sample features acquired in the fasting state as the standard to correct the sample features in the non-fasting state to the feature distribution state of the fasting state.

[0050] Furthermore, the model training module is also used to validate the classification network model based on the constructed validation sample set and collect the validation set results. The validation set results include the prediction scores of the validation samples as autoimmune pancreatitis and pancreatic cancer, as well as the true labels of the validation samples. The system also includes a model calibration module, which is used to establish a temperature model containing a single parameter T. The temperature model is used to scale and calibrate the predicted score of each validation sample, and to construct a negative log-likelihood loss function. The loss function is used to measure the difference between the calibrated probability and the true label. The gradient descent algorithm is used to optimize and adjust the value of T to minimize the loss of the validation sample set, thereby obtaining an optimal temperature value T. The model training module is also used to scale the prediction score of the temperature model when calling the classification network model to perform classification prediction on the multimodal data to be analyzed, and to obtain the calibrated prediction probability by normalizing the scaled score.

[0051] As the system implementation is basically similar to the method implementation, the description is relatively simple, and relevant parts can be found in the description of the method implementation.

[0052] Furthermore, another embodiment of the present invention provides a computer program product storing a computer program that, when executed by a processor, implements the steps described in the above embodiment of the non-invasive assessment method for pancreatic diseases based on multimodal data, for example... Figure 1 Steps S11-S16 are shown.

[0053] Furthermore, another embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the steps described in the above embodiment of the non-invasive assessment method for pancreatic diseases based on multimodal data. Figure 1 Steps S11-S16 are shown.

[0054] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, any of the claimed embodiments can be used in any combination.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A non-invasive assessment method for pancreatic diseases based on multimodal data, characterized in that, The method includes: Acquire the multimodal data to be analyzed, including endoscopic ultrasound images and enhanced CT images; The lesion type is identified from the endoscopic ultrasound images; The gallbladder region is identified in the enhanced CT image data, and the image acquisition status corresponding to the current enhanced CT image data is determined based on the volume of the gallbladder region. The image acquisition status includes fasting status and non-fasting status. DICOM image sequences of the arterial phase, portal venous phase, and delayed phase are sequentially extracted from enhanced CT image data and converted into image volume data respectively. The contours of pancreatic lesion regions are identified from the volume data corresponding to the arterial phase, portal venous phase, and delayed phase, and pancreatic lesion region volume data are obtained. Based on the coordinate matching relationship between pancreatic lesion region volume data and image volume data, the coordinates of lesion regions in pancreatic lesion region volume data are traversed to extract lesion connected regions from the DICOM images of each DICOM image sequence and take the lesion connected regions as the region of interest (ROI) of the lesion. Statistical features, morphological features, texture features, and multiphase enhancement features of the lesion ROI are extracted. Based on the lesion type, the statistical characteristics, morphological characteristics, texture characteristics, and multiphase enhancement characteristics of the lesion ROI, and the image acquisition status, a feature vector of multimodal data is constructed. A pre-trained classification network model is used to classify the feature vectors, and the output multimodal data is used as the predicted probability of autoimmune pancreatitis and pancreatic cancer, so as to realize non-invasive auxiliary diagnosis of pancreatic diseases.

2. The method according to claim 1, characterized in that, The training steps for a classification network model include: A multimodal data sample dataset was constructed, which includes endoscopic ultrasound image samples, enhanced CT image data samples, and corresponding pancreatic disease labels. Based on the endoscopic ultrasound image samples, the characteristics of the lesion type in the samples are obtained; Acquire the sample image acquisition status of enhanced CT image data samples; Statistical features, morphological features, texture features, and multiphase enhancement features of the lesion ROIs in the enhanced CT image data samples are obtained. A sample feature vector is constructed based on the characteristics of the sample lesion type, the statistical characteristics, morphological characteristics, texture characteristics, and multiphase enhancement characteristics of the sample lesion ROI, as well as the sample image acquisition status. A classification model training set is constructed based on the sample feature vectors of each multimodal data sample and the corresponding pancreatic disease labels. A classification network model is trained on the preset machine learning model based on the classification model training set, so that the classification network model outputs the predicted probability of the multimodal data sample as autoimmune pancreatitis and pancreatic cancer based on the input sample feature vector.

3. The method according to claim 2, characterized in that, After constructing the sample feature vector based on the characteristics of the sample lesion type, the statistical characteristics, morphological characteristics, texture characteristics, and multiphasic enhancement characteristics of the sample lesion ROI, and the sample image acquisition status, the method further includes: The sample feature vectors are corrected by using the features of samples acquired in a fasted state as the standard, and correcting the features of samples in non-fasted states to the feature distribution state of the fasted state.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: The classification network model is validated based on the constructed validation sample set, and the validation set results are collected. The validation set results include the prediction scores of the validation samples for autoimmune pancreatitis and pancreatic cancer, as well as the true labels of the validation samples. A temperature model with a single parameter T is established to scale and calibrate the predicted score of each validation sample. A negative log-likelihood loss function is constructed to measure the difference between the calibrated probability and the true label. The gradient descent algorithm is used to optimize and adjust the value of T to minimize the loss of the validation sample set, thus obtaining an optimal temperature value T. When calling the classification network model to perform classification prediction on the multimodal data to be analyzed, a temperature model is used to scale the prediction score, and the normalized score is then used to obtain the calibrated prediction probability.

5. The method according to any one of claims 1-3, characterized in that, The methods for acquiring endoscopic ultrasound images include: Acquire digestive endoscopy images and identify parts of the digestive tract from the digestive endoscopy images; When a designated digestive tract location is identified, guide the endoscope to acquire endoscopic ultrasound images. The acquired endoscopic ultrasound images were used to identify the gallbladder and pancreas regions in order to select endoscopic ultrasound images that included complete gallbladder and pancreas regions.

6. The method according to any one of claims 1-3, characterized in that, The gallbladder region is identified from the enhanced CT image data, and the image acquisition status corresponding to the current enhanced CT image data is determined based on the volume of the gallbladder region, including: A pre-trained gallbladder region recognition model is used to identify the enhanced CT image data to identify the gallbladder region. The gallbladder volume is estimated based on the area of ​​the gallbladder region. When the gallbladder volume is greater than a preset volume threshold, the image acquisition state corresponding to the current enhanced CT image data is determined to be fasting; otherwise, the image acquisition state corresponding to the current enhanced CT image data is determined to be non-fasting. The training steps of the gallbladder location recognition model include: annotating the gallbladder region in the CT image data of the preset abdominal enhanced CT image dataset to obtain the training dataset; and using the training dataset to learn and train the preset convolutional network to obtain the gallbladder location recognition model.

7. The method according to any one of claims 1-3, characterized in that, The extraction of statistical features, morphological features, texture features, and multiphasic enhancement features for each lesion ROI includes: By calculating the average CT value for each ROI, the statistical characteristics of the CT grayscale distribution of the lesion region can be obtained. Extract the three-dimensional geometric attributes of each ROI, and calculate its volume, surface area, compactness, extensibility, sphericity, major axis dimension and minor axis dimension to obtain the morphological characteristics of the lesion area; Extract metrics representing local texture uniformity, including contrast, homogeneity, energy, and entropy. Extract the average CT value for each ROI, calculate the average CT value in the arterial phase and the average CT value in the delayed phase, and calculate the multiphase enhancement feature ΔHU_delay-art based on the average CT value in the arterial phase and the average CT value in the delayed phase, as follows: ΔHU_delay-art = HUart - HUdelay.

8. A non-invasive assessment system for pancreatic diseases based on multimodal data, characterized in that, The system includes: The data acquisition module is used to acquire the multimodal data to be analyzed, including endoscopic ultrasound images and enhanced CT images. An ultrasound lesion identification module is used to identify the type of lesion in the ultrasound endoscopic images; The acquisition status detection module is used to identify the gallbladder region in the enhanced CT image data and determine the image acquisition status corresponding to the current enhanced CT image data based on the volume of the gallbladder region. The image acquisition status includes fasting status and non-fasting status. The feature extraction module is used to sequentially extract DICOM image sequences of the arterial phase, portal venous phase, and delayed phase from enhanced CT image data and convert them into image volume data respectively. The contours of pancreatic lesion regions are identified from the volume data corresponding to the arterial phase, portal venous phase, and delayed phase to obtain pancreatic lesion region volume data. Based on the coordinate matching relationship between the pancreatic lesion region volume data and the image volume data, the coordinates of the lesion region in the pancreatic lesion region volume data are traversed to extract the lesion connected regions from the DICOM images of each DICOM image sequence and take the lesion connected regions as the lesion region of interest (ROI). The statistical features, morphological features, texture features, and multiphase enhancement features of the lesion ROI are extracted. The feature vector construction module is used to construct feature vectors for multimodal data based on the lesion type, statistical characteristics, morphological characteristics, texture characteristics, and multiphase enhancement characteristics of the lesion ROI, as well as the image acquisition status. The classification and prediction module is used to classify the feature vector using a pre-trained classification network model, so as to output multimodal data as the predicted probability of autoimmune pancreatitis and pancreatic cancer, thereby realizing non-invasive auxiliary diagnosis of pancreatic diseases.

9. The system according to claim 8, characterized in that, The system also includes: The training set construction module is used to construct a multimodal data sample dataset, which includes multimodal data samples such as endoscopic ultrasound images, enhanced CT images, and corresponding pancreatic disease labels. The ultrasound lesion identification module is used to obtain the lesion type characteristics of the sample based on the ultrasound endoscopic image sample; The acquisition status detection module is used to obtain the sample image acquisition status of enhanced CT image data samples. The feature extraction module is used to obtain the statistical features, morphological features, texture features and multiphase enhancement features of the lesion ROI from the enhanced CT image data sample; The feature vector construction module is used to construct sample feature vectors based on the sample lesion type characteristics, the statistical characteristics, morphological characteristics, texture characteristics, and multiphase enhancement characteristics of the sample lesion ROI, as well as the sample image acquisition status. The model training module is used to construct a classification model training set based on the sample feature vectors of each multimodal data sample and the corresponding pancreatic disease labels. Based on the classification model training set, a classification network model is trained on a preset machine learning model so that the classification network model outputs the predicted probability of the multimodal data sample as autoimmune pancreatitis and pancreatic cancer based on the input sample feature vector.

10. The system according to claim 9, characterized in that, The system also includes: The feature correction module is used to correct the sample feature vector after constructing the sample feature vector based on the sample lesion type characteristics, the statistical characteristics, morphological characteristics, texture characteristics, and multiphase enhancement characteristics of the sample lesion ROI, as well as the sample image acquisition status. The feature vector is corrected to the feature distribution state of the fasting state by using the sample image acquisition status of the fasting state as the standard.

Images