Pancreatic tumor intelligent analysis method and system based on multi-modal medical image fusion

Abstract

The invention belongs to the technical field of medical image processing, and discloses a pancreatic tumor intelligent analysis method and system based on multi-modal medical image fusion, and the method comprises the steps: carrying out the preprocessing of a tomography image, and carrying out the region sketching operation, and combining a sketching result with a neural network structure to generate a pancreatic tumor region recognition technology. Respectively constructing three-dimensional image processing and diagnosis language output models on the basis of the medical image data and the medical data words; a pancreatic tumor detection area identification technology and a three-dimensional image processing and diagnosis language output model are used as theoretical basis and are combined with a virtual reality technology to build a consultation planning platform. According to the method, the pancreatic tumor area can be accurately identified through preprocessing and area sketching, later misdiagnosis and missed diagnosis are reduced, meanwhile, through the virtual reality technology, doctors and patients can visually observe and analyze the tumor and surrounding tissues thereof in the virtual environment, and the tumor condition can be analyzed more deeply.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a pancreatic tumor intelligent analysis method and system based on multimodal medical image fusion. Background Art

[0002] Pancreatic tumors are malignant or benign tumors that originate from the pancreas. The most common type is pancreatic cancer, especially pancreatic ductal adenocarcinoma, which is a particularly deadly type of cancer. The types of pancreatic tumors mainly include pancreatic ductal adenocarcinoma (PDAC), pancreatic neuroendocrine tumors, cystic tumors, etc. When diagnosing pancreatic tumors, imaging examinations are usually used, including abdominal CT scans, MRI or endoscopic ultrasound (EUS) technology to achieve analytical diagnosis of pancreatic tumors.

[0003] However, most existing medical image diagnosis systems rely solely on medical images to achieve tumor diagnosis, and often ignore the textual descriptions in the reports associated with the images, such as medical terms, diagnostic conclusions, etc. Moreover, as for the images themselves, most current systems rely on single-modality images and fail to achieve fusion diagnosis predictions for multi-parameter and multi-modality images, which in turn reduces the accuracy of tumor condition analysis. Summary of the invention

[0004] The embodiments of the present invention provide a pancreatic tumor intelligent analysis method and system based on multimodal medical image fusion to solve the problem of reducing the accuracy of tumor condition analysis in the prior art.

[0005] In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended to be a general review, nor is it intended to identify key / important components or to delineate the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0006] According to a first aspect of an embodiment of the present invention, a method for intelligent analysis of pancreatic tumors based on multimodal medical image fusion is provided.

[0007] In one embodiment, a pancreatic tumor intelligent analysis method based on multimodal medical image fusion includes the following steps:

[0008] After preprocessing the tomographic images, the region is delineated, and the delineation results are combined with the neural network structure to generate pancreatic tumor region recognition technology;

[0009] Based on medical image data and medical data words, three-dimensional image processing and diagnostic language output models are constructed respectively;

[0010] Pancreatic tumor detection area recognition technology, three-dimensional image processing and diagnostic language output model are combined with virtual reality technology to build a consulting and planning platform.

[0011] In one embodiment, the tomographic image is preprocessed and then a region delineation operation is performed, and the delineation result is combined with a neural network structure to generate a pancreatic tumor region recognition technology, which includes the following steps:

[0012] The tomographic image is segmented using an automatic segmentation model, and a pancreatic location area image is determined based on the segmentation result;

[0013] Extracting a regional image from the multi-parameter tomographic image according to the pancreas position segmentation result, and performing a preprocessing operation on the regional image to outline the tumor area;

[0014] A recognition model is generated based on the tumor area delineated by the multi-parameter tomographic image, and the pancreatic position area image is input into the recognition model to realize pancreatic tumor area detection.

[0015] In one embodiment, the multi-parameter tomographic image includes a plain tomographic image, an enhanced tomographic image, and multi-layer tomographic images with different layer thicknesses.

[0016] In one embodiment, constructing a three-dimensional image processing and a diagnostic language output model based on medical image data and medical data words respectively includes the following steps:

[0017] Obtaining medical features corresponding to medical image data by annotation, and performing enhancement processing on the medical image data after the annotation is completed;

[0018] Train the 3D data processing structure based on medical features and build an image processing model based on the training results;

[0019] Standardize the medical data language based on the medical terminology list and convert the medical data language into numerical form according to the processing results;

[0020] Taking medical data language as input, the preset language model is supervised and trained based on the expression sentences, and the diagnostic language output model is obtained according to the training results.

[0021] In one embodiment, obtaining medical features corresponding to medical image data by annotation and performing enhancement processing on the medical image data after annotation includes the following steps:

[0022] Collecting medical image data from a pre-arranged data acquisition center, and extracting tumor characteristic parameters associated with the medical image data after performing redundant data removal and privacy information masking operations on the medical image data;

[0023] Use manual annotation technology to obtain the medical features corresponding to the medical image data, and compare the medical features with tumor feature parameters to verify the accuracy of manual annotation;

[0024] After the accuracy verification is completed, the size, resolution and grayscale value of the medical image data are unified, and the image quality is enhanced using contrast adjustment technology, and then the noise of the medical image data is removed using a smoothing filter;

[0025] The processed medical image data is stored in a preset data storage management platform.

[0026] In one embodiment, obtaining medical features corresponding to medical image data by manual annotation technology and comparing the medical features with tumor feature parameters to verify the accuracy of manual annotation includes the following steps:

[0027] Use manual and image standard tools to obtain tumor and lesion areas in medical image data as medical features, and use key point detection algorithms to detect feature points of medical features;

[0028] Use digital image processing technology to perform three-dimensional correction on medical feature points, and perform dimensionality reduction processing on medical features based on the correction results to generate medical feature parameters;

[0029] Medical feature parameters are used as test samples, and tumor feature parameters are used as standard samples. The best similarity matrix, distance matrix and comparison threshold are obtained by learning the standard samples.

[0030] The weight values ​​are calculated based on the best similarity matrix and the distance matrix, and the test sample and the standard sample are weighted and combined according to the weight values ​​to obtain the similarity score, and the similarity score is compared with the classification threshold;

[0031] If the similarity score is greater than or equal to the comparison threshold, it means that the accuracy of manual annotation meets the requirements. If the similarity score is less than the comparison threshold, it means that the accuracy of manual annotation does not meet the requirements and manual annotation needs to be performed again.

[0032] In one embodiment, training a three-dimensional data processing structure based on medical features and constructing an image processing model according to the training results includes the following steps:

[0033] Taking the medical data image corresponding to the medical feature as input data, defining the category number corresponding to the medical data image as a supervision label, and inputting the medical data image into the three-dimensional data processing structure;

[0034] Defining a three-dimensional medical image according to the length, width and height of the medical data image in the three-dimensional data processing structure, and performing a convolution operation on the three-dimensional medical image according to the three-dimensional data processing structure to capture local information of the three-dimensional space;

[0035] After the capture is completed, the activation function is used to increase the nonlinear expression ability of the three-dimensional data processing structure, and the pooling layer is used to perform the pooling operation on the three-dimensional medical image;

[0036] The pooling result is flattened into a one-dimensional feature vector and sent to the fully connected layer to obtain the final representation. The output layer is used to convert the final representation result into a category probability distribution to complete the construction of the image processing model.

[0037] In one embodiment, combining pancreatic tumor detection area recognition technology, three-dimensional image processing and diagnostic language output model with virtual reality technology to build a consulting planning platform includes the following steps:

[0038] Use simulation cluster creation technology to build a computing cluster, and combine the computing cluster with pancreatic tumor detection area recognition technology, three-dimensional image processing and diagnostic language output model;

[0039] The computing cluster and virtual reality technology are combined to develop a result output display platform, and the simulation and preview interactive functions are integrated to obtain a consulting analysis platform;

[0040] Input the real-time tomography images into the consultation and analysis platform to obtain the pancreatic tumor analysis results:

[0041] According to a second aspect of an embodiment of the present invention, a pancreatic tumor intelligent analysis system based on multimodal medical image fusion is provided.

[0042] In one embodiment, a pancreatic tumor intelligent analysis system based on multimodal medical image fusion includes:

[0043] A tomographic image recognition unit is used to perform a region delineation operation on the tomographic image after preprocessing, and to combine the delineation result with the neural network structure to generate a pancreatic tumor region recognition technology;

[0044] An image diagnosis basis evaluation unit is used to construct a three-dimensional image processing and a diagnostic language output model based on medical image data and medical data words respectively;

[0045] The consulting and analysis platform building unit is used to combine pancreatic tumor detection area recognition technology, three-dimensional image processing and diagnostic language output model with virtual reality technology to build a consulting planning platform.

[0046] In one embodiment, the tomographic image recognition unit includes:

[0047] A tomographic image segmentation module, used to segment the tomographic image using an automatic segmentation model, and determine a pancreatic position area image based on the segmentation result;

[0048] An image preprocessing module is used to extract a regional image from the multi-parameter tomographic image according to the pancreas position segmentation result, and perform a preprocessing operation on the regional image to outline the tumor area;

[0049] The pancreatic region tumor recognition module is used to generate a recognition model based on the tumor region delineated by the multi-parameter tomographic image, and input the pancreatic location region image into the recognition model to realize pancreatic tumor region detection:

[0050] According to a third aspect of an embodiment of the present invention, a computer device is provided.

[0051] In some embodiments, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0052] According to a fourth aspect of embodiments of the present invention, a computer-readable storage medium is provided.

[0053] In one embodiment, the computer-readable storage medium stores a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0054] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0055] 1. The present invention can accurately identify pancreatic tumor areas through preprocessing and regional delineation, reduce misdiagnosis and missed diagnosis, and use a three-dimensional image processing model to provide more comprehensive information on tumor morphology and surrounding tissue structure, which is helpful for more accurate disease analysis in the later stage. At the same time, the diagnostic language output model constructed based on medical data words can generate standardized diagnostic reports, improve the consistency and readability of medical records, and finally, through virtual reality technology, doctors and patients can intuitively observe and analyze tumors and their surrounding tissues in a virtual environment, which is helpful for a more in-depth analysis of the tumor condition.

[0056] 2. The present invention uses an automatic segmentation model to accurately segment images, which can accurately determine the position and shape of the pancreas, thereby helping to accurately identify tumor areas, and obtain accurate medical features through manual annotation. The enhanced medical image data can better highlight important features. The entire processing flow from medical image data to medical data language not only improves the quality and efficiency of data processing, but also provides strong data support for clinical analysis.

[0057] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0059] Figure 1 is a flow chart of a pancreatic tumor intelligent analysis method based on multimodal medical image fusion according to an exemplary embodiment;

[0060] Figure 2 It is a principle block diagram of a pancreatic tumor intelligent analysis system based on multimodal medical image fusion according to an exemplary embodiment;

[0061] Figure 3 The figure is a schematic diagram showing the structure of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0062] The following description and accompanying drawings fully illustrate the specific embodiments of this article so that those skilled in the art can practice them. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of the embodiments of this article includes the entire scope of the claims, as well as all available equivalents of the claims. Herein, the terms "first", "second", etc. are only used to distinguish one element from another, without requiring or implying any actual relationship or order between these elements. In fact, the first element can also be called the second element, and vice versa. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the structure, device or equipment including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such structure, device or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the structure, device or equipment including the elements. Each embodiment is described in a progressive manner herein, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other.

[0063] The terms "longitudinal", "lateral", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. in this document indicate the orientation or position relationship based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing this document and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In the description of this document, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a mechanical connection or an electrical connection, it can also be the internal communication of two elements, it can be a direct connection, or it can be an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0064] As used herein, the term "plurality" means two or more than two, unless otherwise specified.

[0065] In this document, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.

[0066] In this article, the term "and / or" is a description of the association relationship between objects, indicating that three relationships may exist. For example, A and / or B means: A or B, or, A and B.

[0067] It should be understood that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0068] Each module in the device or system of the present application can be implemented in whole or in part by software, hardware, or a combination thereof. The above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above modules.

[0069] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0070] Figure 1An embodiment of the multimodal medical image fusion pancreatic tumor intelligent analysis method of the present invention is shown.

[0071] In this optional embodiment, the pancreatic tumor intelligent analysis method based on multimodal medical image fusion includes the following steps:

[0072] Step S101, performing a region delineation operation on the tomographic image after preprocessing, and combining the delineation result with the neural network structure to generate a pancreatic tumor region recognition technology;

[0073] Step S103, constructing three-dimensional image processing and diagnostic language output models based on medical image data and medical data words respectively;

[0074] Step S105, combining pancreatic tumor detection area recognition technology, three-dimensional image processing and diagnostic language output model with virtual reality technology to build a consulting planning platform.

[0075] In this optional embodiment, after preprocessing the tomographic image, a regional delineation operation is performed, and the delineation result is combined with a neural network structure to generate a pancreatic tumor region recognition technology. The tomographic image can be segmented using an automatic segmentation model, and the pancreatic position regional image can be determined based on the segmentation result; the regional image is extracted from the multi-parameter tomographic image according to the pancreatic position segmentation result, and a preprocessing operation is performed on the regional image to delineate the tumor region; a recognition model is generated based on the tumor region delineated by the multi-parameter tomographic image, and the pancreatic position regional image is input into the recognition model to realize pancreatic tumor region detection.

[0076] It needs to be explained that in specific applications, the deep learning convolutional neural network can be used on the CT cross-sectional image, and the AttentionUNet automatic segmentation model can be used for image segmentation to determine the position of the pancreas, and based on the gland segmentation result, the local image area is extracted from the multi-parameter CT image, and the deep learning convolutional neural network with multi-parameter image input is used to perform tumor detection on the pancreatic gland area segmented from the medical image, where the multi-parameter CT image refers to a multimodal CT image acquired under different scanning parameters, including plain CT images, enhanced CT images, multi-layer CT images at different layer thicknesses, etc.

[0077] The processing operations are performed on the multi-parameter CT image as follows:

[0078] (1) Multi-parameter image registration: The mutual information method is used to achieve rigid registration of CT images under different parameters, that is, registration is achieved by maximizing the mutual information of the overlapping area;

[0079] (2) Tumor area delineation: The Mask R-CNN deep learning model is used to automatically detect and delineate the tumor area;

[0080] (3) Image resampling: Use linear interpolation to standardize the image resolution;

[0081] (4) Image capture: Based on the results of the region delineation, a mask of the region of interest (ROI) is constructed, i.e., the tumor area is 1 and the non-tumor area is 0. This mask is applied to the image and the area with the mask of 1 is extracted.

[0082] (5) Data standardization: Scale the input data to a standard distribution with a mean of 0 and a standard deviation of 1 so that the input data have similar scale and distribution.

[0083] Specifically, the deep learning convolutional neural network input is continuous 3-layer CT image data, and the output result is the pancreatic tumor detection area in the middle layer, that is, the input is 4-dimensional data (frequency encoding x phase encoding x 3 layers x parameter map), and the output is 2-dimensional data (frequency encoding x phase encoding).

[0084] In this optional embodiment, when constructing three-dimensional image processing and diagnostic language output models based on medical image data and medical data words respectively, the medical features corresponding to the medical image data can be obtained by annotation, and the medical image data can be enhanced after the annotation is completed; the three-dimensional data processing structure is trained based on the medical features, and the image processing model is constructed according to the training results; the medical data language is standardized based on the medical terminology list, and the medical data language is converted into a numerical form according to the processing results; the medical data language is used as input, and the preset language model is supervised and trained based on the expression sentence, and the diagnostic language output model is obtained according to the training results.

[0085] In this optional embodiment, when the medical features corresponding to the medical image data are obtained by annotation and the medical image data is enhanced after the annotation is completed, the medical image data can be collected from a pre-arranged data acquisition center, and the tumor feature parameters associated with the medical image data can be extracted after the medical image data is subjected to redundant data removal and privacy information masking operations; the medical features corresponding to the medical image data are obtained by manual annotation technology, and the medical features are compared with the tumor feature parameters to verify the accuracy of the manual annotation; after the accuracy verification is completed, the size, resolution and grayscale value of the medical image data are unified, and the contrast adjustment technology is used to enhance the image quality and then a smoothing filter is used to remove the noise of the medical image data; the processed medical image data is stored in a preset data storage management platform.

[0086] In this optional embodiment, when using manual annotation technology to obtain medical features corresponding to medical image data, and comparing the medical features with tumor feature parameters to verify the accuracy of manual annotation, manual and image standard tools can be used to obtain tumor and lesion areas in the medical image data as medical features, and a key point detection algorithm can be used to detect feature points of the medical features; digital image processing technology is used to perform three-dimensional correction on the medical feature points, and dimensionality reduction processing is performed on the medical features based on the correction results to generate medical feature parameters; the medical feature parameters are used as test samples and the tumor feature parameters are used as standard samples, and the standard samples are used to learn to obtain the best similarity matrix, distance matrix and comparison threshold; weight values ​​are calculated based on the best similarity matrix and distance matrix, and the test sample and the standard sample are weighted and combined according to the weight values ​​to obtain a similarity score, and the similarity score is compared with the classification threshold; if the similarity score is greater than or equal to the comparison threshold, it means that the accuracy of the manual annotation meets the requirements, and if the similarity score is less than the comparison threshold, it means that the accuracy of the manual annotation does not meet the requirements and manual annotation needs to be performed again.

[0087] It should be explained that when using standard samples (tumor characteristic parameters) to train the similarity measurement model to verify the accuracy of manual annotation, the medical characteristic parameters and tumor characteristic parameters can be standardized in advance to ensure that the data are at the same level and reduce the impact of different measurements. At the same time, a machine learning algorithm is selected to train the similarity measurement, and the tumor characteristic parameters of the standard sample are used as training data to learn the best similarity matrix and distance matrix that describe the similarity between samples; based on the learned similarity matrix and distance matrix, the weight of each feature is calculated, where the weight indicates the importance of each feature in the recognition process, and the medical characteristic parameters of the test sample and the characteristic parameters of the standard sample are weighted and combined according to the calculated weights; based on the weighted synthesized features, the similarity score between the test sample and the standard sample is calculated.

[0088] In this optional embodiment, when training a three-dimensional data processing structure based on medical features and constructing an image processing model according to the training results, a medical data image corresponding to the medical features can be used as input data, the category number corresponding to the medical data image is defined as a supervision label, and the medical data image is input into the three-dimensional data processing structure; a three-dimensional medical image is defined according to the length, width and height of the medical data image in the three-dimensional data processing structure, and a convolution operation is performed on the three-dimensional medical image according to the three-dimensional data processing structure to capture local information in the three-dimensional space; after the capture is completed, an activation function is used to increase the nonlinear expression ability of the three-dimensional data processing structure, and a pooling layer is used to perform a pooling operation on the three-dimensional medical image; the pooling result is flattened into a one-dimensional feature vector and sent to the fully connected layer to obtain the final representation, and the output layer is used to convert the final representation result into a category probability distribution to complete the construction of the image processing model.

[0089] It should be explained that the specific construction of the image processing model includes the steps of training data set and identifying data set processing, cleaning, calibration and preprocessing of existing medical image data, and then building the image processing model, which is the basis of the entire intelligent analysis. The specific implementation steps are as follows:

[0090] (1) Data collection: Obtain medical image data from hospitals, clinics, medical imaging centers, etc., including X-rays, CT scans, MRIs, etc., and ensure that the image formats are unified. At the same time, collect other prior knowledge and necessary information related to the medical image data, such as tumor staging, tumor type, and tumor resectability, which are convenient for doctors to make diagnosis;

[0091] (2) Data cleaning: remove redundant data, remove blurred, unclear or damaged images, remove or cover up patient privacy information (such as name, date of birth, etc.) in the image, and verify the accuracy of the text information such as the diagnosis basis and the standardization of terminology;

[0092] (3) Data calibration: Radiologists or professionals use professional image annotation tools to manually annotate key features in the image (such as tumors, lesions, etc.), and perform data verification to ensure the accuracy and consistency of all annotations. A secondary review is performed when necessary.

[0093] (4) Data preprocessing: unify the image size, resolution, and grayscale value range to achieve image normalization, use filtering, contrast adjustment and other techniques to enhance image quality, improve detail visibility, and use smoothing filters and other techniques to remove noise from the image;

[0094] (5) Data platform construction: Store the cleaned, calibrated, and preprocessed images in a reliable storage system. Common storage solutions include cloud storage, NAS, etc., and design a suitable database architecture. Develop or use an existing data management platform to facilitate data query, retrieval, sharing, and access control.

[0095] On the basis of completing the training data set and identification data set processing steps, an image processing model is further constructed and trained. When training the deep learning model, pancreatic tumor lesion characteristics, metastatic lymph node images, and peripheral vascular invasion characteristic images are used as input data, and the category number of the image is used as a supervisory label. The image model is repeatedly trained. Based on the comparison of different models, this embodiment adopts a network similar to C3D (i.e., 3D Convolutional Network).

[0096] The original C3D network is used for video feature extraction, with 16 frames of images input at a time and the output being the feature expression of this video. This is very similar to the nodule detection scenario, where a set of 2D slices of a 3D image are input at a time and the output is the discrimination result of this image.

[0097] The specific steps of the C3D network are as follows: the input of the C3D network is a three-dimensional medical image, where the three dimensions represent the length, width, and height (i.e., x, y, and z) of the human body image segmentation. Each voxel of the three-dimensional image has a channel representing the grayscale value, and the C3D network is composed of multiple repeated components, each of which contains a convolution layer, an activation function, and a pooling layer. Unlike traditional 2D convolution, the 3D convolution kernel has three dimensions, and convolution operations are performed in the x, y, and z dimensions. This operation enables the model to capture local information in three-dimensional space; each 3D convolution layer is followed by a ReLU activation function to increase the nonlinear expression ability of the network; the pooling layer performs pooling operations in the x, y, and z dimensions. Pooling can use maximum pooling, global pooling, and average pooling. The pooling kernel size can be 2*2*2 or determined according to the specific size of the image. The pooling operation reduces the size of the feature map in three dimensions while retaining important features.

[0098] For a typical C3D network, it can be composed of multiple stacked components (i.e., convolution layer + activation function + pooling layer). High-level features are extracted layer by layer, and the first few layers of 3D convolution layers extract low-level features, such as edges, textures, etc.; the latter few layers of 3D convolution layers extract high-level spatial features, such as tumor morphological patterns and complex spatial relationships in the image. After completing the above convolution operation, it is flattened into a one-dimensional feature vector and sent to the fully connected layer to obtain the final representation, and then through the Softmax (output layer) layer, the feature vector is converted into a category probability distribution for the final image processing decision result.

[0099] After the image model is built, the language output model is trained using the medical terminology associated with the image as input and the diagnostic evidence statement as the supervision label. The specific implementation steps are as follows:

[0100] (1) Data preprocessing: Use a medical terminology table to standardize synonyms and abbreviations, and convert text data into a numerical form that the model can process. Common methods include word embeddings (such as Word2Vec and GloVe) or context-dependent embeddings.

[0101] (2) Model selection: Select one of BioBERT, ClinicalBERT, BlueBERT, etc. as the pre-trained language model;

[0102] (3) Model training: The model is trained with medical terms as input and diagnostic statements as supervision. The cross-entropy loss is selected as the loss function, the Adam optimizer is selected, and appropriate learning rate adjustment strategies are adopted, such as learning rate decay and warm-up.

[0103] (4) Model evaluation: Evaluation indicators: Use accuracy, precision, recall, F1 score and other evaluation indicators to evaluate model performance.

[0104] In this optional embodiment, when the pancreatic tumor detection area recognition technology, three-dimensional image processing and diagnostic language output model are combined with virtual reality technology to build a consulting planning platform, a computing cluster can be built using simulation cluster creation technology, and the computing cluster can be combined with pancreatic tumor detection area recognition technology, three-dimensional image processing and diagnostic language output model; the computing cluster is combined with virtual reality technology to develop a result output display platform, and the simulation and rehearsal interactive functions are integrated to obtain a consulting analysis platform; the real-time tomography image is input into the consulting analysis platform to obtain the pancreatic tumor analysis results.

[0105] It should be explained that the construction of the consultation and analysis platform is based on the comprehensive medical imaging database of pancreatic tumor patients, with pancreatic tumor detection area recognition technology, three-dimensional image processing and diagnostic language output model as core technologies, and a large-scale parallel computing platform as hardware support. Data image processing software, deep learning intelligent analysis software, and diagnostic result output display software are developed, and a prototype system for automatic analysis and tumor staging of pancreatic cancer is designed and implemented. The system is applied in clinical practice to verify the performance of the system in clinical analysis, and finally assist doctors in formulating reasonable treatment plans. The specific steps are as follows:

[0106] (1) Build a high-performance computing cluster to ensure the stability and efficiency of the parallel computing environment. When building a high-performance computing cluster, you can first analyze the amount of data to be processed, computing requirements, and expected load, select server hardware based on requirements, and design the network architecture within the cluster to ensure efficient data transmission and communication;

[0107] At the same time, install servers, storage devices, network devices and other hardware, configure the internal network, install the operating system and cluster management software on each node, then configure the roles and responsibilities of each node, select and install appropriate simulation software to ensure that the simulation software can use the cluster's computing resources for high-performance computing;

[0108] (2) Develop data image processing software to generate detailed anatomical models of the pancreas and surrounding tissues through image processing and 3D reconstruction technology, and use deep learning frameworks such as TensorFlow or PyTorch to develop intelligent analysis software that uses three-dimensional image processing and diagnostic language output models to accurately analyze tumor location, size, and invasion, and provide personalized surgical plan recommendations;

[0109] (3) Combined with virtual reality (VR) technology, we developed software for outputting and displaying diagnostic results, designed a user-friendly interface, and realized the visualization of images and analysis results. At the same time, we integrated interactive functions so that doctors can simulate and rehearse surgical pathways in a virtual environment, optimize surgical strategies, and reduce intraoperative risks. Finally, the platform continuously iterates and optimizes the surgical planning model through real-time data updates and postoperative feedback mechanisms, thereby improving the overall analytical effect and patient prognosis.

[0110] In specific applications, images with diagnostic value can be selected from the comprehensive medical imaging database and clinically collected images of pancreatic cancer patients. According to the NCCN guidelines (National Comprehensive Cancer Network, cancer treatment guidelines organization), senior imaging doctors mark the tumor location, size, metastatic lymph node status, peripheral vascular invasion and other information, and combine the data list of the number of metastatic lymph nodes and TNM staging (a staging system specifically used in the cancer treatment process) with the CT, MRI, US and other labeled image sequences included in each patient to build a training data platform for the artificial intelligence analysis system. This platform not only contains medical images with annotated information and analysis value, but also contains other prior knowledge and necessary information such as tumor staging and tumor resectability, providing an important data foundation for the learning and training of artificial intelligence analysis models. Based on the data platform, the core algorithm module of intelligent analysis is designed to realize intelligent data preprocessing, image-based tumor segmentation, extraction and fusion of analytical features, and output of tumor analysis results.

[0111] Figure 2 An embodiment of the multimodal medical image fusion pancreatic tumor intelligent analysis system of the present invention is shown.

[0112] In this optional embodiment, the pancreatic tumor intelligent analysis system based on multimodal medical image fusion includes:

[0113] A tomographic image recognition unit 201 is used to perform a region delineation operation on the tomographic image after preprocessing, and to combine the delineation result with the neural network structure to generate a pancreatic tumor region recognition technology;

[0114] An image diagnosis basis evaluation unit 203 is used to construct a three-dimensional image processing and a diagnostic language output model based on medical image data and medical data words respectively;

[0115] The consultation and analysis platform building unit 205 is used to build a consultation and planning platform by combining pancreatic tumor detection area recognition technology, three-dimensional image processing and diagnostic language output model with virtual reality technology.

[0116] In this optional embodiment, the tomographic image recognition unit 201 includes:

[0117] A tomographic image segmentation module, used to segment the tomographic image using an automatic segmentation model, and determine a pancreatic position area image based on the segmentation result;

[0118] An image preprocessing module is used to extract a regional image from the multi-parameter tomographic image according to the pancreas position segmentation result, and perform a preprocessing operation on the regional image to outline the tumor area;

[0119] The pancreatic region tumor recognition module is used to generate a recognition model based on the tumor region delineated by the multi-parameter tomographic image, and input the pancreatic position region image into the recognition model to realize pancreatic tumor region detection.

[0120] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store static information and dynamic information data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the above method embodiment are implemented.

[0121] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0122] In addition, the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiment when executing the computer program.

[0123] In addition, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0125] The present invention is not limited to the structures which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An intelligent analysis method for pancreatic tumors based on multimodal medical image fusion. It is characterized in that The pancreatic tumor intelligent analysis method comprises the following steps: After preprocessing the tomographic images, the region is delineated, and the delineation results are combined with the neural network structure to generate pancreatic tumor region recognition technology; Based on medical image data and medical data words, three-dimensional image processing and diagnostic language output models are constructed respectively; Pancreatic tumor detection area recognition technology, three-dimensional image processing and diagnostic language output model are combined with virtual reality technology to build a consulting and planning platform.

2. The pancreatic tumor intelligent analysis method based on multimodal medical image fusion according to claim 1, It is characterized in that The method of performing a region delineation operation after preprocessing the tomographic image and combining the delineation result with the neural network structure to generate a pancreatic tumor region recognition technology includes the following steps: The tomographic image is segmented using an automatic segmentation model, and a pancreatic location area image is determined based on the segmentation result; Extracting a regional image from the multi-parameter tomographic image according to the pancreas position segmentation result, and performing a preprocessing operation on the regional image to outline the tumor area; A recognition model is generated based on the tumor area delineated by the multi-parameter tomographic image, and the pancreatic position area image is input into the recognition model to realize pancreatic tumor area detection.

3. The pancreatic tumor intelligent analysis method based on multimodal medical image fusion according to claim 2, It is characterized in that The multi-parameter tomographic images include plain tomographic images, enhanced tomographic images and multi-layer tomographic images with different layer thicknesses.

4. The pancreatic tumor intelligent analysis method based on multimodal medical image fusion according to claim 1, It is characterized in that The three-dimensional image processing and diagnostic language output models are constructed based on the medical image data and the medical data words, respectively, and include the following steps: Obtaining medical features corresponding to medical image data by annotation, and performing enhancement processing on the medical image data after the annotation is completed; Train the 3D data processing structure based on medical features and build an image processing model based on the training results; Standardize the medical data language based on the medical terminology list and convert the medical data language into numerical form according to the processing results; Taking medical data language as input, the preset language model is supervised and trained based on the expression sentences, and the diagnostic language output model is obtained according to the training results.

5. The pancreatic tumor intelligent analysis method based on multimodal medical image fusion according to claim 4, It is characterized in that The method of obtaining the medical features corresponding to the medical image data by using the annotation method and performing enhancement processing on the medical image data after the annotation is completed comprises the following steps: Collecting medical image data from a pre-arranged data acquisition center, and extracting tumor characteristic parameters associated with the medical image data after performing redundant data removal and privacy information masking operations on the medical image data; Use manual annotation technology to obtain the medical features corresponding to the medical image data, and compare the medical features with tumor feature parameters to verify the accuracy of manual annotation; After the accuracy verification is completed, the size, resolution and grayscale value of the medical image data are unified, and the image quality is enhanced using contrast adjustment technology, and then the noise of the medical image data is removed using a smoothing filter; The processed medical image data is stored in a preset data storage management platform.

6. The pancreatic tumor intelligent analysis method based on multimodal medical image fusion according to claim 5, It is characterized in that The method of obtaining medical features corresponding to the medical image data by using manual annotation technology and comparing the medical features with tumor feature parameters to verify the accuracy of manual annotation includes the following steps: Use manual and image standard tools to obtain tumor and lesion areas in medical image data as medical features, and use key point detection algorithms to detect feature points of medical features; Use digital image processing technology to perform three-dimensional correction on medical feature points, and perform dimensionality reduction processing on medical features based on the correction results to generate medical feature parameters; Medical feature parameters are used as test samples, and tumor feature parameters are used as standard samples. The best similarity matrix, distance matrix and comparison threshold are obtained by learning the standard samples. The weight values ​​are calculated based on the best similarity matrix and the distance matrix, and the test sample and the standard sample are weighted and combined according to the weight values ​​to obtain the similarity score, and the similarity score is compared with the classification threshold; If the similarity score is greater than or equal to the comparison threshold, it means that the accuracy of manual annotation meets the requirements. If the similarity score is less than the comparison threshold, it means that the accuracy of manual annotation does not meet the requirements and manual annotation needs to be performed again.

7. The pancreatic tumor intelligent analysis method based on multimodal medical image fusion according to claim 6, It is characterized in that The method of training the three-dimensional data processing structure based on medical features and constructing an image processing model according to the training results includes the following steps: Taking the medical data image corresponding to the medical feature as input data, defining the category number corresponding to the medical data image as a supervision label, and inputting the medical data image into the three-dimensional data processing structure; Defining a three-dimensional medical image according to the length, width and height of the medical data image in the three-dimensional data processing structure, and performing a convolution operation on the three-dimensional medical image according to the three-dimensional data processing structure to capture local information of the three-dimensional space; After the capture is completed, the activation function is used to increase the nonlinear expression ability of the three-dimensional data processing structure, and the pooling layer is used to perform the pooling operation on the three-dimensional medical image; The pooling result is flattened into a one-dimensional feature vector and sent to the fully connected layer to obtain the final representation. The output layer is used to convert the final representation result into a category probability distribution to complete the construction of the image processing model.

8. The pancreatic tumor intelligent analysis method based on multimodal medical image fusion according to claim 1, It is characterized in that The construction of a consulting and planning platform by combining pancreatic tumor detection area recognition technology, three-dimensional image processing and diagnostic language output model with virtual reality technology includes the following steps: Use simulation cluster creation technology to build a computing cluster, and combine the computing cluster with pancreatic tumor detection area recognition technology, three-dimensional image processing and diagnostic language output model; The computing cluster and virtual reality technology are combined to develop a result output display platform, and the simulation and preview interactive functions are integrated to obtain a consulting analysis platform; The real-time tomography images are input into the consultation and analysis platform to obtain the pancreatic tumor analysis results.

9. A pancreatic tumor intelligent analysis system based on multimodal medical image fusion, It is characterized in that The pancreatic tumor intelligent analysis system includes: A tomographic image recognition unit is used to perform a region delineation operation on the tomographic image after preprocessing, and to combine the delineation result with the neural network structure to generate a pancreatic tumor region recognition technology; An image diagnosis basis evaluation unit is used to construct a three-dimensional image processing and a diagnostic language output model based on medical image data and medical data words respectively; The consulting and analysis platform building unit is used to combine pancreatic tumor detection area recognition technology, three-dimensional image processing and diagnostic language output model with virtual reality technology to build a consulting planning platform.

10. The pancreatic tumor intelligent analysis system based on multimodal medical image fusion according to claim 9, It is characterized in that The tomographic image recognition unit comprises: A tomographic image segmentation module, used to segment the tomographic image using an automatic segmentation model, and determine a pancreatic position area image based on the segmentation result; An image preprocessing module is used to extract a regional image from the multi-parameter tomographic image according to the pancreas position segmentation result, and perform a preprocessing operation on the regional image to outline the tumor area; The pancreatic region tumor recognition module is used to generate a recognition model based on the tumor region delineated by the multi-parameter tomographic image, and input the pancreatic position region image into the recognition model to realize pancreatic tumor region detection.

Images