Method for constructing AI-enabled liver tumor three-dimensional image reconstruction platform
By combining artificial intelligence and artificial methods for three-dimensional image segmentation and augmented reality technology of liver tumors, the accuracy and surgical difficulties in medical image segmentation technology are solved, and more efficient diagnosis and surgical effects of liver tumors are achieved.
Patent Information
- Application Number
- CN202411159292.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-12-17
AI Technical Summary
Medical image segmentation technology faces the problems of small number of labeled samples, low image quality and poor algorithm generalization ability in three-dimensional image reconstruction of liver tumors, resulting in inaccurate diagnosis and difficulty in surgery.
Using artificial intelligence methods combined with artificial methods, manual segmentation is performed through 3DSlicer and Blender, image segmentation is performed using pre-trained diffusion model and nnU-Net model, network training is performed by combining dice and cross entropy loss, and deployed by Next.js and three.js A three-dimensional model, using ARKit for augmented reality projection, assisting doctors in completing tumor cutting surgery.
It improves the accuracy of automatic segmentation of liver tumors and the success rate of surgery, lowers the operating threshold of doctors, enhances the spatial perception ability of doctors, and improves the success rate of surgery.
Smart Images
Figure CN119152151A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of information technology application technology, and specifically relates to a method for building an AI-enabled three-dimensional image reconstruction platform for liver tumors. Background Art
[0002] The AI-enabled 3D liver tumor image reconstruction platform uses artificial intelligence methods to reconstruct tissues, organs and tumors, and deploys the reconstructed model to the server, which can be interactively viewed through a Web application. The virtual model is superimposed on the real image data through software, and augmented reality assists doctors in removing tumors. Corresponding to the above process, the technologies involved in the 3D liver tumor image reconstruction platform include: medical image segmentation technology, interactive platform deployment and augmented reality technology.
[0003] Medical image segmentation is a computer vision technology that aims to automatically or semi-automatically identify and separate regions of interest (ROI), such as organs, tissues, lesions, etc., from medical imaging data for easier and more accurate analysis, interpretation and further processing. The present invention intends to help doctors accurately locate tumor areas through image segmentation to provide more accurate diagnostic information. During surgery, accurate image segmentation can help guide doctors to complete liver tumor resection. Medical image segmentation technology still faces some challenges such as a small number of labeled samples, low image quality, and poor generalization of algorithms. The present invention intends to design an effective automatic liver tumor segmentation method for three-dimensional model reconstruction.
[0004] Implement server deployment of the reconstructed model to support interactive viewing, including model deployment, front-end and back-end development, and configuration of the deployment environment. Select a suitable server (such as Alibaba Cloud, Huawei Cloud, etc.) for deployment. The server side needs to set up the server environment, including installing necessary software (such as Python, related libraries) and configuring the network (such as setting firewall rules to ensure that the API port is accessible). Develop a simple front-end application, you can use HTML / CSS / JavaScript, or frameworks such as React and Vue.js to create an interactive user interface. The front-end should include file upload (or other data input methods), send requests to the server, and display the reconstructed model or results returned by the server. If the model output is a three-dimensional model or requires complex interaction, you can consider using JavaScript libraries such as Three.js and Babylon.js to render and interact with the three-dimensional model on the front end. An interactive three-dimensional model query platform greatly reduces the model query cost for doctors and patients, and is conducive to communication between doctors and patients.
[0005] Augmented Reality (AR) is a technology that enhances our perception of the real world by superimposing computer-generated information (such as visual images, sounds, text, videos, etc.) on the real world. The use of AR technology in intraoperative guidance systems is ushering in a new era of medical diagnosis and surgical treatment. The technology improves surgical accuracy, enhances the doctor's spatial perception ability, and greatly improves patient treatment outcomes. Superimpose a virtual 3D reconstructed model (key markers) in the doctor's field of view to enhance the perception of the real world. Use special AR glasses, helmets, or through screens in the operating room to accurately align the virtual image with the real world. Real-time guidance for doctors to accurately position, cut, and operate during surgery, especially when vision is limited. This technology will increase the success rate of surgery, especially to help young doctors grow rapidly. Summary of the invention
[0006] In order to solve the above problems, the present invention proposes: a method for building an AI-enabled liver tumor three-dimensional image reconstruction platform, comprising the following steps:
[0007] S1. Use manual and AI methods to segment organs and tumors in CT datasets.
[0008] S2, reconstruct the segmented image into a 3D model, and deploy the model to the server backend using Next.js and three.js, so that users can interactively view it through the front-end web application;
[0009] S3. Use the ARKit software development kit to deploy the augmented reality model, and then use augmented reality glasses to project the reconstructed model into the real scene to assist doctors in completing tumor resection surgery.
[0010] Furthermore, in step S1, the artificial method used for medical image segmentation is specifically:
[0011] Using 3DSlicer, the outline of the region of interest is manually drawn on each slice, and then the drawn outline is extended to the entire 3D dataset using an automatic filling algorithm to obtain a complete segmentation result;
[0012] The manually segmented 3D model is exported to STL format, and then Blender is used to render the model. The desired rendering effect is obtained by adjusting the lighting, material, and texture properties to prepare for subsequent visualization and interaction.
[0013] Furthermore, in step S1, the AI method used for medical image segmentation is specifically:
[0014] We augmented the existing data using a diffusion model pre-trained on the abdominal benchmark dataset AbdomenCT-1k, merging the original dataset and the generated dataset into a new augmented dataset.
[0015] The nnU-Net model is trained on the expanded dataset for image segmentation.
[0016] Furthermore, in the AI method used for medical image segmentation, the method of using nnU-Net for image segmentation is as follows:
[0017] Preprocess the CT data set, including cropping, resampling, and normalization steps;
[0018] Utilizes a pool of basic U-Net architectures, automatically configured and trained using a five-fold cross-validation approach;
[0019] Combine Dice and Cross Entropy Loss to train the nnU-Net network:
[0020] L total =L dice +L CE
[0021] The dice loss is calculated based on the softmax output of the network u and the true segmentation map v:
[0022]
[0023] where u is the softmax output of the network, v is the one-hot encoding of the ground truth segmentation map, and both u and v have shape I×K, where i∈I is the number of pixels in the training block or batch, and k∈K is the class;
[0024] Using a patch-based inference strategy, the network accuracy decreases at patch boundaries, giving higher weights to voxels closer to the center during prediction aggregation across patches;
[0025] Perform an ensemble by combining different models, automatically selecting the model that achieves the highest average prospect dice score in cross-validation on the training set, and choosing the best performing model for final submission;
[0026] Use the trained model to generate segmentations at full resolution, and perform post-processing on the segmentation results, applying connected component analysis to ensure segmentation labels are coherent and remove smaller components to improve accuracy.
[0027] Furthermore, in step S2, three.js is used to render the 3D model, and Next.js is used to implement interactive query, specifically:
[0028] The 3D reconstructed model file is uploaded to the server. The server is configured with the necessary Next.js and three.js environments for interactive query. Users access the server model through the IP address and query the model in an interactive way.
[0029] Users use mobile app to access the model by scanning QR code, or interactively query through computer web application.
[0030] Furthermore, in step S3, the three-dimensional reconstruction model is developed and deployed for use in a real-world environment, and the reconstructed image is projected into a real scene in the following manner:
[0031] Use Xcode with ARKit to develop apps, ensuring that the rendering and interaction of models meet the accuracy required by real scenes;
[0032] It is planned to fit and superimpose the intraoperative images with the three-dimensional images, so that doctors can obtain a better field of view during minimally invasive surgery, clearly segment the tumor and organs, and improve the success rate of the operation.
[0033] Compared with the prior art, the above technical solution adopted by the present invention has the advantages of applying AI algorithm to the whole treatment process of preoperative diagnosis, intraoperative guidance and postoperative analysis of liver tumors, and building a three-dimensional image reconstruction platform for liver tumors. The platform mainly embodies the following advantages:
[0034] (1) Using the improved nnU-net, the optimal network architecture is automatically selected according to the specific medical image segmentation task, which reduces the burden of manually adjusting the network structure and makes it easier to achieve model adaptation and optimization;
[0035] (2) Using generative models to expand data sets solves the problem of limited medical data resources;
[0036] (3) Adopt the server-client model and design a user-friendly interface to lower the operation threshold and make it easy and quick to use. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0038] Figure 1 This is the framework diagram of the AI-enabled liver tumor 3D reconstruction platform;
[0039] Figure 2This is a diagram of the manual segmentation method using 3D Slicer;
[0040] Figure 3 This is the improved nnU-net network structure diagram;
[0041] Figure 4 It is a flow chart of server-side deployment and human-computer interaction;
[0042] Figure 5 It is an augmented reality projection effect diagram. Specific implementation methods
[0044] The principles of the present disclosure will be described below with reference to several example embodiments shown in the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that these embodiments are described only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way.
[0045] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment".
[0046] The purpose of the present invention is to propose a method for building a three-dimensional image reconstruction platform for liver tumors, which can help doctors evaluate complex and diverse medical image data, mainly realize automatic positioning and division of the liver and tumors, and ultimately provide doctors with an augmented reality three-dimensional model for high-quality diagnosis.
[0047] In order to achieve the above-mentioned object, the present invention proposes a method for constructing a liver tumor three-dimensional image reconstruction platform, such as Figure 1 As shown, the following steps are included:
[0048] S1. Use manual and AI methods to segment organs and tumors in CT datasets.
[0049] S2, reconstruct the segmented image into a 3D model, and deploy the model to the server backend using Next.js and three.js, so that users can interactively view it through the front-end web application;
[0050] S3. Use the ARKit software development kit to deploy the augmented reality model, and then use augmented reality glasses to project the reconstructed model into the real scene to assist doctors in completing tumor resection surgery.
[0051] In step S1, the artificial method used for medical image segmentation is specifically:
[0052] Using 3D Slicer, manually draw the outline of the region of interest on each slice, and then use the automatic filling algorithm to extend the drawn outline to the entire 3D dataset to obtain a complete segmentation result;
[0053] The manually segmented 3D model is exported to STL format, and then the model is rendered using Blender. The lighting, material, texture and other properties are adjusted to obtain the desired rendering effect, which facilitates subsequent visualization and interaction.
[0054] In step S1, the AI method used for medical image segmentation is specifically:
[0055] We augmented the existing data using a diffusion model pre-trained on the abdominal benchmark dataset AbdomenCT-1k, merging the original dataset and the generated dataset into a new augmented dataset.
[0056] The nnU-Net model is trained on the expanded dataset for image segmentation.
[0057] Furthermore, the way to use nnU-Net for image segmentation is:
[0058] Preprocess the CT data set, including cropping, resampling, normalization and other steps;
[0059] Utilizes a basic U-Net architecture pool (2D U-Net, 3D U-Net, U-Net Cascade), automatically configured and trained using a five-fold cross-validation method;
[0060] Combine Dice and Cross Entropy Loss to train the nnU-Net network:
[0061] L total =L dice +L CE
[0062] The dice loss is calculated based on the softmax output of the network u and the true segmentation map v:
[0063]
[0064] where u is the softmax output of the network, v is the one-hot encoding of the ground truth segmentation map, and both u and v have shape I×K, where i∈I is the number of pixels in the training block or batch, and k∈K is the class;
[0065] Using a patch-based inference strategy, the network accuracy decreases at patch boundaries, giving higher weights to voxels closer to the center during prediction aggregation across patches;
[0066] Perform an ensemble by combining different models, automatically selecting the model that achieves the highest average foreground dice score in cross-validation on the training set to increase segmentation performance and robustness, and select the best performing model for final submission;
[0067] Use the trained model to generate segmentations at full resolution, and perform post-processing on the segmentation results, applying connected component analysis techniques to ensure segmentation labels are coherent and remove smaller components to improve accuracy.
[0068] In step S2, three.js is used to render the 3D model, and Next.js is used to implement interactive query, specifically:
[0069] Upload the 3D reconstructed model file to the server, which is configured with the necessary Next.js and three.js environments for interactive query. Users can access the server model through the IP address and query the model in an interactive way. The model can be enlarged, rotated, stretched, irrelevant organs hidden, and the transparency of organs adjusted;
[0070] Users can use mobile app to access the model by scanning QR code, or interactively query through computer web application. The server-side deployment of the model and interactive query method make it convenient for users to use it anytime and anywhere.
[0071] In step S3, the 3D reconstruction model is developed and deployed for use in a real-world environment, and the reconstructed image is projected onto a real scene in the following manner:
[0072] Use Xcode with ARKit to develop apps, ensuring that the rendering and interaction of models meet the accuracy required by real scenes;
[0073] It is planned to fit and superimpose the intraoperative images with the three-dimensional images, so that doctors can obtain a better field of view during minimally invasive surgery, clearly segment the tumor and organs, and improve the success rate of the operation.
[0074] This embodiment provides a method for building an AI-enabled liver tumor three-dimensional image reconstruction platform, including the following steps:
[0075] Step S1, segmenting organs and tumor parts in the CT data set using manual and AI methods respectively;
[0076] In an example embodiment:
[0077] like Figure 2As shown in the figure, 3D Slicer's Segment Editor segmentation tool is used to manually segment the CT image. First, create a new segmentation node, and then use "Paint" to manually draw the segmentation, or use "Threshold" for threshold-based segmentation. After completing the basic segmentation, the segmentation results can be further edited and optimized. 3DSlicer provides a variety of tools and functions, such as smoothing, filling, trimming, joint segmentation, etc., to improve and adjust the segmentation results.
[0078] like Figure 3 As shown, the diffusion model pre-trained on the abdominal benchmark dataset AbdomenCT-1k is used to augment the existing data, and the original dataset and the generated dataset are merged into a new augmented dataset;
[0079] The expanded data is segmented into tissues, organs and tumors using nn-Unet. The program uses the default U-Net network architecture template to train multiple models. Through cross-validation, the best performing single model or model combination is selected from the trained models. If post-processing can improve the verification performance, post-processing is performed to generate a full-resolution segmentation result.
[0080] Step S2: Reconstruct the segmented image into a three-dimensional model, and use Next.js and three.js to deploy the model to the server backend, so that users can interactively view it through the front-end Web application;
[0081] like Figure 4 As shown, the 3D reconstructed model file is uploaded to the server, which is configured with the necessary Next.js and three.js environments for interactive query. Users can access the server model through the IP address and query the model in an interactive way. The model can be enlarged, rotated, stretched, irrelevant organs hidden, and the transparency of organs adjusted.
[0082] Step S3: Use the ARKit software development kit to deploy the augmented reality model, and then use augmented reality glasses to project the reconstructed model into the real scene to assist doctors in completing the tumor resection surgery;
[0083] Use Xcode to develop applications with the ARKit augmented reality framework to ensure that the rendering and interaction of the model meet the accuracy required by the real scene. The actual effect is as follows Figure 5 shown.
[0084] The above description is only an optional embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
[0085] Although the claims in this application have been formulated with respect to particular combinations of features, it should be understood that the scope of the present disclosure also includes any novel feature or any novel combination of features disclosed herein, whether explicitly or implicitly or in any generalization thereof, whether or not it relates to the same scheme in any claim currently claimed.
Claims
1. A method for building an AI-enabled liver tumor three-dimensional image reconstruction platform. It is characterized in that The steps include: S1. Use manual and AI methods to segment organs and tumors in CT datasets. S2, reconstruct the segmented image into a 3D model, and deploy the model to the server backend using Next.js and three.js, so that users can interactively view it through the front-end web application; S3. Use the ARKit software development kit to deploy the augmented reality model, and then use augmented reality glasses to project the reconstructed model into the real scene to assist doctors in completing tumor resection surgery.
2. The method for building an AI-enabled liver tumor three-dimensional image reconstruction platform according to claim 1, It is characterized in that In step S1, the artificial method used for medical image segmentation is specifically: Using 3DSlicer, the outline of the region of interest is manually drawn on each slice, and then the drawn outline is extended to the entire 3D dataset using an automatic filling algorithm to obtain a complete segmentation result; The manually segmented 3D model is exported to STL format, and then Blender is used to render the model. The desired rendering effect is obtained by adjusting the lighting, material, and texture properties to prepare for subsequent visualization and interaction.
3. According to the method for building an AI-enabled liver tumor three-dimensional image reconstruction platform according to claim 1, It is characterized in that In step S1, the AI method used for medical image segmentation is specifically: We augmented the existing data using a diffusion model pre-trained on the abdominal benchmark dataset AbdomenCT-1k, merging the original dataset and the generated dataset into a new augmented dataset. The nnU-Net model is trained on the expanded dataset for image segmentation.
4. According to the method for building an AI-enabled liver tumor three-dimensional image reconstruction platform according to claim 3, It is characterized in that Among the AI methods used for medical image segmentation, the way to use nnU-Net for image segmentation is: Preprocess the CT data set, including cropping, resampling, and normalization steps; Utilizes a pool of basic U-Net architectures, automatically configured and trained using a five-fold cross-validation approach; Combine Dice and Cross Entropy Loss to train the nnU-Net network: L total =L dice +L CE The dice loss is calculated based on the softmax output of the network u and the true segmentation map v: where u is the softmax output of the network, v is the one-hot encoding of the ground truth segmentation map, and both u and v have shape I×K, where i∈I is the number of pixels in the training block or batch, and k∈K is the class; Using a patch-based inference strategy, the network accuracy decreases at patch boundaries, giving higher weights to voxels closer to the center during prediction aggregation across patches; Perform an ensemble by combining different models, automatically selecting the model that achieves the highest average prospect dice score in cross-validation on the training set, and choosing the best performing model for final submission; Use the trained model to generate segmentations at full resolution, and perform post-processing operations on the segmentation results, applying connected component analysis to ensure segmentation labels are coherent and remove smaller components to improve accuracy.
5. According to the method for building an AI-enabled liver tumor three-dimensional image reconstruction platform according to claim 1, It is characterized in that In step S2, three.js is used to render the 3D model, and Next.js is used to implement interactive query, specifically: The 3D reconstructed model file is uploaded to the server. The server is configured with the necessary Next.js and three.js environments for interactive query. Users access the server model through the IP address and query the model in an interactive way. Users use mobile app to access the model by scanning QR code, or interactively query through computer web application.
6. The method for building an AI-enabled liver tumor three-dimensional image reconstruction platform according to claim 1, It is characterized in that In step S3, the 3D reconstruction model is developed and deployed for use in a real-world environment, and the reconstructed image is projected onto a real scene in the following manner: Use Xcode with ARKit to develop apps, ensuring that the rendering and interaction of models meet the accuracy required by real scenes; It is planned to fit and superimpose the intraoperative images with the three-dimensional images, so that doctors can obtain a better field of view during minimally invasive surgery, clearly segment the tumor and organs, and improve the success rate of the operation.