Medical image three-dimensional visual diagnosis method and system based on artificial intelligence and Unreal Engine

Abstract

The invention discloses a medical image intelligent diagnosis and three-dimensional visualization interaction method and system based on artificial intelligence and Unreal Engine. The system is composed of a data acquisition module, an artificial intelligence analysis module, a three-dimensional reconstruction module, a rendering interaction module and a report generation module. The method comprises the following steps: firstly, carrying out voxel normalization on a DICOM / NIfTI image; the improved 3D U-Net is combined with CBAM attention and WGAN-GP training, and a focus initial density field is output; an AI enhanced implicit voxel reconstruction (AEIVR) algorithm is proposed, an enhanced density field is generated by using position coding + MLP, and grid extraction is completed by using improved Marking Cubes of a dynamic threshold; high-fidelity real-time rendering is realized in combination with a Nand micro polygon and Lumen global illumination in Unreal Engine, and VR / AR interaction and virtual surgery (real-time updating of resection mask driving) are supported. According to the method, the problems of large artifacts and insufficient interactivity and real-time performance of existing two-dimensional output and three-dimensional reconstruction are solved, and the end-to-end segmentation-reconstruction-rendering-interaction-reporting process is realized. On typical CT / MRI data, the segmentation Dice is about 0.90-0.94, the reconstruction voxel resolution is 0.1-0.5 mm, the artifacts are reduced by about 20%-30%, the PSNR is improved by about 3-7 dB, and the rendering interaction delay lt is achieved; the system is suitable for scenes such as lung CT, brain MRI, orthopaedic X-ray and cardiac ultrasound, and can be expanded to preoperative planning, remote consultation and teaching.

Description

I. Technical Field

[0001] This invention relates to the fields of artificial intelligence, medical image processing, and 3D visualization. Specifically, it relates to a method and system that combines an implicit density field inspired by an improved 3D U-Net neural network, a generative adversarial network (GAN), and NeRF (Neural Radiance Field) with Unreal Engine's Nanite micropolygon geometry rendering technology and Lumen global illumination rendering technology to achieve automatic lesion segmentation, high-fidelity 3D reconstruction, and interactive diagnosis of medical images. II. Background Technology

[0002] Current medical imaging diagnostics (such as CT, MRI, X-ray, and ultrasound) mainly rely on manual interpretation by doctors, which is inefficient. For small lesions, manual interpretation is prone to missing them, with a false negative rate of approximately 10%–15% (based on a survey of publicly available literature). Because two-dimensional images cannot visually represent the three-dimensional spatial structure of lesions, the completeness and accuracy of the diagnosis are limited. In recent years, some artificial intelligence (AI)-assisted diagnostic systems have emerged, but they still have many shortcomings:

[0003] Lack of 3D reconstruction and interactive functions: For example, US Patent US20200372648A1 proposes a medical image-assisted diagnostic method based on convolutional neural networks (CNN), which can only segment lesions in two-dimensional images and output probability maps. This system is limited to planar analysis, cannot perform 3D reconstruction, and does not support interactive 3D visualization diagnosis.

[0004] For two-dimensional image enhancement only: For example, patent CN113902921B discloses a GAN-based image enhancement technology that can improve the quality of low-contrast medical images, but it does not involve the application of three-dimensional visualization or high-performance rendering engines, and cannot meet the clinical needs for three-dimensional diagnosis.

[0005] Limitations of traditional 3D reconstruction methods: Some existing 3D reconstruction methods (such as US Patent US20100328305A1) typically employ the Marching Cubes algorithm (isosurface extraction algorithm) with a fixed threshold, which performs poorly when processing low-contrast or high-noise images. For example, when the image signal-to-noise ratio (SNR) is below 20dB, a fixed threshold is insufficient to accurately distinguish between tissue and lesions, often resulting in numerous artifacts and noise, thus limiting reconstruction accuracy and rendering efficiency.

[0006] Insufficient Interaction and Simulation: Some technologies use Transformer models for reconstruction (such as Chinese patent CN111655160A), which can generate three-dimensional models of MRI images, but the interactive functions are limited and do not support advanced applications such as virtual surgical simulation, making it difficult to fully meet clinical needs.

[0007] NeRF methods suffer from poor real-time performance: NeRF (Neural Radiation Field, proposed by Mildenhall et al. at ECCV 2020) uses implicit function representations to reconstruct general 3D scenes, achieving excellent results in the field of computer vision. However, NeRF has extremely high computational complexity, with typical rendering times exceeding 1 second, making it unsuitable for real-time clinical applications in medical imaging.

[0008] In summary, there are currently no technologies or patents that apply Unreal Engine's Nanite and Lumen cutting-edge graphics rendering technologies to the 3D reconstruction and diagnostic interaction of medical images. Therefore, there is an urgent need for a new method that integrates AI high-precision segmentation, implicit 3D reconstruction, and real-time rendering from a game engine to improve the accuracy, efficiency, and interactive experience of medical image diagnosis. III. Summary of the Invention

[0009] This invention aims to provide a comprehensive solution integrating artificial intelligence, medical imaging, and Unreal Engine to overcome the shortcomings of existing technologies and improve the intelligence and visualization level of clinical diagnosis. Specific objectives include: Improving Lesion Segmentation Accuracy: An improved 3D U-Net algorithm combining Convolutional Block Attention Module (CBAM) and Generative Adversarial Network (GAN) is proposed to achieve high-precision automatic segmentation of lesions in medical images, especially improving the recognition rate of tiny lesions with a diameter <5mm and reducing the false negative rate. The AI-Enhanced Implicit Voxel Reconstruction (AEIVR) algorithm, combined with implicit density fields and dynamic thresholds, is employed to improve the accuracy and robustness of 3D reconstruction; enhancing the 3D reconstruction effect: We propose an AI-enhanced Implicit Voxel Reconstruction (AEIVR) algorithm that combines neural network implicit density field representation and dynamic thresholding strategy to improve the accuracy and robustness of 3D reconstruction, ensuring that clear and accurate 3D models can still be obtained under low-contrast and high-noise images.

[0010] Achieve high-fidelity real-time rendering and interaction: Utilizing the latest Nanite virtual geometry technology and Lumen dynamic global illumination technology from Unreal Engine, achieve high-fidelity real-time rendering of 3D lesion models. Support interactive diagnosis and virtual surgical simulation through immersive devices such as VR / AR, providing realistic visual effects and a smooth user experience.

[0011] Intelligent diagnostic report generation: Automatically generates diagnostic reports containing information such as lesion location, volume, boundary features, risk level, and historical image comparison, providing decision support for doctors and improving diagnostic efficiency and clinical application value. IV. Technical Solution

[0012] (I) Overall Architecture The system of this invention comprises five major modules: data input, AI diagnosis, 3D reconstruction, rendering interaction, and auxiliary output. These modules work collaboratively, as described below:

[0013] Data Input Module: Imports medical image data (e.g., DICOM 3.0 or NIfTI format files), supporting multiple modalities such as CT, MRI, X-ray, and ultrasound. Image resolution is preferably no less than 512×512 pixels. Preprocessing of the input data includes format parsing and grayscale normalization, standardizing voxel grayscale values ​​to the [0,1] range to reduce the influence of different devices and conditions. The normalization method uses linear scaling. ,in, This is the original voxel gray value (e.g., the Hounsfield unit HU value of a CT image). and These are the minimum and maximum grayscale values ​​of the image data, respectively. The volumetric data after standardization... It will be used as input for subsequent AI models.

[0014] AI Diagnostic Module: This module utilizes an improved 3D U-Net deep neural network (containing 32 convolutional layers with 3×3 kernels) to extract features and segment lesions from standardized volumetric data. The network incorporates a CBAM attention mechanism (including channel and spatial attention) to enhance focus on key regions. Furthermore, feedback from a generative adversarial network discriminator is fused to improve segmentation robustness. Specifically, the initial segmentation result from the U-Net3D output is element-wise multiplied with the attention mechanism output to obtain the initial density field. , representing the confidence level that each voxel belongs to a lesion. The segmentation model parameters are optimized by introducing the Wasserstein GAN with Gradient Penalty (WGAN-GP) loss function, which takes the form: ,in This represents the label density field of the actual lesions, with lesion areas = 1 and non-lesion areas = 0. This is the gradient of the discriminator with respect to the interpolated samples. This loss ensures that the generated results are close to the true distribution and improves training stability. After adopting the above improvement strategy, the segmentation accuracy (measured by the Dice coefficient, which reflects the overlap between the segmentation results and the ground truth) can reach about 0.90 to 0.94, which is about 10% to 20% higher than the model without attention and adversarial training.

[0015] 3D Reconstruction Module: Based on the initial density field output by the AI ​​module. The AI-Enhanced Implicit Voxel Reconstruction (AEIVR) algorithm proposed in this invention reconstructs two-dimensional image sequences into three-dimensional voxel models. This algorithm combines explicit thresholding segmentation and implicit field representation, which can reduce noise artifacts while preserving details. The specific process includes: first, generating an implicit density field representation, and then performing isosurface extraction and mesh generation (see key technical points below for details). The reconstructed output is a high-precision three-dimensional mesh model (such as OBJ format), providing a data foundation for the rendering module.

[0016] Unreal Engine Rendering Module: Leveraging the powerful graphics rendering capabilities of Unreal Engine, this module performs real-time rendering of the reconstructed 3D models. On one hand, it employs UE5's Nanite technology to optimize the rendering efficiency of high-polygon models. Nanite is a virtualized micropolygon geometry system that can render even models containing more than [a certain number of polygons]. Even individual polygon vertices can be rendered efficiently. On the other hand, by combining UE5's Lumen global illumination system, dynamic lighting effects and global ray tracing are achieved, improving the contrast and visual realism of lesions and surrounding tissues. Through the collaboration of Nanite and Lumen, this invention can achieve real-time interactive rendering of medical image models while maintaining cinematic image quality.

[0017] Interaction Module: Offers diverse interaction methods, including VR (Virtual Reality) / AR (Augmented Reality) devices and traditional PC interfaces. Doctors can wear VR headsets (such as Oculus Rift) or AR smart glasses (such as Microsoft HoloLens) to view and manipulate 3D models in an immersive manner. The system supports operations such as model rotation, scaling, cross-sectional cutting, and transparency adjustment, with a rotation resolution accuracy of up to 0.1°. Furthermore, it provides virtual surgical simulation functionality, allowing doctors to perform operations such as resection and marking on the model in a virtual environment. The system provides real-time feedback on postoperative tissue changes (e.g., the morphology of remaining tissue after the removal of a certain area). The entire interaction process is optimized with a latency of less than 60ms, ensuring that user actions are reflected on the display screen almost in real time, providing a smooth user experience.

[0018] Assisted Diagnostic Output: After completing image analysis and interactive operations, the system automatically generates an auxiliary diagnostic report. The report includes the location, size (volume), shape and boundary characteristics of the discovered lesions, lesion type, and risk level assessment. It also allows for comparative analysis of current results with historical image records. This report can be used by physicians for diagnostic recording and surgical planning, improving the scientific rigor and efficiency of clinical decision-making.

[0019] (II) Key Technologies

[0020] 1. Intelligent Segmentation Algorithm: The lesion segmentation algorithm of this invention employs an improved 3D U-Net deep learning model, combining an attention mechanism and adversarial training to enhance performance. The introduced CBAM attention mechanism (Convolutional Block Attention Module) includes channel attention and spatial attention, adaptively emphasizing feature channels and spatial locations useful for lesion detection, thereby improving the detection rate of small lesions (diameter < 5 mm). For medical images with low contrast and high noise (SNR < 20 dB), this invention incorporates a WGAN-GP discriminator into the segmentation network training to improve the performance of the predicted density field. With real labeling Adversarial training allows the model to better learn noise distribution and edge blurring, improving robustness and generalization ability.

[0021] 2. 3D Reconstruction – AEIVR Algorithm: The AI-Enhanced Implicit Voxel Reconstruction (AEIVR) algorithm proposed in this invention is the core of the 3D reconstruction module. It integrates the advantages of implicit representation and explicit mesh extraction methods, and includes the following steps: Implicit density field generation: The initial density field obtained from AI segmentation. Integrating into the implicit representation of the neural network to obtain a smoother and more detailed density field The specific approach involves employing a Multilayer Perceptron (MLP) model, taking the spatial coordinates of voxels and initial density values ​​as input, to predict the corrected density values. To enhance the MLP's ability to fit spatial details, positional encoding is introduced to embed the coordinates at high frequencies. The above formula applies to three-dimensional coordinates conduct The sine and cosine function transforms of the first level generate a high-dimensional positional encoding vector. This is used to provide multi-scale spatial frequency information. Then, the location code is concatenated with the initial density field value (denoted as...). The input is fed into a 5-layer fully connected MLP network: The MLP has a hidden layer dimension of 256, uses ReLU as the activation function, and outputs... This is the enhanced implicit density field value (which can be understood as the probability or volume density value that the voxel belongs to the lesion). The detail noise correction term learned by GAN is used to restore the subtle texture of the true lesion boundary. Through the above implicit representation, the model can suppress the influence of random noise on the reconstruction while interpolating to complete missing data.

[0022] Voxel to mesh transformation: For the obtained density field An improved Marching Cubes algorithm is used to extract isosurfaces and generate a 3D triangular mesh model. Unlike traditional methods that use a fixed threshold, this invention dynamically calculates the isosurface extraction threshold based on the statistical characteristics of the reconstructed data. To better distinguish the foreground (lesion) from the background: in The global mean of the density field values. Standard deviation, The appropriate value can be selected based on the image noise level and the desired level of reconstruction detail (usually taken as...). This dynamic thresholding strategy adaptively adjusts sensitivity, appropriately lowering the threshold in low-contrast data to capture minute lesions and raising the threshold in high-noise data to filter out stray noise. Marching Cubes mesh vertex calculations use linear interpolation formulas to determine the intersection points of isosurfaces and voxel edges: in and For two adjacent lattice points in a voxel unit, their density values ​​are respectively and , This refers to the position of the threshold isosurface on the line connecting these two grid points. By performing the above calculation on each cubic unit and connecting them to generate triangular patches, a complete lesion surface mesh can be constructed. Tests show that the improved algorithm using dynamic thresholding can reduce artifact noise by approximately 20%–30%, significantly improving the smoothness and accuracy of the reconstructed model.

[0023] Model training loss: To train the implicit reconstruction model (including the aforementioned MLP and threshold optimization module), this invention designs a loss function that comprehensively considers pixel differences and structural similarity: in, is the density field of the real three-dimensional lesion (obtained by fine manual delineation or high-quality images), MSE is the mean squared error loss, and SSIM is the structural similarity index (used to measure the similarity between the reconstructed result and the real object in terms of structure and contrast, with a value range of [0,1]). and The weighting coefficients are determined based on experience. , By simultaneously minimizing pixel-level errors and structural differences, the model can balance global accuracy and local detail fidelity, and the reconstructed 3D density field has higher peak signal-to-noise ratio (PSNR) and SSIM value in quantitative metrics.

[0024] 3. Clinical Interactive Functions: This invention places great emphasis on the operability of clinical applications, and features rich 3D interactive functions, especially virtual surgical simulation. Doctors can directly perform "virtual surgery" operations on the reconstructed 3D model within the visualization interface provided by Unreal Engine, such as using controllers to select and "remove" tumor tissue or add virtual implants. The system simulates tissue removal by applying a user-defined binary mask to the density field. in This represents a user-defined mask function for the excision area, with a value of 0 at the area to be removed and 1 in other areas. This operation is equivalent to clearing the density value of the selected area to zero, thus simulating the state of tissue removal after surgery. After the excision operation is completed, the AI ​​model can perform real-time analysis. The system analyzes and updates parameters such as the extent of residual lesions and recalculates volume, then feeds the results back to the doctor. This human-computer interaction loop allows doctors to assess the impact of different surgical approaches on the lesions and surrounding tissues, improving the scientific rigor and personalization of preoperative planning.

[0025] (III) Comparison of innovative features with existing technologies Compared with existing related technologies, the present invention has the following innovations and advantages in its technical solution: Compared to US20200372648A1, which uses only a 2D CNN to segment medical images and outputs a planar probability map, lacking 3D reconstruction and interactive functions, this invention innovatively introduces an improved 3D U-Net network combined with adversarial learning. This enables the direct acquisition of a 3D lesion density field from 2D images, and the segmentation results can be used to reconstruct the complete 3D lesion morphology. Experiments show that the average Dice coefficient of the 3D segmentation of this invention is approximately 0.90–0.94, which is about 10%–20% more accurate than the traditional 2D methods reported in the literature. Furthermore, through VR / AR rendering interaction using Unreal Engine, this invention provides an immersive 3D diagnostic environment, representing a significant improvement over the 2D static output of US20200372648A1.

[0026] Compared to CN113902921B, which focuses on using GANs to improve the contrast and clarity of two-dimensional medical images without involving any three-dimensional model reconstruction or user interaction, this invention revolves around three-dimensional reconstruction. Based on the segmented two-dimensional region of the lesion, it further generates a high-fidelity three-dimensional model of the lesion using the AEIVR algorithm, achieving a voxel resolution of 0.1mm–0.5mm, realistically reflecting the three-dimensional anatomical structure of the lesion. Furthermore, this invention incorporates Unreal Engine rendering, allowing doctors to observe the lesion from any angle and perform interactive operations, extending its application scope far beyond the image enhancement method described in CN113902921B.

[0027] Compared to US20100328305A1: This patent uses the traditional Marching Cubes algorithm to reconstruct 3D surfaces. Due to the use of a fixed threshold, it is prone to artifacts when dealing with low-contrast or high-noise images, and the reconstructed mesh is often not fine enough, with rendering efficiency limited by the mesh size. This invention introduces a dynamic threshold strategy into the reconstruction algorithm, adaptively adjusting the isosurface extraction standard to achieve optimal reconstruction results under different data conditions. Combined with Unreal Engine's Nanite micropolygon rendering technology, this invention improves rendering efficiency by approximately 20% to 40% compared to traditional engines, and can still display smoothly even models containing millions of triangles. Simultaneously, statistical test results show that the reconstructed model of this invention reduces artifact noise by approximately 20% to 30% (measured by quantitatively calculated noise point cloud area), significantly improving visual quality.

[0028] Compared to CN111655160A: This patent uses a Transformer model to generate a 3D reconstructed model of MRI images, but its output is usually a static 3D model without in-depth design of clinical interactive functions. This invention not only generates accurate 3D models but also fully utilizes the advantages of a game engine to achieve rich user interaction, especially with its pioneering virtual surgical simulation function. Doctors in this system can operate on the 3D model with a low latency of <60ms, achieving near real-time interaction smoothness. This allows for realistic simulation of surgical procedures and immediate AI-assisted feedback, something that existing technologies such as CN111655160A have failed to achieve.

[0029] Compared to NeRF technology: While NeRF can reconstruct complex scenes through implicit network representation, it requires optimized training from multi-view images, and rendering a single frame often takes more than 1 second, making it difficult to meet clinical requirements. The AEIVR algorithm of this invention borrows the implicit representation idea of ​​NeRF but has undergone targeted optimization: it uses medical image prior segmentation guidance to reduce the optimization dimensionality; it introduces dynamic thresholding to quickly extract surfaces; and it utilizes Unreal Engine to accelerate rendering. The final system can complete the rendering and update of a 3D image frame within <60ms, meeting the real-time requirements of clinical applications and being more closely aligned with engineering applications.

[0030] Pioneering Integration: This invention is the first to apply Unreal Engine's Nanite and Lumen rendering technologies to the field of 3D visualization and diagnosis of medical images, breaking through the constraints of traditional medical imaging and interaction. Through the ingenious combination of the AEIVR algorithm and UE5 graphics technology, a high-fidelity 3D reconstruction and immersive interactive diagnosis integrated solution is achieved, filling a gap in existing technologies and demonstrating significant innovation and cutting-edge technology.

[0031] (iv) Beneficial effects

[0032] The present invention, through the above-described technical solution, is expected to produce the following beneficial effects in clinical applications: High-precision segmentation: Utilizing an improved 3D U-Net + attention mechanism + GAN, high-precision automatic segmentation of lesions in various medical images is achieved, with an average Dice segmentation coefficient exceeding 0.90. Compared to manual image interpretation and traditional algorithms, the false negative rate for small lesions (<5mm) is reduced to below 5%, significantly improving the early detection rate of lesions.

[0033] High-fidelity 3D reconstruction: The AEIVR reconstruction algorithm can generate detailed 3D models with a resolution of 0.1–0.5 mm, and the reconstruction results are highly consistent with the real anatomical structures. Compared with the fixed threshold method, artifact noise is reduced by about 20%–30%, and the peak signal-to-noise ratio (PSNR) of the reconstructed model is improved by about 3–7 dB, significantly improving the reliability and diagnostic value of the 3D model.

[0034] High-efficiency real-time rendering: Leveraging Unreal Engine's Nanite and Lumen technologies, this system achieves high-efficiency real-time rendering of large medical models. Compared to traditional medical visualization tools, the rendering frame rate is increased by approximately 20% to 40%, and the interaction latency is stabilized within 60ms, ensuring smooth switching between multiple views and angles and meeting the needs of real-time clinical operations.

[0035] Immersive Interactive Experience: Supports multiple interactive methods such as VR / AR, allowing doctors to observe and manipulate 3D medical images in an immersive manner. The spatial positioning accuracy of VR / AR devices can reach 0.1°. With the interaction of the controller, it can realize operations such as free rotation, magnification, cutting and annotation of lesions, improving the intuitiveness of diagnosis and the precision of surgical planning.

[0036] Wide Applicability: The method and system of this invention can be applied to various medical imaging scenarios, such as CT screening of lung nodules, MRI analysis of brain tumors, X-ray diagnosis of orthopedic fractures, and echocardiographic reconstruction of cardiac structures. The system has good scalability and can be further integrated with telemedicine to transmit high-fidelity three-dimensional diagnostic results in real time via the network, enabling remote consultations and medical teaching demonstrations, thus maximizing social benefits. (v) Description of the attached drawings Figure 1 System overall architecture diagram: This diagram shows the overall architecture of the system of the present invention, including a data input module, an AI diagnosis module, a 3D reconstruction module, an Unreal Engine rendering module, an interaction module, and an auxiliary diagnosis output module.

[0038] Figure 2 AI Diagnostic Module Flowchart: This flowchart depicts the processing flow of the AI ​​diagnostic module, from receiving pre-processed images to outputting the initial density field.

[0039] Figure 3 : AEIVR 3D reconstruction algorithm diagram. This diagram shows the structure of the AEIVR algorithm, including implicit density field generation and voxel-to-mesh conversion.

[0040] Figure 4Unreal Engine Rendering and Interaction Module Diagram: This diagram is a sub-architecture diagram of the system, focusing on the Unreal Engine rendering process and interaction functions.

[0041] Figure 5 Virtual surgery simulation flowchart. This flowchart illustrates the clinical interactive functions, especially the virtual surgery mode. (vi) Detailed Implementation Methods To more clearly illustrate the technical solution of the present invention, the workflow and effects of the system are described below with reference to specific embodiments. However, it should be understood that these embodiments are only for illustrating the present invention and are not intended to limit the scope of the present invention.

[0043] Example 1: Intelligent Diagnosis of Lung CT Images A patient's lung CT image (DICOM format, 512×512 pixel resolution, containing 300 slices) was input into the system of this invention. First, the data input module preprocessed the CT volume data, normalizing the original Hounsfield unit values ​​to [0,1]. Then, the AI ​​diagnostic module called a trained 3D U-Net model to automatically segment lung nodules from the preprocessed data. In this case, the AI ​​successfully detected a tiny nodule approximately 4mm in diameter and output an initial density field. The nodular region The value is close to 1 (high confidence), while other normal lung tissue values ​​are close to 0. Next, the 3D reconstruction module uses the AEIVR algorithm to... Optimization and reconstruction are then performed. First, an enhanced density field is generated using an implicit MLP model combined with location encoding. To address the significant noise in lung imaging, a dynamic threshold was selected. Marching Cubes extraction was performed to obtain a high-precision 3D surface mesh of the nodule. The reconstructed nodule model was output in OBJ format with a voxel resolution of approximately 0.2 mm, accurately depicting the spherical outline and surface details of the nodule. The Unreal Engine rendering module loaded the 3D model, optimized the rendering of the high-polygon mesh using Nanite, and calculated lighting in real time using Lumen, making the shadows and highlights of the nodule clearly visible in the virtual lung. The doctor wore an Oculus Rift VR headset with the model loaded, interacted with the controllers to zoom in on the nodule to view its internal structure, and used a virtual dissection tool to observe the cross-section of the nodule. Throughout the interaction, the system frame rate remained above 60 FPS, and the model rotation and scaling operations had almost no latency. Finally, the system generated an automatic diagnostic report, which showed: a nodule was detected in the right upper lobe subsegment, with a volume of approximately 0.5 cm³, spiculated edges, and uniform density. Combining imaging features and the patient's medical history, the AI ​​assessed the nodule's malignancy risk as high and recommended prompt surgical intervention. The report also included a comparison of the current nodule with the patient's CT scan from a physical examination a year ago, showing that the nodule had grown from 2 mm in diameter to 4 mm. This information provided strong support for the doctor's diagnostic decision.

[0044] Example 2: Three-dimensional reconstruction of brain MRI images and surgical simulation This embodiment uses MRI scan data (NIfTI format 3D volumetric data) from a patient with a brain tumor. First, the system reads the MRI volumetric data and performs standardized preprocessing. Because the grayscale range of MRI differs from that of CT (non-HU values), the system normalizes the data based on its minimum and maximum values. The AI ​​module then applies a trained segmentation model to segment various brain structures, focusing on identifying abnormal tumor regions and generating an initial density field. The segmentation results showed an oval-shaped tumor in the left frontal lobe region. High values ​​are taken in this region, corresponding to clearly defined tumor boundaries. The 3D reconstruction module utilizes the AEIVR algorithm for... Further refinement was carried out, especially considering the high signal-to-noise ratio (SNR) of MRI images, by adjusting the threshold factor. The density was appropriately reduced to 1.2 to capture possible details of tumor invasion. The resulting enhanced density field... After conversion using Marching Cubes, a realistic 3D model of the brain tumor and surrounding brain tissue structures is obtained. To better observe internal relationships, the rendering module uses a semi-transparent material (approximately 50% transparency) for the tumor entity, while highlighting the tumor edges and surrounding important blood vessels to facilitate doctors' identification of the tumor's adjacency with functional areas. In the Unreal Engine visualization interface, doctors load the model through a desktop application and interact with it using a keyboard and mouse. The system supports arbitrary plane cutting: doctors can cut the model along the coronal plane to view the cross-sectional anatomy of the tumor inside the brain in real time. Subsequently, the doctor initiates virtual surgery mode and uses controls to delineate the area of ​​the tumor to be removed. Once the "remove" command is confirmed, the system immediately updates the 3D model. Tumor tissue within the region was removed and recalculated. The simulation of the resection outcome was then performed. The AI ​​module subsequently analyzed the remaining tissue, indicating locations of residual abnormal tissue that might require further treatment. Throughout the surgical simulation, the system maintained an interaction latency of less than 50ms, ensuring the continuity of the surgical procedure. The final surgical simulation report included: the tumor volume was approximately 30 cm³, with well-defined boundaries and no significant transgression of critical functional areas; the estimated integrity of the remaining normal brain tissue after the simulated resection was good. This report can provide a reference for preoperative planning.

[0045] Example 3: Orthopedic X-ray imaging-assisted diagnosis The system of this invention can also function effectively for X-ray imaging (digital DR image, size 2048×2048) of a fracture. First, two X-ray images from different angles are input into the system. After preprocessing and feature alignment, they are combined to generate a coarse three-dimensional bone density field (for single-angle X-rays, a prior anatomical model can also be used to assist in generating initial three-dimensional information). The AI ​​diagnostic module uses a trained bone injury detection model to identify the fracture area in the midshaft of the tibia and outputs the initial density field. The density value at the fracture sutures was significantly reduced. Next, the AEIVR 3D reconstruction algorithm... The system processes the data to generate a 3D model of the tibia and fracture fragments. Since X-ray imaging is projection-based, reconstruction has significant uncertainties. Therefore, the system incorporates anatomical prior constraints and manual correction to adjust the position and angle of the fracture segments, ultimately obtaining an anatomically aligned 3D skeletal model. The rendering module applies a semi-transparent material to the skeletal model and highlights the fracture surface to make the fracture easily identifiable. Doctors view this 3D skeletal model using HoloLens AR glasses, which overlay virtual skeletal images onto the patient's actual limb, facilitating a direct assessment of the fracture. Through gesture interaction, doctors can zoom in on the fracture site and activate measurement tools to automatically calculate the width of the fracture fissure and the distance of bone fragment misalignment. Then, using the system's virtual internal fixation tools, doctors simulate the placement of plates and screws on the model for reduction and fixation. The system adjusts the bone fragment positions in real time based on the screw placement to ensure the rationality of the virtual reduction and provides feedback on the appropriateness of parameters such as screw length and angle. After the simulation was completed, the system generated a report describing the fracture type as a comminuted fracture with 3 bone fragments; recommending the use of a 120mm long plate with 6 screws for fixation; and stating that the bone axis deviation after reduction was <2°. This result provides a quantitative reference for doctors to develop actual surgical plans.

[0046] As can be seen from the above embodiments, the present invention can adapt to various medical imaging scenarios and provide an end-to-end solution from image acquisition, AI analysis, 3D reconstruction, virtual interaction to report generation. While improving diagnostic accuracy, it also provides doctors with new visualization tools and decision support methods.

Claims

1. A medical image processing and diagnosis method based on artificial intelligence and Unreal Engine, characterized in that, include: 1) Acquire and preprocess medical images; 2) The lesions are segmented using an artificial intelligence model to obtain the initial density field; 3) Based on the segmentation results, perform 3D reconstruction to generate a 3D model; 4) Render 3D models using Unreal Engine and provide interactive features; 5) Output a diagnostic report that includes lesion characteristics and risk assessment.

2. The method according to claim 1, characterized in that, The artificial intelligence model described is an improved 3D U-Net combined with the CBAM attention mechanism. Output:

3. The method according to claim 1, characterized in that, The 3D reconstruction employs an AI-enhanced implicit voxel reconstruction (AEIVR) algorithm, which includes position encoding, multilayer perceptron-enhanced density calculation, dynamic threshold isosurface extraction, and improved Marching Cubes mesh generation.

4. The method according to claim 1, characterized in that, Unreal Engine uses Nanite and Lumen technologies for rendering, supports VR / AR interaction, and has an interaction latency of less than 60ms.

5. The method according to claim 1, characterized in that, In the virtual surgery mode, the user defines the excision mask. , System calculation: And the diagnostic results are updated in real time.

6. A medical image processing and diagnostic system based on artificial intelligence and Unreal Engine, characterized in that, include: The system includes a data acquisition module, an artificial intelligence analysis module, a 3D reconstruction module, a rendering and interaction module, and a report generation module.

7. The system according to claim 6, characterized in that, The artificial intelligence analysis module uses the WGAN-GP loss function: L G A N = E D σ 0 - D σ g t + λ ⋅ E | | ∂ D ∂ σ i n t e r p | | 2 - 1 2 ,in This represents the label density field of the actual lesions, with lesion areas = 1 and non-lesion areas = 0.

8. This is the gradient of the discriminator with respect to the interpolated samples. This loss ensures that the generated results are close to the true distribution and improves training stability.

9. A computer-readable storage medium having a program stored thereon that, when run on a processor, performs the method as described in any one of claims 1-5.

Images