Method for reconstructing three-dimensional model based on artificial intelligence by using two-dimensional image and application thereof

Abstract

The invention provides a method for reconstructing a three-dimensional model based on artificial intelligence by using a two-dimensional image and an application thereof, when an implant is required to replace a lesion tissue during an operation, the lesion tissue does not need to be scanned by three-dimensional CT scanning, and only the two-dimensional medical image needs to be shot, so that the three-dimensional model is reconstructed. And a three-dimensional model of the lesion tissue can be obtained through the artificial intelligence model. A doctor can understand the specific condition of a surgical site through the three-dimensional model, then make a targeted surgical scheme, and directly design a corresponding implant and a customized surgical auxiliary tool in the three-dimensional model according to the surgical site. Meanwhile, related auxiliary design of the implant and the surgical auxiliary tool in the manufacturing process through the three-dimensional printing technology can be automatically generated, and finally, the generated design data of the implant and the surgical auxiliary tool are sent to the three-dimensional printing equipment to directly complete printing operation of the implant and the surgical auxiliary tool. The working process can be greatly simplified, the accuracy is improved, and the design requirements of customized medical equipment are met.

Description

Technical Field

[0001] The patent of this invention relates to a method for processing medical images, in particular, a method for reconstructing two-dimensional medical images into three-dimensional models through artificial intelligence. Background Art

[0002] Interdisciplinary collaborations between medicine and engineering are bringing cutting-edge solutions to healthcare systems, including the use of 3D printing to reconstruct human tissue at surgical sites. This concept brings the design and manufacturing of patient-specific implants and tools in-house, reducing the time and cost of 3D medical printing and developing products that better meet diverse clinical needs.

[0003] However, for most medical institutions, the major obstacle is designing a suitable production process to complete the 3D printing of human tissue for surgical sites. Designing artificial tissue to meet both clinical needs and engineering production standards for patient-specific implants or devices is extremely complex, requiring the support of numerous engineering experts with extensive medical knowledge and requiring significant effort and preparation time. Hospitals often lack experienced engineering teams to carry out these tasks. For medical institutions, establishing a qualified engineering team and a good collaborative mechanism between clinical professionals and engineers is extremely costly and time-consuming, often impractical. Furthermore, seeking to partner with external companies with sufficient engineering and manufacturing capabilities also carries potential drawbacks, such as the risk of contamination during long-distance transportation between factories and hospitals, as well as related regulatory issues. Summary of the Invention

[0004] In response to these shortcomings, the present invention has developed a method for reconstructing three-dimensional models based on artificial intelligence using two-dimensional images. The method is used to generate a three-dimensional model of the surgical site through the patient's two-dimensional medical imaging data through the intervention of artificial intelligence. The three-dimensional model is used to assist medical personnel in designing artificial implants and related auxiliary tools, planning surgical plans, etc., thereby improving surgical results and reducing patients' treatment costs.

[0005] The present invention is achieved through the following technical solutions: A method for reconstructing a three-dimensional model using two-dimensional images based on artificial intelligence, comprising: a training data set I, wherein each set of training data includes at least one or more two-dimensional image data and multiple slice image data corresponding to the two-dimensional image along a specified dimension; Inputting the training data set I into the deep learning neural network I for training to obtain a three-dimensional image reconstruction model; Training data set II, where each set of training data includes multiple slice image data, annotated points of specified objects in the slice image data, and converting the annotated points of all slice image data into a three-dimensional model related to the object; Input the training data set II into the deep learning neural network II for training to obtain a three-dimensional model reconstruction model; The two-dimensional image to be predicted is input into the three-dimensional image reconstruction model to obtain three-dimensional image data corresponding to the two-dimensional image to be predicted; then the three-dimensional image data is input into the three-dimensional model reconstruction model to obtain a three-dimensional model of the specified object corresponding to the two-dimensional image to be predicted.

[0006] The single or multiple two-dimensional image data in the training data I are two-dimensional image data obtained by X-ray scanning and / or two-dimensional image data in image data obtained by three-dimensional CT scanning.

[0007] The plurality of slice image data are slice image data obtained by slicing image data obtained by three-dimensional CT scanning along a specified dimension.

[0008] The method further includes a tissue density prediction model for predicting tissue density distribution, which includes: Training set III, wherein each set of training data includes two-dimensional image data and data for tissue density annotation of the two-dimensional image data; Input the training set III into the deep learning neural network III training to obtain the tissue density prediction model; The two-dimensional image to be predicted is input into the tissue density prediction model to obtain the density distribution data of the specified object.

[0009] Among them, the three-dimensional model of the specified object is imported into the three-dimensional graphics processing software, the surgical site is marked on the three-dimensional model, and the fitting surface of the fixture that matches the outer contour of the surgical site is generated by the three-dimensional graphics processing software, and then the remaining part of the fixture is set on the fitting surface through the software.

[0010] The beneficial effects of the present invention are as follows: When a patient needs surgery to replace diseased tissue with an implant, there is no need to use a relatively expensive three-dimensional CT scan to scan the diseased tissue to form a three-dimensional image for three-dimensional printing of the implant. Instead, the artificial intelligence model of the present invention can be used to directly obtain a three-dimensional model of the corresponding diseased tissue, such as a three-dimensional model of the bones at the surgical site, by taking relatively inexpensive two-dimensional medical images, such as X-ray scans. The doctor can use this three-dimensional model to clearly understand the specific conditions of the surgical site and then make a targeted surgical plan. The doctor can also directly design the corresponding implant and customized surgical auxiliary tools for the surgical site directly on the three-dimensional model. At the same time, the doctor can automatically generate relevant auxiliary designs for the production process of implants and surgical auxiliary tools using three-dimensional printing technology. Finally, the design data of the generated implants and surgical auxiliary tools is sent to the three-dimensional printing equipment to directly complete the printing operation of the implants and surgical auxiliary tools. This can greatly simplify the workflow, improve accuracy, and meet the needs of customized medical equipment design. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present invention will be further described below in conjunction with the accompanying drawings: Figure 1 It is a schematic diagram of the architecture of the three-dimensional image reconstruction model; Figure 2 It is a flow chart for preprocessing two-dimensional image data in a three-dimensional image reconstruction model; Figure 3 It is a schematic diagram of the architecture of the 3D model reconstruction model. DETAILED DESCRIPTION

[0012] In order to ultimately convert a two-dimensional image into a three-dimensional model corresponding to a specified object, the present invention adopts two sets of artificial intelligence models: a three-dimensional image reconstruction model and a three-dimensional model reconstruction model.

[0013] The three-dimensional image reconstruction model converts the two-dimensional image to be predicted into corresponding three-dimensional image data, and the three-dimensional model reconstruction model converts the three-dimensional image data formed above into a three-dimensional model.

[0014] Reference Figure 1 and Figure 2 The 3D image reconstruction model uses a neural network with an encoder-converter-decoder structure to reconstruct 3D CT scans from 2D X-ray projections. The neural network is trained using supervised learning, and loss functions such as mean squared loss and structural similarity index (SSIM) are used to evaluate the reconstruction loss.

[0015] The encoder module consists of multiple convolutional blocks, each containing multiple two-dimensional convolutional layers to extract structural information hidden in the X-ray projections. Skip connections are introduced between convolutional layers to ensure the transfer of information between lower and higher layers. The convolutional blocks are arranged sequentially to extract, filter, and refine the collected information.

[0016] The transformer is used to reshape the collected information from 2D to 3D for subsequent 3D reconstruction based on the information extracted from the encoder module. Finally, the decoder module is responsible for building the CT scan prediction data in the form of an array based on the information extracted from the X-ray projections.

[0017] The decoder consists of multiple 3D deconvolution layers arranged in sequence, with the number of channels decreasing towards the output. Throughout the network, layers performing batch normalization and rectified linear units (ReLU) are also added to improve the stability of the network during training.

[0018] The neural network training process involves repeatedly adjusting its parameters to minimize a chosen loss function, such as mean squared loss and structural similarity index (SSIM). The goal is to optimize the network's performance for reconstructing accurate 3D CT scan predictions from 2D X-ray projections.

[0019] To begin the training process, a dataset consisting of pairs of 2D X-ray projections and corresponding ground-truth 3D CT scans is prepared. Ground-truth CT scans are typically acquired using conventional CT scanning techniques. The DICOM images storing the CT scans are preprocessed to extract intensity maps. The intensity values ​​in these maps are converted to Hounsfield units (HU) using the rescale slope and rescale intercept values ​​provided in the DICOM metadata.

[0020] HU = m * SV + b Where SV represents the stored value in the intensity map (i.e., pixel intensity), m represents the DICOM rescaling slope, and b represents the DICOM rescaling intercept. Note that the rescaling slope and rescaling intercept can be found in the DICOM tags (0028, 1053) and (0028, 1052), respectively.

[0021] This normalization ensures that intensity values ​​remain consistent across scans, allowing the network to learn meaningful patterns and relationships.

[0022] To optimize memory usage during training, the intensity maps are resized to the required dimensions. This resizing process reduces computational requirements without significantly sacrificing the information content of the input data. The resized intensity maps are then concatenated into a three-dimensional array representing the CT volume.

[0023] To generate X-ray projection counterparts of the CT scans, we employed a digitally reconstructed radiograph (DRR) approach. This approach projects the 3D CT volume onto a 2D plane to simulate X-ray projections. The generated X-ray projections are then compared with the corresponding 2D X-ray projections in the training dataset.

[0024] During training iterations, the network takes X-ray projections as input and forward-propagates through an encoder-transformer-decoder architecture. The encoder module extracts structural information from the X-ray projections, capturing important features and patterns. The transformer module reshapes this information from 2D to 3D, preparing it for subsequent reconstruction. The decoder module reconstructs 3D CT scan predictions using 3D deconvolution layers, gradually building up more refined predictions.

[0025] The network parameters are optimized by calculating the loss between the predicted and actual 3D CT scan values. Mean squared loss measures the average squared difference between the predicted and true values. The network learns to minimize this loss, making its predictions as close to the ground truth as possible. In raw data units, the root mean squared loss (RMSE) is often used as a more interpretable metric. SSIM considers brightness, contrast, and structural information to assess the structural similarity between the predicted and ground truth images.

[0026] The optimization process typically uses backpropagation and gradient descent algorithms to update the network parameters. The gradient of the loss function with respect to the network parameters is calculated, and the parameters are adjusted accordingly to minimize the loss. This iterative process continues until the network performance reaches a satisfactory level, or converges.

[0027] By optimizing network parameters through this training process, the AI ​​solution aims to improve the diagnostic accuracy and efficiency of volume reconstruction of CT scans from 2D X-ray projections. This advancement has significant implications for various medical fields, as it can improve patient care, treatment planning, and diagnosis.

[0028] Reference Figure 3 The 3D model reconstruction model is a technical solution for binary semantic segmentation of bone tissue in the above CT scans using the U-Net architecture. The input CT scan image is preprocessed to normalize its Hounsfield unit and scale the pixel intensity between 0 and 1.

[0029] The U-Net architecture consists of an encoder and a decoder. The encoder contains multiple convolutional layers to extract features, while the decoder contains multiple deconvolutional layers to upsample the feature maps to the resolution of the original image. Skip connections between the encoder and decoder enable the network to accurately reconstruct the original image. The network output is a binary mask of bone tissue in the CT scan.

[0030] Regarding data availability, we collected CT scan images in DICOM format, and experienced staff accurately labeled the bone tissue within each DICOM slice. The binary image and its corresponding raw DICOM slice were considered a data set. The intensity arrays of the raw DICOM slices were extracted and converted to HU values. After data preparation (ideally, more than 500 data sets), supervised learning was performed using the proposed neural network architecture. The loss functions used for training the network were the Sorensen-Dice (DICE) loss and the mean squared error (MSE).

[0031]

[0032] y refers to the true value (i.e., the area in the image that should be highlighted), and y_pred refers to the predicted value (i.e., the area in the image that is predicted to be highlighted). The network parameters are adjusted to minimize the loss function.

[0033] Tissue density prediction models, specifically deep learning models for bone mineral density (BMD), are used to predict bone mineral density (BMD) from X-ray CT images. We used a V-Net architecture trained on a preprocessed dataset. We collected a set of diverse CT scans of target bones, each matched to precise BMD measurements from dual-energy X-ray absorptiometry (DXA). These images underwent preprocessing, including bone segmentation, image intensity normalization, and resizing to a uniform voxel size.

[0034] The V-Net architecture, renowned for its effectiveness in volumetric segmentation, is now being adapted for BMD prediction. This requires a specialized encoder-decoder structure with skip connections, where the encoder downsamples the input and the decoder restores the resolution. The network utilizes 3D convolutional layers with batch normalization and ReLU activation functions, culminating in an output layer that predicts continuous BMD values.

[0035] Training involves splitting the dataset into training, validation, and test sets, using mean squared error as the loss function and adaptive learning rates using optimization algorithms such as Adam. Model accuracy is evaluated using metrics such as mean absolute error and correlation between predicted and actual BMD values. To ensure that the model's decisions are easily understood, visualization techniques such as Gradient Weighted Class Activation Mapping (Grad-CAM) are also used.

[0036] The trained model is embedded into the platform, providing detailed BMD assessments directly within the clinical workflow, enhancing surgical planning and diagnosis. The software displays the model's BMD predictions and highlights critical areas for the surgeon. This aids in the diagnosis of conditions such as osteoporosis and assists in patient-specific surgical planning by informing decisions about surgical approach and implant type. Metrics based on BMD predictions provide an estimate of bone strength and health. Understanding a patient's BMD is crucial in surgery, especially orthopedic procedures such as joint replacements and spinal surgery. It can help surgeons assess the risk of fracture or complications during and after surgery. BMD-based metrics displayed in the software can help surgeons identify strong and weak bones at increased fracture risk, assisting with appropriate surgical techniques and implant selection, as well as postoperative care and rehabilitation, to ensure optimal patient outcomes.

[0037] According to the three-dimensional model corresponding to the surgical site obtained above, the doctor can clearly understand the specific situation of the surgical site through the three-dimensional model, and then make a targeted surgical plan. The doctor can also directly design the corresponding implant and customized surgical auxiliary tools used in the surgical process directly on the three-dimensional model according to the surgical site. At the same time, the doctor can automatically generate relevant auxiliary designs for the production process of implants and surgical auxiliary tools through three-dimensional printing technology, and finally send the generated design data of implants and surgical auxiliary tools to the three-dimensional printing equipment to directly complete the printing operation of implants and surgical auxiliary tools.

[0038] Regarding the automated generation of customized surgical aids, such as surgical clamps, particularly those used in orthopedic surgery, a technical solution utilizes the U-Net medical image segmentation framework to generate surgical clamp models. The U-Net architecture performs semantic segmentation on bone CT images, accurately extracting bone regions from the scanned images. This segmentation process is crucial for the subsequent clamp model generation step.

[0039] Once the intersection area between the bone and fixture models is obtained, the VTK Delaunay 3D filter is applied to create an unstructured mesh using the Delaunay triangulation algorithm. This mesh provides a common and manipulable structure for further processing.

[0040] To extract surfaces from the generated mesh, we used the VTK Extract Surface filter. This filter identifies the outer surfaces of the mesh, smoothing the intersections. We then applied the VTK Surface Smoothing filter to refine the surfaces, improving their quality and aesthetics. This resulted in a visually appealing shell model suitable for fixture design.

[0041] The jig generation process involves extruding extracted surfaces to create the jig shell. The inventors also designed AutoJIG software, which uses a 3D model of the surgical site to automatically generate customized jigs. Designed specifically for surgical planning, the software provides an intuitive user interface for surgeons and medical specialists. AutoJIG allows users to interactively position screws and cutting planes on the extracted surfaces, defining the desired jig and guide thicknesses. Using Boolean operations such as union and difference, the jig shell, screws, and cutting guide are combined into a complete jig model.

[0042] This jig model follows key design principles to meet the requirements of orthopedic surgery. It consists of a base plate, cutting guides, and holes for screws or K-wires. The base plate provides stability and support, ensuring precise bone positioning. The cutting guide guides surgical tools along a predetermined path for precise bone resection. Holes for screws or K-wires stabilize the bone during surgery.

[0043] AutoJIG software offers adjustable parameters that allow users to customize jig designs. Surgeons can adjust settings such as screw size, cutting guide angle, and jig thickness based on their preferences and surgical needs. User-defined annotations provide instructions and markup to guide the precise fabrication and placement of jig components.

[0044] Visualization of the proposed surgical procedure is another key aspect of the solution. AutoJIG software enables surgeons to visualize the surgical plan based on the designed jig model, helping to assess the feasibility and effectiveness of the surgical approach. This visualization enhances surgical decision-making and allows for adjustments or refinements before the actual surgery.

[0045] In addition, AutoJIG software supports the import of patient bone models and completed fixtures, providing further analysis and modification capabilities. It also provides export functions for easy manufacturing or integration into surgical simulation systems.

[0046] The fixture model generated by the AutoJIG software can be directly printed using a 3D printer to create a fixture used during surgery. To automate the 3D printing process, the inventors have also designed a system for generating the model support structure required for fixture printing. Specifically, this system utilizes a finite cylinder model and Sr critical points to generate the model support structure.

[0047] First, a bounding box is generated for the model input, and the projection of the bounding box on the XY plane and the Z coordinate value (z value) of the bounding box are calculated. On the projection surface, a uniform grid is created according to the spacing parameters entered by the user.

[0048] Generates a finite cylinder at each grid point, with a height equal to the Z value along the positive Z axis from the grid point. Calculates the intersection between the cylinder and the input model.

[0049] Sr=Si / Sc Where Si is the intersection area, Sc is the cross-sectional area of ​​the cylinder, and the ratio Sr represents the degree of deviation of the intersection point.

[0050] The larger the Sr, the higher the defect risk of the support. The critical point of Sr of the support generation is evaluated through testing and measurement of the support generation.

[0051] At the grid points where the Sr value is lower than the critical Sr point, a support point is generated by performing a Boolean subtraction operation between the cylinder and the input model. After iterative processing of the grid points, the support points of the input model are generated.

[0052] This technical solution integrates advanced image segmentation, mesh generation, and Boolean operations to provide a comprehensive software tool for surgical fixture design and visualization. By adhering to the design principles of accuracy, stability, customization, and visualization, the fixture models generated by this solution help improve surgical planning and outcomes in orthopedic surgery.

[0053] The above-mentioned embodiments illustrate the use of a multi-robot collaborative system to manufacture medical products. However, this multi-robot collaborative system can also be applied to perform medical surgeries. Similar to the principles of medical products, the medical products can be replaced with surgical sites on the human body. Different multi-robot collaborations can then be configured to perform the corresponding surgical procedures for different surgical steps.

Claims

1. A method for reconstructing a three-dimensional model using two-dimensional images based on artificial intelligence, characterized in that include: A training data set I, wherein each set of training data includes at least one or more two-dimensional image data and multiple slice image data corresponding to the two-dimensional image along a specified dimension; Inputting the training data set I into the deep learning neural network I for training to obtain a three-dimensional image reconstruction model; Training data set II, where each set of training data includes multiple slice image data, annotated points of specified objects in the slice image data, and converting the annotated points of all slice image data into a three-dimensional model related to the object; Input the training data set II into the deep learning neural network II for training to obtain a three-dimensional model reconstruction model; The two-dimensional image to be predicted is input into the three-dimensional image reconstruction model to obtain three-dimensional image data corresponding to the two-dimensional image to be predicted; then the three-dimensional image data is input into the three-dimensional model reconstruction model to obtain a three-dimensional model of the specified object corresponding to the two-dimensional image to be predicted.

2. The method for reconstructing a three-dimensional model using two-dimensional images based on artificial intelligence according to claim 1, characterized in that: The single or multiple two-dimensional image data in the training data I are two-dimensional image data obtained by X-ray scanning and / or two-dimensional image data in image data obtained by three-dimensional CT scanning.

3. The method for reconstructing a three-dimensional model using two-dimensional images based on artificial intelligence according to claim 1, characterized in that: The plurality of slice image data are slice image data obtained by slicing image data obtained by three-dimensional CT scanning along a specified dimension.

4. The method for reconstructing a three-dimensional model using two-dimensional images based on artificial intelligence according to claim 1, characterized in that: Also included is a tissue density prediction model for predicting tissue density distribution, which includes: Training set III, wherein each set of training data includes two-dimensional image data and data for tissue density annotation of the two-dimensional image data; Input the training set III into the deep learning neural network III training to obtain the tissue density prediction model; The two-dimensional image to be predicted is input into the tissue density prediction model to obtain the density distribution data of the specified object.

5. The method for generating a surgical clamp using the reconstructed three-dimensional model according to claim 1 or 4, characterized in that: The generated three-dimensional model of the specified object is imported into the three-dimensional graphics processing software, the surgical site is marked on the three-dimensional model, and the fitting surface of the fixture that matches the outer contour of the surgical site is generated by the three-dimensional graphics processing software, and then the remaining part of the fixture is set on the fitting surface by the software.

Images