Three-dimensional reconstruction method and system for soft tissue

By combining pre-training U-net model and 3D U-net model, combined with attention mechanism and adversarial training module, the problem of insufficient soft tissue segmentation efficiency and accuracy is solved, and efficient three-dimensional reconstruction of soft tissue is achieved, suitable for individualized virtualization Laminoscopic system.

WO2024245469A1PCT designated stage expired Publication Date: 2024-12-05THE AFFILIATED HOSPITAL OF QINGDAO UNIV

Patent Information

Application Number
PCT/CN2024/109079
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-02
Filing Date
2024-08-01
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

The existing three-dimensional reconstruction methods of soft tissues have insufficient segmentation accuracy and efficiency, especially the segmentation process of tumors or complex soft tissue lesions is long, which affects the treatment progress, and the segmentation accuracy and efficiency of existing deep learning methods are not high. .

Method used

The pre-trained U-net model combines 3D U-net model and attention mechanism for rough and fine segmentation of soft tissues. The generalization ability of the model is enhanced through adversarial training modules, and a three-dimensional reconstruction model is constructed using a body-drawn ray projection algorithm. , combined with visualization technology for model output and clinical correction.

Benefits of technology

It significantly improves the efficiency and accuracy of soft tissue segmentation, improves the reliability and preoperative evaluation effect of three-dimensional reconstruction results, and is suitable for the three-dimensional reconstruction of soft tissue in individualized virtual alascopy system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024109079_05122024_PF_FP_ABST
    Figure CN2024109079_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The present application relates to a three-dimensional reconstruction method and system for soft tissue. The method comprises: acquiring medical image data of soft tissue, performing preprocessing and segmentation on the medical image data, and constructing a three-dimensional reconstruction model of the soft tissue with regard to a result after soft tissue segmentation, wherein the segmentation comprises a rough segmentation step and a fine segmentation step; in the rough segmentation step, a pre-trained U-net model is used to roughly segment the soft tissue; and in the fine segmentation step, an attention mechanism is used to highlight features of the soft tissue, and soft tissue fine segmentation is performed in combination with a 3D U-net model. The method has the advantages of high efficiency and high precision for the reconstruction of a soft tissue model.
Need to check novelty before this filing date? Find Prior Art

Description

Three-dimensional reconstruction method and system for soft tissue

[0001] This application claims priority to Chinese patent application number 2023106498255 filed with the Patent Office of China on June 2, 2023, entitled “Method, system and computer storage medium for three-dimensional reconstruction of soft tissue”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of medical image data processing, and in particular to a method and system for three-dimensional reconstruction of soft tissue based on image segmentation. Background Art

[0003] Since the rise of virtual surgery and 3D visualization for preoperative assessment, the 3D reconstruction of human abdominal soft tissue has been a hot topic. The prerequisite for 3D reconstruction is the accurate segmentation of soft tissue. Current soft tissue segmentation methods often employ morphological analysis strategies, focusing on algorithms that track image grayscale differences. However, these methods lack effective artificial learning mechanisms, and the accuracy of the results obtained cannot guarantee clinical applicability. Later manual corrections are often required to improve the segmentation effect. Especially for tumors or complex soft tissue lesions, the entire segmentation process can take 30 minutes to 4 hours, seriously impacting treatment progress. With the widespread application of artificial intelligence technology, using artificial intelligence models as a medical image segmentation framework to improve soft tissue recognition efficiency and enhance 3D reconstruction accuracy has become a hot topic in the medical field.

[0004] To improve the accuracy and efficiency of soft tissue segmentation, many researchers have conducted extensive research on the application of deep learning to medical image segmentation. For example, in "Gradient-based learning applied to document recognition," LeCun, Yann, et al., Proceedings of the IEEE 86.11 (1998), proposed using a convolutional neural network model to segment images. However, this method is time-consuming and has low segmentation accuracy. CN110570432A discloses a deep learning-based method for liver tumor segmentation in CT images. Specifically, the U-Net network structure is optimized and improved by adding Dropout and Batch Normalization layers to modify the network structure. The VGG-16 network module is used to replace the U-Net encoder to establish an optimized model for accurate segmentation of liver tumor CT images. However, this method uses a single imaging modality analysis method, which cannot comprehensively cover disease types and has limited effectiveness in improving soft tissue segmentation and three-dimensional reconstruction. CN111652886A discloses a liver tumor segmentation method based on an improved U-net network. Before liver tumor segmentation, the liver region is first segmented. A liver segmentation neural network is built using the Keras deep learning framework, with TensorFlow as the backend. The improved U-net-based liver segmentation network is then trained. A liver tumor segmentation network based on the improved U-net is built and trained using the Keras deep learning framework. The improved U-net-based liver segmentation network is then used to segment the liver region from abdominal liver CT images, and then to separate tumor and normal liver tissue from the liver region. This method can eliminate a large number of missegments and reduce the complexity of the network model. However, the segmentation accuracy and efficiency of this method are still low.

[0005] Summary of the Invention

[0006] In response to some problems existing in the prior art, the present application provides a three-dimensional reconstruction method and system for soft tissue.

[0007] In a first aspect, the present application provides a method for three-dimensional reconstruction of soft tissue, comprising the following steps:

[0008] Image data acquisition step: including acquiring medical image data of soft tissue;

[0009] Data preprocessing step: including preprocessing the acquired medical image data to obtain the input image data of the pre-trained U-net model;

[0010] The soft tissue segmentation step based on the U-net network is as follows: the pre-processed image is segmented; the segmentation includes a coarse segmentation step and a fine segmentation step. First, the coarse segmentation step uses a pre-trained U-net model to roughly segment the soft tissue; then, the fine segmentation step uses an attention mechanism to highlight the characteristics of the soft tissue and combines it with a 3D U-net model to perform fine segmentation of the soft tissue;

[0011] Soft tissue three-dimensional reconstruction step: includes constructing a three-dimensional reconstruction model of the soft tissue based on the results of fine soft tissue segmentation.

[0012] In one embodiment, in the image data acquisition step, the medical image data is multimodal medical image data of soft tissue, and is obtained from one or more of computed tomography, magnetic resonance imaging, and positron emission tomography.

[0013] In one embodiment, in the data preprocessing step, the preprocessing includes: using contrast stretching technology to enhance image details; using median filtering technology to eliminate isolated noise points on the image, and combining Gaussian filtering to achieve smooth denoising of the image.

[0014] In one embodiment, in the soft tissue segmentation step based on the U-net network, the coarse segmentation link uses a pre-trained U-net model to extract medical image features and obtain a feature map; the fine segmentation link adds an attention mechanism during the upsampling and downsampling processes of the 3D U-net model to further process the extracted medical image features.

[0015] In one embodiment, more specifically, the attention mechanism adopts a channel attention module M C (F) and spatial attention module M S (F) in series, connect M C (F) and M S The output of (F) is multiplied by the roughly segmented medical image features.

[0016] In one embodiment, during the training of the 3D U-net model, an adversarial training module is used to generate random perturbations, and the perturbations are added to the original samples for training the 3D U-net model.

[0017] In one embodiment, in the soft tissue three-dimensional reconstruction step, a ray casting algorithm based on volume rendering is used to construct a three-dimensional reconstruction model of the soft tissue based on the result of fine soft tissue segmentation.

[0018] In one embodiment, the three-dimensional reconstruction method of soft tissue further includes a visualization output step: utilizing visualization technology and human-computer interaction virtual reality technology to design an interactive unit to output the three-dimensional reconstruction model of the soft tissue.

[0019] In one embodiment, the three-dimensional reconstruction method of soft tissue further includes a model evaluation and correction step: combining clinical physician evaluation and model performance evaluation indicators to correct and improve the three-dimensional reconstruction model of the soft tissue.

[0020] A second aspect of the present application provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the three-dimensional reconstruction method for soft tissue described in any of the foregoing embodiments is implemented.

[0021] A third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon; when the program is executed by a processor, the three-dimensional reconstruction method of soft tissue described in any of the above embodiments is implemented.

[0022] A fourth aspect of the present application provides a three-dimensional reconstruction system for soft tissue, comprising the following modules:

[0023] An image data acquisition module, used for acquiring medical image data of soft tissue;

[0024] The data preprocessing module is used to preprocess the acquired medical image data to obtain the input image data of the pre-trained U-net model;

[0025] A soft tissue segmentation module based on a U-net network is used to segment preprocessed images; it includes a coarse segmentation module and a fine segmentation module. The coarse segmentation module uses a pre-trained U-net model to perform coarse soft tissue segmentation; the fine segmentation module uses an attention mechanism to highlight soft tissue features and combines it with a 3D U-net model to perform fine soft tissue segmentation.

[0026] The soft tissue 3D reconstruction module is used to construct a 3D reconstruction model of the soft tissue based on the results of fine soft tissue segmentation.

[0027] In one embodiment, in the image data acquisition module, the medical image data is multimodal medical image data of soft tissue, and comes from one or more of computed tomography, magnetic resonance imaging, and positron emission tomography.

[0028] In one embodiment, the data preprocessing module includes: using contrast stretching to enhance image detail; using median filtering to eliminate isolated noise points in the image; and combining this with Gaussian filtering to achieve image smoothing and denoising. During the training process of the data preprocessing module, the module may also use interval normalization to adjust the grayscale value range of the image; and resampling to unify the image resolution.

[0029] In one embodiment, in the soft tissue segmentation module based on the U-net network, the coarse segmentation module uses a pre-trained U-net model to extract medical image features and obtain a feature map; the fine segmentation module adds an attention mechanism during the upsampling and downsampling processes of the 3D U-net model to further process the extracted medical image features.

[0030] In one embodiment, more specifically, the attention mechanism uses a channel attention module and a spatial attention module in series, and multiplies the output results of the two with the roughly segmented medical image features.

[0031] In one embodiment, during the training of the 3D U-net model, an adversarial training module is used to generate random perturbations, and the perturbations are added to the original samples for training the 3D U-net model.

[0032] In one embodiment, in the soft tissue 3D reconstruction module, a ray casting algorithm based on volume rendering is used to construct a 3D reconstruction model of the soft tissue based on the result of fine soft tissue segmentation.

[0033] In one embodiment, the three-dimensional reconstruction system of soft tissue further includes a visualization output module for designing an interactive unit using visualization technology and human-computer interaction virtual reality technology to output the three-dimensional reconstruction model of the soft tissue.

[0034] In one embodiment, the three-dimensional reconstruction system of soft tissue further includes a model evaluation and correction module for correcting and improving the three-dimensional reconstruction model of soft tissue in combination with clinical physician evaluation and model performance evaluation indicators.

[0035] The three-dimensional reconstruction method and system of soft tissue in this application can be applied to a personalized virtual laparoscope system, and therefore can also be expressed as a three-dimensional reconstruction method and system of soft tissue in a personalized virtual laparoscope system. The "individualization" mentioned here means that the method and system in this application can be applied to the three-dimensional reconstruction of soft tissue of each patient; the "virtual laparoscope system" mentioned here refers to a virtual laparoscope device, which can be the method and system in this application built on an existing virtual laparoscope device to add its corresponding functions.

[0036] The three-dimensional reconstruction method or system of soft tissue provided in at least one embodiment of the present application, for specific soft tissue lesions, combines multimodal medical images to jointly describe the morphological characteristics of the lesions, and jointly inputs the multimodal information into the artificial intelligence model, which can effectively improve the reliability of the soft tissue three-dimensional reconstruction results and improve the preoperative evaluation effect.

[0037] At least one embodiment of the present application provides a three-dimensional reconstruction method or system for soft tissue, which uses a pre-trained U-net model to perform coarse segmentation of abdominal soft tissue; at the same time, it combines the attention mechanism and the 3D U-net model to perform fine segmentation of soft tissue, thereby improving the efficiency and accuracy of soft tissue segmentation.

[0038] The three-dimensional reconstruction method or system of soft tissue provided by at least one embodiment of the present application involves an adversarial training module that addresses the problem of insufficient original sample data, enhances the original sample space, expands the deep learning perception space, improves the model's understanding and generalization capabilities, and improves the convergence efficiency during the segmentation model learning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] FIG1 is a schematic flow chart of a method for three-dimensional reconstruction of soft tissue according to an embodiment;

[0040] FIG2 is a schematic diagram of a process of a soft tissue segmentation method according to an embodiment;

[0041] FIG3 is a schematic structural diagram of a three-dimensional reconstruction system for soft tissue according to an embodiment;

[0042] FIG4 is a segmentation result of soft tissue (rectal tumor) obtained using an embodiment of the present application;

[0043] FIG5 is a three-dimensional reconstruction result of tissues and organs obtained using an embodiment of the present application. DETAILED DESCRIPTION

[0044] This application relates to an image segmentation method and system using deep learning technology in medical imaging, as well as a three-dimensional reconstruction method and system for soft tissue based on this image segmentation. It should be noted that the features of the embodiments of this application may be combined with each other unless they conflict. This application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0045] The soft tissue in this application mainly refers to tissue organs (such as liver, lungs, kidneys, peritoneal soft tissue, etc.) and structures such as tumors; therefore, in some cases, the soft tissue in this application can also be expressed as tissue organs instead.

[0046] Referring to FIG1 , a first embodiment of the present application provides a method for three-dimensional reconstruction of soft tissue, which may include the following steps:

[0047] S101: Acquisition of medical imaging data. The medical imaging data may be multimodal imaging data, coming from computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), etc. Although medical imaging analysis based on single modality images (such as single CT, MRI, PET, etc.) is the mainstream in the field, there are some defects; for example, although CT is simple to operate and highly repeatable, it is limited in soft tissue characterization; MRI has high clarity for soft tissue and anatomical structures, but lacks metabolic dynamic information; PET has extremely high sensitivity in evaluating tumor biological characteristics (metabolism, malnutrition, etc.), but has poor spatial resolution. In the actual application of this application, a single modality or multimodality joint registration technology will be reasonably adopted according to different lesion types to construct an original sample set for parsing disease-related imaging features. For example, for structures such as the liver, lungs, pancreas, and adrenal glands, CT image data analysis is primarily used; for structures such as bones, muscles, and the brain, MRI image data analysis is primarily used; for early clinical screening, heart, and tumors, PET image data analysis is primarily used; and for clinical cases with complex lesion locations, multimodal joint registration analysis technology is considered to obtain original medical images. In addition, since the medical images described above are generally in the form of images, they can also be referred to as images in this application.

[0048] S102: Data Preprocessing. Raw medical image data often contains interference factors such as noise and motion artifacts, which can adversely affect subsequent human evaluation and AI calculations. Therefore, preprocessing collected medical image data helps optimize image quality, reduce noise, enhance contrast, and improve the overall performance of the framework.

[0049] In this embodiment, contrast stretching technology is first used to enhance image details and solve the problem of blurred image texture features caused by underexposure or overexposure; median filtering technology is used to eliminate isolated noise points in medical images, and Gaussian filtering is combined to achieve smooth denoising of the image.

[0050] During the training of the data preprocessing model, interval normalization technology can also be used to adjust the grayscale value range of the image for joint analysis of the formats of medical images from different machine models; and resampling technology can be used to unify the resolution of medical images, eliminate the uncertainty caused by layer scanning spacing and pixel spacing, and standardize the imaging input pixel format of the artificial intelligence model.

[0051] S103: Soft tissue segmentation based on U-net network. The pre-processed image is segmented. First, a pre-trained U-net model is used to perform rough soft tissue segmentation. Then, an attention mechanism is used to highlight the salient features of the target soft tissue (such as the shape of the liver or tumor). A 3D U-net model is then used to perform fine soft tissue segmentation.

[0052] Furthermore, to address the issue of insufficient original sample data, an adversarial training module was designed into the 3D U-net model. This module simulates perturbations in medical images and incorporates these perturbations into the training samples to enhance the original sample space, enabling adaptive perturbation generation and data augmentation, thereby expanding the deep learning perceptual space. This adversarial training module is used only during the training phase of the 3D U-net model to improve its understanding and generalization capabilities. In the designed 3D U-net model, each convolutional layer is equipped with a Relu activation function and a BatchNorm normalization structure to improve the convergence efficiency of the 3D U-net model learning process for segmentation.

[0053] S104: Three-dimensional reconstruction of soft tissue. Based on the results of fine segmentation of soft tissue, a ray casting algorithm based on volume rendering is used to construct a three-dimensional reconstruction model of soft tissue (tissue organs). First, the grayscale value of a single pixel in the single-layer image (CT, MRI, PET single-modality image or multi-modality joint registration image) after fine segmentation is extracted, and the three-dimensional coordinates are determined based on the spatial adjacency information of the pixel, and saved in the form of a three-dimensional array as volume texture data. The volume texture data is converted into color values ​​and transparency values ​​using linear conversion function technology to clarify the color information and transparency information of different texture coordinates in the three-dimensional volume data field. In this embodiment, filtering technology and clinical information are used to divide the grayscale values ​​of the image into 8-10 groups, and different texture conversion functions are used for different groups to achieve contrast enhancement of different soft tissues. Vector calculation and internal point diffusion technology are used to realize the three-dimensional space conversion of the image plane to ensure the applicability of the ray casting algorithm in three-dimensional space. The converted ray casting algorithm resamples the volume texture data at regular intervals to obtain the color and transparency values ​​at the sampling points. Finally, the color information is accumulated along the algorithmic direction to determine the spatial texture information, completing the 3D reconstruction of the soft tissue. 3D reconstruction technology is relatively mature, so this embodiment can also use other conventional 3D reconstruction models to achieve 3D reconstruction of soft tissue.

[0054] In some embodiments, the three-dimensional reconstruction method of soft tissue may further include the following steps:

[0055] S105: Visual output of soft tissue models. Using visualization technology and human-computer interaction virtual reality technology, an interactive unit is designed to output the 3D reconstruction model of the soft tissue. Users can manually inspect the 3D reconstruction results, including zooming in, zooming out, perspective, marking, and virtual cutting.

[0056] S106: Model evaluation and correction: The three-dimensional reconstructed model can be corrected and improved in combination with the clinical physician's evaluation and the model performance evaluation index.

[0057] As shown in FIG2 , in some embodiments, more specifically, the U-net network soft tissue segmentation method may include the following steps:

[0058] Rough segmentation: When performing rough segmentation of soft tissue, a pre-trained U-net model can be used to process the pre-processed image, extract image features (i.e., medical image features, including morphological features of soft tissue, etc.), and obtain a feature map of the same size as the original image (i.e., the feature map of the image).

[0059] The pre-trained U-net model used in some embodiments of the present application is obtained after image segmentation training on the Carvana dataset, wherein the Carvana dataset includes 5056 training samples involving 16 types of car image segmentation and recognition. By migrating the U-net model initially trained on the Carvana data to the field of medical image segmentation, a rough segmentation result about the contours and edges of the medical image can be given, which provides targeted pixel targets for subsequent fine segmentation, reduces image redundancy, and improves the computational efficiency of the segmentation model. In other words, the 2DU-net model in the prior art is first trained using the Carvana dataset to obtain a 2D U-net model after initial training; then multiple 2D medical images (about soft tissues) are input into the 2D U-net model after initial training to continue training, and the pre-trained U-net model described in the present application is obtained; the pre-trained U-net model is 2D and can preliminarily segment each soft tissue.

[0060] Fine segmentation step: Fine segmentation of soft tissue is performed on the image after rough segmentation; in this step, the fine segmentation of soft tissue is mainly performed by combining the attention mechanism and the 3D U-net model. Medical images are often grayscale images with simple semantics and relatively fixed structures, so high-level semantic features and low-level semantic features are equally important in parsing image information. In some embodiments, with respect to the 3D U-net model, first, the 3D U-net model uses 4 downsampling modules to extract low-level semantic features of medical images. Secondly, the upsampling module of the 3D U-net model continues to use a multi-scale information merging strategy, and continuously fuses the low-level semantic features extracted by the downsampling module during the upsampling process, so that the image after rough segmentation is restored to edge and other information more finely. In the construction of the 3D U-net model, each convolutional layer can also be equipped with a Relu activation function and a BatchNorm normalization structure to solve the covariance offset problem and improve the convergence efficiency during the learning process of the segmentation model.

[0061] There are certain differences between the 3D U-net model in this application and the pre-trained U-net model. Specifically, during the 3D U-net model training process, a 3D open source medical imaging dataset is first used to train the 3D U-net model in the prior art to obtain a 3D U-net model after initial training; then multiple 3D medical images (about soft tissue) are input into the 3D U-net model after initial training to continue training, and the 3D U-net model used for fine segmentation in this embodiment is obtained. It can be seen that the pre-trained U-net model and the 3D U-net model are different at least in dimension, the former is two-dimensional and the latter is three-dimensional. It is worth noting that, whether it is a U-net model or a 3D U-net model, their data sets in training can be selected according to actual conditions and are not limited to the data sets described above.

[0062] This application incorporates an attention mechanism design into the upsampling and downsampling processes of the 3D U-net model, respectively, so that the attention mechanism is integrated into the 3D U-net model for fine segmentation. As a result, the 3D U-net model, as a deep learning model, pays more attention to tissue and organ parts during training, ignoring disturbing parts such as the spine and blood in the image, thereby improving training efficiency.

[0063] The attention mechanism adopts the channel attention module M C (F) and spatial attention module M S (F) Series design, M C (F) and M S The output of (F) is multiplied by the roughly segmented medical image features and assigned a weight to achieve effective attention mechanism fusion.

[0064] The channel attention module is used to perform global maximum pooling (MaxPool) and average pooling (AvgPool) on the feature map of the roughly segmented image in the spatial dimension to compress the spatial size; the obtained global maximum pooling and average pooling results are input into the multi-layer perceptron (MLP) to learn the importance of each channel; the importance of all channels is "added" and input into the Sigmoid function to obtain the final channel attention module M C (F). M C (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0065] Among them, F is the input medical image feature map (i.e., the feature map of the image after rough segmentation); AvgPool represents the global average pooling operation on the feature map F; MaxPool represents the global maximum pooling operation on the feature map F; MLP is a multi-layer perceptron structure; σ represents a nonlinear activation function (such as the Sigmoid function mentioned above).

[0066] The spatial attention module needs to perform global maximum pooling and average pooling on the feature map of the roughly segmented input image in the channel dimension; the global pooling and average pooling output results are spliced ​​according to the channel (f); the final splicing result is input into the Sigmoid function to obtain the spatial attention module M S (F). M S (F)=σ(f([AvgPool(F);MaxPool(F)]))

[0067] Among them, F is the input medical image feature map (i.e., the feature map of the image after rough segmentation); AvgPool represents the global average pooling operation on the feature map F; MaxPool represents the global maximum pooling operation on the feature map F; f represents a feature splicing operation, which is used to vector splice the maximum pooling and average pooling results; σ represents a nonlinear activation function (such as the Sigmoid function mentioned above).

[0068] Furthermore, the problem of insufficient medical image samples during fine segmentation is common in the field. To address this issue, an adversarial training module can be introduced to achieve adaptive perturbation generation and data augmentation, improving the generalization capability of the 3D U-net model. First, the adversarial training module continuously generates random perturbations and adds them to the original training medical images for use in the 3D U-net model training process, aiming to reduce the accuracy of medical image segmentation. Second, the 3D U-net model continuously detects these perturbations, mining high-quality information from the medical images and thereby improving the segmentation accuracy of the 3D U-net model. The interaction between the adversarial training module and the 3D U-net module generates a set of adversarial examples, which play a crucial role in expanding the learning range of the 3D U-net model and enhancing the receptive domain of deep learning. Finally, incorporating these adversarial examples into the training sample enriches the original training sample and compensates for the insufficient number of medical samples.

[0069] Let us return to the 3D U-net model training process described above. After obtaining the 3D U-net model after initial training, multiple 3D medical images (about soft tissue) are first input into the 3D U-net model after initial training to continue training, and a 3D U-net model after secondary training is obtained. Then, the multiple 3D medical images are perturbed using an adversarial training module to form multiple disturbed 3D medical images. These multiple disturbed 3D medical images are input into the 3D U-net model after secondary training to continue training, and the 3D U-net model used for fine segmentation in this embodiment is obtained.

[0070] As shown in FIG3 , a second embodiment of the present application provides a three-dimensional reconstruction system for soft tissue, which includes:

[0071] An image data acquisition module is used to acquire single-modality or multi-modality medical image data, wherein the single-modality or multi-modality image data comes from one or a combination of computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET).

[0072] The data preprocessing module is used to preprocess the collected medical imaging data. In some embodiments, the contrast stretching technology is first used to enhance the image details to solve the problem of blurred image texture features caused by underexposure or overexposure; the median filtering technology is used to eliminate isolated noise points on the medical image, and combined with Gaussian filtering to achieve smooth denoising of the image. During the training process of the data preprocessing module, the interval normalization technology can also be used to adjust the grayscale value range of the image for joint analysis of the formats of medical images from different machine models; and the resampling technology can be used to unify the resolution of medical images, eliminate the uncertainty caused by layer scanning spacing and pixel spacing, and standardize the imaging input pixel format of the artificial intelligence model.

[0073] The soft tissue segmentation module based on the U-net network is used to segment the pre-processed medical images. In some embodiments, the soft tissue segmentation module includes a coarse segmentation module and a fine segmentation module.

[0074] The rough segmentation module is used to roughly segment the soft tissue, and a pre-trained U-net model can be used to extract image features to obtain a feature map of the same size as the original image.

[0075] A fine segmentation module is used to perform fine segmentation of soft tissue on the coarsely segmented image. In some embodiments, the fine segmentation module combines an attention mechanism to improve the upsampling and downsampling processes of the 3D U-net model to perform fine segmentation of soft tissue.

[0076] To address the issue of insufficient raw sample data, an adversarial training module is incorporated into the model to implement adaptive perturbation generation and data augmentation, improving the generalization capabilities of the 3D U-Net model. In constructing the 3D U-Net model, each convolutional layer is equipped with a ReLU activation function and BatchNorm normalization structure to improve the convergence efficiency of the 3D U-Net segmentation model during learning. For more details, please refer to the description of the fine segmentation step above.

[0077] The Soft Tissue 3D Reconstruction Module uses a volume rendering-based ray casting algorithm to construct 3D reconstruction models of tissues and organs based on the results of fine soft tissue segmentation. After extracting grayscale features from medical images, grayscale values ​​are grouped according to research objectives to enhance contrast between different tissues. During algorithm execution, a conversion function is used to assign color and transparency values ​​to different pixels. Resampling techniques are then used to combine these color and resampled values ​​to achieve 3D reconstruction of medical images.

[0078] Visualization Output Module: Utilizing visualization technology and human-computer interaction virtual reality technology, an interactive unit is designed to output 3D reconstruction models of soft tissue. Users can manually inspect the 3D reconstruction results, including zooming in, zooming out, perspective, marking, and virtual cutting.

[0079] Model evaluation and correction module: used to calibrate and improve the above-mentioned 3D reconstruction model based on clinical physician evaluation and model performance evaluation indicators.

[0080] Figure 4 shows the segmentation result of soft tissue (rectal tumor) obtained using an embodiment of the present application. CT images are used as medical images to be processed, and the task involved is a CT image segmentation task of rectal cancer tumors. In the figure, the left side is the original CT image; the middle is the segmentation result; the right side is the expansion based on the segmentation result (clinically helpful for radiotherapy planning); DICE represents the segmentation effect evaluation index. The results show that the soft tissue segmentation model of this embodiment can effectively identify and segment the tumor location and structure in the CT image, and can also expand the segmented results to a certain scale to meet the clinical needs of subsequent tumor radiotherapy tasks.

[0081] Figure 5 shows the 3D reconstruction results of tissues and organs obtained using an embodiment of the present application. These reconstructions involve 3D reconstructions of the lungs, liver, peritoneal soft tissue, and kidneys. Using CT images as the processed images, accurate 3D reconstructions of different tissues and organs were achieved through preprocessing, segmentation, and 3D reconstruction techniques.

[0082] The third embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the three-dimensional reconstruction method of soft tissue described in any of the foregoing embodiments is implemented.

[0083] A fourth embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon; when the program is executed by a processor, the three-dimensional reconstruction method of soft tissue described in any of the above embodiments is implemented.

[0084] It should be noted that for the aforementioned methods and systems, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the implementation methods described in the specification are all preferred or optional implementation methods, and the actions and modules involved are not necessarily required for this application.

[0085] In the several embodiments provided in this application, it should be understood that the disclosed system can be implemented in other ways. For example, the system implementation described above is only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical or other forms.

[0086] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0087] It should be noted that the above detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs.

[0088] The described embodiments are merely descriptions of the preferred embodiments of the present application and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements made to the technical solutions of the present application by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present application.

Claims

1. A method for three-dimensional reconstruction of soft tissue, wherein: The steps include: Image data acquisition step: including acquiring medical image data of soft tissue; Data preprocessing step: including preprocessing the acquired medical image data to obtain input image data for the pre-trained U-net model; The soft tissue segmentation step based on the U-net network: segment the preprocessed image; the segmentation includes a rough segmentation step and a fine segmentation step; first, the rough segmentation step uses a pre-trained U-net model to roughly segment the soft tissue; then the fine segmentation step uses an attention mechanism to highlight the characteristics of the soft tissue, and combines the 3D U-net model to perform fine segmentation of the soft tissue; Soft tissue three-dimensional reconstruction step: including building a three-dimensional reconstruction model of the soft tissue based on the results of fine soft tissue segmentation.

2. The method for three-dimensional reconstruction of soft tissue according to claim 1, wherein: In the image data acquisition step, the medical image data is multimodal medical image data of soft tissue, and comes from one or more of computer tomography, magnetic resonance imaging and positron emission tomography.

3. The method for three-dimensional reconstruction of soft tissue according to claim 1, wherein: In the data preprocessing step, the preprocessing includes: using contrast stretching technology to achieve image detail enhancement; using median filtering technology to eliminate isolated noise points on the image, and combining Gaussian filtering to achieve image smoothing and denoising.

4. The method for three-dimensional reconstruction of soft tissue according to claim 1, wherein: In the soft tissue segmentation step based on the U-net network, the coarse segmentation link uses a pre-trained U-net model to extract medical image features to obtain a feature map; the fine segmentation link adds an attention mechanism during the upsampling and downsampling processes of the 3D U-net model to further process the extracted medical image features.

5. The method for three-dimensional reconstruction of soft tissue according to claim 4, wherein: The attention mechanism uses a channel attention module and a spatial attention module in series, and multiplies the output results of the channel attention module and the spatial attention module with the roughly segmented medical image features.

6. The method for three-dimensional reconstruction of soft tissue according to any one of claims 1 to 5, wherein: During the training process of the 3D U-net model, an adversarial training module is used to generate random perturbations, and the perturbations are added to the original samples for training the 3D U-net model.

7. The method for three-dimensional reconstruction of soft tissue according to any one of claims 1 to 5, wherein: In the soft tissue three-dimensional reconstruction step, a ray casting algorithm based on volume rendering is used to construct a three-dimensional reconstruction model of the soft tissue based on the result of fine segmentation of the soft tissue; The three-dimensional reconstruction method of soft tissue also includes a visualization output step: using visualization technology and human-computer interaction virtual reality technology to design an interactive unit to output the three-dimensional reconstruction model of the soft tissue; and The three-dimensional reconstruction method of soft tissue also includes a model evaluation and correction step: combining the evaluation of clinical physicians and the model performance evaluation index to correct and improve the three-dimensional reconstruction model of the soft tissue.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the three-dimensional reconstruction method of soft tissue according to any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium having a computer program stored thereon; when the program is executed by a processor, the method for three-dimensional reconstruction of soft tissue according to any one of claims 1 to 7 is implemented.

10. A three-dimensional reconstruction system for soft tissue, wherein: Includes the following modules: An image data acquisition module, used for acquiring medical image data of soft tissue; A data preprocessing module is used to preprocess the acquired medical image data to obtain input image data for the pre-trained U-net model; The soft tissue segmentation module based on U-net network is used to segment the preprocessed images; It includes a coarse segmentation module and a fine segmentation module; wherein the coarse segmentation module is used to use a pre-trained U-net model to perform coarse segmentation of soft tissue; the fine segmentation module is used to use an attention mechanism to highlight the features of soft tissue, and to perform fine segmentation of soft tissue in combination with a 3D U-net model; The soft tissue 3D reconstruction module is used to construct a 3D reconstruction model of the soft tissue based on the results of fine soft tissue segmentation.

11. The three-dimensional reconstruction system of soft tissue according to claim 10, wherein: In the image data acquisition module, the medical image data is multimodal medical image data of soft tissue, which comes from one or more of computer tomography, magnetic resonance imaging and positron emission tomography; in the data preprocessing module, the preprocessing includes: using contrast stretching technology to achieve image detail enhancement; using median filtering technology to eliminate isolated noise points on the image, and combining Gaussian filtering to achieve image smoothing and denoising; during the training process of the data preprocessing module, using interval normalization technology to adjust the grayscale value range of the image; and using resampling technology to unify the resolution of the image.

12. The three-dimensional reconstruction system of soft tissue according to claim 10, wherein: In the soft tissue segmentation module based on the U-net network, the coarse segmentation module uses a pre-trained U-net model to extract medical image features and obtain a feature map; the fine segmentation module adds an attention mechanism in the upsampling and downsampling processes of the 3D U-net model to further process the extracted medical image features; more specifically, the attention mechanism uses a channel attention module and a spatial attention module in series to multiply the output results of the channel attention module and the spatial attention module with the medical image features after coarse segmentation.

13. The three-dimensional reconstruction system of soft tissue according to claim 10, wherein: In the soft tissue 3D reconstruction module, a ray casting algorithm based on volume rendering is used to construct a 3D reconstruction model of the soft tissue based on the result of fine segmentation of the soft tissue; The three-dimensional reconstruction system of soft tissue also includes a visualization output module, which is used to design an interactive unit by using visualization technology and human-computer interaction virtual reality technology to output the three-dimensional reconstruction model of the soft tissue; and The three-dimensional reconstruction system of soft tissue also includes a model evaluation and correction module, which is used to correct and improve the three-dimensional reconstruction model of soft tissue in combination with the evaluation of clinical physicians and the model performance evaluation index.

Citation Information

Patent Citations

  • Pancreas CT image segmentation method based on integrated deep convolutional neural network

    CN112116605A

  • CT image lung vessel segmentation method

    CN112258514A

  • Liver CT tumor segmentation and classification method based on deep learning

    CN114494199A

  • Cerebrovascular image segmentation method based on multi-scale attention network

    CN114723698A

  • Brain tumor segmentation method based on brain three-dimensional MRI image design

    CN115496771A

Cited By

  • Multi-angle photography and three-dimensional reconstruction method and system for soft tissue

    CN119540468A

  • Pathological image segmentation method and device based on deep learning and readable storage medium thereof

    CN120031899A

  • Artificial intelligence driven periodontal minimally invasive surgery preoperative model processing system

    CN121839158A