Automatic image segmentation method, system, device and medium

Abstract

The present invention relates to an automatic image segmentation method, system, device and medium, comprising: inputting a target fetal congenital malformation heart cross-section image into a pre-built nn-UNet model for automatic recognition and segmentation of vascular structures to obtain a preliminary segmentation result; based on the preliminary segmentation result, using VTK to perform three-dimensional structure reconstruction to obtain an initial point cloud file; performing gap filling and smoothing operations on the initial point cloud file, and projecting the optimized three-dimensional vascular model into a series of parallel cutting planes according to the actual heart slice spacing; calculating the intersection set of the point cloud data and each cutting plane, and performing topological closure and area filling processing to generate a binary mask to obtain a segmentation result that incorporates a priori three-dimensional geometric morphology of the blood vessels; post-processing the preliminary segmentation result to obtain a final segmentation result of the target fetal congenital malformation heart cross-section image. The present invention can be widely used in the field of image processing.

Description

Technical Field

[0001] The present invention belongs to the field of medical image segmentation, and relates to an automatic image segmentation method, system, device and medium, and in particular to an automatic segmentation method, system, device and medium for anatomical cross-section images of fetal congenital heart malformations. Background Art

[0002] Congenital heart disease (CHD) is one of the most common severe congenital structural malformations worldwide and a leading cause of neonatal mortality. Timely and accurate prenatal diagnosis is crucial for improving the survival rate and quality of life of children with CHD. CHD is often accompanied by complex cardiovascular malformations, such as anomalous pulmonary venous connection, transposition of the great arteries, persistent truncus arteriosus, pulmonary artery stenosis, aortopulmonary window, anomalous origin of the pulmonary artery, interrupted aortic arch, right aortic arch, double aortic arch, anomalous coronary artery drainage, and interrupted inferior vena cava. Accurate identification of these cardiovascular malformations is crucial for clinicians to formulate treatment plans. However, due to the complex structure of the heart and the diverse range of pathological changes, precise segmentation of blood vessels is particularly difficult. Therefore, accurate segmentation of vascular regions is crucial for diagnosis and treatment planning. It helps clinicians accurately identify and locate lesions, enabling more precise diagnosis and treatment. It plays a crucial role in CHD diagnosis, treatment planning, and surgical navigation.

[0003] While traditional manual segmentation methods are simple and straightforward, they are extremely time-consuming and require extensive domain expertise. For example, using the database employed in this invention as an example, manual image annotation and segmentation methods previously required at least five minutes to process each image. Each cardiac imaging dataset contains an average of approximately 600 images, requiring a total of 3,000 minutes to complete. Taking the currently available fetal congenital heart malformation anatomical database as an example, manually segmenting all the data would require at least two years, and the database is still being expanded. Therefore, there is an urgent need to utilize artificial intelligence technology to improve the efficiency of medical image segmentation.

[0004] Traditional methods such as nnU-Net and U-Mamba can be used to segment cross-sectional images of fetal congenital heart malformations. Although both models can be trained in 2D or 3D, 3D training requires the annotation of a large number of complete heart slices. In the medical field, data annotation is extremely expensive, making 3D training difficult to effectively reduce costs. Furthermore, 2D training methods cannot fully utilize the three-dimensional structural information of blood vessels. Experiments have found that vessel recognition may be discontinuous in certain slice locations, meaning that some vessels are not correctly identified in certain slices.

[0005] Blood vessels in medical images are highly spatially connected structures, and their morphology typically exhibits strong continuity between adjacent slices. Based on this physiological and anatomical prior, relying solely on 2D segmentation results can lead to vessel fragmentation or omission due to local information loss or noise interference. Therefore, 2D segmentation methods are susceptible to being limited by the local features of individual slices, making it difficult for the model to correctly identify vessels in the presence of local information loss or noise interference, resulting in discontinuities or fragmentation. However, as three-dimensional structures, blood vessels exhibit strong spatial topological connectivity, and their morphology is typically continuous between adjacent slices. Therefore, segmenting individual slices using 2D methods alone fails to leverage the overall morphological information of the vessels, making it difficult to ensure the stability and consistency of the segmentation results. In contrast, 3D reconstruction can constrain vascular morphology on a global scale, making the segmentation results more consistent with the actual anatomical structure and compensating for the incompleteness of 2D segmentation methods due to insufficient local information. Summary of the Invention

[0006] In response to the above problems, the purpose of the present invention is to provide a method, system, device and medium for automatic segmentation of anatomical cross-sectional images of fetal congenital heart malformations that combines deep learning and uses prior knowledge of vascular three-dimensional features for post-processing, which can achieve high-precision segmentation of cardiac vessels.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a method for automatic image segmentation, comprising the following steps:

[0009] The cross-sectional image of the heart of the target fetus with congenital malformation is input into the pre-built nn-UNet model for automatic recognition and segmentation of vascular structures to obtain preliminary segmentation results.

[0010] Based on the preliminary segmentation results, VTK is used to reconstruct the 3D structure and obtain the initial point cloud file;

[0011] Perform gap filling and smoothing operations on the initial point cloud file to obtain an optimized three-dimensional blood vessel model;

[0012] The optimized three-dimensional vascular model is projected into a series of parallel cutting planes according to the actual heart slice spacing;

[0013] The intersection points of the point cloud data and each cutting plane are calculated, and after topological closure and area filling processing, a binary mask is generated to obtain the segmentation result with the 3D geometric morphology prior of the blood vessels.

[0014] The preliminary segmentation results output by the nn-UNet model were post-processed based on the segmentation results with the addition of 3D geometric morphology priors to obtain the final segmentation results of the cross-sectional image of the heart of the target fetus with congenital malformation.

[0015] Furthermore, the cross-sectional image of the heart of the target fetus with congenital malformation is input into the pre-built nn-UNet model for automatic recognition and segmentation of vascular structures, and a preliminary segmentation result is obtained, including:

[0016] Collecting a preset number of fetal congenital malformation heart cross-sectional images and storing them in a preset format;

[0017] Annotate the preset cardiac vessels in the stored cross-sectional images of the heart of each fetus with congenital malformation and construct a data set;

[0018] Use the dataset to train the pre-built nn-UNet model to obtain a nn-UNet model that meets the preset requirements as an image segmentation model;

[0019] The image segmentation model is used to automatically identify and segment the vascular structure of the target fetus's congenital malformation heart cross-section image, and preliminary segmentation results are obtained.

[0020] Furthermore, the step of collecting a preset number of fetal congenital malformation heart cross-sectional images and storing them in a preset format includes:

[0021] Obtain a preset number of fetal congenital malformation heart cross-sectional image data and store them in PNG format;

[0022] Converting the original color image of the collected fetal heart cross section with congenital malformation into a grayscale image;

[0023] Convert the grayscale image of the cross-section of the heart of a fetus with congenital malformation from the original PNG format to the NII.GZ format commonly used in medical imaging and store it.

[0024] Furthermore, the method of using the dataset to train the pre-built nn-UNet model to obtain the nn-UNet model that meets the preset requirements as the image segmentation model includes:

[0025] Build the nn-UNet model;

[0026] The nn-UNet model includes: a preprocessing module, a U-net network module, and an inference module; the preprocessing module is used to crop, resample, and normalize invalid areas in the cross-sectional image of the fetal heart with congenital malformation; the U-net network module is used to construct a deep neural network to perform semantic segmentation of the heart slice blood vessels; the inference module is used to process the semantic segmentation results using a sliding window to obtain the segmentation results of the heart slice blood vessels;

[0027] Based on the preset training parameters, the constructed nn-UNet model is trained using the data set to obtain a nn-UNet model that meets the preset requirements as an image segmentation model.

[0028] Furthermore, the U-net network module includes an encoder and a decoder;

[0029] The encoder is used to extract deep features of blood vessels in heart slices through multiple convolutional layers and pooling layers, and the activation function in the convolutional layer adopts leaky ReLU function and Instance Norm function;

[0030] The decoder is used to combine shallow and deep features using skip connections and output target segmentation results.

[0031] Furthermore, based on the preliminary segmentation results, VTK is used to perform 3D structure reconstruction to obtain an initial point cloud file, including:

[0032] Based on the set isosurface threshold, VTKFlyingEdges3D is used to extract the isosurface and obtain the structural boundary of the blood vessel;

[0033] Enable ComputeScalarsOn to increase the point cloud density and get the initial point cloud file.

[0034] Furthermore, the gap filling and smoothing operations on the initial point cloud file include:

[0035] vtkImageContinuousDilate3D is used to dilate the initial point cloud file to expand the blood vessel area;

[0036] Use vtkImageContinuousErode3D to perform erosion to restore the original shape of the blood vessels;

[0037] Apply vtkImageGaussianSmooth for Gaussian smoothing.

[0038] In a second aspect, the present invention provides an automatic image segmentation system, comprising:

[0039] The preliminary segmentation module is used to input the cross-sectional image of the heart of the target fetus with congenital malformation into the pre-built nn-UNet model to automatically identify and segment the vascular structure and obtain the preliminary segmentation result;

[0040] The 3D reconstruction module is used to reconstruct the 3D structure based on the preliminary segmentation results using VTK to obtain the initial point cloud file;

[0041] The optimization module is used to perform gap filling and smoothing operations on the initial point cloud file to obtain an optimized three-dimensional blood vessel model;

[0042] A projection module is used to project the optimized three-dimensional vascular model into a series of parallel cutting planes according to the actual heart slice spacing;

[0043] The mask module is used to calculate the intersection points of the point cloud data and each cutting plane, and after topological closure and area filling processing, it generates a binary mask to obtain the segmentation result that incorporates the 3D geometric morphology prior of the blood vessels;

[0044] The post-processing module is used to post-process the preliminary segmentation results output by the nn-UNet model based on the segmentation results with the addition of the 3D geometric morphology prior of the blood vessels, and obtain the final segmentation results of the cross-sectional image of the heart of the fetus with congenital malformation to be segmented.

[0045] In a third aspect, the present invention provides a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any method.

[0046] In a fourth aspect, the present invention provides a computing device comprising: one or more processors and a memory, wherein the memory stores one or more programs and is configured to be executed by the one or more processors, and the one or more programs include instructions for executing any method.

[0047] The present invention has the following advantages due to the adoption of the above technical solution:

[0048] 1. This invention reduces annotation costs. Traditional 3D training methods rely heavily on fully annotated 3D data, while this invention uses 2D image data for training, reducing data annotation costs. It also combines VTK for 3D reconstruction and, through projection optimization strategies, feeds the 3D reconstruction results back to 2D slices to correct the original segmentation and improve overall accuracy and stability.

[0049] 2. By constructing a cardiovascular malformation dataset, the present invention can more accurately identify abnormal vascular structures and help doctors better formulate diagnosis and treatment plans, as CHD is accompanied by a variety of complex vascular malformations.

[0050] 3. Efficient post-processing optimization strategy. This paper uses VTK to perform 3D reconstruction, smoothing optimization, and interpolation completion of blood vessels, leveraging anatomical prior knowledge to improve segmentation consistency. By supplementing 3D topological information in the post-processing stage, the segmentation results are more consistent, overcoming the limitations of 2D methods.

[0051] Therefore, the present invention can be widely applied in the field of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. Throughout the drawings, the same reference numerals are used to denote the same components. In the drawings:

[0053] Figure 1 This is a flow chart of the automatic image segmentation method provided by an embodiment of the present invention;

[0054] Figure 2 This is a comparison chart of the segmentation effect provided by an embodiment of the present invention and other existing methods;

[0055] Figure 3 is an IoU indicator table of the segmentation result provided by an embodiment of the present invention;

[0056] Figure 4 This is a Dice indicator table of the segmentation result provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present invention.

[0058] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0059] Blood vessels in medical images are highly spatially connected structures, and their morphology typically exhibits strong continuity between adjacent slices. Based on this physiological and anatomical prior, relying solely on 2D segmentation results can lead to vessel fragmentation or omission due to local information loss or noise interference. Therefore, 2D segmentation methods are susceptible to being limited by the local features of individual slices, making it difficult for the model to correctly identify vessels in the presence of local information loss or noise interference, resulting in discontinuities or fragmentation. However, as three-dimensional structures, blood vessels exhibit strong spatial topological connectivity, and their morphology is typically continuous between adjacent slices. Therefore, segmenting individual slices using 2D methods alone fails to leverage the overall morphological information of the vessels, making it difficult to ensure the stability and consistency of the segmentation results. In contrast, 3D reconstruction can constrain vascular morphology on a global scale, making the segmentation results more consistent with the actual anatomical structure and compensating for the incompleteness of 2D segmentation methods due to insufficient local information.

[0060] Based on the above analysis, some embodiments of the present invention provide an automatic image segmentation method designed to leverage the 3D geometric morphology of blood vessels to inversely guide 2D segmentation results. Specifically, the method includes: automatically identifying and separating vascular structures using a nnU-Net training model to obtain a preliminary segmentation result; then, based on this segmentation result, 3D reconstruction of the vascular structures is performed using VTK. The initial point cloud data is optimized through gap filling and smoothing to generate a high-quality 3D vascular model; then, the 3D model is projected axially into a series of parallel cutting planes based on the actual cardiac slice spacing. By calculating the intersection set between the point cloud and each cutting plane, a 2D binary mask containing a 3D geometric morphology prior is generated using a topological closure and region filling algorithm. This mask is then used to guide post-processing of the preliminary nnU-Net results, resulting in a more accurate final segmentation result. The advantage of this inverse projection method lies in its ability to effectively map 3D structural information into 2D images. This method, while constrained by spatial consistency, corrects errors in the original segmentation caused by noise, occlusion, or structural complexity, significantly improving the anatomical plausibility and boundary continuity of the segmentation. This invention effectively solves the problems of insufficient segmentation accuracy, poor identification of vascular connectivity, and loss of reconstruction details in traditional segmentation methods, significantly improves the integrity and accuracy of vascular segmentation, reduces the cost and error of manual intervention, has good application prospects, and is suitable for fields such as auxiliary diagnosis of cardiovascular diseases.

[0061] Correspondingly, in other embodiments of the present invention, an automatic image segmentation system, device and medium are provided.

[0062] Example 1

[0063] like Figure 1As shown, this embodiment provides an automatic image segmentation method, which includes the following steps:

[0064] S1. Input the target fetal congenital malformation heart cross-section image into the pre-built nn-UNet model to automatically identify and segment the vascular structure and obtain the preliminary segmentation result;

[0065] S2. Based on the preliminary segmentation results, VTK (Visualization Toolkit) is used to perform 3D structure reconstruction to obtain the initial point cloud file;

[0066] S3, performing gap filling and smoothing operations on the initial point cloud file to obtain an optimized three-dimensional blood vessel model;

[0067] S4, projecting the optimized three-dimensional vascular model into a series of parallel cutting planes according to the actual heart slice spacing;

[0068] S5. Calculate the intersection points of the point cloud data and each cutting plane, perform topological closing and region filling processing, generate a binary mask, and obtain a segmentation result that incorporates the 3D geometric morphology prior of the blood vessels;

[0069] S6. Post-process the preliminary segmentation results output by the nn-UNet model based on the segmentation results that incorporate the 3D geometric morphology prior of the blood vessels to obtain the final segmentation result of the cross-sectional image of the heart of the target fetus with congenital malformation.

[0070] Furthermore, the above step S1 includes the following steps:

[0071] S11, collecting a preset number of fetal congenital malformation heart cross-sectional images and storing them in a preset format;

[0072] S12. Labeling preset heart vessels in the stored cross-sectional images of the heart of each fetus with congenital malformation, and constructing a data set;

[0073] S13. Using the data set to train the pre-built nn-UNet model, a nn-UNet model that meets preset requirements is obtained as an image segmentation model;

[0074] S14. Use the image segmentation model to automatically identify and segment the vascular structure of the cross-sectional image of the heart of the target fetus with congenital malformation to obtain a preliminary segmentation result.

[0075] Furthermore, the above step S11 includes the following steps:

[0076] S111, obtaining a preset number of fetal congenital malformation heart cross-sectional image data and storing them in PNG format;

[0077] S112. Using OpenCV (Open Computer Vision Library), convert the original color image of the cross-section of the fetal heart with congenital malformation into a grayscale image;

[0078] S113. Use Nibabel (a medical imaging data processing library) and PIL (a Python imaging library, now Pillow) to convert grayscale images of fetal cardiac cross-sections with congenital malformations from the original PNG format to the NII.GZ format commonly used in medical imaging and store them to ensure data format consistency.

[0079] In this embodiment, only OpenCV, nibabel and PIL library tools are used as examples for introduction, but the present invention is not limited thereto.

[0080] Furthermore, in the above step S12, when labeling the preset heart vessels in the stored image, at least four major heart vessels need to be labeled. In this embodiment, the four major heart vessels include: the aorta, the descending aorta, the pulmonary artery, and the superior vena cava.

[0081] Furthermore, in step S12, when labeling the pre-defined cardiac vessels in the stored cross-sectional images of each fetal congenital malformation, the LabelMe tool was first used to assign labels 1, 2, 3, and 4 to the four major vessels: the aorta, pulmonary artery, descending aorta, and superior vena cava. The corresponding JSON annotation files were then generated. The labelme_json_to_dataset command was then used to batch convert the labeled data into a folder containing the original images, labeled images, and category names. The color labeled images were then extracted. Finally, nibabel and PIL were used to convert the data into the NII.GZ format to match the image data.

[0082] In this embodiment, the constructed dataset includes annotations of more than 300 cardiac slices of more than 20 hearts. One of the hearts is finely annotated, with a slice annotated every 120 mm. Four hearts are annotated with more than 20 slices each, and the remaining hearts are annotated with 5 slices each.

[0083] Furthermore, the above step S13 includes the following steps:

[0084] S131, build nn-UNet model;

[0085] S132. Based on preset training parameters, the constructed nn-UNet model is trained using the data set to obtain a nn-UNet model that meets the preset requirements as an image segmentation model.

[0086] Furthermore, in step S131, the nn-UNet model constructed in this embodiment includes a preprocessing module, a U-net network module, and an inference module. The preprocessing module is used to crop, resample, and normalize invalid regions in the cross-sectional images of the fetal heart with congenital malformations; the U-net network module is used to construct a deep neural network for semantic segmentation of the heart slice vessels; and the inference module is used to process the semantic segmentation results using a sliding window to obtain segmentation results of the heart slice vessels.

[0087] Furthermore, the preprocessing module includes a cropping module, a resampling module and a normalization module; wherein the cropping module is used to perform crop processing on the input image data to crop out invalid areas and reduce the computational burden; the resampling module is used to resample the cropped image, for example, third-order spline interpolation can be used to adjust the image, and the nearest neighbor interpolation method can be used to adjust the segmentation label; the normalization module is used to use z-score standardization to normalize the data with a mean of 0 and a variance of 1 according to the channel dimension to improve the convergence speed and generalization ability of the model.

[0088] Furthermore, in this embodiment, the U-net network module improves upon the traditional U-net network and comprises an encoder and a decoder. The encoder is a contracting path, extracting deep features of cardiac vascular structures through multi-layer convolution and pooling operations, enhancing understanding of vascular structures at different scales. The decoder is an expanding path, utilizing skip connections to combine shallow and deep features to achieve high-resolution vascular segmentation, improving the recognition of small vessels, and ultimately outputting the target segmentation result. During backpropagation, the U-net network learns based on the differences between the output result and the actual segmentation target. In this embodiment, the initial number of feature map channels is set to 32, which is doubled with each downsampling and halved with each upsampling. To balance computational performance and memory consumption, the maximum number of feature map channels for a 2D U-net is 512.

[0089] In this embodiment, the encoder structure is similar to that of the traditional U-net network structure, which includes 5 groups of convolutional layers and 4 pooling layers. Each convolutional layer is used to extract features of the input image data in sequence to generate a new feature map; each pooling layer is used to downsample and encode each feature map, and the activation function in each convolutional layer is changed from ReLU to leaky ReLU and the Batch Norm is changed to Instance Norm; the decoder includes 4 deconvolution layers, 4 groups of convolutional layers and 4 copy and cut layers; each deconvolution layer is used to deconvolve the decoder output feature map; each convolution layer is used to extract features from the deconvolution result to produce a new feature map; the copy and cut layer is used to crop the feature map output by each layer of the encoder and splice it with the feature map of the corresponding part of the decoder to obtain the final image segmentation result.

[0090] Furthermore, during the back-propagation process, the U-net network learns based on the difference between the output result and the actual segmentation target. The loss function consists of the sum of cross entropy and Dice loss, and the loss is calculated for the deep supervision output.

[0091] Furthermore, in the inference module, when a sliding window is used to predict the segmentation results of the target vascular structure in the cardiac slice, the window size is consistent with the patch size used during training, and adjacent windows overlap by half. Gaussian importance weighting is used to perform weighted aggregation on the softmax results. By performing mirrored inference along all axes, test-time augmentation (TTA) is achieved to further improve the segmentation performance.

[0092] Furthermore, in step S132 above, the U-net network was trained for 1000 iterations using stochastic gradient descent with Nesterov momentum (μ = 0.99). The initial learning rate was set to 0.01, and a Poly decay strategy was employed. Data sampling employed a 66.7% random sampling strategy with a 33.3% foreground oversampling strategy, ensuring that each batch contained at least one foreground patch, with a batch size of 2. Data augmentation techniques such as rotation, scaling, Gaussian noise, Gaussian blur, brightness and contrast adjustment, low-resolution simulation, gamma correction, and mirroring were dynamically applied during training.

[0093] Furthermore, in the above step S2, when VTK is used to perform 3D structure reconstruction, the following steps are included:

[0094] S21. Based on the set isosurface threshold, use VTKFlyingEdges3D to perform isosurface extraction to obtain the structural boundary of the blood vessel; in this embodiment, the isosurface threshold can be set to 160 or 170;

[0095] S22. Enable ComputeScalarsOn to increase the point cloud density and obtain the initial point cloud file.

[0096] Furthermore, the above step S3 includes the following steps:

[0097] S31. Filling gaps in the initial point cloud file.

[0098] To fill small gaps in the 3D structure caused by missing vascular segmentation, this example first uses vtkImageContinuousDilate3D to dilate the initial point cloud file to expand the vascular region and fill any broken areas caused by segmentation errors. vtkImageContinuousErode3D is then used to erode the vascular region to restore its original shape and avoid distortion caused by excessive dilation. The dilation and erosion kernel sizes are set to (7,7,3) and (5,5,3), respectively, to accommodate the spatial distribution characteristics of the vessels.

[0099] S32. Perform a smoothing operation on the filled point cloud file.

[0100] After morphological processing, Gaussian smoothing was performed using vtkImageGaussianSmooth to reduce noise in the segmentation results and improve the continuity of the vessel boundaries. The standard deviation was set to 5.0 and the radius factor was set to (2.0, 2.0, 2.0) to ensure that the smoothing operation could remove small-scale noise without affecting the overall vessel structure.

[0101] To further optimize the vascular surface morphology, this paper introduces the vtkWindowedSincPolyDataFilter for smoothing, setting the number of iterations to 5 and enabling boundary smoothing and feature edge smoothing. The feature angle and edge angle are set to 180.0 to minimize angularity. The model data is normalized using the NonManifoldSmoothingOn and NormalizeCoordinatesOn methods to ensure the consistency of the reconstructed vascular structure in 3D space.

[0102] Furthermore, in step S6, the initial segmentation results output by the nn-UNet model are post-processed based on the segmentation results incorporating the 3D vascular geometry prior. This involves comparing the optimized segmentation results with GroundTruth (the true labels, obtained manually) to test the Dice Similarity Coefficient (Dice Similarity Coefficient) and Intersection over Union (IoU). The Dice coefficient is a measure of the similarity between two sets. In medical image segmentation, it is used to assess the degree of consistency between the predicted results and the true labels. It ranges from [0 to 1], with values ​​closer to 1 indicating better segmentation results.

[0103] Example 2

[0104] like Figure 2 and Figure 3 As shown, the present invention optimizes the segmentation results and improves the coherence and stability of the overall vascular network.

[0105] The present invention uses connected component analysis to remove isolated pseudo-segmented regions. Experimental results show that some slices (such as Figure 2 In 1 and 2), there are small, discrete regions that do not conform to the anatomical characteristics of real blood vessels and are often caused by model misidentification. By setting an appropriate area threshold, the present invention effectively removes these falsely segmented regions, thereby reducing interference from non-vascular structures.

[0106] This invention combines information from adjacent slices to correct vascular connectivity, compensating for disconnections caused by missing information. For example, in the experimental results for slice 3, some vascular regions were not correctly segmented on a single slice, but continuous vascular structures were detected at corresponding locations in adjacent slices. This invention addresses this issue by introducing a correction strategy based on three-dimensional spatial consistency constraints. This strategy uses projection and interpolation to complete the missing regions, maintaining more natural longitudinal connectivity for the vessels. Experimental results demonstrate that this method effectively reduces the incidence of vascular disconnection and improves overall coherence.

[0107] This paper uses morphological filtering to optimize segmented edges. Preliminary results show that some vascular edges appear jagged or irregular, deviating from the smooth tubular structure of real blood vessels. To address this, the paper experiments with different morphological processing strategies, including edge smoothing and morphological dilation, to reduce irregular edge sharp corners and improve the continuity of vascular contours. Results show that this method significantly improves the smoothness of vascular structures, making segmentation results more consistent with the anatomical features of medical images.

[0108] Overall, the post-processing strategies employed demonstrated promising optimization results in the experiments. By removing pseudo-segmented regions, enhancing vascular connectivity, and optimizing edge morphology, the present invention effectively improved the stability of segmentation results and reduced fragmentation caused by noise interference or local model misjudgment. These improvements not only enhanced segmentation accuracy but also strengthened the integrity of the vascular network, providing more reliable data support for subsequent vascular analysis and 3D reconstruction.

[0109] like Figure 4 As shown, this embodiment uses IoU and Dice coefficient as the main indicators to compare the segmentation performance of the known methods nnUNet, U-Mamba and SAM_Med_2D and the four methods of the present invention.

[0110] In terms of overall segmentation accuracy, nnUNet and U-Mamba performed relatively closely, but some differences remained in detail. In contrast, SAM_Med_2D's segmentation results were more ambiguous in the vascular edge regions. This may be due to the fact that its pre-training data primarily came from medical imaging such as MRI (magnetic resonance imaging), CT (computed tomography), and ultrasound. The distribution of the current dataset differs significantly from these imaging data, limiting the model's generalization ability. Furthermore, the relatively small sample size of this dataset may have limited the model's learned features, further impacting accurate segmentation of edge regions, particularly exhibiting significant uncertainty in vascular details.

[0111] Using the automatic segmentation method for fetal anatomical cross-section images of congenital heart malformations proposed in the present invention, the present invention has currently established a database of fetal anatomical cross-sections of CHD in the second and third trimesters (22-32 weeks) with a layer thickness of 60um, including endocardial cushion defects, tricuspid valve hypoplasia, anomalous pulmonary venous drainage, hypoplastic left heart, tetralogy of Fallot, transposition of the great arteries, persistent truncus arteriosus, pulmonary artery stenosis, aortopulmonary window, anomalous origin of the pulmonary artery, cardiac rhabdomyoma, atrial myxoma, atrial ectasia, interrupted aortic arch, right aortic arch, double aortic arch, anomalous coronary artery drainage, heterogeneous malformation and interrupted inferior vena cava, etc., covering more than 90% of congenital heart malformations. The CHD anatomical database's continuous display, multi-directional reconstruction, and arbitrary-angle rotation capabilities allow ultrasound physicians to quickly access a wide range of different congenital heart malformations, mastering the cross-sectional and spatial imaging characteristics of different fetal CHDs. Using virtual fetal echocardiography, the database improves the accuracy of prenatal CHD diagnosis and prognosis. Currently, the images in the fetal CHD anatomical database are not yet fully segmented or reconstructed. Each CHD case contains 500-700 cross-sectional images, with a 60µm slice thickness and 8688×5792 pixels. Therefore, automated image segmentation using artificial intelligence (AI) is required to improve accuracy and efficiency.

[0112] Example 3

[0113] The above-mentioned embodiment 1 provides an automatic image segmentation method. Correspondingly, this embodiment provides an automatic image segmentation system. The system provided in this embodiment can implement the automatic image segmentation method of embodiment 1. The system can be implemented by software, hardware, or a combination of software and hardware. For example, the system can include integrated or separate functional modules or functional units to perform the corresponding steps in each method of embodiment 1. Since the system of this embodiment is basically similar to the method embodiment, the description process of this embodiment is relatively simple. For relevant details, please refer to the partial description of embodiment 1. The embodiment of the system provided in this embodiment is merely illustrative.

[0114] The automatic image segmentation system provided in this embodiment includes:

[0115] The preliminary segmentation module is used to input the cross-sectional image of the heart of the target fetus with congenital malformation into the pre-built nn-UNet model to automatically identify and segment the vascular structure and obtain the preliminary segmentation result;

[0116] The 3D reconstruction module is used to reconstruct the 3D structure based on the preliminary segmentation results using VTK to obtain the initial point cloud file;

[0117] The optimization module is used to perform gap filling and smoothing operations on the initial point cloud file to obtain an optimized three-dimensional blood vessel model;

[0118] A projection module is used to project the optimized three-dimensional vascular model into a series of parallel cutting planes according to the actual heart slice spacing;

[0119] The mask module is used to calculate the intersection points of the point cloud data and each cutting plane, and after topological closure and area filling processing, it generates a binary mask to obtain the segmentation result that incorporates the 3D geometric morphology prior of the blood vessels;

[0120] The post-processing module is used to post-process the preliminary segmentation results output by the nn-UNet model based on the segmentation results with the addition of the 3D geometric morphology prior of the blood vessels, and obtain the final segmentation results of the cross-sectional image of the heart of the fetus with congenital malformation to be segmented.

[0121] Example 4

[0122] This embodiment provides a processing device corresponding to the automatic image segmentation method provided in this embodiment 1. The processing device can be a processing device used for a client, such as a mobile phone, laptop computer, tablet computer, desktop computer, etc., to execute the method of embodiment 1.

[0123] The processing device includes a processor, a memory, a communication interface, and a bus. The processor, memory, and communication interface are connected via the bus to facilitate communication between them. The memory stores a computer program executable on the processor. When the processor executes the computer program, it executes the automatic image segmentation method provided in Example 1.

[0124] Preferably, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0125] Preferably, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited here.

[0126] Example 5

[0127] The automatic image segmentation method of this embodiment 1 can be specifically implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing the automatic image segmentation method described in this embodiment 1.

[0128] Computer readable storage media can be tangible devices that hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any combination thereof.

[0129] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A method for automatic image segmentation, characterized in that: The following steps are involved: The cross-sectional image of the heart of the target fetus with congenital malformation is input into the pre-built nn-UNet model for automatic recognition and segmentation of vascular structures to obtain preliminary segmentation results. Based on the preliminary segmentation results, VTK is used to reconstruct the 3D structure and obtain the initial point cloud file; Perform gap filling and smoothing operations on the initial point cloud file to obtain an optimized three-dimensional blood vessel model; The optimized three-dimensional vascular model is projected into a series of parallel cutting planes according to the actual heart slice spacing; The intersection points of the point cloud data and each cutting plane are calculated, and after topological closure and area filling processing, a binary mask is generated to obtain the segmentation result with the 3D geometric morphology prior of the blood vessels. The initial segmentation results output by the nn-UNet model are post-processed based on the segmentation results that incorporate the 3D geometric morphology prior of the blood vessels to obtain the final segmentation results of the cross-sectional image of the heart of the target fetus with congenital malformation. The cross-sectional image of the heart of the target fetus with congenital malformation is input into the pre-built nn-UNet model for automatic recognition and segmentation of vascular structures to obtain preliminary segmentation results, including: Collecting a preset number of fetal congenital malformation heart cross-sectional images and storing them in a preset format; Annotating preset cardiac and vascular structures in the stored cross-sectional images of the hearts of each fetus with congenital malformation and constructing a data set; wherein the preset cardiac and vascular structures include the aorta, descending aorta, pulmonary artery and superior vena cava; Use the dataset to train the pre-built nn-UNet model to obtain a nn-UNet model that meets the preset requirements as an image segmentation model; The image segmentation model is used to automatically identify and segment the vascular structure of the target fetus's congenital malformation heart cross-section image, and preliminary segmentation results are obtained; Based on the preliminary segmentation results, VTK is used to reconstruct the three-dimensional structure and obtain the initial point cloud file, including: Based on the set isosurface threshold, VTKFlyingEdges3D is used to extract the isosurface and obtain the structural boundary of the blood vessel; Enable ComputeScalarsOn to increase the point cloud density and obtain the initial point cloud file; The gap filling and smoothing operations on the initial point cloud file include: vtkImageContinuousDilate3D is used to dilate the initial point cloud file to expand the blood vessel area; Use vtkImageContinuousErode3D to perform erosion to restore the original shape of the blood vessels; Apply vtkImageGaussianSmooth for Gaussian smoothing.

2. The automatic image segmentation method according to claim 1, wherein: The collecting of a preset number of fetal congenital malformation heart cross-sectional images and storing them in a preset format includes: Obtain a preset number of fetal congenital malformation heart cross-sectional image data and store them in PNG format; Converting the original color image of the collected fetal heart cross section with congenital malformation into a grayscale image; Convert the grayscale image of the cross-section of the heart of a fetus with congenital malformation from the original PNG format to the NII.GZ format commonly used in medical imaging and store it.

3. The automatic image segmentation method according to claim 1, wherein: The method of using the dataset to train the pre-built nn-UNet model to obtain the nn-UNet model that meets the preset requirements as the image segmentation model includes: Build the nn-UNet model; The nn-UNet model includes: a preprocessing module, a U-net network module, and an inference module; the preprocessing module is used to crop, resample, and normalize invalid areas in the cross-sectional image of the fetal heart with congenital malformation; the U-net network module is used to construct a deep neural network to perform semantic segmentation of the heart slice blood vessels; the inference module is used to process the semantic segmentation results using a sliding window to obtain the segmentation results of the heart slice blood vessels; Based on the preset training parameters, the constructed nn-UNet model is trained using the data set to obtain a nn-UNet model that meets the preset requirements as an image segmentation model.

4. The automatic image segmentation method according to claim 3, wherein: The U-net network module includes an encoder and a decoder; The encoder is used to extract deep features of blood vessels in heart slices through multiple convolutional layers and pooling layers, and the activation function in the convolutional layer adopts leaky ReLU function and Instance Norm function; The decoder is used to combine shallow and deep features using skip connections and output target segmentation results.

5. An automatic image segmentation system, characterized in that: include: The preliminary segmentation module is used to input the cross-sectional image of the heart of the target fetus with congenital malformation into the pre-built nn-UNet model to automatically identify and segment the vascular structure and obtain the preliminary segmentation result; The 3D reconstruction module is used to reconstruct the 3D structure based on the preliminary segmentation results using VTK to obtain the initial point cloud file; The optimization module is used to perform gap filling and smoothing operations on the initial point cloud file to obtain an optimized three-dimensional blood vessel model; A projection module is used to project the optimized three-dimensional vascular model into a series of parallel cutting planes according to the actual heart slice spacing; The mask module is used to calculate the intersection points of the point cloud data and each cutting plane, and after topological closure and area filling processing, it generates a binary mask to obtain the segmentation result that incorporates the 3D geometric morphology prior of the blood vessels; A post-processing module is used to post-process the preliminary segmentation results output by the nn-UNet model based on the segmentation results that incorporate the 3D geometric morphology prior of the blood vessels, and obtain the final segmentation results of the cross-sectional image of the heart of the fetus with congenital malformation to be segmented; The cross-sectional image of the heart of the target fetus with congenital malformation is input into the pre-built nn-UNet model for automatic recognition and segmentation of vascular structures to obtain preliminary segmentation results, including: Collecting a preset number of fetal congenital malformation heart cross-sectional images and storing them in a preset format; Annotating preset cardiac and vascular structures in the stored cross-sectional images of the hearts of each fetus with congenital malformation and constructing a data set; wherein the preset cardiac and vascular structures include the aorta, descending aorta, pulmonary artery and superior vena cava; Use the dataset to train the pre-built nn-UNet model to obtain a nn-UNet model that meets the preset requirements as an image segmentation model; The image segmentation model is used to automatically identify and segment the vascular structure of the target fetus's congenital malformation heart cross-section image, and preliminary segmentation results are obtained; Based on the preliminary segmentation results, VTK is used to reconstruct the three-dimensional structure and obtain the initial point cloud file, including: Based on the set isosurface threshold, VTKFlyingEdges3D is used to extract the isosurface and obtain the structural boundary of the blood vessel; Enable ComputeScalarsOn to increase the point cloud density and obtain the initial point cloud file; The gap filling and smoothing operations on the initial point cloud file include: vtkImageContinuousDilate3D is used to dilate the initial point cloud file to expand the blood vessel area; Use vtkImageContinuousErode3D to perform erosion to restore the original shape of the blood vessels; Apply vtkImageGaussianSmooth for Gaussian smoothing.

6. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any one of the methods of claims 1 to 4 .

7. A computing device, characterized in that include: One or more processors and a memory, wherein the memory stores one or more programs and is configured to be executed by the one or more processors, wherein the one or more programs include instructions for executing any one of the methods according to claims 1 to 4.

Images