Automatic image segmentation method, system, equipment and medium

Abstract

The invention relates to an automatic image segmentation method, system and device and a medium, and the method comprises the steps: inputting a target fetus congenital malformation heart cross section image into a pre-constructed nn-UNet model, and carrying out the automatic recognition and segmentation of a blood vessel structure, and obtaining a preliminary segmentation result; based on the preliminary segmentation result, performing three-dimensional structure reconstruction by using VTK to obtain an initial point cloud file; performing gap filling and smoothing operation on the initial point cloud file, and projecting the optimized three-dimensional blood vessel model into a series of parallel cutting planes according to the actual heart slice spacing; calculating an intersection point set of the point cloud data and each cutting plane, carrying out topological closing and region filling processing, generating a binary mask, and obtaining a segmentation result added with the blood vessel three-dimensional geometrical shape prior; and post-processing the preliminary segmentation result to obtain a final segmentation result of the congenital malformation heart cross section image of the target fetus. The method can be widely applied to the field of image processing.

Description

Technical Field

[0001] The present invention belongs to the field of medical image segmentation, and relates to an image automatic segmentation method, system, device and medium, in particular to an automatic segmentation method, system, device and medium for anatomical cross-sectional images of fetal congenital heart malformations. Background Art

[0002] Congenital Heart Disease (CHD) is one of the most common severe congenital structural malformations globally and is a major cause of neonatal death. Timely and accurate prenatal diagnosis is crucial for improving the survival rate and quality of life of children. CHD is often accompanied by complex cardiovascular malformations, such as anomalous pulmonary venous drainage, transposition of the great arteries, persistent truncus arteriosus, pulmonary artery stenosis, aortopulmonary window, anomalous origin of the pulmonary artery, interruption of the aortic arch, right aortic arch, double aortic arch, anomalous drainage of the coronary artery, and interruption of the inferior vena cava, etc. The precise identification of these cardiovascular malformations is very important for clinicians to formulate treatment plans. However, due to the complex cardiac structure and diverse pathological changes, the precise segmentation of blood vessels is particularly difficult. Therefore, accurately segmenting the blood vessel region is of great significance for the diagnosis and formulation of treatment plans, which can help doctors accurately identify and locate the lesion area, thereby achieving more accurate diagnosis and treatment, and playing a crucial role in CHD diagnosis, treatment planning, and surgical navigation.

[0003] Although the traditional manual segmentation method is simple and direct, its process is extremely time-consuming and requires a large amount of professional domain knowledge. For example, taking the database adopted by the present invention as an example, when previously using the manual method for image annotation and segmentation, the processing time for each image was at least 5 minutes, and each cardiac image data on average contained about 600 images, totaling 3000 minutes to complete. Taking the existing anatomical database of fetal congenital heart malformations as an example, it takes at least 2 years to manually segment all the data, and this database is still continuously expanding. Therefore, there is an urgent need to use artificial intelligence technology to improve the efficiency of medical image segmentation.

[0004] For anatomical cross-sectional images of fetal congenital heart malformations, traditional methods such as nnU-Net and U-Mamba can be used for segmentation. Although both of these models can choose 2D or 3D training methods, the 3D training method requires the annotation of a large number of complete cardiac slices. In the medical field, the cost of data annotation is extremely high, making it difficult to effectively reduce costs using 3D training. And using the 2D training method cannot fully utilize the three-dimensional structural information of blood vessels. Experiments have found that there may be a situation where blood vessel recognition is discontinuous at some slice positions, that is, some blood vessels are not correctly recognized in some slices.

[0005] Blood vessels in medical images are highly spatially connected structures, and their morphology usually has strong continuity between adjacent slices. Based on this physiological and anatomical prior, if only relying on 2D segmentation results, blood vessel breaks or omissions may occur due to local information loss or noise interference. Therefore, relying solely on 2D methods for blood vessel segmentation is easily restricted by the local features of individual slices, making it difficult for the model to correctly identify blood vessels in the case of local information loss or noise interference, and thus resulting in discontinuous or broken situations. As a three-dimensional structure, blood vessels have strong topological connectivity in space, and their morphology is usually continuous between adjacent slices. Therefore, segmenting individual slices only through 2D methods cannot utilize the overall morphological information of blood vessels, making it difficult to ensure the stability and coherence of the segmentation results. On the contrary, with the help of three-dimensional reconstruction, the morphology of blood vessels can be constrained on a global scale, making the segmentation results more in line with the actual anatomical structure and compensating for the incompleteness problems caused by insufficient local information in 2D segmentation methods. Summary of the Invention

[0006] Aiming at the above problems, the purpose of the present invention is to provide an automatic segmentation method, system, device and medium for anatomical cross-sectional images of fetal congenital heart malformations that combines deep learning and uses the prior of three-dimensional features of blood vessels for post-processing, which can achieve high-precision segmentation of cardiac blood vessels.

[0007] To achieve the above purpose, the present invention adopts the following technical solutions: In the first aspect, the present invention provides an image automatic segmentation method, including the following steps: Input the target fetal congenital malformation heart cross-sectional image into a pre-constructed nn-UNet model for automatic recognition and segmentation of blood vessel structures to obtain a preliminary segmentation result; Based on the preliminary segmentation result, use VTK for three-dimensional structure reconstruction to obtain an initial point cloud file; Perform gap filling and smoothing operations on the initial point cloud file to obtain an optimized three-dimensional blood vessel model; Project the optimized three-dimensional blood vessel model into a series of parallel cutting planes according to the actual cardiac slice spacing; Calculate the intersection set of the point cloud data and each cutting plane, and after performing topological closure and region filling processing, generate a binary mask to obtain a segmentation result with the prior of three-dimensional geometric morphology of blood vessels; Based on the segmentation result with the prior of three-dimensional geometric morphology of blood vessels, perform post-processing on the preliminary segmentation result output by the nn-UNet model to obtain the final segmentation result of the target fetal congenital malformation heart cross-sectional image.

[0008] Further, inputting the cross-sectional image of the target fetus's congenital malformation heart into the pre-constructed nn-UNet model for automatic recognition and segmentation of blood vessel structures to obtain a preliminary segmentation result, including: Collect a preset number of cross-sectional images of the fetus's congenital malformation heart and store them in a preset format; Annotate the preset heart blood vessels in each stored cross-sectional image of the fetus's congenital malformation heart and construct a data set; Use the data set to train the pre-constructed nn-UNet model to obtain an nn-UNet model that meets the preset requirements as an image segmentation model; Use the image segmentation model to automatically recognize and segment the blood vessel structures of the cross-sectional image of the target fetus's congenital malformation heart to obtain a preliminary segmentation result.

[0009] Further, the collecting a preset number of cross-sectional images of the fetus's congenital malformation heart and storing them in a preset format includes: Obtain the cross-sectional image data of a preset number of the fetus's congenital malformation hearts and store them in the PNG format; Convert the original color image of the cross-sectional image of the fetus's congenital malformation heart into a grayscale image; Convert the grayscale image of the cross-sectional image of the fetus's congenital malformation heart from the original PNG format to the commonly used NII.GZ format in medical imaging and store it.

[0010] Further, the using the data set to train the pre-constructed nn-UNet model to obtain an nn-UNet model that meets the preset requirements as an image segmentation model includes: Construct an nn-UNet model; The nn-UNet model includes: a preprocessing module, a U-net network module, and an inference module; the preprocessing module is used to crop, resample, and normalize the data of the invalid regions in the cross-sectional image of the fetus's congenital malformation heart; the U-net network module is used to construct a deep neural network for semantic segmentation of the blood vessels of the heart slices; the inference module is used to process the semantic segmentation result by using a sliding window to obtain the segmentation result of the blood vessels of the heart slices; Based on the preset training parameters, use the data set to train the constructed nn-UNet model to obtain an nn-UNet model that meets the preset requirements as an image segmentation model.

[0011] Further, the U-net network module includes an encoder and a decoder; The encoder is used to extract deep features of blood vessels in cardiac slices through multiple convolutional layers and pooling layers, and the activation functions in the convolutional layers adopt the leaky ReLU function and the Instance Norm function; The decoder is used to combine shallow and deep features using skip connections and output the target segmentation result.

[0012] Further, based on the preliminary segmentation result, VTK is used for three-dimensional structure reconstruction to obtain an initial point cloud file, including: Based on a set isosurface threshold, VTK Flying Edges 3D is used for isosurface extraction to obtain the structural boundary of the blood vessels; Enable ComputeScalarsOn to increase the point cloud density to obtain the initial point cloud file.

[0013] Further, the gap filling and smoothing operations on the initial point cloud file include: Use vtkImageContinuousDilate3D to dilate the initial point cloud file to expand the blood vessel area; Use vtkImageContinuousErode3D for erosion to restore the original shape of the blood vessels; Apply vtkImageGaussianSmooth for Gaussian smoothing.

[0014] In a second aspect, the present invention provides an image automatic segmentation system, including: A preliminary segmentation module for inputting a cross-sectional image of a target fetal congenital malformation heart into a pre-constructed nn-UNet model for automatic recognition and segmentation of blood vessel structures to obtain a preliminary segmentation result; A three-dimensional reconstruction module for performing three-dimensional structure reconstruction using VTK based on the preliminary segmentation result to obtain an initial point cloud file; An optimization module for performing gap filling and smoothing operations on the initial point cloud file to obtain an optimized three-dimensional blood vessel model; A projection module for projecting the optimized three-dimensional blood vessel model into a series of parallel cutting planes according to the actual cardiac slice spacing; A mask module for calculating the intersection set of the point cloud data and each cutting plane, and after performing topological closure and region filling processing, generating a binary mask to obtain a segmentation result with the prior of the three-dimensional geometric shape of the blood vessels added; A post-processing module for post-processing the preliminary segmentation result output by the nn-UNet model based on the segmentation result with the prior of the three-dimensional geometric shape of the blood vessels added to obtain the final segmentation result of the cross-sectional image of the fetal congenital malformation heart to be segmented.

[0015] In a third aspect, the present invention provides a computer-readable storage medium storing one or more programs, the one or more programs including instructions which, when executed by a computing device, cause the computing device to execute any of the methods.

[0016] In a fourth aspect, the present invention provides a computing device, comprising: one or more processors and a memory, wherein the memory stores one or more programs and is configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the methods.

[0017] Due to the above technical solutions adopted by the present invention, it has the following advantages: 1. The present invention reduces the annotation cost. The traditional 3D training method relies heavily on fully annotated three-dimensional data, while the present invention uses 2D image data for training, reducing the data annotation cost. At the same time, VTK is combined for three-dimensional reconstruction, and through the projection optimization strategy, the result of three-dimensional reconstruction is fed back to the 2D slice to correct the original segmentation, improving the overall accuracy and stability.

[0018] 2. The present invention constructs a dataset of congenital heart vascular malformations. Since CHD is accompanied by various complex vascular malformations, it can more accurately identify abnormal vascular structures and help doctors better formulate diagnosis and treatment plans.

[0019] 3. An efficient post-processing optimization strategy. The present invention performs three-dimensional reconstruction, smoothing optimization and interpolation completion of blood vessels through VTK, and uses anatomical prior knowledge to improve the segmentation coherence. Three-dimensional topological information is supplemented in the post-processing stage to make the blood vessel segmentation result more coherent, making up for the limitations of 2D methods.

[0020] Therefore, the present invention can be widely applied to the field of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Throughout the drawings, the same reference numerals are used to represent the same components. In the drawings: Figure 1 is a flowchart of the image automatic segmentation method provided by an embodiment of the present invention; Figure 2 is a comparison diagram of the segmentation effect and other existing methods provided by an embodiment of the present invention; Figure 3 is an IoU index table of the segmentation result provided by an embodiment of the present invention; Figure 4 is a Dice index table of the segmentation result provided by an embodiment of the present invention. Detailed implementation mode

[0022] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the described embodiments of the present invention fall within the scope of protection of the present invention.

[0023] It should be noted that the terms used herein are only for describing specific implementation modes and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they specify the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0024] Blood vessels in medical images are highly spatially connected structures, and their morphology usually has strong continuity between adjacent slices. Based on this physiological and anatomical prior, if only relying on 2D segmentation results, blood vessel breaks or omissions may occur due to local information loss or noise interference. Therefore, relying solely on 2D methods for blood vessel segmentation is easily restricted by the local features of individual slices, making it difficult for the model to correctly identify blood vessels in the case of local information loss or noise interference, and thus resulting in discontinuous or broken situations. As a three-dimensional structure, blood vessels have strong topological connectivity in space, and their morphology is usually continuous between adjacent slices. Therefore, segmenting independent slices only by 2D methods cannot utilize the overall morphological information of blood vessels, making it difficult to ensure the stability and coherence of the segmentation results. On the contrary, with the help of three-dimensional reconstruction, the morphology of blood vessels can be constrained on a global scale, making the segmentation results more in line with the actual anatomical structure and compensating for the incompleteness problem caused by insufficient local information in 2D segmentation methods.

[0025] Based on the above analysis, in some embodiments of the present invention, an automatic image segmentation method is provided, aiming to use the three-dimensional geometric shape of blood vessels to inversely guide the results of two-dimensional segmentation. Specifically, the method includes: automatically identifying and separating blood vessel structures through the nnU-Net training model to obtain a preliminary segmentation result; subsequently, based on this segmentation result, using VTK to perform three-dimensional reconstruction of the blood vessel structure, and optimizing the initial point cloud data through gap filling and smoothing processing to generate a high-quality three-dimensional blood vessel model; then, according to the actual cardiac slice spacing, projecting the three-dimensional model along the axis into a series of parallel cutting planes, and by calculating the intersection set of the point cloud and each cutting plane, combining topological closure and region filling algorithms, generating a two-dimensional binary mask containing three-dimensional geometric shape priors to guide the post-processing of the preliminary results of nn-UNet, so as to obtain a more accurate final segmentation result. The advantage of this back-projection method is that it can effectively map three-dimensional structural information into two-dimensional images, thereby correcting errors in the original segmentation caused by noise, occlusion, or complex structures under the constraint of spatial consistency, and significantly improving the anatomical rationality and boundary continuity of the segmentation. The present invention effectively solves problems such as insufficient segmentation accuracy, poor recognition of blood vessel connectivity, and loss of reconstruction details in traditional segmentation methods, significantly improves the integrity and accuracy of blood vessel segmentation, reduces the cost and error of manual intervention, has good application prospects, and is applicable to fields such as the auxiliary diagnosis of cardiovascular diseases.

[0026] Correspondingly, in other embodiments of the present invention, an automatic image segmentation system, device, and medium are provided.

[0027] Embodiment 1 As Figure 1 shown, this embodiment provides an automatic image segmentation method, which includes the following steps: S1. Input the cross-sectional image of the target fetal congenital malformation heart into a pre-constructed nn-UNet model for automatic recognition and segmentation of blood vessel structures to obtain a preliminary segmentation result; S2. Based on the preliminary segmentation result, use VTK (Visualization Toolkit) to perform three-dimensional structure reconstruction to obtain an initial point cloud file; S3. Perform gap filling and smoothing operations on the initial point cloud file to obtain an optimized three-dimensional blood vessel model; S4. Project the optimized three-dimensional blood vessel model into a series of parallel cutting planes according to the actual cardiac slice spacing; S5. Calculate the intersection set of the point cloud data and each cutting plane, and after performing topological closure and region filling processing, generate a binary mask to obtain a segmentation result with the prior of the three-dimensional geometry of blood vessels; S6. Post-process the preliminary segmentation result output by the nn-UNet model based on the segmentation result incorporating the prior of the three-dimensional geometric shape of blood vessels to obtain the final segmentation result of the cross-sectional image of the target fetal congenital malformation heart.

[0028] Further, in the above step S1, the following steps are included: S11. Collect a preset number of cross-sectional images of fetal congenital malformation hearts and store them in a preset format; S12. Label the preset heart blood vessels in each of the stored cross-sectional images of fetal congenital malformation hearts and construct a data set; S13. Use the data set to train the pre-constructed nn-UNet model to obtain an nn-UNet model that meets the preset requirements as an image segmentation model; S14. Use the image segmentation model to automatically identify and segment the blood vessel structure of the cross-sectional image of the target fetal congenital malformation heart to obtain a preliminary segmentation result.

[0029] Further, in the above step S11, the following steps are included: S111. Obtain the cross-sectional image data of a preset number of fetal congenital malformation hearts and store them in the PNG format; S112. Use OpenCV (Open Source Computer Vision Library) to convert the obtained original color image of the cross-sectional fetal congenital malformation heart into a grayscale image; S113. Use the Nibabel (Medical Image Data Processing Library) and PIL (Python Imaging Library, now Pillow) libraries to convert the grayscale image of the cross-sectional fetal congenital malformation heart from the original PNG format to the commonly used NII.GZ format in medical imaging and store it to ensure the consistency of the data format.

[0030] In this embodiment, only the tools of OpenCV, nibabel, and PIL libraries are taken as examples for introduction, but it is not limited thereto.

[0031] Further, in the above step S12, when labeling the preset heart blood vessels in the stored images, at least four main blood vessels of the heart need to be labeled. In this embodiment, the four main blood vessels of the heart include: aorta, descending aorta, pulmonary artery, and superior vena cava.

[0032] Further, in the above step S12, when labeling the preset cardiac blood vessels in the stored cross-sectional images of the congenital malformed fetal hearts, first, the LabelMe tool is used to complete the labeling. Labels 1, 2, 3, and 4 are respectively assigned to the aorta, pulmonary artery, descending aorta, and superior vena cava, and the corresponding JSON annotation files are generated. Then, with the help of the labelme_json_to_dataset command, the annotation data is batch-converted into a folder containing the original images, label images, and class names, and the color label images are further extracted. Finally, nibabel and PIL are used to process it into the NII.GZ format to match the image data.

[0033] In this embodiment, the constructed dataset includes the annotation of more than 300 cardiac slices of more than twenty hearts. One of the hearts is finely annotated, with one slice annotated every 120 millimeters, more than 20 slices annotated for each of the four hearts, and 5 slices annotated for each of the remaining hearts.

[0034] Further, in the above step S13, the following steps are included: S131. Construct an nn-UNet model; S132. Based on the preset training parameters, use the dataset to train the constructed nn-UNet model to obtain an nn-UNet model that meets the preset requirements as the image segmentation model.

[0035] Further, in the above step S131, the nn-UNet model constructed in this embodiment includes a preprocessing module, a U-net network module, and an inference module. Among them, the preprocessing module is used to crop, resample, and normalize the data of the invalid regions in the cross-sectional images of the congenital malformed fetal hearts; the U-net network module is used to construct a deep neural network for semantic segmentation of the blood vessels in the cardiac slices; the inference module is used to process the semantic segmentation results using a sliding window to obtain the segmentation results of the blood vessels in the cardiac slices.

[0036] Further, the preprocessing module includes a cropping module, a resampling module, and a normalization module; among them, the cropping module is used to perform Crop (cropping) processing on the input image data to crop the invalid regions and reduce the computational burden; the resampling module is used to resample the cropped image. For example, the third-order spline interpolation can be used to adjust the image, and the nearest neighbor interpolation method can be used to adjust the segmentation label; the normalization module is used to perform z-score normalization, normalizing the data with a mean of 0 and a variance of 1 in the channel dimension to improve the convergence speed and generalization ability of the model.

[0037] Furthermore, in this embodiment, the U-net network module is improved based on the traditional U-net network. It includes an encoder and a decoder. Among them, the encoder is a contracting path, which is used to extract the deep features of the blood vessels in the heart slice through multi-layer convolution and pooling operations, enhancing the understanding of blood vessel structures at different scales. The decoder is an expanding path, which is used to combine shallow and deep features using skip connections to achieve high-resolution blood vessel segmentation, improve the recognition ability of small blood vessels, and finally output the target segmentation result. During the backpropagation process, the U-net network learns based on the difference between the output result and the actual segmentation target. In this embodiment, the initial number of channels of the feature map is set to 32, which doubles during each downsampling and halves during upsampling. To balance computational performance and memory consumption, the maximum number of channels of the 2D U-Net is 512.

[0038] In this embodiment, the encoder structure is similar to that of the traditional U-net network. It includes 5 groups of convolutional layers and 4 pooling layers. Each convolutional layer is used to sequentially extract features from the input image data to generate a new feature map. Each pooling layer is used to perform downsampling encoding on each feature map. The activation function in each convolutional layer is changed from ReLU to leaky ReLU, and Batch Norm is changed to Instance Norm. The decoder includes 4 transposed convolutional layers, 4 groups of convolutional layers, and 4 copy and crop layers. Each transposed convolutional layer is used to perform transposed convolution on the decoder output feature map. Each convolutional layer is used to extract features from the transposed convolution result to produce a new feature map. The copy and crop layer is used to crop the feature maps output by each layer of the encoder and splice them with the corresponding part of the feature map of the decoder to obtain the final image segmentation result.

[0039] Furthermore, during the backpropagation process, the U-net network learns based on the difference between the output result and the actual segmentation target. The loss function is composed of the sum of cross entropy and Dice loss, and the loss is calculated for the deep supervision output.

[0040] Furthermore, in the inference module, when using a sliding window to predict the segmentation result of the target blood vessel structure in the heart slice, the window size is the same as the patch size used during training, and adjacent windows overlap by half. Gaussian importance weighting is used to perform weighted aggregation on the softmax result. Through mirror inference along all axes, test-time augmentation (TTA) is achieved to further improve the segmentation performance.

[0041] Further, in the above step S132, during the training process, the U-net network performs 1000 rounds of iteration, uses stochastic gradient descent with Nesterov momentum (μ = 0.99), sets the initial learning rate to 0.01, and adopts the Poly strategy for decay. Data sampling adopts a strategy of 66.7% random sampling and 33.3% foreground oversampling to ensure that each batch contains at least one foreground patch, and the batch size is 2. During the training process, data augmentation techniques such as rotation, scaling, Gaussian noise, Gaussian blur, brightness contrast adjustment, low-resolution simulation, gamma correction, and mirroring are dynamically applied.

[0042] Further, in the above step S2, when performing 3D structure reconstruction using VTK, it includes: S21. Based on the set isosurface threshold, use VTKFlyingEdges3D to extract the isosurface to obtain the structural boundary of the blood vessels; in this embodiment, the isosurface threshold can be set to 160 or 170; S22. Enable ComputeScalarsOn to increase the point cloud density to obtain an initial point cloud file.

[0043] Further, in the above step S3, it includes the following steps: S31. Fill the gaps in the initial point cloud file.

[0044] To fill the small gaps in the 3D structure caused by missing blood vessel segmentation, when filling the gaps in the initial point cloud file in this embodiment, first use vtkImageContinuousDilate3D for dilation to expand the blood vessel area and fill the broken areas that may be caused by segmentation errors; then use vtkImageContinuousErode3D for erosion to restore the original shape of the blood vessels and avoid morphological distortion caused by excessive dilation. The sizes of the dilation and erosion kernels are set to (7,7,3) and (5,5,3) respectively to adapt to the spatial distribution characteristics of the blood vessels.

[0045] S32. Smooth the filled point cloud file.

[0046] After morphological processing, apply vtkImageGaussianSmooth for Gaussian smoothing to reduce the noise in the segmentation result and improve the continuity of the blood vessel boundary. The standard deviation is set to 5.0, and the radius factor is set to (2.0,2.0,2.0) to ensure that the smoothing operation can remove small-scale noise without affecting the overall structure of the blood vessels.

[0047] To further optimize the surface morphology of blood vessels, the present invention introduces vtkWindowedSincPolyDataFilter for smoothing, sets the number of iterations to 5, and enables functions such as Boundary Smoothing and Feature Edge Smoothing. The feature angles and edge angles are set to 180.0 to minimize sharp corners. At the same time, the model data is normalized through NonManifoldSmoothingOn and NormalizeCoordinatesOn to ensure the consistency of the reconstructed blood vessel structure in three-dimensional space.

[0048] Furthermore, in the above step S6, post-processing the preliminary segmentation result output by the nn-UNet model based on the segmentation result incorporating the prior of the three-dimensional geometric morphology of blood vessels means comparing the optimized segmentation result with the GroundTruth (true label, obtained through manual annotation), and testing the Dice (Dice Similarity Coefficient) and IoU (Intersection over Union). Among them, the Dice coefficient is an index to measure the similarity of two sets, and is used in medical image segmentation to evaluate the degree of agreement between the prediction result and the true annotation. The value range is [0,1], and the closer the value is to 1, the better the segmentation effect.

[0049] Embodiment 2 As Figure 2 and Figure 3 shown, the present invention optimizes the segmentation result and improves the coherence and stability of the overall blood vessel network.

[0050] The present invention uses connected component analysis to remove isolated pseudo-segmentation regions. The experimental results show that there are discrete small regions in some slices (such as Figure 2 1, 2 in

[0051] The present invention corrects blood vessel connections by combining information from adjacent slices to make up for the breakage problems caused by missing information. For example, in the experimental results of slice 3, some blood vessel regions were not correctly segmented on a single slice, but continuous blood vessel structures were detected in the corresponding positions of its adjacent slices. The present invention solves this problem by introducing a correction strategy based on three-dimensional spatial consistency constraints, and complements the missing regions through projection and interpolation methods, so that the blood vessels maintain a more natural connectivity longitudinally. The experimental results show that this method effectively reduces the incidence of blood vessel breakage and improves the overall coherence.

[0052] The present invention uses morphological filtering methods to optimize the segmentation edges. In the preliminary results, some blood vessel edges showed serrations or irregular protrusions, which deviated from the smooth tubular structure of real blood vessels to a certain extent. For this reason, the present invention experimented with different morphological processing strategies, including edge smoothing and morphological dilation operations, to reduce irregular edge sharp corners and improve the continuity of the blood vessel contour. The results show that this method significantly improves the smoothness of the blood vessel structure, making the segmentation results more in line with the anatomical features of medical images.

[0053] Generally speaking, the post-processing strategies adopted all showed good optimization effects in the experiments. By removing pseudo-segmentation regions, enhancing blood vessel connectivity, and optimizing edge morphology, the present invention effectively improves the stability of the segmentation results and reduces breakage phenomena caused by noise interference or local misjudgment of the model. These improvements not only improve the accuracy of segmentation, but also enhance the integrity of the blood vessel network, providing more reliable data support for subsequent blood vessel analysis and three-dimensional reconstruction.

[0054] As Figure 4 shown, this embodiment uses the IoU and Dice coefficients as the main indicators to compare the segmentation performance of the four methods of the known methods nnUNet, U-Mamba, SAM_Med_2D and the present invention.

[0055] In terms of the overall segmentation accuracy, the performances of nnUNet and U-Mamba are relatively close, but there are still certain differences at the detailed level. In contrast, the segmentation results of SAM_Med_2D in the blood vessel edge region are relatively blurred. This may be because its pre-trained data mainly comes from medical images such as MRI (Magnetic Resonance Imaging), CT (Computed Tomography), and ultrasound, while the distribution of the current dataset is quite different from these image data, resulting in limited generalization ability of the model. In addition, the sample size of this dataset is relatively small, and there may be deficiencies in the features learned by the model, further affecting the precise segmentation of the edge region, especially showing great uncertainty in the blood vessel details.

[0056] Using the automatic segmentation method of fetal congenital heart malformation anatomical cross-sectional images proposed by the present invention, the present invention has currently established a CHD fetal anatomical cross-sectional database with a slice thickness of 60um in the middle and late pregnancy (22-32 weeks), including endocardial cushion defect, Ebstein anomaly, anomalous pulmonary venous drainage, hypoplastic left heart, tetralogy of Fallot, transposition of the great arteries, persistent truncus arteriosus, pulmonary stenosis, aortopulmonary window, anomalous origin of the pulmonary artery, cardiac rhabdomyoma, atrial myxoma, atrial aneurysm, interruption of the aortic arch, right aortic arch, double aortic arch, anomalous coronary artery drainage, isomerism malformation and interruption of the inferior vena cava, etc., covering more than 90% of congenital heart malformations. By using the continuous display, multi-directional reconstruction and arbitrary angle rotation functions of the CHD anatomical database, ultrasound doctors can be exposed to a large number of different types of congenital heart malformations in a short period of time, master the tomographic and spatial image characteristics of different types of fetal congenital heart diseases, virtual fetal echocardiography technology, improve the prenatal diagnosis accuracy rate of CHD, and improve the prognosis. At present, the images of the fetal congenital heart malformation anatomical database have not been segmented and three-dimensionally reconstructed. The anatomical database of each fetal congenital heart malformation case contains 500-700 cross-sectional images with a slice thickness of 60um and a pixel of 8688×5792. It is necessary to use artificial intelligence methods for automatic image segmentation to improve accuracy and efficiency.

[0057] Example 3 The above Example 1 provides an image automatic segmentation method. Correspondingly, this embodiment provides an image automatic segmentation system. The system provided in this embodiment can implement the image automatic segmentation method of Example 1, and the system can be implemented in a software, hardware or a combination of software and hardware manner. For example, the system can include integrated or separate functional modules or functional units to execute the corresponding steps in each method of Example 1. Since the system of this embodiment is basically similar to the method embodiment, the description process of this embodiment is relatively simple, and the relevant parts can refer to the partial description of Example 1. The embodiment of the system provided in this embodiment is only illustrative.

[0058] The image automatic segmentation system provided in this embodiment includes: A preliminary segmentation module, configured to input the cross-sectional image of the target fetal congenital malformation heart into a pre-constructed nn-UNet model for automatic recognition and segmentation of blood vessel structures, and obtain a preliminary segmentation result; A three-dimensional reconstruction module, configured to perform three-dimensional structure reconstruction using VTK based on the preliminary segmentation result to obtain an initial point cloud file; An optimization module, configured to perform gap filling and smoothing operations on the initial point cloud file to obtain an optimized three-dimensional blood vessel model; A projection module, configured to project the optimized three-dimensional blood vessel model into a series of parallel cutting planes according to the actual cardiac slice spacing; A mask module is used to calculate the intersection set of the point cloud data and each cutting plane, and after performing topological closure and region filling processing, generate a binary mask to obtain a segmentation result with the prior of the three-dimensional geometric shape of blood vessels added. A post-processing module is used to post-process the preliminary segmentation result output by the nn-UNet model based on the segmentation result with the prior of the three-dimensional geometric shape of blood vessels added to obtain the final segmentation result of the cross-sectional image of the fetal congenital malformation heart to be segmented.

[0059] Example 4 This example provides a processing device corresponding to the image automatic segmentation method provided in Example 1. The processing device can be a processing device for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the method of Example 1.

[0060] The processing device includes a processor, a memory, a communication interface, and a bus. The processor, the memory, and the communication interface are connected through the bus to complete mutual communication. A computer program that can run on the processor is stored in the memory, and when the processor runs the computer program, it executes the image automatic segmentation method provided in Example 1.

[0061] Preferably, the memory can be a high-speed random access memory (RAM: Random Access Memory), and may also include a non-volatile memory, such as at least one disk memory.

[0062] Preferably, the processor can be various types of general-purpose processors such as a central processing unit (CPU), a digital signal processor (DSP), etc., which are not limited here.

[0063] Example 5 The image automatic segmentation method of Example 1 can be specifically implemented as a computer program product. The computer program product can include a computer-readable storage medium, on which computer-readable program instructions for executing the image automatic segmentation method described in Example 1 are uploaded.

[0064] The computer-readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. The computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination of the above.

[0065] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0066] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or the combination of blocks.

[0067] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that realize the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or the combination of blocks.

[0068] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable devices provide steps for realizing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or the combination of blocks.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: still can modify the specific implementation manners of the present invention or make equivalent replacements, and any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the protection scope of the claims of the present invention.

Claims

1. An automatic image segmentation method, characterized in that, It includes the following steps: Input the cross-sectional image of the congenital malformed heart of the target fetus into the pre-constructed nn-UNet model for automatic recognition and segmentation of blood vessel structures to obtain a preliminary segmentation result; Based on the preliminary segmentation result, use VTK for three-dimensional structure reconstruction to obtain an initial point cloud file; Perform gap filling and smoothing operations on the initial point cloud file to obtain an optimized three-dimensional blood vessel model; Project the optimized three-dimensional blood vessel model into a series of parallel cutting planes according to the actual heart slice spacing; Calculate the intersection set of the point cloud data and each cutting plane, and after performing topological closure and region filling processing, generate a binary mask to obtain a segmentation result with the prior of the three-dimensional geometric shape of the blood vessels added; Post-process the preliminary segmentation result output by the nn-UNet model based on the segmentation result with the prior of the three-dimensional geometric shape of the blood vessels added to obtain the final segmentation result of the cross-sectional image of the congenital malformed heart of the target fetus.

2. The automatic image segmentation method according to claim 1, wherein, The step of inputting the cross-sectional image of the congenital malformed heart of the target fetus into the pre-constructed nn-UNet model for automatic recognition and segmentation of blood vessel structures to obtain a preliminary segmentation result includes: Collect a preset number of cross-sectional images of the congenital malformed heart of the fetus and store them in a preset format; Label the preset heart blood vessel structures in each of the stored cross-sectional images of the congenital malformed heart of the fetus and construct a data set; Use the data set to train the pre-constructed nn-UNet model to obtain an nn-UNet model that meets the preset requirements as an image segmentation model; Use the image segmentation model to automatically recognize and segment the blood vessel structures in the cross-sectional image of the congenital malformed heart of the target fetus to obtain a preliminary segmentation result.

3. The automatic image segmentation method according to claim 2, wherein The step of collecting a preset number of cross-sectional images of the congenital malformed heart of the fetus and storing them in a preset format includes: Obtain the cross-sectional image data of a preset number of congenital malformed hearts of the fetus and store them in the PNG format; Convert the original color image of the cross-section of the congenital malformed heart of the fetus into a grayscale image; Convert the grayscale image of the cross-section of the congenital malformed heart of the fetus from the original PNG format to the commonly used NII.GZ format in medical imaging and store it.

4. The automatic image segmentation method according to claim 2, wherein The step of using the data set to train the pre-constructed nn-UNet model to obtain an nn-UNet model that meets the preset requirements as an image segmentation model includes: Construct an nn-UNet model; The nn-UNet model includes a preprocessing module, a U-net network module, and an inference module; the preprocessing module is used to crop, resample, and normalize the data of the invalid regions in the cross-sectional image of the congenital malformed heart of the fetus; the U-net network module is used to construct a deep neural network for semantic segmentation of the blood vessels in the heart slices; the inference module is used to process the semantic segmentation result by using a sliding window to obtain the segmentation result of the blood vessels in the heart slices; Based on the preset training parameters, use the data set to train the constructed nn-UNet model to obtain an nn-UNet model that meets the preset requirements as an image segmentation model.

5. An automatic image segmentation method according to claim 4, characterized in that, The U-net network module includes an encoder and a decoder; The encoder is used to extract deep features of blood vessels in cardiac slices through multiple convolutional layers and pooling layers, and the activation functions in the convolutional layers adopt the leaky ReLU function and the Instance Norm function; The decoder is used to combine shallow and deep features by using skip connections and output the target segmentation result.

6. The automatic image segmentation method according to claim 1, characterized in that Based on the preliminary segmentation result, three-dimensional structure reconstruction is performed using VTK to obtain an initial point cloud file, including: Based on a set isosurface threshold, isosurface extraction is performed using VTK Flying Edges3D to obtain the structural boundary of the blood vessels; Enable ComputeScalarsOn to increase the point cloud density to obtain the initial point cloud file.

7. The automatic image segmentation method according to claim 1, characterized in that The operations of gap filling and smoothing on the initial point cloud file include: Use vtkImageContinuousDilate3D to dilate the initial point cloud file to expand the blood vessel area; Use vtkImageContinuousErode3D for erosion to restore the original shape of the blood vessels; Apply vtkImageGaussianSmooth for Gaussian smoothing.

8. An automatic image segmentation system, characterized in that, Including: A preliminary segmentation module for inputting a cross-sectional image of a target fetal congenital malformation heart into a pre-constructed nn-UNet model for automatic recognition and segmentation of blood vessel structures to obtain a preliminary segmentation result; A three-dimensional reconstruction module for performing three-dimensional structure reconstruction using VTK based on the preliminary segmentation result to obtain an initial point cloud file; An optimization module for performing gap filling and smoothing operations on the initial point cloud file to obtain an optimized three-dimensional blood vessel model; A projection module for projecting the optimized three-dimensional blood vessel model into a series of parallel cutting planes according to the actual cardiac slice spacing; A mask module for calculating the intersection set of the point cloud data and each cutting plane, and generating a binary mask after topological closure and region filling processing to obtain a segmentation result with the prior of the three-dimensional geometric shape of the blood vessels; A post-processing module for post-processing the preliminary segmentation result output by the nn-UNet model based on the segmentation result with the prior of the three-dimensional geometric shape of the blood vessels to obtain the final segmentation result of the cross-sectional image of the target fetal congenital malformation heart to be segmented.

9. A computer-readable storage medium storing one or more programs, characterized in that, The one or more programs include instructions that, when executed by a computing device, cause the computing device to execute any of the methods described in claims 1 to 7.

10. A computing device, characterized in that, Including: One or more processors and a memory, where the memory stores one or more programs and is configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the methods described in claims 1 to 7.

Images