Image processing system and image processing method
By removing calcified areas in the image processing system and using image completion and segmentation models, the anatomical structure covered by calcified parts in medical image data is solved, and the problem of inaccurate segmentation results in the prior art is achieved, and high accuracy is achieved anatomical segmentation.
Patent Information
- Application Number
- CN202310999877.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-09
- Publication Date
- 2025-02-18
AI Technical Summary
The prior art is difficult to accurately segment the anatomical structure covered by calcified parts in medical image data, resulting in inaccurate segmentation results and ineffective analysis and measurement.
An image processing system is used to remove calcified areas and use an image completion model to complete the calcified image data. Then, the target anatomy structure is segmented from the completed image data using an image segmentation model.
Accurate segmentation of the anatomical structure covered by calcified parts in medical image data is achieved, the accuracy and reliability of the segmentation results are improved, and the anatomical structure can be more accurately analyzed and measured.
Smart Images
Figure CN119477783A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an image processing system and an image processing method capable of segmenting anatomical structures covered by occlusion patterns in medical image data. Background Art
[0002] For the segmentation of anatomical structures in medical image data, the prior art includes segmentation methods using deep learning networks, methods using structure fitting or geometric reconstruction, and the like.
[0003] However, as human body functions change, many anatomical structures may become calcified. When the anatomical structures in medical image data are covered by occlusion patterns such as calcified parts, it is difficult to present the true anatomical structure information on the medical image, and it is difficult to perform effective segmentation using the above-mentioned methods in the existing technology.
[0004] The following is an example of segmenting the aortic valve contained in medical image data. Figure 22 In FIG, the left figure shows an image containing a normal aortic valve, and the right figure shows an image in which the normal aortic valve is segmented and represented by a mask (the grid portion in the figure). Figure 22 As shown, the normal aortic valve can be accurately segmented using existing segmentation methods.
[0005] like Figure 23 As shown in the left figure, when the aortic valve is calcified, the aortic valve is covered with calcium, and its complete anatomical structure is difficult to observe. When the aortic valve is segmented using the existing segmentation method, Figure 23 As shown in the right image, the aortic valve structure cannot be accurately segmented. The reason is as follows.
[0006] Mainstream segmentation technologies or models target normal structures, but calcification data will have bright calcium areas, which are significantly inconsistent with the image intensity distribution of normal data. Therefore, current mainstream technologies or models will have poor results when segmenting calcification data.
[0007] Therefore, although the existing segmentation methods can segment the calcified aortic valve, the structure of the segmented aortic valve is inaccurate, and it is impossible to accurately analyze and measure it, and it is also impossible to accurately analyze its calcification degree.
[0008] To address this issue, deep learning methods using data from calcified aortic valves have been considered. However, deep learning is data-driven and requires a large amount of labeled data. Data samples from calcified aortic valves are scarce and difficult to obtain in large quantities. Adding a small amount of calcified aortic valve data to the training set will not significantly improve the accuracy of the training model and may even negatively impact the accuracy of normal data.
[0009] Moreover, the labeling of data on calcified aortic valves is labor-intensive, difficult, and labor-intensive, and the accuracy and consistency of the labels cannot be guaranteed.
[0010] Therefore, conventionally, there is no technology for segmenting the aortic valve using a model using data of a calcified aortic valve.
[0011] In addition, techniques that segment the calcified aortic valve region and related structures as a whole are also considered. Figure 24 As shown in the figure, the irregular edges of the calcified aortic valve region in the left image also lead to irregular edges in the segmentation result in the right image. This makes it difficult for doctors to accurately understand the original structure of the aortic valve. The irregularity of calcification also leads to inconsistent segmentation results, making subsequent clinical analysis difficult. Summary of the Invention
[0012] Therefore, in view of the above situation, the present invention provides an image processing system and an image processing method that can accurately segment anatomical structures covered by occlusion patterns in medical image data.
[0013] The image processing system of the present invention processes medical image data with occlusion patterns, and includes: an occlusion pattern area removal unit, which removes the occlusion pattern area in the above-mentioned medical image data to generate non-occlusion pattern data; an image data completion and segmentation unit, which uses normal medical image data and / or erased image data obtained by erasing the normal medical image data through a random mask to train an image completion model and an image segmentation model, or a target anatomical structure acquisition model, and uses the above-mentioned image completion model to complete the image information of the above-mentioned non-occlusion pattern data to generate completed medical image data, and then uses the above-mentioned image segmentation model to segment the target anatomical structure corresponding to the above-mentioned occlusion pattern from the above-mentioned completed medical image data, or, based on the above-mentioned target anatomical structure acquisition model, directly obtains the target anatomical structure corresponding to the above-mentioned occlusion pattern from the above-mentioned non-occlusion pattern data; and a display unit, which displays the above-mentioned target anatomical structure.
[0014] The apparatus may further include a completion determination unit configured to determine whether to perform image information completion using the image completion model based on a feature of the unobstructed pattern data.
[0015] It can also be configured to further include: a completed image evaluation unit that evaluates whether the target anatomical structure is obtained by segmentation by the above-mentioned image segmentation model based on the characteristics of the above-mentioned completed medical image data.
[0016] It may also be configured such that a user evaluates the supplemented medical image data to determine whether to obtain the target anatomical structure by segmentation using the image segmentation model.
[0017] It can also be constructed to include: an occlusion pattern area determination unit, which determines the area of interest in the occlusion pattern area based on the anatomical information of the abnormal part of the medical image data with the occlusion pattern, and the image data completion and segmentation unit selects a method for obtaining the target anatomical structure based on the anatomical information of the area of interest.
[0018] It can also be constructed so that the above-mentioned image data completion and segmentation unit generates the above-mentioned completed medical image data through a deep learning network. In the deep learning network, the above-mentioned erased image data is used as input, the above-mentioned normal medical image data is used as the true value to train the above-mentioned image completion model, and the above-mentioned unobstructed pattern data is input into the above-mentioned image completion model to generate the above-mentioned completed medical image data.
[0019] The above-mentioned deep learning network can also be configured as a generative adversarial network.
[0020] It can also be constructed such that, in the above-mentioned image data completion and segmentation unit, when training the above-mentioned target anatomical structure acquisition model, the above-mentioned erased image data is used as input, and the above-mentioned normal medical image data is used as the true value to learn relevant low-dimensional and high-dimensional completion features through a self-supervised learning network, and the above-mentioned learned relevant low-dimensional and high-dimensional completion features are applied to the deep learning network for obtaining the above-mentioned target anatomical structure through the sharing operation of the encoder's weights.
[0021] The medical image data may be two-dimensional or three-dimensional data of various modalities.
[0022] The shielding pattern may be a calcified anatomical structure.
[0023] The occlusion pattern region may be a region where image noise exists or a region where a treatment device is implanted, and the occlusion pattern region removing unit may remove the occlusion pattern region selected by the user to generate non-occlusion pattern data.
[0024] The image processing method of the present invention processes medical image data with occlusion patterns, including: an occlusion pattern area removal step, removing the occlusion pattern area in the above-mentioned medical image data to generate non-occlusion pattern data; an image data completion and segmentation step, using normal medical image data and / or erased image data obtained by erasing the normal medical image data through a random mask to train an image completion model and an image segmentation model, or a target anatomical structure acquisition model, using the above-mentioned image completion model to complete the image information of the above-mentioned non-occlusion pattern data to generate completed medical image data, and then using the above-mentioned image segmentation model to segment the target anatomical structure corresponding to the above-mentioned occlusion pattern from the above-mentioned completed medical image data, or, based on the above-mentioned target anatomical structure acquisition model, directly obtaining the target anatomical structure corresponding to the above-mentioned occlusion pattern from the above-mentioned non-occlusion pattern data; and a display step, displaying the above-mentioned target anatomical structure.
[0025] It can also be constructed to further include: a completion decision step, which determines whether to use the above-mentioned image completion model to complete the image information based on the characteristics of the above-mentioned unobstructed pattern data.
[0026] It can also be constructed to include: a completed image evaluation step, which evaluates whether the target anatomical structure is obtained by segmentation by the above-mentioned image segmentation model based on the characteristics of the above-mentioned completed medical image data.
[0027] It may also be configured such that a user evaluates the supplemented medical image data to determine whether to obtain the target anatomical structure by segmentation using the image segmentation model.
[0028] It can also be constructed to include: an occlusion pattern area determination step, determining the area of interest in the occlusion pattern area based on the anatomical information of the abnormal part of the medical image data with the occlusion pattern, and in the above-mentioned image data completion and segmentation step, selecting a method for obtaining the above-mentioned target anatomical structure based on the anatomical information of the above-mentioned area of interest.
[0029] It can also be constructed so that in the above-mentioned image data completion and segmentation step, the above-mentioned completed medical image data is generated by a deep learning network, in which the above-mentioned erased image data is used as input, the above-mentioned normal medical image data is used as the true value to train the above-mentioned image completion model, and the above-mentioned unobstructed pattern data is input into the above-mentioned image completion model to generate the above-mentioned completed medical image data.
[0030] The above-mentioned deep learning network can also be configured as a generative adversarial network.
[0031] It can also be constructed such that, in the above-mentioned image data completion and segmentation step, when training the above-mentioned target anatomical structure acquisition model, the above-mentioned erased image data is used as input, and the above-mentioned normal medical image data is used as the true value to learn relevant low-dimensional and high-dimensional completion features through a self-supervised learning network, and the above-mentioned learned relevant low-dimensional and high-dimensional completion features are applied to the deep learning network for obtaining the above-mentioned target anatomical structure through the sharing operation of the encoder's weights.
[0032] The medical image data may be two-dimensional or three-dimensional data of various modalities.
[0033] The shielding pattern may be a calcified anatomical structure.
[0034] The occlusion pattern region may be a region where image noise exists or a region where a therapeutic device is implanted, and in the occlusion pattern region removing step, the occlusion pattern region selected by the user is removed to generate non-occlusion pattern data.
[0035] Effects of the Invention
[0036] According to the image processing system and the image processing method of the present invention, the anatomical structure covered by the occlusion pattern in the medical image data can be accurately segmented. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 This is a block diagram of the structure of the image processing system according to the first embodiment.
[0038] Figure 2 This is a flowchart showing the training and inference of the calcified region extraction model.
[0039] Figure 3 FIG. 1 is a diagram showing segmentation of a calcified area from valve image data having the calcified area.
[0040] Figure 4 is a flowchart showing the training and inference of the image completion model.
[0041] Figure 5 FIG. 1 is a diagram showing the completion of valve image data from which the calcified area has been removed to form a complete valve.
[0042] Figure 6 is a flowchart showing the training and inference of an image segmentation model.
[0043] Figure 7 The diagram shows valve image data after the calcified region is completed using the image completion model, and a valve mask output for the completed valve image data.
[0044] Figure 8These are Dice distribution maps of the valve segmented using the conventional segmentation method and the segmentation method using the image processing system according to the first embodiment.
[0045] Figure 9 1 and 2 are Hausdorff distance distribution maps of the valve segmented using the conventional segmentation method and the segmentation method using the image processing system according to the first embodiment.
[0046] Figure 10 This is a diagram comparing the segmentation effect when the calcified valve data is directly segmented for a moderately calcified valve and the segmentation effect of the image processing system according to the first embodiment.
[0047] Figure 11 This is a diagram comparing the segmentation effect when the calcified valve data is directly segmented for a severely calcified valve and the segmentation effect of the image processing system according to the first embodiment.
[0048] Figure 12 This is a flowchart of the image processing method according to the first embodiment.
[0049] Figure 13 This is a block diagram of the structure of the image processing system according to the second embodiment.
[0050] Figure 14 3 is a diagram illustrating the training process of the target anatomical structure acquisition model.
[0051] Figure 15 This is a flowchart of the image processing method according to the second embodiment.
[0052] Figure 16 This is a block diagram of the structure of the image processing system according to the third embodiment.
[0053] Figure 17 This is a flowchart of the image processing method according to the third embodiment.
[0054] Figure 18 This is a block diagram of the structure of the image processing system according to the fourth embodiment.
[0055] Figure 19 This is a flowchart of the image processing method according to the fourth embodiment.
[0056] Figure 20 This is a block diagram of the structure of the image processing system according to the fifth embodiment.
[0057] Figure 21 This is a flowchart of the image processing method according to the fifth embodiment.
[0058] Figure 22 A diagram showing an example of segmenting the aortic valve included in medical image data.
[0059] Figure 23 FIG. 1 is a diagram showing an example of segmenting a calcified aortic valve using a conventional segmentation method.
[0060] Figure 24 FIG. 1 is a diagram showing an example of segmenting the calcified aortic valve region and related structures as a whole. DETAILED DESCRIPTION
[0061] Hereinafter, the image processing system and the image processing method according to the present invention will be described with reference to the accompanying drawings.
[0062] The image processing system of the present invention is composed of multiple functional modules and can be installed as software in a standalone computer or other device with a CPU (central process unit) and memory, or distributed across multiple devices, with a processor executing the various functional modules of the image processing system stored in memory. Alternatively, the system can be implemented in hardware as a circuit capable of performing the various functions of the image processing system. The circuitry implementing the image processing system can transmit, receive, or collect data via a network such as the Internet.
[0063] Furthermore, the image processing system of the present invention can be installed at the site of medical image acquisition to process the images on-site. Furthermore, the image processing system can be directly installed in a medical image acquisition device such as a CT (Computed Tomography) device or an MRI (Magnetic Resonance Imaging) device as part of the device.
[0064] In the first to fourth embodiments, an example of segmenting the aortic valve (hereinafter sometimes referred to as the valve) by image processing is described using image data of a calcified aortic valve. However, the present invention is not limited to this example and can also be applied to image processing of image data of other calcified anatomical structures or to image processing of medical image data containing occlusion pattern areas.
[0065] Hereinafter, various embodiments of the present invention will be described in detail.
[0066] First embodiment
[0067] Reference Figures 1 to 12 A first embodiment of the present invention will be described.
[0068] Figure 1 This is a block diagram of the structure of the image processing system according to the first embodiment.
[0069] like Figure 1As shown, the image processing system 1 includes a calcified anatomical structure region removing unit 11 , an image data complementing and segmenting unit 12 , and a display unit 13 .
[0070] The calcified anatomical structure region removing unit 11 removes the calcified anatomical structure region from the input medical image data having the calcified anatomical structure region (occlusion pattern region) to generate calcification-free anatomical structure data.
[0071] The medical image data may be data of various modalities such as CT data and MRI data, and the medical image data may be either two-dimensional image data or three-dimensional image data.
[0072] A calcified anatomical region is a region where the anatomical structure is calcified.
[0073] The calcified anatomical structure region removing unit 11 includes a calcified region extraction model 111 and a removing unit 112 .
[0074] The calcification region extraction model 111 is a model for extracting calcification regions on anatomical structures. The calcification region extraction model 111 can be implemented using AI (Artificial Intelligence) technologies such as deep learning, or using a traditional threshold segmentation algorithm.
[0075] The following describes an example of using deep learning, such as a Unet segmentation network, to implement the calcified region extraction model 111 .
[0076] like Figure 2 As shown, in the training stage, in step S1, the input medical image data containing calcification areas is pre-segmented, and the pre-segmentation can be performed using various existing segmentation methods. In step S2, the labels of the calcification areas pre-segmented in step S1 are fine-tuned to prepare the ground truth (GT). The task of labeling the calcification areas to obtain the true value is low in difficulty and labor cost. Moreover, in steps S1 and S2, the pre-segmented calcification mask can be obtained using traditional algorithms to improve the doctor's labeling efficiency when obtaining the true value. Then, in step S3, the segmentation model is trained. In step S4, the trained model is evaluated and prepared for the AI model.
[0077] In the inference phase, in step S5, new calcification data is prepared, i.e., medical image data with calcified anatomical regions is input. Then, in step S6, the calcification region extraction model 111 trained in the training phase is used to segment (extract) the calcified regions from the input medical image data.
[0078] Figure 3The diagram shows how the calcified region (hatched region in the right figure) is segmented from valve image data having the calcified region (left figure).
[0079] The removal unit 112 is used to remove the calcified area extracted by the calcified area extraction model 111, for example, Figure 5 As shown in the left figure of , the calcified area extracted is removed by setting the brightness value of the area to 0 and forming a black area.
[0080] The image data complementation and segmentation unit 12 includes an image complementation model 121 and an image segmentation model 122 .
[0081] The image completion model 121 is a model used to complete the non-calcified anatomical structure data after the calcification area is removed to generate completed medical image data. It is a model obtained by training the erased image data obtained by erasing normal medical image data using a random mask.
[0082] Normal medical image data refers to medical image data without occlusion pattern areas such as calcified anatomical structure areas.
[0083] The completion method of the image completion model 121 can be any image completion method, but based on the current research results in the completion field, it is preferred to use a completion method using a deep learning network.
[0084] When the image completion model 121 uses the completion method using a deep learning network for completion, in the training stage, the erased image data obtained by erasing the normal medical image data through a random mask is used as input, and the normal medical image data is used as the true value for training. In the inference stage, the calcification-free anatomical structure data after removing the calcified area is input into the image completion model 121 to generate completed medical image data that completes the removed calcified area.
[0085] Since the calcified areas on the anatomical structure are irregular and of varying sizes, and there are also cases of severe calcification, that is, there is a problem that the calcified areas to be removed are large, we consider using a completion method based on a conditional generative adversarial network (GAN) to train the image completion model 121. Figure 4 To illustrate the case of using the conditional GAN-based completion method to train the image completion model 121.
[0086] During the training stage, in step S11, medical image data with normal valve structure is selected as the model training set. In step S12, at least a portion of the valve area in the medical image data is erased using a random mask. Then, in step S13, the medical image data with at least a portion of the valve area erased in step S12 is used as the input of the training set, and the medical image data with normal valve structure selected in step S11 is used as the true value. In step S14, model training for the completion of the erased area is performed. Finally, in step S15, the trained model is evaluated for accuracy.
[0087] During the training phase, a large amount of existing medical image data with normal valve structure is used to train the image completion model. A random mask is generated by a preset random seed, and the valve area is randomly erased to generate multiple randomly erased image data, thereby improving the accuracy of completion.
[0088] In the inference stage, in step S16, medical image data (in this embodiment, image data with a valve (also referred to as valve image data)) with the calcified area removed by the calcified anatomical structure area removal unit 11 is obtained, and then, in step S17, the removed calcified area is completed using the image completion model 121 trained in the training stage to restore the complete valve structure in the image data.
[0089] Figure 5 The figure shows how valve image data with calcified areas removed (left figure) is complemented to form a complete valve (right figure).
[0090] The image segmentation model 122 is a model for segmenting medical image data with complete anatomical structures and is a model trained using normal medical image data. In the present invention, the image segmentation model 122 is used to segment the medical image data completed by the image completion model 121.
[0091] In this embodiment, the Unet segmentation network is used to train the image segmentation model 122, but other deep learning networks can also be used for training.
[0092] like Figure 6 As shown, in the training stage, in step S21, valve annotation is performed on normal medical image data, and then, in step S22, a model for segmenting valve structure using AI technology is trained, and finally, in step S23, the trained model is evaluated for accuracy and prepared.
[0093] In the inference stage, in step S24, the valve image data after the calcification area is completed by the image completion model 121 is obtained. Then, in step S25, the completed valve image data obtained in step S24 is segmented using the image segmentation model 122 trained in the training stage to segment the valve structure. In step S26, a mask is output for the segmented valve.
[0094] Figure 7 The diagram shows valve image data after the calcified region is completed by the image completion model 121 (left figure) and a valve mask output for the completed valve image data in step S26 of the inference phase (right figure).
[0095] The display unit 13 displays the target anatomical structure acquired by the image data complementation and segmentation unit 12. Specifically, in this embodiment, the display unit 13 displays a mask of the valve segmented by the image segmentation model 122.
[0096] Next, use Figures 8 to 11 The effect of valve segmentation generated by the image processing system 1 will be described.
[0097] Using the segmentation model trained based on normal medical image data, the calcified valve data and the calcified valve data after the image processing system 1 of this embodiment are segmented directly, and then compared with the true value of the aortic valve. The performance of the segmentation results is as follows: Figure 8 as well as Figure 9 The test set consists of 11 sets of two-dimensional cross-sectional images of the center position of typical calcified valves.
[0098] Figure 8 is the Dice distribution diagram of the test set, Figure 9 is the Hausdorff distance distribution of the test set.
[0099] exist Figure 8 In the figure, the dotted line shows the segmentation effect of the image processing system 1 of the present invention, and the solid line shows the segmentation effect when the calcified valve data is directly segmented. In the Dice distribution diagram, the closer the Dice value is to 1, the better the segmentation effect. Figure 8 It can be seen that the segmentation effect of the image processing system 1 of the present invention is better.
[0100] exist Figure 9 In FIG, the dot-dashed line shows the segmentation effect of the image processing system 1 of the present invention, and the solid line shows the segmentation effect when the calcified valve data is directly segmented. In the Hausdorff distance distribution diagram, the closer the value is to 0, the better the segmentation effect. Figure 9 It can also be seen that the segmentation effect of the image processing system 1 of the present invention is better.
[0101] Figure 10 This is a diagram comparing the segmentation effect when the calcified valve data is directly segmented for a moderately calcified valve and the segmentation effect of the image processing system 1 of the present invention. Figure 11 This is a diagram comparing the segmentation effect of a severely calcified valve when the calcified valve data is directly segmented and the segmentation effect of the image processing system 1 of the present invention.
[0102] exist Figure 10 In the figure, the upper left figure is the image data of the moderately calcified valve, the lower left figure is a figure showing the true value of the segmentation mask of the moderately calcified valve, the upper middle figure is the same image data as the upper left figure, the lower middle figure is a figure showing the valve mask obtained when the calcified valve data is directly segmented using a segmentation model trained based on normal medical image data, the upper right figure is a figure after the moderately calcified valve image data in the upper left figure is supplemented by the image processing system 1, and the lower right figure is a figure showing the mask output after the supplemented figure is segmented by the image processing system 1. The Dice value of the valve segmentation in the lower middle figure is 0.19, and the Dice value of the valve segmentation in the lower right figure is 0.81, that is, the segmentation effect of the image processing system 1 of the present invention is better. Of course, from Figure 10 It can also be clearly seen that the segmentation mask of the lower right figure is closer to the true value of the segmentation mask shown in the lower left figure.
[0103] Therefore, according to Figure 10 It can be seen that the image processing system 1 of the present invention can produce a good segmentation optimization effect on moderate calcification data.
[0104] exist Figure 11 In the figure, the upper left part is the image data of a severely calcified valve, the lower left part is a figure showing the true value of the segmentation mask of the severely calcified valve, the upper middle part is the same image data as the upper left part, the lower middle part is a figure showing the valve mask obtained when the calcified valve data is directly segmented using a segmentation model trained based on normal medical image data, the upper right part is a figure after the image data of the severely calcified valve in the upper left part is supplemented by the image processing system 1, and the lower right part is a figure showing the mask output after the supplemented image is segmented by the image processing system 1. The Dice value of the valve segmentation in the lower middle part is 0.24, and the Dice value of the valve segmentation in the lower right part is 0.79, that is, the segmentation effect of the image processing system 1 of the present invention is better. Of course, from Figure 11 It can also be clearly seen that the segmentation mask of the lower right figure is closer to the true value of the segmentation mask shown in the lower left figure.
[0105] So far, it is very difficult to segment the data of severe calcification using existing segmentation methods. Figure 11 It can be seen that the image processing system 1 of the present invention can also produce better segmentation optimization effects on data with severe calcification.
[0106] Below, refer to Figure 12 The image processing method of this embodiment will be described.
[0107] In step S101 , for input medical image data having a calcified anatomical structure region (occlusion pattern region), ie, valve image data, the calcified anatomical structure region is removed to generate calcification-free anatomical structure data.
[0108] In step S102, normal medical image data is used to train an image completion model and an image segmentation model, and the image completion model is used to complete the image information of the non-calcified anatomical structure data to generate completed medical image data. Thereafter, the image segmentation model is used to segment the target anatomical structure corresponding to the calcified anatomical structure, i.e., the valve structure, from the completed medical image data.
[0109] In step S103 , the valve structure segmented in step S102 is displayed.
[0110] To address the problem of small sample size of calcification data in training the segmentation model, in this embodiment, calcification data does not need to be used for training during the processing of the image processing system 1 , and a better segmentation effect of the calcification data can be obtained.
[0111] In this embodiment, by using a large amount of normal medical image data for image completion training, the target anatomical structure features can be effectively learned, so that the target anatomical structure covered by the calcified area can be completed. The completed data is then segmented using a conventional segmentation model to obtain excellent segmentation accuracy.
[0112] Moreover, in this embodiment, the segmentation results obtained by the image processing system 1 can be used as pre-segmentation results for doctors to refer to when annotating calcification data. Doctors do not need to annotate calcification data from scratch, which can improve annotation efficiency.
[0113] Moreover, in this embodiment, the calcified area and the target anatomical structure can be separated, which can solve the problem that the edges of the segmentation results are irregular when the calcified area and the target anatomical structure are segmented as a whole, making subsequent clinical analysis difficult. It can avoid the interference of the calcified area and more accurately obtain the structural characteristics of the target anatomical structure.
[0114] Second embodiment
[0115] Below, refer to Figures 13-15 The second embodiment will be described.
[0116] The image processing system 1a of the second embodiment differs from the image processing system 1 of the first embodiment in the structure of the image data complementation and segmentation unit. The other structures are the same. Below, only the different parts are described, and the same parts are given the same reference numerals as in the first embodiment, and repeated descriptions are omitted.
[0117] like Figure 13 As shown, the image processing system 1 a includes a calcified anatomical structure region removing unit 11 , an image data complementing and segmenting unit 12 a , and a display unit 13 .
[0118] The image data complementing and segmenting unit 12 a includes a target anatomical structure acquisition model 123 .
[0119] The target anatomical structure acquisition model 123 is a model that directly acquires the target anatomical structure corresponding to the calcified anatomical structure from non-calcified anatomical structure (non-occlusion pattern) data, and is a model obtained by training erased image data obtained by erasing normal medical image data using a random mask.
[0120] Figure 14 The training process of the target anatomical structure acquisition model 123 is shown. During training, erased image data obtained by erasing normal medical image data using a random mask is input into the encoder of a self-supervised learning network (SSLNet). The normal medical image data is used as the true value, and the decoder of the self-supervised learning network outputs a completed image that completes the erased portion. Furthermore, during training, erased image data obtained by erasing normal medical image data using a random mask is input into the encoder of a segmentation network. The normal medical image data is used as the true value, and the decoder of the segmentation network outputs a segmentation mask of the target anatomical structure corresponding to the portion erased by the random mask. The segmentation network here can be any deep learning segmentation network.
[0121] Moreover, during the training process, relevant low-dimensional and high-dimensional completion features are learned through a self-supervised learning network in the completion based on the self-supervised learning network, and the learned relevant low-dimensional and high-dimensional completion features are applied to the segmentation network for obtaining the target anatomical structure through the sharing operation of the encoder weights.
[0122] In this way, when the target anatomical structure acquisition model 123 is trained using image data of a normal aortic valve, the structure of the aortic valve (segmentation mask of the aortic valve) that completes at least a portion of the erased aortic valve can be output by inputting image data in which at least a portion of the aortic valve is erased.
[0123] In the inference phase, when the valve data of the anatomical structure without calcification after the calcification area is input to the target anatomical structure acquisition model 123 , the structure of the valve can be output (directly acquired).
[0124] Below, refer to Figure 15 The image processing method according to the second embodiment will be described.
[0125] In step S201 , for input medical image data having a calcified anatomical structure region (occlusion pattern region), ie, valve image data, the calcified anatomical structure region is removed to generate calcification-free anatomical structure data.
[0126] In step S202, a target anatomical structure acquisition model is trained using randomly erased normal medical image data, and based on the target anatomical structure acquisition model, the target anatomical structure corresponding to the calcified anatomical structure, i.e., the valve structure, is directly acquired from the non-calcified anatomical structure data generated in step S201.
[0127] In step S203 , the valve structure segmented in step S202 is displayed.
[0128] In this embodiment, during the training phase of the target anatomical structure acquisition model 123, the completion features learned by the self-supervised learning network are shared with the segmentation network in the form of encoder weights, so that the segmentation network includes both completion and segmentation functions. Therefore, during the inference phase, the image data completion and segmentation unit 12a does not require an image completion model, which can simplify the workflow of the inference phase and enable the anatomical structure covered by the calcified area to be directly obtained from the data after the calcified area is removed.
[0129] Third embodiment
[0130] Below, refer to Figures 16-17 A third embodiment will be described.
[0131] like Figure 16 As shown, the image processing system 1b includes a calcified anatomical structure region removing unit 11, an image data complementing and segmenting unit 12, a display unit 13, a complement determining unit 14, and a complemented image evaluating unit 15. The functions of the calcified anatomical structure region removing unit 11, the image data complementing and segmenting unit 12, and the display unit 13 are the same as those of the corresponding parts in the first embodiment, and therefore their description is omitted.
[0132] The complement determination unit 14 determines whether to perform image information complementation using the image complement model 121 in the image data complementation and segmentation unit 12 based on the characteristics of the non-calcified anatomical structure (non-occlusion pattern) data.
[0133] The characteristics of the calcification-free anatomical structure data include the number and distribution of removed calcified areas and other relevant characteristics. When processing image data of a calcified aortic valve, the other relevant characteristics include, for example, the size and shape of the heart chamber.
[0134] For example, a threshold value may be set for the number of calcified regions removed. When the number of calcified regions removed is less than the threshold value, the complement determination unit 14 determines that subsequent image information complementation is possible. When the number of calcified regions removed is greater than the threshold value, the complement determination unit 14 determines that subsequent image information complementation is not possible. In other words, when the characteristics of the anatomical structure data without calcifications (characteristics of the calcified regions removed) meet the specified index conditions, it is determined that subsequent image information complementation is possible. When the characteristics of the anatomical structure data without calcifications do not meet the specified index conditions, it is determined that subsequent image information complementation is not possible.
[0135] This can reduce the risk of failing to complete image information using the image completion model 121 .
[0136] Alternatively, the complement determination unit 14 may not be provided, and the user may determine whether to perform subsequent image information complementation based on the characteristics of the non-calcified anatomical structure data. For example, if the user determines that the number of calcified regions removed from the non-calcified anatomical structure data is small and their distribution is concentrated, the image complementation model 121 may be used to determine that image information complementation can be performed. If the user determines that the number of calcified regions removed from the non-calcified anatomical structure data is large and their distribution is dispersed, the image complementation model 121 may be used to determine that image information complementation cannot be performed.
[0137] The supplemented image evaluation unit 15 evaluates whether the target anatomical structure has been segmented using the image segmentation model 122 based on the characteristics (quality) of the supplemented medical image data. If the supplemented medical image data is of acceptable quality, the supplemented image evaluation unit 15 determines to proceed with the subsequent structure segmentation process. If the supplemented medical image data is of unacceptable quality, the supplemented image evaluation unit 15 determines not to proceed with the subsequent structure segmentation process. When processing image data of a calcified aortic valve, for example, the quality of the supplemented medical image data may be the continuity of the valve. If the valve is continuous, the supplemented medical image data is considered to be of acceptable quality. If the valve is discontinuous, the supplemented medical image data is considered to be of unacceptable quality.
[0138] This can reduce the risk of being unable to segment the target anatomical structure using the image segmentation model 122 .
[0139] Alternatively, the supplemented image evaluation unit 15 may not be provided, and the user may evaluate the supplemented medical image data to determine whether to segment the target anatomical structure using the image segmentation model. If the user determines that the supplemented medical image data is of acceptable quality, the user determines that the target anatomical structure is to be segmented using the image segmentation model. If the user determines that the supplemented medical image data is of unacceptable quality, the user determines that the process of segmenting the target anatomical structure using the image segmentation model is not to be performed.
[0140] Below, refer to Figure 17 The image processing method according to the third embodiment will be described.
[0141] In step S301 , for input medical image data having a calcified anatomical structure region (occlusion pattern region), ie, valve image data, the calcified anatomical structure region is removed to generate calcification-free anatomical structure data.
[0142] In step S302, it is determined whether the characteristics of the non-calcified anatomical structure data meet the specified index conditions. When the specified index conditions are not met ("No" in S302), the processing of the image processing system 1b is terminated. When the specified index conditions are met ("Yes" in S302), the trained image completion model is used to complete the image information of the non-calcified anatomical structure data to generate completed medical image data (step S303).
[0143] In step S304, it is determined whether the quality of the completed medical image data is qualified. When it is determined to be unqualified ("No" in S304), the processing of the image processing system 1b is terminated. When it is determined to be qualified ("Yes" in S304), the trained image segmentation model is used to segment the valve structure from the completed medical image data (step S305).
[0144] In step S306 , the valve structure segmented in step S305 is displayed.
[0145] In this embodiment, the risk of image data completion and segmentation failure can be reduced, and the target anatomical structure can be reliably segmented.
[0146] Fourth embodiment
[0147] Below, refer to Figures 18-19 A fourth embodiment will be described.
[0148] like Figure 18As shown, the image processing system 1c includes a calcified anatomical structure region removing unit 11, an image data complementing and segmenting unit 12, a display unit 13, a complement determining unit 14, a complemented image evaluating unit 15, and a calcified anatomical structure region identifying unit 16. The functions of the components other than the calcified anatomical structure region identifying unit 16 are the same as those of the corresponding components in the third embodiment, and therefore their description is omitted.
[0149] The calcified anatomical structure region identifying unit 16 identifies a region of interest (ROI) within the calcified anatomical structure region based on anatomical information of an abnormal part in the input medical image data having the calcified anatomical structure region (occlusion pattern region). The image data completion and segmentation unit 12 selects a method for acquiring the target anatomical structure based on the anatomical information of the ROI.
[0150] When processing image data of a calcified aortic valve, the majority of the calcified area of the aortic valve occurs on the aortic arch side, rather than on the left ventricular outflow tract (LVOT) side. In this case, the calcified anatomical structure region determination unit 16 determines a region of interest (ROI) within the calcified anatomical structure region, namely the aortic arch region, based on the anatomical information of the calcified area of the valve. The image data completion and segmentation unit 12 then selects a method for acquiring the valve structure based on the anatomical information (size, morphology, etc.) of the aortic arch region. For example, the image data completion and segmentation unit 12 may select a deep learning method or use a moderate calcification model or a severe calcification model for completion and segmentation.
[0151] Below, refer to Figure 19 An image processing method according to the fourth embodiment will be described.
[0152] Steps S403 to S408 in the image processing method of the fourth embodiment correspond to steps S301 to S306 in the image processing method of the third embodiment, respectively, and therefore description thereof will be omitted.
[0153] In step S401 , a region of interest in a calcified anatomical structure region is determined based on anatomical information of a calcified site in input medical image data. In this embodiment, the aortic arch region in the calcified valve region is determined.
[0154] In step S402, a method for obtaining the target anatomical structure, i.e., the valve structure, is selected based on the anatomical information of the region of interest, i.e., the aortic arch region. For example, when it is determined that the aortic arch region is moderately calcified, the moderate calcification model is selected to perform the subsequent steps S403 to S408.
[0155] In this embodiment, a method and a model for acquiring a target anatomical structure are appropriately selected based on anatomical information of a region of interest, thereby enabling accurate segmentation of the target anatomical structure.
[0156] Fifth embodiment
[0157] Below, refer to Figures 20-21 A fifth embodiment will be described.
[0158] like Figure 20 As shown, the image processing system 1d includes an occlusion pattern region removal unit 11d, an image data complementation and segmentation unit 12, a display unit 13, a complementation determination unit 14, and a complemented image evaluation unit 15. The functions of the components other than the occlusion pattern region removal unit 11d are identical to those of the corresponding components in the third embodiment, and therefore their description is omitted.
[0159] The occlusion pattern region removing unit 11d removes the occlusion pattern region selected by the user and generates non-occlusion pattern data. The occlusion pattern region is, for example, a region where image noise exists or a region where a treatment device is implanted.
[0160] The occlusion pattern region removing unit 11 d includes an occlusion pattern region extraction model 111 d and a removing unit 112 d .
[0161] The occlusion pattern region extraction model 111d is a model for extracting the occlusion pattern region selected by the user. The occlusion pattern region extraction model 111d can be implemented using AI technologies such as deep learning or a traditional threshold segmentation algorithm.
[0162] The removing unit 112 d is used to remove the occlusion pattern region extracted by the occlusion pattern region extraction model 111 d . For example, the removing unit 112 d may remove the occlusion pattern region by setting the brightness value of the extracted occlusion pattern region to 0, thereby forming a black region.
[0163] Below, refer to Figure 21 The image processing method according to the fifth embodiment will be described.
[0164] In step S501, the user selects an occlusion pattern area such as an area where a treatment device is implanted in the medical image data, and removes the occlusion pattern area to generate medical image data without an occlusion pattern.
[0165] In step S502, it is determined whether the characteristics of the data of the unobstructed pattern meet the specified index conditions. When the specified index conditions are not met ("No" in S502), the processing of the image processing system 1d is terminated. When the specified index conditions are met ("Yes" in S502), the trained image completion model is used to complete the image information of the data of the unobstructed pattern to generate completed medical image data (step S503).
[0166] In step S504, it is determined whether the quality of the completed medical image data is qualified. When it is determined to be unqualified ("No" in S504), the processing of the image processing system 1d is terminated. When it is determined to be qualified ("Yes" in S504), the trained image segmentation model is used to segment the target anatomical structure that is partially or completely obscured by the occlusion pattern from the completed medical image data (step S505).
[0167] In step S506 , the target anatomical structure segmented in step S505 is displayed.
[0168] In this embodiment, anatomical structures obscured by an occlusion pattern region, such as a region with image noise or a region where a therapeutic device is implanted, can be reliably segmented.
[0169] The components of the image processing systems described in the above embodiments are conceptual and functional, and need not necessarily be physically configured as shown. That is, the specific forms of distribution and integration of each system are not limited to those shown; all or part of them can be functionally or physically distributed and integrated in arbitrary units, corresponding to various loads and usage conditions. Furthermore, all or any part of the processing functions performed in each system can be implemented by a CPU and programs parsed and executed by the CPU, or as hardware using wired logic.
[0170] Furthermore, the image processing system described in each of the above embodiments can be implemented by executing a pre-prepared program on a computer such as a personal computer or workstation. This program can be distributed via a network such as the Internet. Furthermore, this program can be recorded on a computer-readable non-volatile recording medium such as a hard disk, floppy disk (FD), CD-ROM, MO, or DVD, and executed by a computer by reading from the recording medium.
[0171] As described above, although various embodiments of the present invention have been described, these embodiments are shown as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other ways, and various omissions, substitutions, and modifications can be made without departing from the scope of the invention. These embodiments and their variations are included in the scope and spirit of the invention and are included in the invention described in the technical solution and its equivalent.
Claims
1. An image processing system for processing medical image data having an occlusion pattern, characterized in that: include: An occlusion pattern region removing unit, which removes the occlusion pattern region in the medical image data to generate non-occlusion pattern data; An image data completion and segmentation unit uses normal medical image data and / or erased image data obtained by erasing the normal medical image data through a random mask to train an image completion model and an image segmentation model, or a target anatomical structure acquisition model, uses the above-mentioned image completion model to complete the image information of the above-mentioned unobstructed pattern data to generate completed medical image data, and then uses the above-mentioned image segmentation model to segment the target anatomical structure corresponding to the above-mentioned occlusion pattern from the above-mentioned completed medical image data, or directly obtains the target anatomical structure corresponding to the above-mentioned occlusion pattern from the above-mentioned unobstructed pattern data based on the above-mentioned target anatomical structure acquisition model; as well as The display unit displays the target anatomical structure.
2. The image processing system according to claim 1, wherein: Also includes: The completion determination unit determines whether to perform image information completion using the image completion model according to the characteristics of the unoccluded pattern data.
3. The image processing system according to claim 1, wherein: Also includes: The completed image evaluation unit evaluates whether the target anatomical structure is obtained by segmentation by the above-mentioned image segmentation model based on the characteristics of the above-mentioned completed medical image data.
4. The image processing system according to claim 1, wherein: The user evaluates the above-mentioned completed medical image data to determine whether to obtain the target anatomical structure by segmentation using the above-mentioned image segmentation model.
5. The image processing system according to claim 1, wherein: Also includes: The occlusion pattern region determining unit determines a region of interest in the occlusion pattern region according to anatomical information of the abnormal part of the medical image data having the occlusion pattern. The image data completion and segmentation unit selects a method for obtaining the target anatomical structure according to the anatomical information of the region of interest.
6. The image processing system according to claim 1, wherein: The image data completion and segmentation unit generates the completed medical image data through a deep learning network. In the deep learning network, the erased image data is used as input, and the normal medical image data is used as the true value to train the image completion model. The above-mentioned unobstructed pattern data is input into the above-mentioned image completion model to generate the above-mentioned completed medical image data.
7. The image processing system according to claim 6, characterized in that: The above deep learning network is a generative adversarial network.
8. The image processing system according to claim 1, wherein: In the above-mentioned image data completion and segmentation unit, when training the above-mentioned target anatomical structure acquisition model, the above-mentioned erased image data is taken as input, and the above-mentioned normal medical image data is taken as the true value to learn the relevant low-dimensional and high-dimensional completion features through a self-supervised learning network, and the above-mentioned learned relevant low-dimensional and high-dimensional completion features are applied to the deep learning network used to obtain the above-mentioned target anatomical structure through the sharing operation of the encoder's weights.
9. The image processing system according to claim 1, wherein: The above-mentioned medical image data are two-dimensional or three-dimensional data of various modalities.
10. The image processing system according to any one of claims 1 to 9, characterized in that: The above-mentioned occlusion pattern is a calcified anatomical structure.
11. The image processing system according to claim 1, wherein: The above-mentioned occlusion pattern area is an area where image noise exists or an area where a treatment device is implanted. The occlusion pattern region removing unit removes the occlusion pattern region selected by a user to generate non-occlusion pattern data.
12. An image processing method for processing medical image data having an occlusion pattern, characterized in that: include: An occluded pattern area removal step, removing the occluded pattern area in the medical image data to generate non-occluded pattern data; an image data completion and segmentation step, using normal medical image data and / or erased image data obtained by erasing the normal medical image data through a random mask to train an image completion model and an image segmentation model, or a target anatomical structure acquisition model, using the image completion model to complete the image information of the unobstructed pattern data to generate completed medical image data, and then using the image segmentation model to segment the target anatomical structure corresponding to the occluded pattern from the completed medical image data, or directly acquiring the target anatomical structure corresponding to the occluded pattern from the unobstructed pattern data based on the target anatomical structure acquisition model; as well as In a displaying step, the target anatomical structure is displayed.
13. The image processing method according to claim 12, characterized in that: Also includes: The completion decision step determines whether to use the image completion model to complete the image information according to the characteristics of the unobstructed pattern data.
14. The image processing method according to claim 12, wherein: Also includes: The completed image evaluation step evaluates whether the target anatomical structure is obtained by segmentation by the above-mentioned image segmentation model based on the characteristics of the above-mentioned completed medical image data.
15. The image processing method according to claim 12, characterized in that: The user evaluates the above-mentioned completed medical image data to determine whether to obtain the target anatomical structure by segmentation using the above-mentioned image segmentation model.
16. The image processing method according to claim 12, characterized in that: Also includes: The occlusion pattern region determining step is to determine the region of interest in the occlusion pattern region according to the anatomical information of the abnormal part of the medical image data having the occlusion pattern. In the image data completion and segmentation step, a method for obtaining the target anatomical structure is selected based on the anatomical information of the region of interest.
17. The image processing method according to claim 12, characterized in that: In the above image data completion and segmentation step, the above completed medical image data is generated by a deep learning network. In the deep learning network, the erased image data is used as input, and the normal medical image data is used as the true value to train the image completion model. The above-mentioned unobstructed pattern data is input into the above-mentioned image completion model to generate the above-mentioned completed medical image data.
18. The image processing method according to claim 17, characterized in that: The above deep learning network is a generative adversarial network.
19. The image processing method according to claim 12, wherein: In the above-mentioned image data completion and segmentation step, when training the above-mentioned target anatomical structure acquisition model, the above-mentioned erased image data is used as input, and the above-mentioned normal medical image data is used as the true value to learn the relevant low-dimensional and high-dimensional completion features through a self-supervised learning network, and the above-mentioned learned relevant low-dimensional and high-dimensional completion features are applied to the deep learning network used to obtain the above-mentioned target anatomical structure through the sharing operation of the encoder's weights.
20. The image processing method according to claim 12, wherein: The above-mentioned medical image data are two-dimensional or three-dimensional data of various modalities.
21. The image processing method according to any one of claims 12 to 20, characterized in that: The above-mentioned occlusion pattern is a calcified anatomical structure.
22. The image processing method according to claim 12, characterized in that: The above-mentioned occlusion pattern area is an area where image noise exists or an area where a treatment device is implanted. In the occlusion pattern region removing step, the occlusion pattern region selected by the user is removed to generate non-occlusion pattern data.