Image processing method, image processing device and storage medium

Abstract

The invention provides an image processing method, an image processing apparatus, and a storage medium capable of improving the segmentation accuracy of a valve. The image processing method comprises the following steps: acquiring anatomical structure information of a valve from a medical image containing the valve, wherein the anatomical structure information comprises at least one of a feature point region mask, geometric shape information, contour information and field information of the valve; according to the anatomical structure information, a loss function containing the weight of the anatomical structure information and the weight of the anatomical structure information in the loss function are set, and a segmentation network used for segmenting the valve is trained through the loss function; and processing the medical image by using the trained segmentation network to generate a predicted valve mask as a segmentation result.

Description

Technical Field

[0001] The present invention relates to valve segmentation in medical images, and more specifically, to an image processing method, an image processing device and a storage medium capable of improving the segmentation accuracy of valves. Background Art

[0002] In the past, the following image segmentation technologies have been known: using preoperative CT scan images to automatically segment valves such as the aortic valve, it is possible to better analyze the valve size and provide more accurate quantitative indicators, thereby enabling better planning of surgery or evaluation of treatment effects. For example, in transcatheter aortic valve implantation (TAVI), image segmentation of the aortic valve helps provide an artificial valve suitable for the patient.

[0003] Moreover, in the task of medical image segmentation, deep learning is one of the most important means and has been widely used due to its end-to-end characteristics and superior performance.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent document 1: CN107292887B

[0007] Patent document 2: US10600185B2

[0008] Non-patent literature

[0009] Non-patent document 1: Attention-Guided Decoder in Dilated Residual Network for Accurate Aortic Valve Segmentation in 3D CT Scans (authored by Fan, B., Tomii, N., Tsukihara, H., Maeda, E., Yamauchi, H., Nawata, K.,... & Ono, M. Published in Machine Learning and Medical Engineering for Cardiovascular Health and Intravascular Imaging and Computer Assisted Stenting:First International Workshop,MLMECH 2019,and8th Joint International Workshop,CVII-STENT 2019,Held in Conjunction withMICCAI 2019,Shenzhen,China,October 13,2019,Proceedings 1(pp.121-129).SpringerInternational Publishing.)

[0010] Non-patent literature 2: Semi-supervised Segmentation of Liver Using Adversarial Learning with Deep Atlas Prior (Authors: Zheng, H., Lin, L., Hu, H., Zhang, Q., Chen, Q., Iwamoto, Y., ... & Wu, J. Published in Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part VI 22 (pp. 148-156). Springer International Publishing.) Summary of the invention

[0011] Although existing deep learning-based segmentation methods directly learn the segmentation mask of the valve, it is difficult to learn the anatomical relationship between the valve itself, between valves, and between valves and blood vessels. Therefore, the segmentation results are very likely to not match the anatomical structure prior (hereinafter sometimes referred to as "anatomical structure information") as anatomical knowledge. Taking the aortic valve as an example, the anatomical knowledge has the following four clinical anatomical characteristics: (1) The lowest point and the junction point of the valve should be located on the valve; (2) The valve is a thin film structure, usually without holes; (3) The valve intersects with the blood vessel wall, and the intersection is a continuous curve; (4) Each valve divides the internal area of ​​the blood vessel, and these areas (also called domains) should have a mutually exclusive relationship. However, the prediction results of previous neural networks are often inconsistent with the above-mentioned clinical anatomical structure and cannot match the above-mentioned prior knowledge. For example, the feature points of the predicted valve do not match those of the real valve, and the predicted valve has holes and incorrect morphology, as shown in Figure 1 (a); the intersection of the predicted valve and the real blood vessel is a discrete point or does not intersect, as shown in Figure 1 (b); one of the predicted valves invades the territory of another valve, as shown in Figure 1 (c).

[0012] The present invention is completed in order to solve the above-mentioned problems. It provides a fine segmentation method of the valve based on anatomical structure prior. By including anatomical structure prior screening steps, extraction steps and application steps, it can effectively solve the problem of mismatch between the segmentation mask and the anatomical structure of the valve in the valve segmentation task. The anatomical structure prior screening step selects the anatomical structure prior (anatomical structure information) unique to the valve, and the anatomical structure prior extraction step performs label preprocessing to obtain the above-mentioned anatomical structure prior. The anatomical structure prior application step introduces anatomical information during training through network design, loss design or post-processing, thereby improving the overall and local segmentation accuracy of the valve.

[0013] Specifically, as a means for solving the above-mentioned problems, one of the technical solutions of the present invention is an image processing method, comprising the following steps: obtaining anatomical structure information of a valve in a medical image containing the valve, the anatomical structure information including at least one of a feature point area mask, geometric morphology information, contour information, and domain information of the valve; according to the anatomical structure information, setting a loss function containing the weight of the anatomical structure information and the weight of the anatomical structure information in the loss function, and training a segmentation network for segmenting the valve through the loss function; and using the trained segmentation network to process the medical image to generate a predicted valve mask as a segmentation result.

[0014] As a means for solving the above-mentioned problems, another technical solution of the present invention is an image processing method, comprising the following steps: obtaining anatomical structure information of a valve in a medical image containing a valve, the anatomical structure information including at least one of a feature point heat map, geometric morphology information, contour information, and domain information of the valve; according to the anatomical structure information, setting a loss function including at least one of the anatomical structure information and a mask of the valve, and training a segmentation network with multiple output branches through the loss function, the output branches outputting at least one of the predicted anatomical structure information and a mask of the valve; and using the trained segmentation network to process the medical image to generate a predicted valve mask and anatomical structure information.

[0015] As a means for solving the above-mentioned problems, another technical solution of the present invention is an image processing method, comprising the following steps: obtaining anatomical structure information of a valve in a medical image containing a valve, the anatomical structure information including at least one of feature point information, geometric morphology information, contour information, and domain information of the valve, the feature point information being a feature point region mask or a feature point heat map; setting a loss function for training a segmentation network for segmenting the valve according to the anatomical structure information, setting a classification network for the segmentation network to train the segmentation network, and setting a loss function for training the classification network and a weight of the loss function according to the anatomical structure information; and processing the medical image using the trained segmentation network to generate a predicted valve mask as a segmentation result.

[0016] As a means for solving the above-mentioned problems, another technical solution of the present invention is an image processing device, comprising: an anatomical structure information acquisition unit, which acquires anatomical structure information of a valve in a medical image containing the valve, and the anatomical structure information includes at least one of a feature point area mask, geometric morphology information, contour information, and domain information of the valve; a training unit, which sets a loss function including weights of the anatomical structure information and the weights of the anatomical structure information in the loss function according to the anatomical structure information, and trains a segmentation network for segmenting the valve through the loss function; and an image processing unit, which processes the medical image using the trained segmentation network to generate a predicted valve mask as a segmentation result.

[0017] As a means for solving the above-mentioned problems, another technical solution of the present invention is an image processing device, comprising: an anatomical structure information acquisition unit, which acquires anatomical structure information of a valve in a medical image containing the valve, and the anatomical structure information includes at least one of a feature point heat map, geometric morphology information, contour information, and domain information of the valve; a training unit, which sets a loss function including at least one of the anatomical structure information and a mask of the valve according to the anatomical structure information, and trains a segmentation network with multiple output branches through the loss function, and the output branches output at least one of the predicted anatomical structure information and the mask of the valve; and an image processing unit, which uses the trained segmentation network to process the medical image to generate a predicted valve mask and anatomical structure information.

[0018] As a means for solving the above-mentioned problems, another technical solution of the present invention is an image processing device, comprising: an anatomical structure information acquisition unit, which acquires anatomical structure information of a valve in a medical image containing the valve, the anatomical structure information including at least one of feature point information, geometric morphology information, contour information, and domain information of the valve, and the feature point information is a feature point region mask or a feature point heat map; a training unit, which sets a loss function for training a segmentation network for segmenting the valve according to the anatomical structure information, and also sets a classification network for the segmentation network to train the segmentation network, and sets a loss function for training the classification network and a weight of the loss function according to the anatomical structure information; and an image processing unit, which uses the trained segmentation network to process the medical image to generate a predicted valve mask as a segmentation result.

[0019] As a means for solving the above-mentioned problems, another technical aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to execute the above-mentioned image processing method.

[0020] According to the present invention, it is possible to provide an image processing method, an apparatus, and a storage medium that improve the accuracy of segmentation of the entire valve and parts thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] FIG. 1 is a schematic diagram showing a valve predicted by the prior art.

[0022] Figure 2 It is a flowchart for explaining the training and inference process of the valve segmentation network of the present invention.

[0023] Figure 3 This is a flowchart for explaining the training and inference process of the valve segmentation network of Example 1.

[0024] FIG. 4 is a schematic diagram showing the extraction of anatomical structure information.

[0025] Figure 5 This is a schematic diagram for explaining the training of the valve segmentation network in Example 1.

[0026] Figure 6 This is a flowchart for explaining the training and inference process of the valve segmentation network of Example 2.

[0027] Figure 7 is a heat map of characteristic points of the valve of the present invention.

[0028] Figure 8 This is a schematic diagram for explaining the training of the valve segmentation network in Example 2.

[0029] Fig. 9 This is a flowchart for explaining the training and inference process of the valve segmentation network of Example 3.

[0030] Fig.10 This is a schematic diagram for explaining the training of the valve segmentation network in Example 3.

[0031] Fig.11 This is a flowchart for explaining the training and inference process of the valve segmentation network of Example 4.

[0032] Fig.12 It is a schematic diagram comparing the segmentation results of the prior art and the present invention.

[0033] Fig.13 It is a schematic diagram comparing the segmentation results of the prior art and the present invention. DETAILED DESCRIPTION

[0034] Hereinafter, embodiments of the present invention will be described based on the drawings.

[0035] In general, if Figure 2 As shown, the training process of the valve segmentation network of the present invention includes the following steps: a step of acquiring a medical image and a label containing a valve (S100); a step of preprocessing the image (S200); an anatomical structure prior screening step, screening (determining) a specific anatomical structure prior (anatomical structure information) (S300); an anatomical structure prior extraction step, extracting the anatomical structure prior by preprocessing the label (S400); an anatomical structure prior application step, introducing anatomical information through network design and loss design, and training the valve segmentation network (S500); and a step of outputting the valve segmentation network (S600). The reasoning process of the valve segmentation network of the present invention includes the following steps: a step of inputting an image containing a valve (S100'); a step of preprocessing the image (S200'); a step of predicting the valve using a trained anatomical structure prior-based segmentation network (S300'); and a step of outputting the final segmentation result (S400').

[0036] Several embodiments based on the above main idea are described in detail below.

[0037] (Example 1)

[0038] Below, refer to Figure 3 , Example 1 of the present invention is described. This Example 1 introduces anatomical structure priors by designing a perceptual loss function and segments the aortic valve. In other words, this Example 1 uses a valve segmentation network based on anatomical structure priors to predict the segmentation results of the valve.

[0039] The training process of the valve segmentation network involved in Example 1 includes the following steps.

[0040] First, a medical image (usually a CT image) containing a valve and a label are obtained. The label may include a feature point label, a valve (aortic valve) label, a blood vessel label (aortic root label), etc., which are input by the user (S100). Then, the CT image is preprocessed as required, such as adjusting the resolution and grayscale range (S200). These two steps are conventional practices in the art, so detailed descriptions are omitted.

[0041] Next, the anatomical structure prior is screened (S300). Here, the anatomical structure prior of the valve can be selected by the user. In this embodiment, the anatomical structure prior of the aortic valve is screened (determined) as at least one of feature point region mask, geometric morphology information, contour information, and domain information.

[0042] Then, the anatomical structure prior is extracted (S400). In the first embodiment, the feature points, valve and blood vessel labels are preprocessed to extract at least one of the selected anatomical structure priors, namely, feature point region mask, geometric morphology information, contour information and domain information.

[0043] In more detail, feature points (also called key points) refer to points on the valve that have obvious anatomical features (i.e., features that are easily distinguished from other parts in a medical scan image) and have a relatively stable relative position relationship with the segmented object (i.e., the position relationship between the feature points and the segmented object will not change significantly due to individual differences in the scanned body), which are of clinical significance and include the lowest point of the valve, the connection point between valves, etc., and the feature point area mask can be generated by the feature point coordinates. The obtained feature point area mask is shown in the 3D schematic diagram of Figure 4 (a). The geometric morphology refers to the morphological features of the valve, such as no holes, wide at the top and narrow at the bottom, etc., which can be generated by extracting the center plane of the valve (as shown in Figure 4 (b)), and the obtained geometric morphology information is, for example, the shape of the valve. The contour information can be generated by extracting the intersection line between the valve label and the blood vessel label, or by fitting the feature point coordinates on the valve with the blood vessel label (as shown in Figure 4 (c)), or by fitting the feature point coordinates on the valve with the blood vessel label and other related algorithms. The area information can be generated by the valve label and the blood vessel label (as shown in (d) of FIG. 4 ).

[0044] Then, the anatomical structure prior is applied (S500). In this embodiment 1, an anatomical structure-aware loss function is designed based on the anatomical structure prior, and a segmentation network (also referred to as a segmentation model) used as a generator for segmenting the valve is trained.

[0045] The segmentation network of the valve uses a common convolutional neural network, such as the U-Net model. In addition, the network is trained using conventional segmentation losses, such as CE loss and DICE loss. During training, Figure 5 As shown, the medical image is input into the segmentation network, and the output of the segmentation network is the predicted segmentation result, namely the valve mask.

[0046] Here, a combination of one or more anatomical structure priors is used to modify their weights in CE loss and DICE loss, thereby designing the perceptual loss function as shown in the following formula.

[0047]

[0048] Among them, w represents the weight of the anatomical structure prior, y represents the actual valve segmentation result, Represents the predicted valve segmentation result, and i represents the i-th one of the four anatomical structure priors: feature point region mask, geometric morphology information, contour information, and domain information.

[0049] In more detail, feature point loss can be designed. The feature point area mask includes part of the valve and part of the background. By giving it a larger weight, the network can focus on learning the features of this area, so that clinically significant points can be included in the valve. Valve geometry loss can also be designed. The valve center plane is located inside the valve. By giving it a larger weight, the network's attention can be optimized, thereby preventing the generation of erroneous valve structures and improving segmentation accuracy. Valve contour loss can also be designed. The intersection line is located at the edge of the valve. By setting the weight of the intersection line, the extraction of edge information can be enhanced to ensure the correct structural relationship between the valve and the blood vessel. Valve area loss can also be designed. For example, the aortic valve divides the aortic root into three areas, which can provide valve area information. By increasing the weight of mis-segmentation within the area, the problem of intervalvular area invasion can be improved.

[0050] For example, if we want the network to focus on learning geometric features, that is, we want the network to focus on avoiding the generation of erroneous valve structures (such as holes), the anatomical structure is a priori filtered as geometric information. In this case, i in the above formula is 2, so the formula becomes:

[0051]

[0052] In this case, during the training process, the medical image is first input into the segmentation network to predict the segmentation result of the valve (valve mask), and then the difference between the predicted result and the actual result is calculated. At this time, if more holes are seen in the extracted geometric information, the weight is designed to be larger, otherwise the weight is designed to be smaller. Then, the difference is multiplied by the weight, and the result of the multiplication is used as the loss function value. Then, the gradient of the network parameters is calculated using the loss function value to determine the next optimization direction of the network parameters. After that, the optimization step size of the network parameters is adjusted by the learning rate, thereby updating the segmentation network. Then, the next round of prediction training is carried out. Repeat the above training until the number of training iterations meets the specified conditions or the loss function value no longer decreases significantly.

[0053] In addition, the above example is a case where the anatomical structure prior is filtered only as geometric morphological information. When more than one anatomical structure prior is filtered out, for example, the case where geometric morphological information and domain information are filtered out, i is 2 and 4, and the above formula becomes:

[0054]

[0055] Finally, the valve segmentation model (segmentation network) is output, thereby completing the training of the model (S600).

[0056] The inference process (ie, application process) of the valve segmentation network involved in Example 1 includes the following steps.

[0057] First, an image containing a valve (usually a CT image) is input (S100'), and then the image is preprocessed such as resolution adjustment and grayscale range adjustment (S200'). These two steps are conventional practices in the art, so detailed descriptions are omitted.

[0058] Next, the segmentation network trained in S600 based on anatomical structure priors is used to predict the segmentation of the valve ( S300 ′).

[0059] Finally, the predicted valve mask is output as the final segmentation result (S400'). The predicted valve mask is, for example, Fig.11 shown.

[0060] (Example 2)

[0061] Below, refer to Figure 6 , Example 2 of the present invention is described. This Example 2 designs a network output and optimizes the valve segmentation result based on the network output, thereby introducing an anatomical structure prior to segment the aortic valve. Compared with Example 1, it can also predict the anatomical structure prior.

[0062] The training process of the valve segmentation network involved in Example 2 includes the following steps.

[0063] First, as in Example 1, an image and a label containing a valve are obtained (S100), the CT image is preprocessed (S200), the anatomical structure prior is screened (S300), and the anatomical structure prior is extracted (S400). Different from Example 1, as the anatomical structure prior, Example 1 uses a feature point region mask, while Example 2 uses a feature point heat map generated by a distribution function and feature point coordinates (such as Figure 7 ).

[0064] Then, the anatomical structure prior is applied (S500). Different from the first embodiment, in the second embodiment, a segmentation loss function and a loss function of the anatomical structure prior are also designed, and the segmentation network is trained accordingly.

[0065] Here, common convolutional neural networks such as U-Net are used as the basic segmentation model, such as Figure 8 As shown, compared with Example 1, the input part is the same, both are medical images, but the output part adds a decoding branch or an output branch of the segmentation network, so that in addition to outputting the predicted valve mask like Example 1, the segmentation network can also output at least one of the feature point heat map, geometric shape, contour information and domain information.

[0066] Moreover, when training the segmentation network, common segmentation losses, such as Cross Entropy (CE) and DICE loss, are used to train the segmentation branches of valve mask, geometric information, contour information and domain information. In addition, common regression losses (such as MSE) are also used to train the branch of feature point heat map. The formula is as follows.

[0067] loss=loss1+loss2

[0068]

[0069]

[0070] Among them, y1 represents the true valve mask (valve segmentation result), y i (i∈2,3,4) represents one of the real geometric information, contour information and domain information. represents one of the predicted geometric information, contour information and domain information, y5 represents the real feature point heat map, represents the predicted feature point heat map, i represents the i-th one in the geometric information, contour information and domain information, w i Indicates whether to use the i-th information.

[0071] For example, when the segmentation network is made to optimize the valve segmentation result using only the domain information, the above formula becomes:

[0072]

[0073] For example, when the segmentation network is made to output only feature point heatmaps, the above formula becomes:

[0074]

[0075] For example, when the segmentation network is made to output the area information and the feature point heat map, the above formula becomes:

[0076]

[0077] Finally, the valve segmentation model is output, thereby completing the model training ( S600 ).

[0078] The inference process (ie, application process) of the valve segmentation network involved in Example 2 includes the following steps.

[0079] First, similar to the first embodiment, an image including a valve is input ( S100 ′), and then the image is preprocessed ( S200 ′).

[0080] Next, the segmentation network trained in S600 is used to predict valve and anatomical structure priors ( S300 ′).

[0081] Finally, the final segmentation result is output (S400'). Here, in addition to outputting the predicted valve mask in the same manner as in Example 1, the anatomical structure prior is also output. Furthermore, the anatomical structure prior information output by the segmentation network can be used to optimize the segmentation result of the aortic valve and output it. For example, the geometry of the segmentation result of the valve can be optimized by fusion or other methods based on the geometric information output by the segmentation network. Alternatively, the problem of discontinuity of the intersection line between the valve and the blood vessel can be optimized by other methods such as fusion or extension based on the contour information output by the segmentation network. Alternatively, the problem of area encroachment between valves can be optimized by other methods such as removing connected domains based on the area information output by the segmentation network. Alternatively, the segmentation result can be optimized by determining whether the feature point is on the valve based on the feature point coordinates obtained from the feature point heat map output by the segmentation network, and if not, the valve can be extended.

[0082] Therefore, according to this embodiment 2, compared with the prior art, the network also learns the features of feature points, geometric shapes, contours and domain information, so that the features learned by the segmentation network can be more robust.

[0083] According to the second embodiment, when designing the segmentation network, a loss function of the anatomical structure prior is also designed, so that the anatomical structure prior can be introduced, and a fine valve segmentation network can be obtained through training. Moreover, compared with the output of only the valve in the first embodiment, the second embodiment outputs the valve and the anatomical structure prior, and the anatomical structure prior information output by the segmentation network can be used to optimize the valve segmentation result, and a more practical final output can be obtained.

[0084] (Example 3)

[0085] Below, refer to Fig. 9 , Example 3 of the present invention is described. As a supplement to Example 1, Example 3 introduces anatomical structure priors and segments the aortic valve by designing an adversarial loss function.

[0086] The training process of the valve segmentation network involved in Example 3 includes the following steps.

[0087] First, similar to Example 1, an image and a label containing a valve are obtained (S100), the CT image is preprocessed (S200), the anatomical structure prior is screened (S300), and the anatomical structure prior is extracted (S400). Different from Example 1, Example 1 uses a feature point region mask as the anatomical structure prior, while Example 3 uses a feature point heat map or a feature point region mask.

[0088] Then, the anatomical structure prior is applied (S500). Different from Example 1, in this Example 3, an adversarial loss function is also used to introduce anatomical prior information and train a segmentation network that matches the anatomical knowledge.

[0089] Here, similar to Example 1, a common convolutional neural network such as U-Net is used as the basic segmentation model of the valve, and a common segmentation loss such as Cross Entropy (CE) and DICE loss is used when training the segmentation network. Fig.10 As shown, different from Example 1, a classification network (so-called discriminator; classification model) is also introduced. That is, in addition to the segmentation loss, the loss of the classification network (Discriminator) is designed a priori according to the anatomical structure and added to the training of the segmentation network. Therefore, the loss of the segmentation network includes three items, as shown in the following formula, where y represents the actual valve segmentation result, represents the predicted valve segmentation result, and ω represents the weight of the classification network loss.

[0090]

[0091] Here, the classification network is a simple convolutional neural network, such as resnet18, which is input with a set of data consisting of the extracted anatomical structure prior and the valve mask, where the segmentation result (valve mask) can be true or predicted by the segmentation network, and the output represents the possibility that the segmentation mask is true. The goal of the classification network is to predict the true segmentation as 1 as much as possible and the predicted segmentation as 0. In other words, when the extracted anatomical structure prior and the predicted segmentation result are input as a set of data to the classification network, the classification network is trained to output "0" representing "false" as much as possible, and when the extracted anatomical structure prior and the true segmentation result are input as a set of data to the classification network, the classification network is trained to output "1" representing "true" as much as possible. In addition, only CE loss is used to train the classification network, as shown in the following formula, where D represents the classification network (i.e., the discriminator).

[0092]

[0093] Since the goal of the segmentation network is to make the classification network output true when the "predicted segmentation mask pair" is input to the classification network, that is, the output can be as close to the real mask as possible, so as to be more consistent with the anatomical structure prior.

[0094] Finally, the valve segmentation model is output, thereby completing the model training ( S600 ).

[0095] The inference process (ie, application process) of the valve segmentation network involved in Example 3 includes the following steps.

[0096] First, similar to the first embodiment, an image including a valve is input ( S100 ′), and then the image is preprocessed ( S200 ′).

[0097] Next, the segmentation network trained in S600 based on anatomical structure priors is used to predict the segmentation of the valve ( S300 ′).

[0098] Finally, the final segmentation result is output ( S400 ′).

[0099] Compared with Example 1, in the anatomical structure prior application step, Example 3 uses adversarial loss to introduce anatomical structure prior, which can obtain a segmentation network that better meets anatomical knowledge.

[0100] (Example 4)

[0101] Below, refer to Fig.11 , Example 4 of the present invention is described. As a supplement to Example 1, Example 4 is different from Example 1 only in that, in the anatomical structure prior screening step (S300), the valve is firstly coarsely segmented automatically using traditional methods, machine learning or existing technologies to obtain a coarse segmentation result, and then, image features are calculated based on the coarse segmentation result of the aortic valve, and then, based on the calculated image features, a visual evaluation of the segmentation result of the existing technology is performed, and based on the evaluation result, the anatomical structure is priori screened (determined) as at least one of feature point region mask, geometric morphology information, contour information, and domain information.

[0102] Here, for example, the valve is roughly segmented using the existing technology to obtain Fig.12 The rough segmentation result on the left is then used to calculate the so-called distance map as the image feature. The user finds holes on the valve (or the number of holes is greater than a specified threshold) on the distance map, so the anatomical structure is intentionally determined as geometric information a priori, thereby optimizing the network's attention and making the network focus on learning the features of geometric features. This can prevent the generation of valve structures with erroneous morphology and improve segmentation accuracy. Alternatively, for example, if the user finds that the result of the rough segmentation is that the number of feature points located on the real valve is less than a specified threshold, the anatomical structure is determined a priori as a feature point region mask, so that the network focuses on learning the features of the feature point region mask, so that points with clinical significance can be included in the valve.

[0103] According to this fourth embodiment, a more suitable anatomical structure prior can be selected and given a greater weight, thereby further improving the accuracy of valve segmentation.

[0104] The present invention has the following advantages.

[0105] First, according to Example 1, since the loss function is designed based on the feature point information (feature point area mask), the accuracy (precision) of the key parts of the segmentation mask can be improved. In particular, after weighting the feature point information, the segmentation mask predicted by the model can contain more feature points.

[0106] In order to compare with the prior art that does not weight the feature point information, 48 experiments were conducted using the segmentation network trained by Example 1 of the present invention. Specifically, in each experiment, a total of 6 feature points were marked on the three predicted valves, and the number of these feature points falling on the real valves was counted. The results are shown in the first column of the table below. It can be seen that the accuracy rate is 4.1% higher than before. In addition, the lowest point (Nadir points) can be calculated on the predicted valve, and the distance between the lowest point and the real lowest point is measured. The results are shown in the second column of the table below. It can be seen that compared with the past, the error is reduced by 0.01 mm on average.

[0107] Landmark in mask(total 288 points) Nadir points mean distance(mm) Prior art 120 1.491±1.659 The present invention 132 1.484±1.166

[0108] Moreover, if Fig.12 As shown, since the loss function is designed according to the geometric morphology information of the valve, the occurrence of erroneous valve segmentation results such as valves containing holes in the prior art can be effectively reduced, and valves of different shapes in a cardiac cycle can be more accurately predicted, thereby improving the accuracy of the overall segmentation mask.

[0109] In addition, since the loss function is designed based on the contour information of the valve, the shape of the connection between the valve and the blood vessel can be correct, reducing the segmentation results that are disconnected or inconsistent with the anatomical structure. Fig.13 As shown, since the loss function is designed according to the domain characteristics between valves, the problem of valves encroaching on each other's domains in the prior art can be reduced.

[0110] In addition, according to Example 2, since the network is designed so that it can output valve structure information, the segmentation result of the valve is optimized by using the network output in the inference stage, which can make the predicted valve mask more accurate and closer to the real valve (result). For example, the predicted valve mask morphology is more correct and has no holes. In addition, according to Example 3, since the adversarial loss function is used to introduce the anatomical structure prior and train the segmentation model, the segmentation model can generate a segmentation mask that conforms to the valve anatomical structure without increasing the additional model running time. In addition, according to Example 4, a more suitable anatomical structure prior can be selected based on the rough segmentation result, and it can be given a greater weight, thereby further improving the accuracy of valve segmentation.

[0111] Furthermore, the present invention may be implemented as a device using the above-mentioned image processing method, may be implemented as a program for causing a computer to execute the above-mentioned method, or may be implemented as a storage medium storing the program.

[0112] In addition, the above embodiments may also be used in combination with each other. For example, the feature of designing a perceptual loss function by modifying the weights of CE loss and DICE loss in embodiment 1 may be applied to other embodiments, and the feature of screening anatomical structure priors by performing rough segmentation in advance in embodiment 4 may be applied to other embodiments.

[0113] Furthermore, the valve of the present invention is applicable not only to the aortic valve but also to the pulmonary valve, upper and lower limb venous valves, and the like.

[0114] Although several embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the subject matter of the invention. These embodiments and their variations are included in the scope and subject matter of the invention, and are included in the scope of the invention described in the claims and their equivalents.

Claims

1. An image processing method, characterized in that: The steps include: Acquire anatomical structure information of the valve in a medical image containing the valve, the anatomical structure information including at least one of a feature point region mask, geometric morphology information, contour information, and domain information of the valve; According to the anatomical structure information, a loss function including the weight of the anatomical structure information and the weight of the anatomical structure information in the loss function is set, and a segmentation network for segmenting the valve is trained by the loss function; as well as The medical image is processed using a trained segmentation network to generate a predicted valve mask as a segmentation result.

2. The image processing method according to claim 1, characterized in that: The anatomical structure information is obtained by preprocessing at least one of the feature point labels, valve labels and blood vessel labels annotated by a user on the medical image.

3. The image processing method according to claim 1 or 2, characterized in that: The loss functions corresponding to the anatomical structure information are the CE loss function and the DICE loss function.

4. The image processing method according to claim 2, characterized in that: In the anatomical structure information, the feature point area mask is generated by the coordinates of the feature points on the valve, the geometric morphology information is generated by extracting the center plane of the valve, the contour information is generated by extracting the intersection line between the valve label and the blood vessel label, or by fitting the feature point coordinates on the valve and the blood vessel label, and the field information is generated by the valve label and the blood vessel label.

5. An image processing method, characterized in that: The steps include: Acquire anatomical structure information of the valve in a medical image containing the valve, the anatomical structure information including at least one of a feature point heat map, geometric morphology information, contour information, and domain information of the valve; According to the anatomical structure information, a loss function including at least one of the anatomical structure information and a mask of the valve is set, and a segmentation network having a plurality of output branches is trained by the loss function, wherein the output branches output at least one of the predicted anatomical structure information and a mask of the valve; as well as The medical image is processed using a trained segmentation network to generate predicted valve mask and anatomical structure information.

6. The image processing method according to claim 5, characterized in that: The mask and anatomical structure information of the valve output by the segmentation network are used to optimize post-processing related to the segmentation result of the valve.

7. The image processing method according to claim 5 or 6, characterized in that: The loss function further includes the weight of the anatomical structure information. In the step of setting the loss function, the weight of the anatomical structure information in the loss function is also set.

8. The image processing method according to claim 5 or 6, characterized in that: The branch of the segmentation network for outputting the feature point heat map is trained using MSE as the regression loss function.

9. The image processing method according to claim 5 or 6, characterized in that: The loss functions corresponding to the geometric morphology information, contour information, domain information, and valve mask are the CE loss function and the DICE loss function, which are used to train the branch of the segmentation network for outputting the geometric morphology information, contour information, domain information, and valve mask.

10. The image processing method according to claim 5, characterized in that: The anatomical structure information is obtained by preprocessing at least one of the feature point labels, valve labels and blood vessel labels annotated by a user on the medical image.

11. The image processing method according to claim 10, characterized in that: In the anatomical structure information, the feature point heat map is generated by the distribution function and the feature point coordinates, the geometric morphology information is generated by extracting the center plane of the valve, the contour information is generated by extracting the intersection line between the valve label and the blood vessel label, or by fitting the feature point coordinates on the valve and the blood vessel label, and the field information is generated by the valve label and the blood vessel label.

12. An image processing method, characterized in that: The steps include: Acquire anatomical structure information of the valve in a medical image containing the valve, the anatomical structure information including at least one of feature point information, geometric morphology information, contour information, and domain information of the valve, wherein the feature point information is a feature point region mask or a feature point heat map; According to the anatomical structure information, a loss function for training a segmentation network for segmenting the valve is set, a classification network is set for the segmentation network to train the segmentation network, and a loss function for training the classification network and a weight of the loss function are set according to the anatomical structure information; as well as The medical image is processed using a trained segmentation network to generate a predicted valve mask as a segmentation result.

13. The image processing method according to claim 12, characterized in that: The loss function of the segmentation network used to segment the valve further includes the weight of the anatomical structure information. In the step of setting the loss function, the weight of the anatomical structure information in the loss function is also set.

14. The image processing method according to claim 12 or 13, characterized in that: The loss function used to train the classification network is the CE loss function.

15. The image processing method according to claim 12 or 13, characterized in that: The classification network is resnet18.

16. The image processing method according to claim 12 or 13, characterized in that: The loss functions corresponding to the anatomical structure information are the CE loss function and the DICE loss function.

17. The image processing method according to claim 12 or 13, characterized in that: The anatomical structure information is obtained by preprocessing at least one of the feature point labels, valve labels and blood vessel labels annotated by a user on the medical image.

18. The image processing method according to claim 17, characterized in that: In the anatomical structure information, the feature point area mask is generated by the coordinates of the feature points on the valve, the feature point heat map is generated by the distribution function and the feature point coordinates, the geometric morphology information is generated by extracting the center plane of the valve, the contour information is generated by extracting the intersection line between the valve label and the blood vessel label, or by fitting the feature point coordinates on the valve and the blood vessel label, and the field information is generated by the valve label and the blood vessel label.

19. The image processing method according to any one of claims 1, 5 and 12, characterized in that: The segmentation network is a U-net network.

20. The image processing method according to any one of claims 1, 5 and 12, characterized in that: The valve is an aortic valve, a pulmonary valve or a valve of the veins of the upper and lower limbs.

21. The image processing method according to any one of claims 1, 5 and 12, characterized in that: In the step of acquiring the anatomical structure information of the valve, the image is automatically roughly segmented for the valve, and the anatomical structure information is determined according to the result of the rough segmentation.

22. An image processing device, characterized in that: have: An anatomical structure information acquisition unit, which acquires anatomical structure information of the valve from a medical image containing the valve, the anatomical structure information including at least one of a feature point region mask, geometric morphology information, contour information, and domain information of the valve; A training unit, which sets a loss function including a weight of the anatomical structure information and a weight of the anatomical structure information in the loss function according to the anatomical structure information, and trains a segmentation network for segmenting the valve by using the loss function; as well as The image processing unit processes the medical image using the trained segmentation network to generate a predicted valve mask as a segmentation result.

23. An image processing device, characterized in that: have: An anatomical structure information acquisition unit, which acquires anatomical structure information of the valve from a medical image containing the valve, wherein the anatomical structure information includes at least one of a feature point heat map, geometric morphology information, contour information, and domain information of the valve; A training unit, which sets a loss function including at least one of the anatomical structure information and a mask of the valve according to the anatomical structure information, and trains a segmentation network having a plurality of output branches through the loss function, wherein the output branches output the predicted at least one of the anatomical structure information and the mask of the valve; as well as The image processing unit processes the medical image using the trained segmentation network to generate predicted valve mask and anatomical structure information.

24. An image processing device, characterized in that: have: an anatomical structure information acquisition unit, which acquires anatomical structure information of the valve from a medical image containing the valve, wherein the anatomical structure information includes at least one of feature point information, geometric morphology information, contour information, and domain information of the valve, wherein the feature point information is a feature point region mask or a feature point heat map; A training unit, which sets a loss function for training a segmentation network for segmenting the valve according to the anatomical structure information, further sets a classification network for the segmentation network to train the segmentation network, and sets a loss function for training the classification network and a weight of the loss function according to the anatomical structure information; as well as The image processing unit processes the medical image using the trained segmentation network to generate a predicted valve mask as a segmentation result. 25 . A computer-readable storage medium storing a program for causing a computer to execute the image processing method according to claim 1 .

Images