Image processing apparatus and image processing method

Abstract

To provide a technique capable of accurately and efficiently extracting a heart valve ring in a medical image.SOLUTION: An image processing apparatus according to an embodiment includes an image acquisition unit, an annulus key point extraction unit, an annulus calibration unit, and an annulus curve acquisition unit. The image acquisition unit acquires a plurality of three dimensional medical images. The annulus key point extraction unit is configured to extract annulus key points based on the plurality of three dimensional medical images by using a trained annulus key point prediction model. The annulus calibration unit is configured to calibrate the annulus key points to supplement at least part of the annulus key points. The annulus curve obtaining unit is configured to perform fitting according to the corrected annulus key points to obtain an annulus curve.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The embodiments disclosed in this specification and the drawings relate to an image processing device and an image processing method. [Background technology]

[0002] Currently, in the field of medical image processing, extraction of the morphology of heart valve annuli from medical images contributes to doctors' diagnosis of abnormalities. In particular, extraction results for the tricuspid valve annulus in the heart can be used to evaluate the annulus' open / close state and measure its morphology, allowing for pathological diagnosis. This can be used to plan surgical procedures. Therefore, extraction of heart valve annuli from medical images has great clinical value.

[0003] Furthermore, when extracting the cardiac annulus from a medical image, the medical image may have poor contrast or may contain noise or artifacts, making it difficult to accurately extract the cardiac annulus.

[0004] Another conventional technique involves a physician manually extracting the heart valve annulus from cardiac images. However, this method requires the physician to extract the annulus' feature points one by one. This method also requires the physician to have extensive experience and advanced skills, and the operation is complex and time-consuming for the physician. Therefore, there is a pressing need for a technique that can extract the heart valve annulus from medical images fully automatically, accurately, and efficiently.

[0005] Another conventional non-manual cardiac valve annulus extraction technique is to extract the bicuspid valve annulus from 3D transesophageal ultrasound images using deep learning. However, this method is not suitable for extracting the tricuspid valve annulus because the tricuspid valve is thinner than the bicuspid valve, resulting in lower contrast in ultrasound images. Furthermore, this method performs two-dimensional classification on ultrasound images and uses an algorithm to reconstruct the 3D annulus extraction results, so it is not inherently a 3D annulus extraction. Furthermore, CT images have poorer contrast than ultrasound images. Furthermore, CT images typically have enhancements for the bicuspid valve region, while CT images with enhancements for the tricuspid valve region are rare, resulting in poor contrast in the tricuspid valve region in CT images and many artifacts. Therefore, this method is not suitable for extracting the tricuspid valve annulus on a 3D scale based on CT images.

[0006] Another conventional non-manual cardiac annulus extraction technique is to use deep learning to detect annulus keypoints in two-dimensional MR images of the four-chamber long axis of the heart. However, if the cardiac annulus data has poor contrast in some regions or problems with noise and artifacts, annulus extraction based solely on keypoints is affected by regional image quality, resulting in large prediction errors for annulus keypoints in some regions, resulting in poor annulus extraction accuracy. Furthermore, this method detects annulus keypoints in two-dimensional images, and the position of the annulus in two-dimensional images does not reflect the morphological information of the annulus. Therefore, this method cannot obtain three-dimensional morphological information of the annulus, and requires a separate three-dimensional reconstruction algorithm.

[0007] Therefore, there is a need for a technology that can automatically, accurately, and efficiently extract cardiac valve annuli on a three-dimensional scale based on various images, particularly CT images. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] Andreassen BS, Veronesi F, Gerard O, et al. "Mitral annulus segmentation using deep learning in 3-D transesophageal echocardiography" IEEE J Biomed Health Inform. 2020 Apr;24(4):994-1003. [Non-patent document 2] Kerfoot E, King CE, Ismail T, et al. "Estimation of cardiac valve annuli motion with deep learning" Statistical Atlases and Computational Models of the Heart. MandMs and EMIDEC Challenges - 11th International Workshop, STACOM 2020, Held in Conjunction with MICCAI 2020, Revised Selected Papers:146-155. Summary of the Invention [Problem to be solved by the invention]

[0009] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to provide a technology that can extract cardiac valve annuli accurately and efficiently from medical images. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems. [Means for solving the problem]

[0010] An image processing device according to an embodiment includes an image acquisition unit, an annulus keypoint extraction unit, an annulus calibration unit, and an annulus curve acquisition unit. The image acquisition unit acquires a plurality of 3D medical images. The annulus keypoint extraction unit extracts annulus keypoints based on the plurality of 3D medical images using a trained annulus keypoint prediction model. The annulus calibration unit calibrates the annulus keypoints to supplement at least a portion of the annulus keypoints. The annulus curve acquisition unit performs fitting based on the calibrated annulus keypoints to acquire an annulus curve. Equipped with [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram illustrating an example of an image processing apparatus according to the first embodiment. [Figure 2] FIG. 2 is a flowchart illustrating an example of an image processing method of the image processing apparatus according to the first embodiment. [Figure 3A] FIG. 3A is an exemplary diagram showing a specific example of the image processing device according to the first embodiment. [Figure 3B] FIG. 3B is an exemplary diagram showing a specific example of the image processing device according to the first embodiment. [Figure 3C] FIG. 3C is an exemplary diagram showing a specific example of the image processing device according to the first embodiment. [Figure 3D] 3D is an exemplary diagram showing a specific example of the image processing device according to the first embodiment. [Figure 3E] FIG. 3E is an exemplary diagram showing a specific example of the image processing device according to the first embodiment. [Figure 4] FIG. 4 is a block diagram illustrating an example of an image processing apparatus according to the second embodiment. [Figure 5] FIG. 5 is a flowchart illustrating an example of an image processing method of the image processing device according to the second embodiment. [Figure 6A] FIG. 6A is an exemplary diagram showing a specific example of the image processing device according to the second embodiment. [Figure 6B]FIG. 6B is an exemplary diagram showing a specific example of the image processing device according to the second embodiment. [Figure 6C] FIG. 6C is an exemplary diagram showing a specific example of the image processing device according to the second embodiment. [Figure 6D] FIG. 6D is an exemplary diagram showing a specific example of the image processing device according to the second embodiment. [Figure 6E] FIG. 6E is an exemplary diagram showing a specific example of the image processing device according to the second embodiment. [Figure 6F] FIG. 6F is an exemplary diagram showing a specific example of the image processing device according to the second embodiment. [Figure 6G] FIG. 6G is an exemplary diagram showing a specific example of the image processing device according to the second embodiment. [Figure 6H] FIG. 6H is an exemplary diagram showing a specific example of the image processing device according to the second embodiment. [Figure 6I] FIG. 6I is an exemplary diagram showing a specific example of the image processing device according to the second embodiment. [Figure 6J] FIG. 6J is an exemplary diagram showing a specific example of the image processing device according to the second embodiment. [Figure 6K] FIG. 6K is an exemplary diagram showing a specific example of the image processing device according to the second embodiment. [Figure 6L] FIG. 6L is an exemplary diagram showing a specific example of the image processing device according to the second embodiment. [Figure 7] FIG. 7 is a flowchart illustrating an example of the operation of extracting an annulus region image by the image processing device according to the second embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of the operation of annulus calibration performed by the image processing device according to the second embodiment. [Figure 9] FIG. 9 is a flowchart illustrating an example of an image processing method of the image processing device according to the first modification of the second embodiment. [Figure 10] FIG. 10 is a flowchart illustrating an example of an image processing method of an image processing device according to Modification 2 of the second embodiment. [Figure 11]FIG. 11 is an exemplary diagram showing an application example of annulus keypoint extraction in the image processing apparatus according to the third embodiment. [Figure 12] FIG. 12 is an exemplary diagram showing a specific example of the image processing device according to the fourth embodiment. [Figure 13A] FIG. 13A is an illustration of an example of extracting the tricuspid annulus based on the prior art. [Figure 13B] FIG. 13B is an illustration of a specific example of extracting the tricuspid annulus based on this embodiment. [Figure 14A] FIG. 14A is an illustration of an example of extracting the tricuspid annulus based on the prior art. [Figure 14B] FIG. 14B is an illustration of a specific example of extracting the tricuspid annulus based on this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] The present application has been made in consideration of the above-mentioned problems of the prior art, and aims to provide an image processing device and image processing method that can automatically and accurately extract cardiac valve annuli based on 3D medical images, reduce the operational burden on the user, improve operability, directly obtain predicted results of 3D valve annuli without the need for image reconstruction, obtain 3D morphological characteristics of the valve annuli, accurately extract cardiac valve annuli even from medical images with poor contrast or high noise, and improve the reliability of cardiac valve annuli extraction.

[0013] According to an embodiment, an image processing device is provided, comprising an image acquisition unit that acquires a plurality of 3D medical images; an annulus keypoint extraction unit that extracts annulus keypoints based on the plurality of 3D medical images using a trained annulus keypoint prediction model; an annulus calibration unit that calibrates the annulus keypoints to supplement at least some of the annulus keypoints; and an annulus curve acquisition unit that performs fitting based on the calibrated annulus keypoints to acquire an annulus curve.

[0014] In addition, according to an embodiment, an image processing device is provided, which further comprises an annulus region image extraction unit that divides each of the plurality of 3D medical images into a right atrial region and a right ventricular region, and extracts an area of ​​a predetermined size centered on the center of gravity of the overlapping region between the right atrial region and the right ventricular region as an annulus region image, and the annulus keypoint extraction unit extracts annulus keypoints based on the plurality of annulus region images using a trained annulus keypoint prediction model.

[0015] In addition, according to an embodiment, an image processing device is provided, and in the process of training based on a training image group to obtain a trained annulus keypoint prediction model, an overall training image classification process is performed to obtain a classified training image group. In the overall training image classification process, it is determined whether each training image group satisfies a predetermined condition, and the training images that do not satisfy the predetermined condition are pre-processed or excluded from model training, thereby obtaining a classified training image group.

[0016] Furthermore, according to an embodiment, an image processing device is provided, in which, in the process of performing training based on a training image group to obtain a trained annulus keypoint prediction model, an overall training image classification process is performed, followed by an annulus region image extraction process to obtain an annulus region image group, in which the right atrial region and right ventricular region are respectively divided for each classified training image group, and an area of ​​a predetermined size centered on the center of gravity of the overlapping region between the right atrial region and the right ventricular region is extracted as the annulus region image.

[0017] Furthermore, according to an embodiment, an image processing device is provided, and in the process of training based on a training image group to obtain a trained annulus keypoint prediction model, an annulus region image extraction process is performed, and then a region image classification process is performed to obtain a classified annulus region image group. In the region image classification process, it is determined whether image processing is necessary for each annulus region image group, and image processing is performed if image processing is necessary.

[0018] According to an embodiment, an image processing apparatus is also provided, and in the process of training based on a set of training images to obtain a trained annulus keypoint prediction model, training is performed based on shape constraints.

[0019] In addition, according to an embodiment, an image processing apparatus is provided, which performs curve fitting based on pre-marked tricuspid annulus keypoints as shape constraints to obtain a tricuspid annulus fitting curve, uniformly resamples the tricuspid annulus fitting curve to obtain keypoint uniform sampling labels as input for point-to-point distance loss in model training, and densely resamples the tricuspid annulus fitting curve to obtain geometric shape labels as input for shape constraint loss in model training.

[0020] In addition, according to an embodiment, an image processing apparatus is provided, and in the process of performing training based on a training image set to obtain a trained annulus keypoint prediction model, training is performed based on an annulus boundary thermogram prediction subtask.

[0021] According to an embodiment, an image processing device is provided, in which the annulus calibration unit calibrates the annulus keypoints by removing at least some of the annulus keypoints and replenishing at least some of the removed annulus keypoints by interpolation.

[0022] According to an embodiment, there is also provided an image processing device, wherein the annulus calibration unit performs calibration so as to supplement new annulus points as the annulus key points by interpolation.

[0023] In addition, according to an embodiment, an image processing method is provided, comprising: an image acquisition step of acquiring a plurality of 3D medical images; an annulus keypoint extraction step of extracting annulus keypoints based on the plurality of 3D medical images using a trained annulus keypoint prediction model; an annulus calibration step of calibrating the annulus keypoints to supplement at least some of the annulus keypoints; and an annulus curve acquisition step of performing fitting based on the calibrated annulus keypoints to acquire an annulus curve.

[0024] The image processing device and image processing method according to the embodiments of the present application can automatically and accurately extract cardiac valve annuli based on 3D medical images, reducing the operational burden on the user and improving operability. It is possible to directly obtain predicted results for 3D valve annuli without the need for image reconstruction, obtain 3D morphological characteristics of the annuli, accurately extract cardiac valve annuli even from medical images with poor contrast or high noise, and improve the reliability of cardiac valve annuli extraction.

[0025] The image processing device and image processing method according to the present application will be described in detail below with reference to the drawings. The drawings are illustrative or conceptual diagrams, and the dimensions of each part, the dimensional ratios between parts, etc., are not necessarily the same as those in reality. Even when the same part is shown, the dimensions and ratios between parts may be different depending on the drawing. In the specification and drawings of the present application, the same elements as those previously described in the drawings will be denoted by the same symbols, and detailed descriptions will be appropriately omitted.

[0026] (First embodiment) First, the configuration of an image processing device 100 according to the first embodiment will be described with reference to Figures 1 to 3E. Note that although the image processing device includes various components, Figure 1 shows only components related to the technical idea of ​​the present application, and other components are omitted.

[0027] As shown in FIG. 1, the image processing device 100 according to the first embodiment includes an image acquisition unit 101, an annulus key point extraction unit 102, an annulus calibration unit 103, and an annulus curve acquisition unit 104.

[0028] The image acquisition unit 101 may be configured, for example, with a component or module having an information processing function such as a CPU or MCU in order to acquire a plurality of medical images. The image acquisition unit 101 may acquire a plurality of medical images from a medical imaging device via a communication method, or may acquire a plurality of medical images by reading a storage device in which a plurality of medical images are previously stored, or may further read a plurality of medical images processed by another component.

[0029] Here, the medical image may be a medical image such as a CT image, an ultrasound image, or an MRI image taken by a medical image diagnostic device such as a CT (Computed Tomography) device, an ultrasound diagnostic device, or an MRI (Magnetic Resonance Imaging) device. That is, the medical image may be a three-dimensional image or a two-dimensional image, but a three-dimensional image is preferable.

[0030] The annulus keypoint extraction unit 102 may be configured with a component or module with information processing capabilities, such as a CPU or MCU, to extract annulus keypoints based on the plurality of 3D medical images using a trained annulus keypoint prediction model. Here, the annulus keypoints are also referred to as predicted annulus keypoints.

[0031] Furthermore, the trained annulus keypoint prediction model is, for example, a trained deep learning model obtained by previously training the deep learning model based on medical images.

[0032] Here, the trained annulus keypoint prediction model is obtained by training a deep learning model based on, for example, a group of multiple training images of cardiac region images and markings (GT, Ground Truth) in which the position of the annulus (e.g., tricuspid annulus) is artificially indicated in advance.

[0033] Here, when multiple 3D medical images acquired by the image acquisition unit 101 are input into the trained annulus keypoint prediction model, a prediction result is obtained. This prediction result is, for example, a predicted annulus keypoint. The prediction result may further include multiple predicted annulus keypoints and multiple heat maps corresponding to the predicted annulus keypoints. Here, the value of each pixel point in each heat map is the probability value of predicting that pixel point as an annulus keypoint. Furthermore, the position of the pixel point in the heat map with the largest probability value is recognized as the position of the predicted annulus keypoint.

[0034] The annulus calibration unit 103 may be composed of a component or module with information processing capabilities, such as a CPU or MCU, to calibrate the annulus keypoints so as to supplement at least some of the annulus keypoints. For example, the annulus calibration unit 103 can calibrate the annulus keypoints by removing at least some of the annulus keypoints extracted by the annulus keypoint extraction unit 10 and supplementing at least some of the removed annulus keypoints by interpolation. The annulus calibration unit 103 can also calibrate the annulus keypoints by supplementing them with new annulus keypoints.

[0035] The annulus curve acquisition unit 104 may be composed of components or modules with information processing functions such as a CPU or MCU to perform fitting based on the annulus key points calibrated by the annulus calibration unit 103 and acquire the annulus curve.

[0036] An image processing method applied by the image processing device 100 according to the first embodiment will be described below. Fig. 2 is a flowchart illustrating an example of the image processing method of the image processing device according to the first embodiment. Figs. 3A to 3E are illustrative diagrams showing specific examples of the image processing device according to the first embodiment.

[0037] As shown in Fig. 2, after starting processing, the image processing device 100 acquires a plurality of medical images by the image acquisition unit 101 in step S100, and then proceeds to step S200. For example, a cardiac region image as shown in Fig. 3A is acquired. Here, Fig. 3A shows only one cardiac region image, and other cardiac region images are omitted.

[0038] Next, in step S200, the image processing device 100 uses the annulus keypoint extraction unit 102 to extract annulus keypoints using the trained annulus keypoint prediction model based on the multiple 3D medical images acquired by the image acquisition unit 101 in step S100, and then proceeds to step S300. Here, for example, annulus keypoints such as those shown in FIG. 3B are acquired, with each point in FIG. 3B representing one annulus keypoint. Furthermore, heat maps corresponding to each annulus keypoint can be simultaneously acquired (see FIG. 3C). While only three heatmaps are shown in FIG. 3C for illustrative purposes, the number of heatmaps is the same as the number of annulus keypoints extracted by the annulus keypoint extraction unit 102.

[0039] Next, in step S300, the image processing device 100 calibrates the annulus keypoints by using the annulus calibration unit 103 to add at least some of the annulus keypoints, and then proceeds to step S400. Here, the annulus calibration unit 103 removes at least some of the annulus keypoints and calibrates the annulus keypoints by adding at least some of the removed annulus keypoints by interpolation. The calibrated annulus keypoints are shown in FIG. 3D.

[0040] Next, in step S400, the image processing device 100 performs fitting using the calibrated annulus keypoints with the annulus curve acquisition unit 104 to acquire the annulus curve, and then ends the processing. Here, fitting is performed using the calibrated annulus keypoints as shown in Fig. 3D to acquire the annulus curve as shown in Fig. 3E.

[0041] According to the image processing device 100 of the first embodiment, a trained annulus keypoint prediction model is used to extract annulus keypoints based on multiple acquired 3D medical images. The annulus keypoints are then calibrated to supplement at least some of the annulus keypoints, and fitting is performed based on the calibrated annulus keypoints to obtain an annulus curve. This allows for automatic extraction of cardiac annuli based on 3D medical images, reducing the user's operational burden and improving operability compared to conventional techniques that required manual operation by a physician. Furthermore, by calibrating the predicted annulus keypoints, inappropriate annulus keypoints that may appear when extracting annulus keypoints using medical images with poor contrast or high noise are eliminated, allowing for accurate extraction of cardiac annuli and improving the reliability of cardiac annulus extraction. Furthermore, fitting based on the calibrated annulus keypoints allows for the acquisition of annulus curves. This allows for direct 3D annulus prediction results and the acquisition of 3D morphological characteristics of the annulus through curve fitting alone, without the need for complex processing such as image reconstruction.

[0042] (Second embodiment) First, the configuration of an image processing device 100A according to the second embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram illustrating an example of the image processing device according to the second embodiment.

[0043] 4, the image processing device 100A according to this embodiment includes an image acquisition unit 101, annulus keypoint extraction unit 102, annulus calibration unit 103, and annulus curve acquisition unit 104, and further includes annulus region image extraction unit 101A. Here, the image acquisition unit 101, annulus calibration unit 103, and annulus curve acquisition unit 104 are substantially the same as the image acquisition unit 101, annulus calibration unit 103, and annulus curve acquisition unit 104 according to the first embodiment, respectively, and detailed description thereof will be omitted.

[0044] Furthermore, in this embodiment, the annulus region image extraction unit 101A included in the image processing device 100A may be configured, for example, by a component or module with an information processing function, such as a CPU or MCU, to extract annulus region images from each of multiple 3D medical images acquired by the image acquisition unit 101. Here, similar to the first embodiment, multiple CT images of the cardiac region are acquired by the image acquisition unit 101, and the unit is configured to extract the tricuspid annulus. In this case, a method for extracting tricuspid annulus region images from each of the multiple CT images of the cardiac region may include, for example, dividing each of the multiple CT images of the cardiac region into a right atrial region and a right ventricular region, and extracting, as the annulus region image, a region of a predetermined size centered on the center of gravity of the overlapping region between the right atrial region and the right ventricular region.

[0045] In addition, in this embodiment, the annulus keypoint extraction unit 102 provided in the image processing device 100A extracts annulus keypoints using a trained annulus keypoint prediction model based on multiple annulus region images extracted by the annulus region image extraction unit 101A.

[0046] An image processing method applied by the image processing device 100A according to the second embodiment will be described below. FIG. 5 is a flowchart illustrating an example of the image processing method of the image processing device 100A according to the second embodiment. FIGS. 6A to 6J are illustrative diagrams showing a specific example of the image processing device 100A according to the second embodiment. FIG. 7 is a flowchart illustrating an example of the operation of extracting an annulus region image by the image processing device 100A according to the second embodiment. FIG. 8 is a flowchart illustrating an example of the operation of calibrating an annulus by the image processing device 100A according to the second embodiment.

[0047] As shown in Fig. 5, after starting processing, the image processing device 100A acquires a plurality of medical images by the image acquisition unit 101 in step S100, and then proceeds to step S100A. For example, a cardiac region image as shown in Fig. 6A is acquired. Here, Fig. 6A shows only one cardiac region image, and other cardiac region images are omitted.

[0048] Next, in step S100A, the annulus region image extraction unit 101A extracts an annulus region image from each of the multiple three-dimensional medical images acquired by the image acquisition unit 101, and then the process proceeds to step S200A.

[0049] Here, Fig. 7 shows the operational flow of extracting an image of an annulus region in step S100A, that is, the processes of steps S100A1 to S100A3 in Fig. 7 may be substituted for the process of step S100A in Fig. 5. Hereinafter, the operation of extracting an image of an annulus region by image processing device 100A according to the second embodiment will be described with reference to Fig. 7.

[0050] 7, in step S100A1, the annulus region image extraction unit 101A of the image processing device 100A segments (extracts) at least the right atrium and the right ventricle using, for example, a four-term division mixed algorithm for the multiple CT images of the cardiac region acquired by the image acquisition unit 101, and then proceeds to step S100A2. Here, as shown in FIG. 6B, in addition to the extracted right atrium and right ventricle, the left atrium and left ventricle are also set as extracted.

[0051] Next, in step S100A2, the annulus region image extraction unit 101A of the image processing device 100A expands the right atrium and right ventricle based on the results extracted in step S100A1, and then obtains an overlapping region between the expanded right atrium and the expanded right ventricle. Then, the process proceeds to step S100A3. For example, the right atrium in the image shown in FIG. 6A is expanded to obtain an expanded image of the right atrium shown in FIG. 6C, and the right ventricle in the image shown in FIG. 6A is expanded to obtain an expanded image of the right ventricle shown in FIG. 6D. Based on the expanded right atrium image and the expanded right ventricle image, the overlapping region indicated by the cross-sectional lines in FIG. 6E is obtained. This overlapping region is the region where the tricuspid valve is located. This overlapping region may also be referred to as a region of interest.

[0052] Next, in step S100A3, the annulus region image extraction unit 101A of the image processing device 100A extracts an annulus region image centered on the center of gravity of the overlapping region, and then proceeds to step S200A. Here, the annulus region image extraction unit 101A calculates the center of gravity of the overlapping region based on the overlapping region obtained in step S100A2, and then extracts an image region of a predetermined size as the annulus region image by cropping the overlapping region using the center of gravity as the cropping center. Here, the predetermined size may be preset or may be changed as needed, and only needs to satisfy the requirement that the image region contain the organ or tissue to be segmented, such as the tricuspid annulus. Here, for example, by extracting an image region of the size indicated by the block shown in FIG. 6F, the annulus region image shown in FIG. 6G is obtained.

[0053] Furthermore, in this embodiment, the annulus region image extraction unit 101A of the image processing device 100A extracts annulus region images for each of the multiple 3D medical images acquired by the image acquisition unit 101, and therefore typically obtains the same number of annulus region images as the multiple 3D medical images acquired by the image acquisition unit 101. In Fig. 6G, only one annulus region image is shown as an example.

[0054] Returning to FIG. 5, next, in step S200A, the image processing device 100A causes the annulus keypoint extraction unit 102 to extract annulus keypoints using the trained annulus keypoint prediction model based on the multiple annulus region images extracted by the annulus region image extraction unit 101A in step S100A. The process then proceeds to step S300. Here, for example, the annulus keypoints shown in FIG. 6H are acquired. At this time, heat maps corresponding to each annulus keypoint, as shown in FIG. 6I, are also simultaneously acquired. Similarly, the number of acquired heat maps is the same as the number of annulus keypoints extracted by the annulus keypoint extraction unit 102.

[0055] Next, in step S300, the image processing device 100A causes the annulus calibration unit 103 to calibrate the annulus keypoints so as to supplement at least some of the annulus keypoints, and then proceeds to step S400. Here, the annulus calibration unit 103 calibrates the annulus keypoints so as to supplement at least some of the annulus keypoints. The calibrated annulus keypoints are shown in FIG. 6J.

[0056] Fig. 8 shows the flow of the annulus keypoint calibration operation in step S300. That is, the processes of steps S301 to S304 in Fig. 8 may be performed instead of the process of step S300 in Fig. 5. Hereinafter, the annulus keypoint calibration operation of image processing device 100A according to the second embodiment will be described with reference to Fig. 8.

[0057] 5 and 8, in step S301, the annulus calibration unit 103 of the image processing device 100A extracts the positions of predicted annulus keypoints for each of the heat maps corresponding to the annulus keypoints shown in FIG. 6I, acquired in step S200A. The process then proceeds to step S302. The value of each pixel in the heat map represents the probability that the pixel will actually be located on the annulus, and the position of the pixel with the highest probability in the heat map is recognized as the position of the predicted annulus keypoint. Thus, predicted annulus keypoints are extracted for each of the heat maps (FIG. 6H).

[0058] Next, in step S302, the annulus calibration unit 103 of the image processing device 100A uses the overlap region acquired in step S100A2 as a region of interest, and for each of the acquired heat maps, determines whether the predicted annulus keypoint in that heat map is a false positive. The heat map is post-processed according to the determination result, and then the process proceeds to step S303. Here, a false positive refers to, for example, a pixel point that is not actually on the annulus, i.e., a point that is extracted as a predicted annulus keypoint, even though it is not a valve annulus keypoint.

[0059] The following method can be used to determine whether a predicted annulus keypoint in each heat map is a false positive: First, using the region of interest as a mask, calculations are performed on each heat map for which predicted annulus keypoints were found in step S301. If the position of the predicted annulus keypoint in the heat map falls within the range of the region of interest, the corresponding predicted annulus keypoint in the heat map is determined to be a false positive. On the other hand, if the position of the predicted annulus keypoint in the heat map falls outside the range of the region of interest, the corresponding predicted annulus keypoint in the heat map is determined to be a false positive and is not considered a predicted annulus keypoint.

[0060] Furthermore, post-processing is performed on heat maps in which predicted annulus keypoints are determined to be false positives. For example, this post-processing involves resetting all pixel values ​​(i.e., probability values) that fall outside the range corresponding to the region of interest in the heat map to 0, and maintaining the values ​​(i.e., probability values) of image points that fall within the range corresponding to the region of interest. Note that this post-processing is merely an example and is not limiting; for example, all pixel values ​​can be reset to a predetermined value as needed.

[0061] Next, in step S303, the annulus calibration unit 103 of the image processing device 100A calibrates the predicted annulus keypoints determined to be false positives in step S302, generates and selects new predicted points, and then proceeds to step S304. Calibrating the predicted annulus keypoints determined to be false positives in step S302 and generating new predicted points involves, for example, extracting the pixel value (i.e., the point with the highest probability value) in a post-processed heat map of the predicted annulus keypoints determined to be false positives in step S302 as the new predicted point. The center of gravity of multiple predicted annulus keypoints other than the predicted annulus keypoints determined to be false positives in step S302, i.e., the multiple predicted annulus keypoints not determined to be false positives, is calculated, and the distance Dnew between the new predicted point and the calculated center of gravity is calculated. The distance between each of the multiple predicted annulus keypoints other than those determined to be false positives in step S302 and the new predicted point is calculated, and the predicted keypoint closest to the new predicted point, i.e., the nearest predicted keypoint, is extracted from the multiple predicted annulus keypoints other than those determined to be false positives in step S302. The distance Dnear between the nearest predicted keypoint and the center of gravity is calculated. The absolute distance difference between the distance Dnew and the distance Dnear, i.e., the absolute value of the difference between the distance Dnear and the distance Dnew, is used to determine whether to retain the new predicted point as a predicted keypoint or to discard the new predicted point. For example, if the absolute value is below a predetermined threshold such as 0.1, the new predicted point is retained. That is, the new predicted point is selected as a predicted keypoint instead of a false positive. On the other hand, if the absolute value is above a predetermined threshold such as 0.1, the new predicted point is discarded and not used as a predicted keypoint.

[0062] Next, in step S304, the annulus calibration unit 103 of the image processing device 100A interpolates at least some of the predicted keypoints, and then proceeds to step S400. For example, fitting can be performed on all of the predicted annulus keypoints obtained in step S303 using a B-spline curve or the like to obtain an overall annulus curve, and new predicted keypoints can be added using equidistant interpolation or the like. Alternatively, at least some of the predicted keypoints can be added by supplementing the false positives that have already been removed in the area where the false positives have been removed.

[0063] The interpolation method is not limited to B-spline interpolation or equidistant interpolation. The process of refilling at least some of the predicted keypoints by interpolation in step S304 can be performed regardless of whether false positives have been removed. If false positives have been removed, the number of refilled predicted keypoints can be set equal to, greater than, or less than the number of removed false positives, thereby calibrating the predicted annulus keypoints.

[0064] Next, in step S400, the image processing device 100 performs fitting using the calibrated annulus keypoints with the annulus curve acquisition unit 104 to acquire the annulus curve, and then ends the processing. For example, fitting is performed using the calibrated annulus keypoints shown in Fig. 6J to acquire the annulus curve shown in Fig. 6K.

[0065] The image processing device 100A according to the second embodiment achieves the technical effects of the first embodiment and includes an annulus region image extraction unit 101A. Therefore, the image processing device extracts annulus region images from multiple acquired 3D medical images and extracts annulus keypoints based on the annulus region images using a trained annulus keypoint prediction model. This reduces the range of images to be processed, avoids unnecessary calculations on irrelevant regions, and reduces the amount of calculation, thereby improving the overall processing speed of the image processing device.

[0066] Furthermore, the image processing device 100A according to the second embodiment can acquire three-dimensional morphological characteristics of the annulus by efficiently and accurately obtaining the curve of the entire annulus. For example, it can calculate the three-dimensional circumference, longest and shortest diameters of the annulus ((a) in FIG. 6L), spatial height of the annulus ((b) in FIG. 6L), and projected area of ​​the annulus on a predetermined plane ((c) in FIG. 6L). Providing these morphological characteristics to the user can be of great reference value in clinical applications, for example.

[0067] (Modification 1 of the second embodiment) The configuration of the image processing device 100B according to the first modification of the second embodiment will be described below.

[0068] The only difference between the configuration of the image processing device 100B according to Variation 1 of the second embodiment and the configuration of the image processing device 100A according to the second embodiment is that the image acquisition unit 101 further includes a medical image classification function. Specifically, after acquiring 3D medical images, the image acquisition unit 101 further classifies the medical images to acquire 3D medical images suitable for the annulus extraction task and provide them to the subsequent annulus keypoint extraction unit 102, etc.

[0069] Here, the classification process for medical images may involve, for example, classification based on image enhancement conditions. Specifically, in a task of segmenting and extracting a tricuspid valve annulus, if an acquired 3D medical image is an image in which only the tricuspid valve region has been enhanced or cleared, i.e., right ventricle enhancement, it is suitable for this task and is retained. On the other hand, if an acquired 3D medical image is an image in which only the bicuspid valve region has been enhanced or cleared, i.e., left ventricle enhancement, the bicuspid valve region in this medical image is used to roughly locate or calibrate the detection of the tricuspid valve annulus. On the other hand, if an acquired 3D medical image is an image in which neither the bicuspid valve region nor the tricuspid valve region has been enhanced, this image may be discarded, and image processing may be performed on it to make it suitable for this task, or it may be classified based on other indicators.

[0070] In addition, the classification of medical images can be based on whether or not an artificial implant is present in the image. Specifically, if an artificial implant is present in the image, the image is discarded directly. If an artificial implant is not present, the image can be classified based on other indicators.

[0071] In addition, the classification process of medical images may be, for example, a process of classifying medical images based on the size of the FOV of the medical image. Specifically, if the FOV of a medical image exceeds a set threshold, the image may be discarded directly, or may be retained after being cropped, or may be classified based on other indicators. If the FOV of a medical image meets a threshold, the image may be retained or classified based on other indicators.

[0072] Furthermore, the classification process for medical images may include, for example, a process of classifying medical images based on the magnitude of the overall noise of the medical images. Specifically, if the overall noise of the medical images is too large, the medical images may be retained after the overall noise is reduced, or may be classified based on other indicators. If the medical images do not have the problem of excessive overall noise, the medical images may be retained or classified based on other indicators.

[0073] Furthermore, in this modified example, the medical image classification process performed by the image acquisition unit 101 on three-dimensional medical images is not limited to the classification process for each medical image described above, but may be a classification process for other medical images if it is possible to remove three-dimensional medical images that are not suitable for subsequent processing and select three-dimensional medical images that are suitable for subsequent processing, or to improve three-dimensional medical images that are not suitable for subsequent processing into three-dimensional medical images that are suitable for subsequent processing.

[0074] Furthermore, the classification processes of each medical image described above may be applied independently, or at least one or more classification processes of medical images may be applied in combination. Furthermore, when the classification processes of each medical image described above are applied in combination, they may be executed in parallel, or may be executed sequentially in a predetermined order as necessary.

[0075] An image processing method applied by the image processing device 100B according to this modification will be described below. Fig. 9 is a flowchart illustrating an example of the image processing method of the image processing device according to the modification 1 of the second embodiment.

[0076] Comparing the operational flow of image processing device 100B according to this modified example shown in Fig. 9 with the operational flow of image processing device 100A according to the second embodiment shown in Fig. 5, the processes of steps S100 and S100A to S400 are the same, and the only difference is that this modified example further performs the process of step S100B. Therefore, here, where necessary, the description of steps S100 and S100A to S400 will be omitted, and step S100B will be described in detail.

[0077] In step S100B, the image acquisition unit 101 of the image processing device 100B performs a medical image classification process on the acquired medical images.

[0078] Specifically, the image acquisition unit 101 determines whether each of the acquired medical images satisfies a predetermined condition. Here, the predetermined condition may be determined in advance according to the segmentation task. For example, the predetermined condition may include whether the medical image satisfies a predetermined image enhancement condition, whether an artificial implant is present in the medical image, whether the FOV size of the medical image reaches a predetermined range, whether the overall noise level of the medical image falls within a predetermined noise range, etc. The above conditions may be determined in a predetermined order or in parallel.

[0079] In the case of parallel judgments, if it is determined that the predetermined conditions are met, the process proceeds to step S100A. On the other hand, if it is determined that the predetermined conditions are not met, the following processing methods are available, for example: (1) Discard this medical image and make a judgment on the next medical image. (2) Process this medical image so that it meets the predetermined conditions, and then return to step S100B and make a judgment again until the predetermined conditions are met and judgment is made on the next medical image. (3) Process this medical image so that it meets the predetermined conditions, and then return to step S100B and make a judgment again, and if the predetermined conditions are still not met after judgment has been made a predetermined number of times, discard this medical image and make a judgment on the next medical image.

[0080] Furthermore, when judgments are made in a predetermined order, if it is determined that the predetermined conditions are met, judgment is made on the next condition, and if all the predetermined conditions are met, the process proceeds to step S100A. On the other hand, if it is determined that the predetermined conditions are not met, the following processing methods are available, for example: (1) Discard this medical image, and make judgment on the next medical image. (2) Process this medical image so that it meets the predetermined conditions, and then return to step S100B to make judgment again. If the predetermined conditions are still not met after judgments have been made a predetermined number of times, discard this medical image, and make judgment on the next medical image.

[0081] The image processing device 100B according to the first variant of the second embodiment achieves the technical effects of the first and second embodiments described above, and furthermore, the image acquisition unit 101 performs a medical image classification process on the multiple medical images acquired, so that it is possible to remove images that are not suitable for the image processing task, or to improve inappropriate images into appropriate images, thereby avoiding a reduction in accuracy due to the introduction of inappropriate images.

[0082] (Modification 2 of the second embodiment) The configuration of an image processing device 100C according to Modification 2 of the second embodiment will be described below.

[0083] The only difference between the configuration of the image processing device 100C according to Modification 2 of the second embodiment and the configuration of the image processing device 100B according to Modification 1 is that the annulus region image extraction unit 101A further includes a region image classification function. Specifically, the annulus region image extraction unit 101A acquires 3D medical images suitable for the annulus extraction task and further classifies the extracted annulus region images so as to provide them to the subsequent annulus keypoint extraction unit 102, etc.

[0084] Here, classification of the annulus region image may be performed, for example, based on whether or not an annulus is present. Specifically, it is determined whether or not an annulus is present in the annulus region image. If an annulus is not present, the image is discarded and a determination is made on the next annulus region image. On the other hand, if an annulus is present, a determination is made on the next annulus region image, or the process may proceed to step S200A.

[0085] Furthermore, classification of the annulus region image may be performed based on image quality, for example. Specifically, statistical features of a target region in the annulus region image may be calculated, and a template matching method may be used to determine whether defects such as noise or artifacts exist in the target region. If no noise or artifacts exist, the process proceeds to step S200A. On the other hand, if defects such as noise or artifacts exist, the annulus region image may be discarded, or an improvement process may be performed on the annulus region image so that its image quality reaches a predetermined level.

[0086] The improvement process varies depending on the type of defect. For example, for images with poor contrast, contrast enhancement, edge enhancement, filter algorithms, etc. may be used to enhance the contrast between the valve and surrounding tissue. For images with noisy contrast, an edge-protection filter algorithm may be used to remove noise while preserving image details and edge structure. For images with metal artifacts, a metal artifact compensation algorithm may be used to detect and repair metal regions to reduce or eliminate the metal artifacts.

[0087] Furthermore, in this modified example, the classification process of the annulus region image performed by the annulus region image extraction unit 101A on the annulus region image is not limited to the classification process of each annulus region image described above, and classification process of other annulus region images may be performed if annulus region images unsuitable for subsequent processing can be removed and annulus region images suitable for subsequent processing can be selected, or annulus region images unsuitable for subsequent processing can be improved into annulus region images suitable for subsequent processing.

[0088] The classification process for each of the above-described annulus region images may be applied independently, or at least one or more classification processes for the annulus region images may be applied in combination. When the classification processes for the above-described annulus region images are applied in combination, they may be executed in parallel, or may be executed sequentially in a predetermined order as necessary.

[0089] An image processing method applied by the image processing device 100C according to this modification will be described below. Fig. 10 is a flowchart illustrating an example of the image processing method of the image processing device according to Modification 2 of the second embodiment.

[0090] Comparing the operational flow of the image processing device 100C according to this modification shown in Fig. 10 with the operational flow of the image processing device 100B according to the second embodiment shown in Fig. 9, the processes of steps S100 to S100A and S200A to S400 are the same, and the only difference is that this modification further performs the process of step S100C. Therefore, the description of steps S100 and S100A to S400 will be omitted here as necessary.

[0091] In step S100C, the annulus region image extraction unit 101A of the image processing device 100B performs classification processing on the extracted annulus region images.

[0092] The classification process of the annulus region image performed by the annulus region image extraction unit 101A on the further extracted annulus region image is similar to the classification process of the medical images performed on the multiple medical images acquired by the image acquisition unit 101 according to the first modified example, and since it is only necessary to use the same flow as in the first modified example and make equal replacements as necessary, a description thereof will be omitted here.

[0093] The image processing device 100C according to Modification 2 of the second embodiment achieves the technical effects of the above-described first and second embodiments and Modification 1, and the annulus region image extraction unit 101A further performs classification processing on the extracted annulus region images, thereby eliminating images that are unsuitable for the image processing task or improving inappropriate images into appropriate images, thereby avoiding a decrease in accuracy due to the introduction of inappropriate images. Furthermore, because classification processing is performed on both medical images and annulus region images, the accuracy of images acquired by the image processing device can be further improved.

[0094] (Third embodiment) The configuration of the image processing device 100D according to the third embodiment will be described below.

[0095] When comparing the configuration of the image processing device 100D according to the third embodiment with the configurations of the image processing devices according to the first embodiment, the second embodiment, and variants 1 and 2, the only difference is that the trained annulus keypoint prediction model used by the annulus keypoint extraction unit 102 when extracting annulus keypoints based on the multiple 3D medical images is obtained by training based on shape constraints.

[0096] In a typical deep learning model training process, a pre-artificially generated marking (GT, Ground Truth) and multiple medical images are acquired and used to train the deep learning model. Specifically, the medical image is input to the deep learning network as an input image (input volume), and a prediction result (predication) is obtained. The loss function (LOSS) value, loss, is used to measure the degree of deviation between the prediction result (predication) and GT (Ground Truth).

[0097] In the training process of the trained annulus keypoint prediction model according to this embodiment, first, uniformly sampled keypoint labels and geometric shape labels for the tricuspid annulus are constructed. Specifically, curve fitting is performed based on the acquired tricuspid annulus keypoints of the marking GT (FIG. 11(a)), to obtain a tricuspid annulus fitting curve (FIG. 11(b)). For example, B-spline curve fitting or other fitting methods may be used. Then, the tricuspid annulus fitting curve is uniformly resampled to obtain uniformly sampled keypoint labels (FIG. 11(c)), which serve as input for the point-to-point distance loss during model training. Furthermore, the tricuspid annulus fitting curve is densely resampled to obtain geometric shape labels (FIG. 11(d)), which serve as input for the shape constraint loss (e.g., weighted Hausdorff distance loss) during model training.

[0098] Next, we train a tricuspid annulus keypoint detection model with shape constraints. There are many methods for shape constraints, such as shape-aware weighted Hausdorff distance loss, adaptive wing loss, and boundary-aware neural network architecture. Here, we use shape-aware weighted Hausdorff distance loss.

[0099] Specifically, during training to obtain a trained annulus keypoint prediction model, the prediction of annulus keypoints is constrained using MSE loss, focal loss, and shape-aware weighted Hausdorff distance loss based on shape constraints. For example, the loss function used is expressed as follows:

[0100]

number

[0101] Of the multiple terms in the above equation (1), the first term represents the default MSE loss, the second term represents the loss term using uniformly sampled labels, and the third term represents the loss term based on shape constraints.

[0102] In this way, the trained annulus keypoint prediction model according to this embodiment considers the distance between the heat map of predicted keypoints and the geometric shape labels shown in Figure 11(d) when predicting annulus keypoints, and penalizes points that are far from the annulus (geometric shape label) but have high prediction probabilities, thereby constraining the overall shape during keypoint prediction. Furthermore, uniformly sampled keypoint location labels may also be combined to simultaneously penalize points that are close to the corresponding keypoints but have low prediction probabilities, thereby maintaining the accuracy of annulus keypoint prediction.

[0103] In this way, the image processing device 100D according to the third embodiment uses a trained annulus keypoint prediction model obtained by training based on shape constraints when extracting annulus keypoints. This maintains the accuracy of annulus keypoint prediction based on uniform sampling labels, and constrains the overall shape during keypoint prediction based on geometric shape labels, improving the overall accuracy of tricuspid annulus keypoint prediction and efficiently resolving the problem of large deviations in keypoint detection in poor contrast or noisy areas.

[0104] (Fourth embodiment) The configuration of the image processing device 100E according to the fourth embodiment will be described below.

[0105] Comparing the configuration of the image processing device 100E according to the fourth embodiment with the configuration of the image processing device 100D according to the third embodiment, the difference is that the trained annulus keypoint prediction model used by the annulus keypoint extraction unit 102 when extracting annulus keypoints based on multiple 3D medical images is obtained by training using other shape constraints in addition to the shape constraints according to the third embodiment, such as adaptive wing loss, and by training based on the annulus boundary heat map prediction subtask.

[0106] Specifically, during training to obtain a trained annulus keypoint prediction model, we first construct keypoint uniform sampling labels and geometric shape labels for the tricuspid annulus, similar to the third embodiment.

[0107] Next, a tricuspid annulus keypoint detection model with shape constraints is trained. Specifically, in addition to applying the shape constraint as in the third embodiment, the loss function also introduces other shape constraints, such as adaptive wing loss.

[0108] Next, we train an annulus keypoint detection model with an annulus boundary prediction subtask. Specifically, we generate an annulus curve by curve fitting based on the predicted keypoints obtained during the training process, and then generate a distance transform diagram as an annulus boundary heat map, as shown in Figure 12, based on the annulus curve obtained by this fitting. We consider constraining the prediction of annulus keypoints based on the annulus boundary of the annulus boundary heat map.

[0109] In this way, the image processing device 100E according to the fourth embodiment achieves the technical effects of the third embodiment, and in addition, when extracting annulus keypoints, it uses a trained annulus keypoint prediction model obtained by training based on shape constraints and annulus boundary constraints. This not only enables prediction of annulus keypoints but also prediction of annulus boundary lines, thereby enabling more accurate prediction of annulus keypoints and more accurate segmentation and extraction of the annulus even in areas with poor contrast or large deviations in keypoint detection in noisy regions.

[0110] (Control experiment) Below, we describe a control experiment in which the tricuspid annulus was extracted using the same input image under the same conditions using the conventional technique and the method according to this embodiment, which incorporates annulus point calibration and shape constraint-based extraction of annulus keypoints.

[0111] (Control experiment 1) The tricuspid annulus was extracted from a cardiac CT image in which the contrast between the mesenchyme and the area near the anterior leaflet was poor and the intensity was similar to that of the main artery area using both the conventional technique and the method according to this embodiment (e.g., the fourth embodiment). As a result, the extraction results shown in Figures 13A and 13B were obtained.

[0112] FIG. 13A is an illustration of a specific example of a tricuspid annulus extracted based on the prior art, and FIG. 13B is an illustration of a specific example of a tricuspid annulus extracted based on this embodiment.

[0113] (Control experiment 2) For a cardiac CT image in which "metal artifacts exist around the annulus region where a pacemaker was left unattended," the tricuspid annulus was extracted using both the conventional technique and the method according to this embodiment (e.g., the fourth embodiment). As a result, the extraction results shown in Figures 14A and 14B were obtained.

[0114] FIG. 14A is an illustration of a specific example of a tricuspid annulus extracted based on the prior art, and FIG. 14B is an illustration of a specific example of a tricuspid annulus extracted based on this embodiment.

[0115] As can be seen from the above control experiment results, this embodiment realizes accurate fully automatic extraction of cardiac valve annuli in 3D medical images, and can directly obtain 3D annulus prediction results to obtain 3D annulus morphology features. Furthermore, a trained keypoint prediction model based on shape constraints can be used to improve the prediction of abnormalities in keypoints occurring in poor contrast or noise areas of the cardiac valve annulus. Furthermore, a trained keypoint prediction model based on the global annulus shape constraints can be used to improve the global prediction of the annulus shape, improve the predicted abnormality data, and improve the stability of annulus detection.

[0116] (Variation) In the above description of each embodiment, the annulus keypoint extraction unit 102 extracts annulus keypoints using a trained annulus keypoint prediction model. However, this is not limited to this example. The image processing device according to this embodiment may also acquire a trained annulus keypoint prediction model by having a separate trained annulus keypoint prediction model acquisition unit.

[0117] Furthermore, the operations of extracting an annulus region image and calibrating annulus keypoints according to the second embodiment are not limited to the above-described embodiment, and may be applied to other embodiments and modified examples of the present application.

[0118] In addition, although the second embodiment exemplifies extraction of the tricuspid annulus based on a CT image of the cardiac region, the present invention is not limited to this example and may be applied to segmentation and extraction of the aortic annulus and the bicuspid annulus. In this case, the overlapping region may be extracted and modified accordingly.

[0119] In addition, in the second embodiment, an example was given of a method in which the right atrium and right ventricle were divided from an image, the division results were dilated, and then an overlapping region was obtained based on the dilated right atrium and the dilated right ventricle. However, this is not limited to this, and for example, another method may be used in which an overlapping region of both the right atrium and right ventricle is obtained based on the divided right atrium and right ventricle.

[0120] Furthermore, in the second embodiment and variants 1 and 2, we have described the extraction of annulus region images in the process of extracting annulus curves, the classification process of medical images, and the classification process of annulus region images. However, these processes can be applied to the training process using a trained annulus keypoint prediction model used when extracting annulus keypoints based on multiple 3D medical images, making it possible to extract annulus keypoints using a trained annulus keypoint prediction model with higher accuracy and efficiency.

[0121] According to at least one of the embodiments described above, it is possible to provide a technique that can extract cardiac valve annulus in a medical image with high accuracy and efficiency.

[0122] Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0123] 100~100E: Image processing equipment 101: Image acquisition unit 101A: Valve annulus region image extraction unit 102: Valve annulus keypoint extraction unit 103: Annulus calibration unit 104: Valve ring curve acquisition part

Claims

1. an image acquisition unit that acquires a plurality of three-dimensional medical images; an annulus keypoint extraction unit that extracts annulus keypoints based on the plurality of 3D medical images using a trained annulus keypoint prediction model; an annulus calibration unit that calibrates the annulus key points so as to supplement at least some of the annulus key points; an annulus curve acquisition unit that performs fitting based on the calibrated annulus keypoints to acquire an annulus curve; An image processing device comprising:

2. a valve annulus region image extraction unit that divides each of the plurality of three-dimensional medical images into a right atrial region and a right ventricular region, and extracts, as a valve annulus region image, a region of a predetermined size centered on the center of gravity of an overlapping region between the right atrial region and the right ventricular region; The image processing device according to claim 1 , wherein the annulus keypoint extraction unit extracts the annulus keypoints based on a plurality of the annulus region images using the trained annulus keypoint prediction model.

3. 3. The image processing device according to claim 1, wherein the trained annulus keypoint prediction model is trained based on a classified training image group obtained by an overall training image classification process, in which, during a training process based on a training image group, it is determined whether each training image group satisfies a predetermined condition, and training images that do not satisfy the predetermined condition are preprocessed or excluded from model training.

4. 4. The image processing device of claim 3, wherein the trained annulus keypoint prediction model is trained based on a group of annulus region images obtained by annulus region image extraction processing, which divides the right atrial region and the right ventricular region into separate regions for each of the classified training images after the overall training image classification processing, and extracts, as annulus region images, regions of a predetermined size centered on the center of gravity of overlapping regions between the right atrial region and the right ventricular region.

5. 5. The image processing device according to claim 4, wherein the trained annulus keypoint prediction model is trained based on a group of classified annulus region images obtained by a region image classification process that determines whether image processing is necessary for each group of annulus region images after the annulus region image extraction process, and performs image processing if image processing is necessary.

6. The image processing device according to claim 1 , wherein the trained annulus keypoint prediction model is trained based on shape constraints during training based on a set of training images.

7. 7. The image processing device of claim 6, wherein the shape constraint comprises: performing curve fitting based on pre-marked tricuspid annulus keypoints to obtain a tricuspid annulus fitting curve; uniformly resampling the tricuspid annulus fitting curve to obtain keypoint uniform sampling labels as input for distance loss between points in model training; and densely resampling the tricuspid annulus fitting curve to obtain geometric shape labels as input for shape constraint loss during model training.

8. The image processing device according to claim 7 , wherein the trained annulus keypoint prediction model is trained based on an annulus boundary heatmap prediction subtask during training based on the training image group.

9. The image processing device according to claim 1 , wherein the annulus calibration unit calibrates the annulus keypoints by removing at least some of the annulus keypoints and replenishing at least some of the removed annulus keypoints by interpolation.

10. The image processing device according to claim 1 , wherein the annulus calibration unit performs calibration by supplementing new annulus points as the annulus key points by interpolation.

11. an image acquisition step of acquiring a plurality of three-dimensional medical images; an annulus keypoint extraction step of extracting annulus keypoints based on the plurality of 3D medical images using a trained annulus keypoint prediction model; an annulus calibration step of calibrating the annulus key points so as to supplement at least some of the annulus key points; an annulus curve acquisition step of performing fitting based on the calibrated annulus keypoints to acquire an annulus curve; An image processing method comprising:

Images